Python3.5多进程原理与用法实例分析

1. 多进程简介

多进程是一种方式，通过启动多个进程并发执行不同的任务，来提高程序的运行效率。Python中提供了多种方式实现多进程，包括subprocess、multiprocessing、os.fork()等，其中multiprocessing是Python自带的一个多进程模块，其封装了底层的C语言模块_multiprocessing，提供了更加易用的多进程编程接口。

2. 多进程基础

2.1 创建进程

创建多个进程可以使用Process类。示例如下：

from multiprocessing import Process

def my_func(arg1, arg2):
    print("arg1:", arg1)
    print("arg2:", arg2)

if __name__ == '__main__':
    p = Process(target=my_func, args=(1, "hello"))
    p.start()
    p.join()

2.2 进程通信

多个进程之间可以进行通信，Python中提供了多种进程通信方式，包括Queue、Pipe、Manager等。示例如下：

from multiprocessing import Process, Queue

def worker(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(item)

if __name__ == '__main__':
    q = Queue()
    p = Process(target=worker, args=(q,))
    p.start()

    q.put("hello")
    q.put("world")
    q.put(None)

    p.join()

3. 示例说明

3.1 示例1 - 计算圆周率

基于multiprocessing模块，计算圆周率。示例如下：

from multiprocessing import Process, Queue
import time

def calculate_pi(part_id, total_parts, queue):
    start = int((part_id - 1) * (1 / float(total_parts)) * 1000000 + 1)
    end = int(part_id * (1 / float(total_parts)) * 1000000)

    pi_sum = 0
    sign = 1

    for i in range(start, end + 1):
        pi_sum += sign * (1 / float(i))
        sign *= -1

    queue.put(pi_sum)

if __name__ == '__main__':
    t1 = time.time()
    total_parts = 10
    queue = Queue()

    processes =[Process(target=calculate_pi, args=(i, total_parts, queue)) for i in range(1, total_parts+1)]

    for p in processes:
        p.start()

    results = []
    for p in processes:
        p.join()
        results.append(queue.get())

    t2 = time.time()

    print("pi:", sum(results) * 4)
    print("elapsed time:", t2 - t1)

3.2 示例2 - 多进程下载文件

基于multiprocessing模块，使用多进程下载文件。示例如下：

from multiprocessing import Process, Queue
import os
import requests
import time

def download_file(url, file_path):
    response = requests.get(url, stream=True)
    with open(file_path, 'wb') as f:
        for chunk in response.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)

if __name__ == '__main__':
    urls = [
        'http://mirror.centos.org/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-Everything-2009.iso',
        'http://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso',
        'https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.9.0-amd64-netinst.iso',
        'https://www.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/13.0/FreeBSD-13.0-RELEASE-amd64-disc1.iso'
    ]
    file_names = [os.path.basename(url) for url in urls]

    t1 = time.time()

    queue = Queue()
    processes = [Process(target=download_file, args=(url, file_name)) for url, file_name in zip(urls, file_names)]

    for p in processes:
        p.start()

    for p, file_name in zip(processes, file_names):
        p.join()
        print(file_name, "finished downloading.")

    t2 = time.time()

    print("total elapsed time:", t2 - t1)

以上是两个使用多进程的示例，分别演示了计算圆周率和多进程下载文件的场景。适当使用多进程，可以提高Python程序的运行效率。