In Python programming, multiprocessing is a powerful parallel processing technique that can significantly improve program performance and efficiency. multiprocessing module in the Python standard library provides two main ways of multiprocessing: Process and Pool. in this article, we will introduce in detail the use of these two ways, features and applications in real projects. This article describes in detail how to use these two methods, their features, and their application in real projects.
Introduction to Process and Pool
1、 Process
The Process class is a core class in the multiprocessing module used to create a separate process. Each Process object can execute a function or method to perform tasks in a separate process.
2、 Pool
The Pool class is another important class in the multiprocessing module for creating a pool of processes that can perform multiple tasks in parallel.The Pool class provides various methods to manage the processes in the pool, such as apply(), map(), apply_async(), and so on.
usage example
1. Use the Process class
Here is a simple example showing how to create and start a process using the Process class:
import multiprocessing
import time
def worker(name):
print(f"Worker {name} is starting.")
(2)
print(f"Worker {name} is exiting.")
if __name__ == "__main__":
p1 = (target=worker, args=("A",))
p2 = (target=worker, args=("B",))
()
()
()
()
In this example, a worker function is defined that will perform some tasks in the process. Then two Process objects are created, each specifying the worker function as the target function and passing in different parameters. Finally, the process is started by calling the start() method and waiting for the process execution to complete with the join() method.
2, using the Pool class
Here is a simple example showing how to create and use a process pool using the Pool class:
import multiprocessing
import time
def worker(name):
print(f"Worker {name} is starting.")
(2)
print(f"Worker {name} is exiting.")
if __name__ == "__main__":
pool = (processes=2)
(worker, ["A", "B"])
()
()
In this example, a process pool is created using the Pool class, specifying a maximum number of processes of 2. The map() method is then used to assign tasks to the processes in the pool for execution. Finally, the pool is closed by calling the close() method, and the join() method is called to wait for all the processes to finish executing.
application scenario
1. Parallel computing
When large-scale data processing or computationally intensive tasks are required, the use of multiple processes can significantly increase the speed at which a program runs. For example, assuming that the sum of the squares of a set of numbers needs to be calculated, the task can be assigned to multiple processes in parallel and the results can then be summarized.
Here is a simple example:
import multiprocessing
def square_sum(numbers):
return sum(x ** 2 for x in numbers)
if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
num_processes = 4
pool = (processes=num_processes)
chunk_size = len(numbers) // num_processes
chunks = [numbers[i:i+chunk_size] for i in range(0, len(numbers), chunk_size)]
results = (square_sum, chunks)
total_sum = sum(results)
print("Total square sum:", total_sum)
()
()
In this example, the list of numbers is divided into 4 sub-lists, then the process pool is used to calculate the sum of squares of each sub-list in parallel, and finally the results are aggregated to get the total sum of squares. This can greatly speed up the calculation, especially when the data size is large.
2. IO-intensive tasks
In tasks that require a large number of IO operations, such as file reads and writes, network requests, etc., the use of multiprocessing can avoid IO blocking and improve the response speed of the program. For example, assuming that multiple files need to be downloaded at the same time, the download task for each file can be assigned to different processes to execute in parallel, thus improving the download efficiency.
Here is a simple example:
import multiprocessing
import requests
def download_file(url, filename):
response = (url)
with open(filename, 'wb') as f:
()
print(f"Downloaded {filename} from {url}")
if __name__ == "__main__":
urls = [
("/", ""),
("/", ""),
("/", "")
]
num_processes = len(urls)
pool = (processes=num_processes)
(download_file, urls)
()
()
In this example, a list of tuples consisting of the URLs and filenames of the files to be downloaded is passed to the starmap() method, which then executes the download tasks in parallel using the process pool. This allows multiple files to be downloaded at the same time, improving download efficiency.
3 Parallel tasking
When a large number of mutually independent tasks need to be processed, multiple processes can be used to process these tasks in parallel. For example, assuming that a batch of image files need to be processed, including operations such as compression, resizing, watermarking, etc., the processing tasks for each image file can be assigned to different processes in parallel, thus increasing the processing speed.
Here is a simple example:
import multiprocessing
from PIL import Image
def process_image(filename):
img = (filename)
# Perform image processing operations such as compression, resizing, watermarking, etc.
((300, 300))
(f"processed_{filename}")
print(f"Processed {filename}")
if __name__ == "__main__":
filenames = ["", "", ""]
num_processes = len(filenames)
pool = (processes=num_processes)
(process_image, filenames)
()
()In this example, a list of image filenames to be processed is passed to the map() method, and then the image processing tasks are executed in parallel using the process pool. This allows multiple images to be processed at the same time, increasing processing speed.
summarize
This article introduces two types of multiprocessing in Python: Process and Pool, and demonstrates their basic usage through sample code. It also discusses the application scenarios of multiprocessing in real projects, including parallel computing, IO-intensive tasks and parallel task processing. Multi-processing is a powerful parallel processing technique in Python, which can improve the performance and efficiency of programs, especially when dealing with large-scale data or IO-intensive tasks with obvious advantages.
This article on Python multi-processing Process and Pool usage details of the article is introduced to this, more related Python multi-processing content, please search for my previous posts or continue to browse the following related articles I hope you will support me in the future!