SoFunction
Updated on 2024-11-16

Detailed method of using multiprocessing in python

multiprocessing

multiprocessing is a module in the Python standard library for implementing multiprocess programming. It provides a simple and efficient way to utilize the power of multi-core processors to speed up program execution and increase system throughput by executing tasks in multiple processes simultaneously.

Here are some common operations using the multiprocessing module:

  • Create the process:

Use the Process class to create a process object, specifying the function or method to be executed.
Use the start() method of the Process class to start the process.

  • Inter-process communication:
    • Use the Queue class to implement inter-process queue communication.
    • Use the Pipe class to implement inter-process pipeline communication.
    • Data sharing between processes is implemented using shared memory (Value and Array).
  • Process Management:
    • Use the join() method of the Process class to wait for the process to finish.
    • Terminate the process using the terminate() method of the Process class.

process

​(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

Parameter Description:

  • group defaults to None (not currently used)
  • target represents the object of the call, i.e., the task executed by the child process.
  • name is the process name
  • args calls the tuple of positional arguments of the object, args=(value1, value2, ...)
  • kwargs calls the object's dictionary, kwargs={key1:value1, key2:value2, ...}
  • daemon indicates if the process is a daemon, boolean value

Methodology Introduction:

  • () starts the process and calls the run() method in the child process
  • () method that runs when the process is started and must be implemented when customizing the
  • () Force the termination of the process without cleanup, if the Process creates a child process, it will cause the process to become a zombie process
  • () Blocking the process so that the master process waits for it to terminate
  • () Same as terminate()
  • Process.is_alive() determine whether the process is alive, if alive, return True
  • () closes the process object and cleans up resources, returning an error if the process is still running

()

() is a class in the multiprocessing module that implements a queue (Queue) for inter-process communication. It provides a secure way for data to be shared between multiple processes. The main features of the () class include:

  • Security: () is thread-safe and can be used in multiple processes simultaneously without worrying about data contention or inconsistency issues.
  • First-In-First-Out (FIFO): It follows the FIFO principle, which ensures that elements added to the queue are removed in the order in which they were added.
  • Blocking operations: when the queue is empty, using the get() method to get elements from the queue will block the process until there are available elements in the queue. When the queue is full, using the put() method to add elements to the queue will block the process until the queue has free space.
import multiprocessing
def worker(queue):
    data = ()  # Getting data from the queue
    # Processing data
if __name__ == '__main__':
    queue = ()
    process = (target=worker, args=(queue,))
    ()
    (data)  # Add data to the queue
    ()

In the above example, a () object is first created and then that queue object is passed as an argument to the worker() function of the child process. In the child process, the get() method is used to get data from the queue for processing. In the master process, use the put() method to add data to the queue. The use of () enables inter-process communication and collaboration by allowing data to be passed securely between multiple processes. This is useful for scenarios such as parallel computing, task distribution and processing.

Take the previous example of point and click bandwidth

Seven nodes are not repeated to take two, C72 is also 21 groups, that is, 21 cycles, each cycle sleep5 seconds, serial is 21x5 = 105 seconds, 21 threads in parallel for 5 seconds.

import multiprocessing
import time
import random
def get_oobw_parallel(node_names):
    results = []
    for i in range(0, len(node_names) - 1):
        for j in range(i + 1, len(node_names)):
            result = get_oobw(node_names[i], node_names[j])
            (result)
    return results
def get_oobw(node_name1, node_name2):
    # Execute the logic of get_oobw
    # ...
    (5)
    latency, bandwidth = round((100.0, 200.0), 4), round((100.0, 200.0), 4)
    result = (node_name1, node_name2, latency, bandwidth)
    return result
start_time = ()
node_names = ["cn1", "cn2", "cn3", "cn4", "cn5", "cn6", "cn7"]  # Fill in your list of node names
results = get_oobw_parallel(node_names)
for result in results:
    node_name1, node_name2, latency, bandwidth = result
    print(node_name1, node_name2, latency, bandwidth)
end_time = ()
# Calculate execution time
execution_time = end_time - start_time
print("Program execution time:", execution_time, "Seconds.")

import multiprocessing
import time
import random
def get_oobw_parallel(node_names):
    results = []
    processes = []
    result_queue = ()
    for i in range(0, len(node_names) - 1):
        for j in range(i + 1, len(node_names)):
            process = (target=get_oobw, args=(node_names[i], node_names[j], result_queue))
            ()
            (process)
    for process in processes:
        ()
    while not result_queue.empty():
        result = result_queue.get()
        (result)
    return results
def get_oobw(node_name1, node_name2, result_queue):
    # Execute the logic of get_oobw
    # ...
    (5)
    latency, bandwidth = round((100.0, 200.0), 4), round((100.0, 200.0), 4)
    result = (node_name1, node_name2, latency, bandwidth)
    result_queue.put(result)
    # return latency, bandwidth
start_time = ()
node_names = ["cn1", "cn2", "cn3", "cn4", "cn5", "cn6", "cn7"]  # Fill in your list of node names
results = get_oobw_parallel(node_names)
for result in results:
    node_name1, node_name2, latency, bandwidth = result
    print(node_name1, node_name2, latency, bandwidth)
end_time = ()
# Calculate execution time
execution_time = end_time - start_time
print("Program execution time:", execution_time, "Seconds.")

to this article on the use of python multiprocessing to this article, more related to the use of python multiprocessing content, please search for my previous articles or continue to browse the following related articles I hope you will support me in the future more!