SoFunction
Updated on 2024-11-10

An in-depth look at an example of multiprocess module usage in Python

What is multiprocessing

Multiprocessing is a concept in computer programming that can also be described as a technique that can be used to achieve parallelism and execute tasks concurrently using multiple CPU cores or processors. This technique is particularly useful for CPU-constrained tasks, where parallel execution can improve the performance of a program.Python provides themultiprocessmodule to handle multiple processes.

Multiprocessing is a computer programming and execution paradigm that involves the simultaneous execution of multiple processes or programs by utilizing multiple CPU cores or processors on a single computer or multiple computers, depending on the problem or setup we are working on.

Multiprocessing is a form of parallelism designed to improve program performance and efficiency by allowing tasks to execute concurrently, thereby making efficient use of available computing resources.

Several components of multiprocessing are discussed below.

  • 1. Processes: Each task or unit of work or process is encapsulated in a separate process, which is an independent unit of program execution. These processes can run concurrently and independently of each other.

  • 2. Parallel execution: Multiple processes (e.g., computation, data processing or multiple request processing) can be executed in parallel through multiple processes, thus performing tasks simultaneously.

  • 3. Resource utilization: multiprocessing is a technique for efficiently utilizing the processing power of modern computers, especially multi-core processors (e.g., quad-core or octa-core), by assigning tasks to multiple CPU cores.

  • 4. Isolation: In multiprocessing, processes are isolated from each other. This means that the operation or failure of one process does not directly affect other processes. This isolation of processes enhances the stability of the system.

  • 5. Inter-process communication (IPC): Inter-process communication mechanisms (e.g., shared memory, pipes, queues and sockets) allow processes to exchange data and coordinate their activities.

  • 6. Scalability: Multiprocessing can be scaled to utilize the power of many processors or even distributed computing clusters to solve computationally intensive problems.

Difference between multithreading and multiprocessing

Imagine you and two assistants cooking in the kitchen. Multithreading is like having those two assistants in the same kitchen completing a task. This would definitely speed up the process, with two people completing a task at the same time. These assistants working in the same kitchen can share the same ingredients, can communicate with each other, and can easily share tasks. But it can also get messy, and they may bump into each other while working, thus ruining all progress.

Multi-processing is like having two helpers, working in two different kitchens, using their own ingredients and personal space. These helpers can work independently without interfering with each other, but if they need to share ingredients or talk, they must use the phone.

From this example, we can learn that multithreading is a good technique for tasks where the assistant does not need to move around a lot because the processor does not need to use a lot of resources. However multi-processing is better suited for tasks that require a lot of movement, or tasks that technically need to use a lot of the kernel's resources at the same time.

If we define it in a single sentence, multithreading is primarily about achieving concurrency and allowing tasks to overlap in time, while multiprocessing is about achieving true parallelism, as each process runs independently on a different CPU core or processor.

The first step in executing multiprocessing in python is to import the module.

import multiprocessing

We are going to run a program using multiple processes. Let's create a sample user-defined program and see how it works:

def test():
  print("This is my sample program")

test()

# Output:
# This is my sample program

To call this program using multiple processes, we will use the following code.

import multiprocessing

def test01():
  print("This is a sample multiprocessing program")

if __name__ == "__main__":
  m = (target=test01)
  print("this is my main program")

  # start the program
  ()

  # stop/terminate the program and release the resource
  ()

In the code above, we have passed themain function uses multiprocessing to call the program we want to work with. And it makes sure that the code only runs when the script is executed directly, not imported as a module.

We can also use multiprocessing in other functions to perform some logical operations, an example of which is given below.

import multiprocessing

def square(n):
  return n**2

if __name__ == '__main__':
  with (processes=5) as pool:
    out = (square, [12,2,3,5,23,14,26,8,4,6])
    print(out)

# Output:
# [144, 4, 9, 25, 529, 196, 676, 64, 16, 36]

Parallel Execution Functions

Running multiple processes

We can run multiple processes on multiple kernels using multiprocessing techniques. We create and start multiple processes, each running the function "workers" with a set of arguments (as shown in the code below). We use "args" to specify the list of arguments to be used by the function.

In the following code, we use the concept of multiprocessing, using 4 processes to handle the program. To do this, we use the multiprocessing module in python'sProcess Methods.

def workers(num):
    print(f"Worker number {num}")

if __name__ == "__main__":
    processes = []
    for i in range(4):
        process = (target=workers, args=(i,))
        (process)
        ()

    for process in processes:
        ()

Output:
Worker number 0
Worker number 1
Worker number 2
Worker number 3

Sharing data between processes

utilization

is a class in the Python programming language that consists of themultiprocessing module, which we use to perform multiprocessing in python. Here's theQueue Used to facilitate communication and data exchange between different processes in a multi-process program.

Some of the main functions and uses are listed below:

  • - Inter-process communication: It allows multiple processes to exchange data securely by using a common data structure (queue). Each process can put data into the queue or retrieve data from it.

  • - Thread-safe: It is designed in such a way that it is thread-safe and process-safe, which means that it can be used safely in a multi-threaded or multi-process environment without race conditions or data corruption.

  • - The FIFO queue always follows the FIFO principle, which means that the first item in the queue will also be the first item out.

  • - Blocking operation: When a process attempts to retrieve an item from an empty queue or insert an item into a filled queue, the operation blocks until conditions change. This process/behavior is useful for inter-process synchronization.

When using Python for parallel or concurrent processing, the is a very useful tool, especially in situations where multiple processes need to share and coordinate data.

The following cell gives an example of how to use Example of a multi-process queue: a producer process produces data or puts data into a queue, and a consumer process consumes data from the queue. A multiprocess queue ensures the secure exchange of data between two processes.

import multiprocessing
import time
#defining our function we want to apply multiprocessing on
#01 the producer function to add elements in the queue
def producer(q):
  for item in range(5):
    (item)
    print(f"Produced:  {item}")
#02 consumer function to get elements from the queue
def consumer(q):
  while True:
    item = ()
    if item is None:
      break
    print(f"Consumed: {item}")
if __name__ == "__main__":
  #creating a multiprocessing queue
  q = ()
  #creating the producer and consumer processes
  producer_process = (target=producer, args=(q,))
  consumer_process = (target=consumer, args=(q,))
  #starting the processes
  producer_process.start()
  consumer_process.start()
  #finish the producer, signal the consumer to exit
  producer_process.join()
  (None) #signaling the consumer about no more data in the queue
  consumer_process.join()
Output:
Produced:  0
Produced:  1
Consumed: 0Produced:  2
Produced:  3Consumed: 1
Produced:  4
Consumed: 2
Consumed: 3
Consumed: 4

process pool

Using Classes

is another class provided by the multiprocessing module in python. It is used to create and manage a pool of processes that can perform tasks in parallel. This class simplifies the process of parallelizing tasks and assigning them to multiple processes.

Below is a list of some of the features and uses.

  • - Used to implement parallel processing in python. It allows us to execute multiple tasks concurrently, taking full advantage of multi-core CPUs and potentially improving the performance of CPU-related operations.

  • - Task assignment: we can use Assign tasks in a pool of work processes. It abstracts the management of work processes and makes parallel work easier.

  • - Resource Management: The process pool automatically manages the creation and destruction of worker processes based on the available CPU cores. We can specify the number of worker processes to be used and the pool will handle the rest.

  • - Load balancing: A process pool efficiently distributes tasks among the worker processes, ensuring that each worker process receives an equal amount of work. This load balancing is essential for optimizing parallel processing.

  • - Results collection: We can collect results from work processes. This is particularly useful when tasks produce results that need to be aggregated or further processed.

  • - Context Manager: It can also act as a context manager, ensuring that pools are properly closed and working processes terminated when no longer needed.

The following cell shows a sample code that uses a simple function that returns the square of a given number to multiprocess a process.

def square(x):
  return x**2 
if __name__=='__main__':
  #create a multiprocessing pool with 4 worker processes
  with (processes=4) as pool:
    #distribute the tasks among worker processes and collect results
    results = (square, [2,5,3,2,1,7,8,5,6,2,2,3])
    #print the results
    print(results)
Output:
[4, 25, 9, 4, 1, 49, 64, 25, 36, 4, 4, 9]

The above is an in-depth exploration of Python multi-process usage examples of details, more information about Python multi-process please pay attention to my other related articles!