Instructions for using the python multiprocess execution method apply

Introduction to apply_async

python in the same thread multiple times to execute the same method, the method execution time-consuming and each time the execution process and the results do not affect each other, if only in the main process, the efficiency will be very low, so the use of (processes=n) and its apply_async () method to improve the parallelism of the program to improve the program's execution efficiency, in which processes=n is the number of processes in parallel execution of the program.

Succinct code for use of apply_async

import multiprocessing
#method is a method that is called multiple times
def method(param):
 pass
if __name__ == '__main__':
 pool = (processes=5)
 params= ['param1', 'param2', 'param3', 'param4', 'param5']
 for param in params:
  pool.apply_async(method, args=(param, )) 
 ()

Usage Summary:

apply_async is asynchronous non-blocking, do not have to wait for the current process execution is complete, at any time to follow up on the operating system scheduling to process switching, that is, parallel execution of multiple processes, to improve the efficiency of program execution.

Addendum: Documenting the map and apply_async methods for python multiprocessing pools

Problems encountered

When I was learning python multiprocessing, I encountered problems with methods running on the process receiving multiple arguments and multiple results, and now after studying it I'll summarize it here

() multiparameter task

Passing in a map method with more than one parameter does not have the desired effect, like the following

def job(x ,y):
 return x * y
if __name__ == "__main__":
 pool = ()
 res = (job, 2, 3)
 print res

So it can only be done by wrapping the method with multiple parameters and running the wrapped method in the process as follows

def job(x ,y):
 return x * y
def job1(z):
 return job(z[0], z[1])
if __name__ == "__main__":
 pool = ()
 res = (job1, [(2, 3), (3, 4)])
 print res

This has the effect of passing multiple parameters

ps: if you need to get more than one result you can pass in more than one tuple in a list

Pool.apply_async() outputs multiple iteration results

When using methods that receive multiple arguments using the apply_async() method, it is sufficient to define multiple arguments normally in the task method, with the arguments passed in as a tuple

But passing multiple values to the apply_async() method to get multiple iteration results will report an error because the method can only receive one value, so you can put the method into a list generator as follows

def job(x):
 return x * x
if __name__ == "__main__":
 pool ()
 res = [pool.apply_async(target=job, (i,)) for i in range(3)]
 print [() for r in res]

python 3 provides the methods startmap and startmap_async.