learning from The Fluent Python
1.
import os import time import sys import requests POP20_CC = ('CN IN US ID BR PK NG BD RU JP ' 'MX PH VN ET EG DE IR TR CD FR').split() BASE_URL = '/data/flags' DEST_DIR = './' def save_flag(img, filename): # Save images path = (DEST_DIR, filename) with open(path, 'wb') as fp: (img) def get_flag(cc): # Get the image url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=()) resp = (url) return def show(text): # Print information print(text, end=' ') () def download_many(cc_list): for cc in sorted(cc_list): image = get_flag(cc) # Get show(cc) # Print save_flag(image, () + '.gif') # Save return len(cc_list) def main(download_many): t0 = () count = download_many(POP20_CC) elapsed = () - t0 msg = '\n{} flags downloaded in {:.2f}s' print((count, elapsed)) # Timing information # ---- Implementing multithreaded downloads using classes from concurrent import futures MAX_WORKERS = 20 # Maximum number of threads used def download_one(cc): image = get_flag(cc) show(cc) save_flag(image, () + '.gif') return cc def download_many_1(cc_list): workers = min(MAX_WORKERS, len(cc_list)) with (workers) as executor: # Instantiate the ThreadPoolExecutor class with the number of threads working; # executor.__exit__ method calls the (wait=True) method. # It'll block the thread until all the threads have finished # res = (download_one, sorted(cc_list)) The # download_one function is called concurrently in multiple threads; # The map method returns a generator, so you can iterate over it and get the values returned by each function. return len(list(res)) if __name__ == '__main__': # main(download_many) # 24 seconds main(download_many_1) # 3 unit of angle or arc equivalent one sixtieth of a degree
2. Futures
You should not normally create your own periodicals
Can only be instantiated by the concurrency framework (or asyncio) Reason: A term object represents something that will happen eventually, and its execution has already been scheduled. Therefore, instances are created only when something is scheduled to be handed off to a subclass
For example, the argument to the () method is a callable object, and a call to this method will schedule the incoming callable object and return a term
def download_many_2(cc_list): cc_list = cc_list[:5] with (max_workers=3) as executor: to_do = [] for cc in sorted(cc_list): future = (download_one, cc) # method schedules the execution time of the callable object. # and then returns an object indicating the pending operation. to_do.append(future) # Stored issues msg = 'Scheduled for {}: {}' print((cc, future)) results = [] for future in futures.as_completed(to_do): # as_completed function outputs an issue at the end of an issue run res = () # Results of acquisitions msg = '{} result: {!r}' print((future, res)) (res) return len(results)
exports: Scheduled for BR: <Future at 0x22da99d2d30 state=running> Scheduled for CN: <Future at 0x22da99e1040 state=running> Scheduled for ID: <Future at 0x22da99e1b20 state=running> Scheduled for IN: <Future at 0x22da99ec520 state=pending> Scheduled for US: <Future at 0x22da99ecd00 state=pending> CN <Future at 0x22da99e1040 state=finished returned str> result: 'CN' BR <Future at 0x22da99d2d30 state=finished returned str> result: 'BR' ID <Future at 0x22da99e1b20 state=finished returned str> result: 'ID' IN <Future at 0x22da99ec520 state=finished returned str> result: 'IN' US <Future at 0x22da99ecd00 state=finished returned str> result: 'US' 5 flags downloaded in 3.20s
3. Blocking I/O and GIL
The CPython interpreter is not inherently thread-safe, so there is a global interpreter lock (GIL) that allows only one thread to be used to execute Python bytecode at a time. As a result, a Python process cannot normally use more than one CPU core at a time.
All functions in the standard library that perform blocking I/O operations release GIL while waiting for the operating system to return the result. This means that multithreading is possible at the Python language level, and I/O-intensive Python programs can benefit from it: while a Python thread waits for a network response, the blocking I/O function releases GIL and runs another thread (network downloads, file reads and writes are IO-intensive).
4. Use of modules to initiate processes
This module achieves true parallel computing because it uses the ProcessPoolExecutor class to distribute work among multiple Python processes. So if you need to do CPU-intensive processing, use this module to bypass the GIL and utilize all available CPU cores!
Click to view: Process, thread conceptual differences
Using modules makes it particularly easy to convert thread-based solutions to process-based ones.
The value of ProcessPoolExecutor is in CPU-intensive jobs.
Above is the details of python using period objects to deal with concurrency tutorial, more information about python period objects to deal with concurrency please pay attention to my other related articles!