SoFunction
Updated on 2024-12-16

Python tutorial on handling concurrency using period objects

learning from The Fluent Python

1.

import os
import time
import sys
import requests
POP20_CC = ('CN IN US ID BR PK NG BD RU JP ' 'MX PH VN ET EG DE IR TR CD FR').split()
BASE_URL = '/data/flags'
DEST_DIR = './'
def save_flag(img, filename):  # Save images
    path = (DEST_DIR, filename)
    with open(path, 'wb') as fp:
        (img)
def get_flag(cc):  # Get the image
    url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=())
    resp = (url)
    return 
def show(text):  # Print information
    print(text, end=' ')
    ()
def download_many(cc_list):
    for cc in sorted(cc_list):
        image = get_flag(cc)  # Get
        show(cc)  # Print
        save_flag(image, () + '.gif')  # Save
    return len(cc_list)
def main(download_many):
    t0 = ()
    count = download_many(POP20_CC)
    elapsed = () - t0
    msg = '\n{} flags downloaded in {:.2f}s'
    print((count, elapsed))  # Timing information
# ---- Implementing multithreaded downloads using classes
from concurrent import futures
MAX_WORKERS = 20  # Maximum number of threads used
def download_one(cc):
    image = get_flag(cc)
    show(cc)
    save_flag(image, () + '.gif')
    return cc
def download_many_1(cc_list):
    workers = min(MAX_WORKERS, len(cc_list))
    with (workers) as executor:
        # Instantiate the ThreadPoolExecutor class with the number of threads working;
        # executor.__exit__ method calls the (wait=True) method.
        # It'll block the thread until all the threads have finished #
        res = (download_one, sorted(cc_list))
        The # download_one function is called concurrently in multiple threads;
        # The map method returns a generator, so you can iterate over it and get the values returned by each function.
    return len(list(res))
if __name__ == '__main__':
    # main(download_many) # 24 seconds
    main(download_many_1)  # 3 unit of angle or arc equivalent one sixtieth of a degree

2. Futures

You should not normally create your own periodicals

Can only be instantiated by the concurrency framework (or asyncio) Reason: A term object represents something that will happen eventually, and its execution has already been scheduled. Therefore, instances are created only when something is scheduled to be handed off to a subclass

For example, the argument to the () method is a callable object, and a call to this method will schedule the incoming callable object and return a term

def download_many_2(cc_list):
    cc_list = cc_list[:5]
    with (max_workers=3) as executor:
        to_do = []
        for cc in sorted(cc_list):
            future = (download_one, cc)
            # method schedules the execution time of the callable object.
            # and then returns an object indicating the pending operation.
            to_do.append(future) # Stored issues
            msg = 'Scheduled for {}: {}'
            print((cc, future))
        results = []
        for future in futures.as_completed(to_do):
            # as_completed function outputs an issue at the end of an issue run
            res = () # Results of acquisitions
            msg = '{} result: {!r}'
            print((future, res))
            (res)
    return len(results)
exports:
Scheduled for BR: <Future at 0x22da99d2d30 state=running>
Scheduled for CN: <Future at 0x22da99e1040 state=running>
Scheduled for ID: <Future at 0x22da99e1b20 state=running>
Scheduled for IN: <Future at 0x22da99ec520 state=pending>
Scheduled for US: <Future at 0x22da99ecd00 state=pending>
CN <Future at 0x22da99e1040 state=finished returned str> result: 'CN'
BR <Future at 0x22da99d2d30 state=finished returned str> result: 'BR'
ID <Future at 0x22da99e1b20 state=finished returned str> result: 'ID'
IN <Future at 0x22da99ec520 state=finished returned str> result: 'IN'
US <Future at 0x22da99ecd00 state=finished returned str> result: 'US'
5 flags downloaded in 3.20s

3. Blocking I/O and GIL

The CPython interpreter is not inherently thread-safe, so there is a global interpreter lock (GIL) that allows only one thread to be used to execute Python bytecode at a time. As a result, a Python process cannot normally use more than one CPU core at a time.

All functions in the standard library that perform blocking I/O operations release GIL while waiting for the operating system to return the result. This means that multithreading is possible at the Python language level, and I/O-intensive Python programs can benefit from it: while a Python thread waits for a network response, the blocking I/O function releases GIL and runs another thread (network downloads, file reads and writes are IO-intensive).

4. Use of modules to initiate processes

This module achieves true parallel computing because it uses the ProcessPoolExecutor class to distribute work among multiple Python processes. So if you need to do CPU-intensive processing, use this module to bypass the GIL and utilize all available CPU cores!

Click to view: Process, thread conceptual differences

Using modules makes it particularly easy to convert thread-based solutions to process-based ones.

The value of ProcessPoolExecutor is in CPU-intensive jobs.

Above is the details of python using period objects to deal with concurrency tutorial, more information about python period objects to deal with concurrency please pay attention to my other related articles!