Detailed explanation of the code for batch downloading of Python files

Overview

This tool is a batch download file script written in Python. It reads the download task list from the configuration file and downloads the file in parallel, while displaying the download progress. This tool uses the requests library to make HTTP requests, uses the tqdm library to display download progress, and uses the configparser library to read configuration files.

rely

Before using the tool, make sure that the following libraries are installed:

requests
tqdm

You can install the required libraries using the following command:

pip install requests tqdm

Configuration File

Create a configuration file (e.g.), which contains a list of download tasks. The configuration file uses the INI format, and each download task has a separate section, each section containsurlandpathTwo fields.

Sample configuration file：

[file1]
url = /
path = downloads/

[file2]
url = /
path = downloads/

[file3]
url = /
path = downloads/

Run the script

Save the following code asbatch_download.py：

import os
import requests
from  import ThreadPoolExecutor
from tqdm import tqdm
import configparser
import argparse

def download_file(task):
    """
     Download a single file and save it to a specified path and display the download progress
     """
    url = task['url']
    path = task['path']
    
    try:
        ((path), exist_ok=True)
        
        with (url, stream=True) as r:
            r.raise_for_status()
            total_size = int(('content-length', 0))
            with open(path, 'wb') as f, tqdm(
                total=total_size, unit='B', unit_scale=True, desc=(path)
            ) as pbar:
                for chunk in r.iter_content(chunk_size=8192):
                    (chunk)
                    (len(chunk))
        
        print(f'{path} Download completed')
    except Exception as e:
        print(f'download {url} fail: {e}')

def read_config(config_path):
    """
     Read the download task list from the configuration file
     """
    config = ()
    (config_path)
    tasks = []
    for section in ():
        url = config[section]['url']
        path = config[section]['path']
        ({'url': url, 'path': path})
    return tasks

def main(config_path):
    """
     Main function, read configuration files and download files in parallel
     """
    tasks = read_config(config_path)
    
    with ThreadPoolExecutor(max_workers=5) as executor:
        (download_file, tasks)

if __name__ == '__main__':
    parser = (description='Batch Download File Tool')
    parser.add_argument('config', type=str, help='Configuration file path')
    args = parser.parse_args()
    
    main()

How to use

Create a configuration file: Created according to the above exampleFile and modify the download task as needed.
Run the script: Run the following command on the command line to specify the configuration file path:

python batch_download.py

Code function description

Import library

import os
import requests
from  import ThreadPoolExecutor
from tqdm import tqdm
import configparser
import argparse

download_fileFunction: Download a single file and save it to the specified path, and display the download progress.

def download_file(task):
    url = task['url']
    path = task['path']
    
    try:
        ((path), exist_ok=True)
        
        with (url, stream=True) as r:
            r.raise_for_status()
            total_size = int(('content-length', 0))
            with open(path, 'wb') as f, tqdm(
                total=total_size, unit='B', unit_scale=True, desc=(path)
            ) as pbar:
                for chunk in r.iter_content(chunk_size=8192):
                    (chunk)
                    (len(chunk))
        
        print(f'{path} Download completed')
    except Exception as e:
        print(f'download {url} fail: {e}')

read_configFunction: Read the download task list from the configuration file.

def read_config(config_path):
    config = ()
    (config_path)
    tasks = []
    for section in ():
        url = config[section]['url']
        path = config[section]['path']
        ({'url': url, 'path': path})
    return tasks

mainFunction: Read the configuration file and download the file in parallel.

def main(config_path):
    tasks = read_config(config_path)
    
    with ThreadPoolExecutor(max_workers=5) as executor:
        (download_file, tasks)

Command line parameter analysis:useargparseResolve command line parameters and specify the configuration file path.

if __name__ == '__main__':
    parser = (description='Batch Download File Tool')
    parser.add_argument('config', type=str, help='Configuration file path')
    args = parser.parse_args()
    
    main()

Things to note

Please make sure that the URL in the configuration file is a valid download link.
If an error occurs during the download process, the script will output the corresponding error message.

This is the introduction to this article about the detailed explanation of the code for Python batch download files. For more related contents of Python batch download files, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!