Detailed explanation of various ways to implement timing tasks in Python

Preface

There are many ways to implement timing tasks in Python, from simple single-threaded timers to complex distributed task scheduling systems. The following is a detailed introduction to Python timing tasks, covering implementation methods, advantages and disadvantages, and scope of application in different scenarios.

1. Basic timing task tools

1. ()

()It is the simplest timer method, and the timing effect is achieved by pausing the program for a period of time.

Sample code

import time

while True:
    print("Execute tasks...")
    (5)  # Execute every 5 seconds

advantage

• Simple and easy to use, suitable for lightweight tasks.
• No additional dependency library is required.

shortcoming

• Blocks the main thread and cannot perform other tasks at the same time.
• Complex scheduling rules (such as running at some point every day) are not supported.

Applicable scenarios

• Simple loop tasks do not require complex scheduling logic.

2. 

is a class in the Python standard library used to execute a function after a specified time.

Sample code

import threading

def task():
    print("Execute tasks...")

# Create a timer and execute the task function after 5 secondstimer = (5, task)
()

advantage

• Non-blocking, can be run in the background.
• Suitable for one-time scheduled tasks.

shortcoming

• Not suitable for cyclical tasks.
• If the task is not completed, it may cause resource leakage.

Applicable scenarios

• One-time delay task.

2. Advanced timed task tools

3. schedule library

scheduleIt is a lightweight third-party library that supports flexible task scheduling rules.

Install

pip install schedule

Sample code

import schedule
import time

def job():
    print("Execute tasks...")

# Execute every 10 seconds(10).(job)

# Execute once every day at 10:30().("10:30").do(job)

while True:
    schedule.run_pending()  # Check if there are tasks that need to be executed    (1)  # Avoid excessive CPU usage

advantage

• Supports multiple scheduling rules (seconds, minutes, hours, days, etc.).
• Simple syntax and easy to get started.

shortcoming

• Single threaded operation is not suitable for high concurrency scenarios.
• Distributed task scheduling is not supported.

Applicable scenarios

• Lightweight periodic task scheduling.

4. APScheduler library

APSchedulerIt is a powerful task scheduling library that supports complex scheduling rules and persistence.

Install

pip install apscheduler

Sample code

from  import BlockingScheduler

def job():
    print("Execute tasks...")

# Create a schedulerscheduler = BlockingScheduler()

# Add a task: execute every 5 secondsscheduler.add_job(job, 'interval', seconds=5)

# Add tasks: execute once every day at 10:30scheduler.add_job(job, 'cron', hour=10, minute=30)

# Start the scheduler()

Features

• Scheduling type：
  • interval: Execute tasks at fixed time intervals.
  • cron: Linux-likecronExpressions support complex time rules.
  • date: Perform a task on a specific date and time.
• Storage backend: Supports various storage methods such as memory and database, suitable for persistent tasks.
• Actuator：
  • ThreadPoolExecutor: Multi-threaded tasks.
  • ProcessPoolExecutor: Multi-process execution of tasks.

advantage

• Powerful and supports complex scheduling rules.
• Support task persistence, and after restarting, unfinished tasks can be continued.
• Strong scalability and suitable for production environments.

shortcoming

• Relatively complex and high learning cost.
• For simple tasks, it may seem too heavyweight.

Applicable scenarios

• Tasks that require complex scheduling rules.
• Tasks that require persistence or high reliability.

3. Distributed task scheduling tool

5. Celery

Celery is a distributed task queue framework that is widely used in large-scale distributed systems.

Install

pip install celery

Sample code

from celery import Celery
from datetime import timedelta

app = Celery('tasks', broker='redis://localhost:6379/0')

@
def job():
    print("Execute tasks...")

# Configure timing tasks.beat_schedule = {
    'run-every-10-seconds': {
        'task': '',
        'schedule': timedelta(seconds=10),
    },
}

if __name__ == '__main__':
    ()

Features

• Broker: Supports Redis, RabbitMQ and other message queues as task middleware.
• Worker: Multiple Workers can handle tasks in parallel.
• Beat: Built-in timed task scheduler, supports periodic tasks.

advantage

• Distributed architecture, suitable for large-scale systems.
• Supports asynchronous tasks and timing tasks.
• Strong scalability and supports dynamic addition of tasks.

shortcoming

• The configuration is complicated and a message queue is required.
• The learning curve is steep.

Applicable scenarios

• Task scheduling in high concurrency, distributed systems.
• Tasks that require asynchronous processing.

6. Airflow

Airflow is an open source workflow management platform designed for complex task scheduling.

Install

pip install apache-airflow

Sample code

from airflow import DAG
from .python_operator import PythonOperator
from datetime import datetime, timedelta

def job():
    print("Execute tasks...")

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 1, 1),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'example_dag',
    default_args=default_args,
    schedule_interval=timedelta(seconds=10),
)

task = PythonOperator(
    task_id='example_task',
    python_callable=job,
    dag=dag,
)

if __name__ == "__main__":
    ()

Features

• DAG（Directed Acyclic Graph）: Define dependencies between tasks through DAG.
• Web UI: Provides a visual task monitoring interface.
• Plugin support: Supports a variety of plug-ins and is highly extensible.

advantage

• Strong workflow management capabilities.
• Visual interface for easy monitoring and debugging.
• Supports complex dependencies.

shortcoming

• Complex configuration and high startup cost.
• The learning curve is steep.

Applicable scenarios

• Complex ETL process.
• Data pipeline and batch task scheduling.

4. System-level timing tasks

7. Cron (Linux system timing task)

Cron is a timing task tool that comes with Linux systems and can be configured through the command line.

Example

editcrontabdocument:

crontab -e

Add the following:

*/5 * * * * /usr/bin/python3 /path/to/

explain

• */5: Perform every 5 minutes.
• /usr/bin/python3: Specify the Python interpreter path.
• /path/to/: script path.

advantage

• System-level timing tasks, stable and reliable.
• Not dependent on Python libraries.

shortcoming

• The configuration is not intuitive enough and error-prone.
• Complex task scheduling is not supported.

Applicable scenarios

• Simple timing tasks at the system level.

5. Summary and comparison

6. Choose suggestions

• Simple task: If it is just a simple periodic task, it is recommended to use itscheduleorAPScheduler。
• Distributed tasks: If distributed scheduling is required, it is recommended to useCelery。
• Complex workflow: If complex task dependencies are involved, it is recommended to useAirflow。
• System level: If it is running on Linux system, it is recommended to use it directlyCron。

This is the end of this article about various ways to implement timing tasks in Python. For more related content on Python implementation timing tasks, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!