1. Introduction to the backoff library
backoff is a Python library for implementing the retry mechanism, which automatically retry failed operations through exponential backoff or other policies. It simplifies the writing of retry logic through decorators, and is suitable for error-prone scenarios such as network requests and database connections.
Core functions
- Automatic retry: Automatically retry when the function throws a specified exception.
- Backoff strategy: Supports multiple retry interval strategies (such as exponential backoff and fixed interval).
- Flexible configuration: can set the maximum number of retry times, maximum time, custom retry conditions, etc.
2. The principle of on_exception decorator
backoff.on_exception is the core decorator of the backoff library, which is used to trigger a retry when a function throws a specified exception. Its working principle is as follows:
2.1 Core logic
Exception capture: When the decorated function throws the specified exception type, a retry is triggered.
Backoff policy: Calculate the retry interval time based on the configured policy (such as ).
Retry condition: Decide whether to continue retry based on max_tries or max_time.
Retry execution: After waiting for the specified interval, call the function again. If it still fails in the end, an exception is thrown.
2.2 Core parameters
@backoff.on_exception(
wait_gen=, # Backoff strategy exception=(Exception,), #Exception type (tuple) that needs to be caught max_tries=3, # Maximum number of retries (including first call) max_time=30, # Maximum retry time (seconds) jitter=None, # Random jitter (prevent concurrent requests from retrying simultaneously) giveup=lambda e: False, # Optional: Customize the conditions for giving up retry logger=None, # Logger on_backoff=None, # Callback function during retry on_giveup=None, # Give up the callback function during retry factor=1, # Multiple factor of backoff strategy **kwargs #Other parameters)
3. Common back-off strategies
Backoff provides multiple backoff strategies, specified through the wait_gen parameter:
3.1 Exponential backoff ()
Policy: The interval of each retry is factor * 2^(n-1), where n is the number of retryes.
Applicable scenarios: temporary failures such as network requests, API calls, etc.
Example:
@backoff.on_exception(, RequestException, max_tries=5)
def fetch_data(url):
return (url)
3.2 Fixed interval ()
Policy: The interval for each retry is fixed to the value specified by the interval parameter.
Applicable scenarios: Scenarios that require stable intervals (such as retry every 10 seconds).
Example:
@backoff.on_exception(
,
KeyError,
max_tries=3,
interval=10 # 10 seconds interval each time)
def process_data(data):
return data["key"]
3.3 Linear backoff ()
Strategy: The interval increases linearly with the number of retry times, and the formula is factor * n.
Example:
@backoff.on_exception(
,
ConnectionError,
max_tries=5,
factor=2 # Increase 2 seconds per interval)
def connect_db():
return connect()
4. How to use and examples
4.1 Basic usage
import backoff
import requests
from import RequestException
@backoff.on_exception(, RequestException, max_tries=5)
def get_api_data(url):
response = (url)
response.raise_for_status() # Trigger exception (such as 4xx/5xx) return ()
try:
data = get_api_data("")
except RequestException as e:
print(f"Final failure: {e}")
4.2 Specifying multiple exception types
@backoff.on_exception(
,
(TimeoutError, ConnectionError),
max_tries=3
)
def fetch_data():
# TimeoutError or ConnectionError may be thrown pass
4.3 Retry at fixed intervals
@backoff.on_exception(
,
KeyError,
max_tries=3,
interval=2 # 2 seconds interval each time)
def process_dict(data):
return data["missing_key"] # Trigger KeyError4.4 Coupled with max_time to limit the total time
@backoff.on_exception(
,
Exception,
max_time=30 # Maximum retry time 30 seconds)
def unreliable_function():
# May try again multiple times in 30 seconds pass
5. Advanced usage
5.1 Custom retry conditions (givenup)
Define the condition to abandon retry through the giveup parameter (stop retry when True is returned):
def giveup_on_404(exception):
return getattr(exception, "status_code", 0) == 404
@backoff.on_exception(
,
RequestException,
giveup=giveup_on_404
)
def get_data(url):
response = (url)
response.raise_for_status()
return ()
5.2 Logging (on_backoff and on_giveup)
Record retry information through the callback function:
def log_backoff(details):
print(f"Try again {details['tries']} Second-rate，wait {details['wait']} Second")
def log_giveup(details):
print(f"放弃Try again: {details['value']}")
@backoff.on_exception(
,
Exception,
on_backoff=log_backoff,
on_giveup=log_giveup
)
def my_function():
pass
6. Typical application scenarios
HTTP request retry:
@backoff.on_exception(, )
def get_http_response(url):
return (url)
Database connection:
@backoff.on_exception(
,
,
max_tries=5,
interval=1
)
def connect_db():
return ("")
File operation:
@backoff.on_exception(
,
FileNotFoundError,
max_tries=3
)
def read_file(path):
return open(path).read()
7. Precautions and best practices
1. Avoid infinite retry:
Always set max_tries or max_time to prevent dead loops.
@backoff.on_exception(, Exception, max_tries=5)
2. Choose the right strategy:
- Exponential backoff: Applicable to network requests (reduces concurrency pressure).
- Fixed interval: suitable for scenarios where stable intervals are required (such as retry every 10 seconds).
3. Clear the exception type:
Only retryable exceptions (such as ConnectionError) are caught to avoid catching global Exceptions.
4. Log logs:
Record retry information through on_backoff and on_giveup for easy debugging.
5. Handle unrecoverable errors:
Use the giveup callback to skip specific errors (such as 404 Not Found).
This is the introduction to this article about python using backoff to achieve automatic retry of exceptions. For more related content on automatic retry of python exceptions, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!