Python Automates Multiple Methods of Splitting Logs by Date

introduction

Log management is crucial when developing and operating Python applications. Logs not only help us debug problems, but also can be used for monitoring and performance analysis. However, if all logs are written to a single file, long-term operation will lead to excessive log files that are difficult to manage and retrieve.

This article will introduce a variety of ways to automate dividing logs by date, including:

• Shell redirect + date command (simple and direct)
• Cron timing tasks + log rotation (automated management)
• logrotate tool (suitable for production environments)
• Date segmentation of Python built-in log module (code level control)

Whether you are a developer or an operation and maintenance engineer, you can find a suitable solution.

1. Why do you need to split logs by date

1.1 Problems with single log files

Too large files: Long-term running applications may generate GB-level logs, affecting read and write performance.

Retrieval Difficulty: It is time-consuming to find a day's log in a single large file.

Backup and Cleanup Trouble: Cannot archive or delete old logs by date.

1.2 Advantages of splitting logs by date

Easy to manage: One file per day, such as app_2024.

Quick location problem: directly view the log file for a certain day.

Automated maintenance: can be combined with compression and regular cleaning strategies.

2. Method 1: Shell redirect + date naming (quickly get started)

2.1 Basic Commands

Use date directly in the startup command to generate a log file name with date:

nohup python3  > "app_$(date +\%Y-\%m-\%d).log" &

illustrate:

• date +\%Y-\%m-\%d Generate the current date, such as 2024-06-10.
• nohup ensures that the process is running in the background and will not stop even if the terminal is closed.

2.2 Advanced scripts

If you need to restart the application every day, you can write a script start_app.sh:

#!/bin/bash
LOG_DIR="/var/log/myapp"
mkdir -p "$LOG_DIR"
LOG_FILE="$LOG_DIR/app_$(date +\%Y-\%m-\%d).log"
nohup python3  > "$LOG_FILE" 2>&1 &

Optimization points:

• Specify the log directory (/var/log/myapp).
• 2>&1 Redirects standard error (stderr) to the log file as well.

Operation mode:

chmod +x start_app.sh
./start_app.sh

3. Method 2: Cron timed tasks (automated management)

3.1 Generate new logs every day using Cron

If the application runs for a long time but wants to switch log files every day, you can use cron + kill and restart:

# Edit Cron Taskscrontab -e

Add the following (toggle logs every day at midnight):

0 0 * * * /bin/bash -c 'kill $(pgrep -f "python3 ") && nohup python3 > /var/log/myapp/app_$(date +\%Y-\%m-\%d).log 2>&1 &'

illustrate:

• 0 0 * * * means execution at 00:00 every day.
• kill $(pgrep -f "python3") Stop the running process first.

3.2 Combined with log compression

Add log compression in Cron task:

0 0 * * * /bin/bash -c 'kill $(pgrep -f "python3 ") && gzip /var/log/myapp/app_$(date -d "yesterday" +\%Y-\%m-\%d).log && nohup python3 > /var/log/myapp/app_$(date +\%Y-\%m-\%d).log 2>&1 &'

Optimization points:

gzip compresses the previous day's logs, saving space.

4. Method 3: Use logrotate (production environment recommendation)

4.1 Installation and Configuration

logrotate is a log management tool built into Linux, suitable for long-term applications.

Create configuration file /etc//myapp:

/var/log/myapp/ {
    daily              # Rotate every day    rotate 30          # Keep 30 days of log    dateext            # Use date as suffix    compress           # Compress old logs    missingok          # If the log does not exist, no error will be reported    notifempty         # Empty logs do not rotate    copytruncate       # Clear the original file after copying (avoid restarting the application)}

Manual test:

logrotate -vf /etc//myapp

4.2 Combining Python logging module

If the application uses Python's logging module, you can configure splitting by date:

import logging
from  import TimedRotatingFileHandler

log_handler = TimedRotatingFileHandler(
    "", when="midnight", interval=1, backupCount=30
)
log_handler.suffix = "%Y-%m-%"
(handlers=[log_handler], level=)

illustrate:

• When="midnight" Switch logs every day.
• backupCount=30 Keep logs for up to 30 days.

5. Method 4: Python built-in log rotation (code-level control)

If you don't want to rely on external tools, you can use Python directly:

import logging
from  import TimedRotatingFileHandler

#Configuration loglogger = ("myapp")
handler = TimedRotatingFileHandler(
    "", when="D", interval=1, backupCount=7, encoding="utf-8"
)
(("%(asctime)s - %(levelname)s - %(message)s"))
(handler)
()

# Sample log("Application started")

Parameter description:

• when="D" is divided by day (also supports H hours and M minutes).
• backupCount=7 Keep the log of the last 7 days.

6. Comparison and summary

Recommended plan

• Development/Test Environment → Shell redirect or Python built-in logs.
• Production environment → logrotate + log compression.

7. Conclusion

Log management is an important part of application operation and maintenance. This article introduces four methods of dividing logs by date, covering from simple shell commands to production-level logrotate configurations.

Your choice depends on:

• Is it necessary to intrude code (Python log module vs external tools).
• Whether automated compression and cleaning are required (logrotate optimal).
• Whether to allow restarting the process (the Cron solution requires restart).

This is the end of this article about the various methods of Python automation dividing logs by date. For more related content of Python dividing logs, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!