SoFunction
Updated on 2024-11-14

Python script to process log files in real time

This Python script is used to monitor the contents of real-time files, such as Error or time out fields can be customized; is my first real Python script, I feel still relatively bloated, but I intend to put it on the blog to record a little bit (still a beginner, the gods do not spray ha), and I really hope that the bloggers can be able to point out a little bit more (now record) Each time the file size value is output to a file, and there are shell commands nested inside, these I think can be optimized out of it, just now I do not know how to do); alerts are based on zabbix, custom templates are executed once in 120s

#!/usr/local/bin/python3.5
###Destription: Read log messages in real time
###Author: Danny Deng
###Datetime: 2016-11-17
import re,time,subprocess,os,linecache
##### defines the log file
file_name = "/usr/local/nginx/logs/"
file_number = "/usr/local/zabbix_agent/"
j = int(0)
seek = int(0)
## Judgment process: whether the file exists --- to determine the existence of the file to store the size of the log --- to determine the number size and the size of filesize
### Define a function to read the contents of a file by line
def readline():
####if judgment seek is greater than 0, greater than the value of the assignment, otherwise the initial 0
while True:
###### defines the file, reads each line based on the seek value, and assigns each tell to seek
with open(file_name,'r') as f:
global seek
#seek = seek
(seek)
data = ()
if data:
seek = ()
yield data
else:
######Python variables converted to shell variables
global file_number
['seek'] = str(seek)
['file_number'] = str(file_number)
('echo $seek > $file_number')
('chown  $file_number')
return
def func_for():
j = int(0)
for i in readline():
f_find = (r"check time out", i,flags=)
if "check time out" in f_find:
j += 1
##### does not output 0, there is a value to output the number of times that error matches the value
try:
print(j)
except NameError:
print(int("0"))
### Determine if a log file exists
if (file_name):
### Determine if a file exists that stores the size of the file contents
if (file_number):
If #### exists, read the file size and assign it to seek_number.
seek_number = int((file_number, 1))
#### then proceeds to determine the stored file size versus the current file size (to determine if the file was regenerated)
if (file_name) >= seek_number and seek_number > 0:
seek = seek_number
func_for()
### If the file is new, seek is assigned to 0.
else:
#open(arg1, "a+").write("0")
#seek = int((file_number, 1))
seek = int(0)
func_for()
####file_number If ####file_number does not exist, create a new one and assign the seek variable to 0.
else:
#open(file_number, "a+").write("0")
#seek = int((file_number, 1))
['file_number'] = str(file_number)
('echo 0 > $file_number')
('chown  $file_number')
func_for()
else:
print("Error")
quit()

The above is a small introduction to the Python script real-time processing log file script, I hope to help you, if you have any questions please leave me a message, I will reply to you in a timely manner. Here also thank you very much for your support of my website!