"It's a lot of work to manually check the logs every day."
"The time it takes to create reports doesn't allow for development to progress."
Do you have any of these concerns?
In this article,A tool that uses Python to automatically analyze logs, create reports with graphs, and send them by email.Here's how to develop it from scratch.
Pandas, Matplotlib, and email sending functionBy combining these, anyone can easily automate their business processes.
By reading this article,Free yourself from tedious manual work and dramatically improve your work efficiencyYou can do this.
What is a log analysis tool?
conclusion
A log analysis tool is a program that automatically reads the operation records of servers and applications and analyzes anomalies and trends.
reason
In particular, for web services and business systems, logs are essential for understanding access status and error information.
However, manually checking them every day is a huge burden.
Examples
For example, if you extract the "number of errors occurring per day" or "number of errors by type" from the error log,
This will be used to improve the system.
point
- Log analysis enables early detection of problems
- Reduces the burden of manual work
- Visualizing data makes it easy to report to stakeholders
Python libraries used
conclusion
We will use the following libraries, all of which are free to use.
List of libraries used
- pandas: A library for data analysis
- matplotlib: A graphing library
- smtplib/email: Standard library for sending email
- datetime: for date processing
These can be easily installed with pip:
1 | pip install pandas matplotlib |
Reading and preprocessing log files
conclusion
Read the CSV log file using pandas and extract and format the necessary columns.
Specific Code
1 | import pandas as pd df = pd.read_csv( "access_log.csv" ) df[ "datetime" ] = pd.to_datetime(df[ "datetime" ]) df[ "date" ] = df[ "datetime" ].dt.date error_df = df[df[ "status" ] > = 500 ] # Status code 500 or more = error |
Creating graphs with Matplotlib
conclusion
The number of errors is aggregated by day and visualized in a bar graph.
Specific Code
1 | import matplotlib.pyplot as plt daily_error = error_df.groupby( "date" ).size() plt.figure(figsize = ( 10 , 5 )) daily_error.plot(kind = "bar" ) plt.title( "Number of errors by day" ) plt.xlabel( "Date" ) plt.ylabel( "Number of errors" ) plt.tight_layout() plt.savefig( "error_graph.png" ) |
How to automatically send emails
conclusion
Python standard smtplib
and email
You can send emails with images using
Specific Code
1 | import smtplib from email.message import EmailMessage msg = EmailMessage() msg[ "Subject" ] = "Today's Log Report" msg[ "From" ] = "your_email@example.com" msg[ "To" ] = "receiver@example.com" msg.set_content( "Please check the attached image." ) with open ( "error_graph.png" , "rb" ) as f: msg.add_attachment(f.read(), maintype = "image" , subtype = "png" , filename = "error_graph.png" ) with smtplib.SMTP_SSL( "smtp.gmail.com" , 465 ) as smtp: smtp.login( "your_email@example.com" , "your_password" ) smtp.send_message(msg) |
Completed code
Here is the code that does all the work in one go:
import pandas as pd import matplotlib.pyplot as plt import smtplib from email.message import EmailMessage # Read and format logs df = pd.read_csv("access_log.csv") df["datetime"] = pd.to_datetime(df["datetime"]) df["date"] = df["datetime"].dt.date error_df = df[df["status"] >= 500] # Aggregate errors and create a graph daily_error = error_df.groupby("date").size() plt.figure(figsize=(10, 5)) daily_error.plot(kind="bar") plt.title("Number of errors by day") plt.xlabel("Date") plt.ylabel("Number of errors") plt.tight_layout() plt.savefig("error_graph.png") # Send Email msg = EmailMessage() msg["Subject"] = "Today's Log Report" msg["From"] = "your_email@example.com" msg["To"] = "receiver@example.com" msg.set_content("Please check the attached image.") with open("error_graph.png", "rb") as f: msg.add_attachment(f.read(), maintype="image", subtype="png", filename="error_graph.png") with smtplib.SMTP_SSL("smtp.gmail.com", 465) as smtp: smtp.login("your_email@example.com", "your_password") smtp.send_message(msg)
summary
In this article, we created a tool in Python to automate the following:
- Reading and formatting log files
- Error aggregation and graph visualization
- Automatic email delivery
This tool will greatly improve the efficiency of your daily reporting tasks.will be done.
As an application,Output to PDF and send Slack notificationsis also possible.
Please try incorporating this into your work.