I created a log analysis and automatic report creation tool using Python [Pandas × Matplotlib × Email sending]

programming

"It's a lot of work to manually check the logs every day."
"The time it takes to create reports doesn't allow for development to progress."
Do you have any of these concerns?

In this article,A tool that uses Python to automatically analyze logs, create reports with graphs, and send them by email.Here's how to develop it from scratch.

Pandas, Matplotlib, and email sending functionBy combining these, anyone can easily automate their business processes.
By reading this article,Free yourself from tedious manual work and dramatically improve your work efficiencyYou can do this.

What is a log analysis tool?

conclusion

A log analysis tool is a program that automatically reads the operation records of servers and applications and analyzes anomalies and trends.

reason

In particular, for web services and business systems, logs are essential for understanding access status and error information.
However, manually checking them every day is a huge burden.

Examples

For example, if you extract the "number of errors occurring per day" or "number of errors by type" from the error log,
This will be used to improve the system.

point

  • Log analysis enables early detection of problems
  • Reduces the burden of manual work
  • Visualizing data makes it easy to report to stakeholders

Python libraries used

conclusion

We will use the following libraries, all of which are free to use.

List of libraries used

  • pandas: A library for data analysis
  • matplotlib: A graphing library
  • smtplib/email: Standard library for sending email
  • datetime: for date processing

These can be easily installed with pip:

1

Reading and preprocessing log files

conclusion

Read the CSV log file using pandas and extract and format the necessary columns.

Specific Code

1
= ] = ] = = ] >= 500]

Creating graphs with Matplotlib

conclusion

The number of errors is aggregated by day and visualized in a bar graph.

Specific Code

1
= =(10, 5=)

How to automatically send emails

conclusion

Python standard smtplib and email You can send emails with images using

Specific Code

1
= ] = ] = ] = (, ===, 465,

Completed code

Here is the code that does all the work in one go:

import pandas as pd import matplotlib.pyplot as plt import smtplib from email.message import EmailMessage # Read and format logs df = pd.read_csv("access_log.csv") df["datetime"] = pd.to_datetime(df["datetime"]) df["date"] = df["datetime"].dt.date error_df = df[df["status"] >= 500] # Aggregate errors and create a graph daily_error = error_df.groupby("date").size() plt.figure(figsize=(10, 5)) daily_error.plot(kind="bar") plt.title("Number of errors by day") plt.xlabel("Date") plt.ylabel("Number of errors") plt.tight_layout() plt.savefig("error_graph.png") # Send Email msg = EmailMessage() msg["Subject"] = "Today's Log Report" msg["From"] = "your_email@example.com" msg["To"] = "receiver@example.com" msg.set_content("Please check the attached image.") with open("error_graph.png", "rb") as f: msg.add_attachment(f.read(), maintype="image", subtype="png", filename="error_graph.png") with smtplib.SMTP_SSL("smtp.gmail.com", 465) as smtp: smtp.login("your_email@example.com", "your_password") smtp.send_message(msg)

summary

In this article, we created a tool in Python to automate the following:

  • Reading and formatting log files
  • Error aggregation and graph visualization
  • Automatic email delivery

This tool will greatly improve the efficiency of your daily reporting tasks.will be done.
As an application,Output to PDF and send Slack notificationsis also possible.
Please try incorporating this into your work.

Copied title and URL