Note: This is an old project of mine about 7 years ago. It's also my first Python project. I'm aware that the code quality isn't great. It sucks actually...
Laziness: AI GENERATED CONTENT
This project is a multi-threaded and multi-process application that listens to network traffic, parses it, writes logs, and interacts with an Elasticsearch database.
The main.py script is the entry point of the application. It reads the configuration from config.ini, establishes a connection to an Elasticsearch database, and starts multiple processes and threads based on the port mappings defined in the configuration file.
Each port mapping starts a UdpListener process (from cli_listener.py) and a LogWriter process (from cli_log_writer.py). The UdpListener listens for network traffic on a specific port and puts the data into a multiprocessing queue (rqueue). The LogWriter process writes logs to different files based on the type of log (e.g., AUDIT, URL, PDENY, etc.).
For each port mapping, multiple Consumer threads (from cli_consumer_huawei.py) are also started. Each Consumer thread takes data from the rqueue, parses it using various parsers (from the parsers directory), writes to the database using various database writers (from the db_writers directory), and puts any logs into another multiprocessing queue (wqueue), which the LogWriter process writes to the log files.
The signer.py script appears to be a separate utility for signing and archiving files. It also reads its configuration from config.ini. Helpers
The file_helper and error_helper modules in the helpers directory are used throughout the application for file operations and error handling, respectively.
Python - The programming language used
Elasticsearch - The database used
This project is licensed under the MIT License