refactor: Implement Celery Tasks with Redis and RabbitMQ Integration"

[esloch]

Description:

1. Implementation of Celery Tasks:

   • Two new modules, tasks_collectors.py and tasks_async.py, have been added to the project to implement Celery tasks for asynchronous data collection and processing.
   • The tasks_collectors.py module defines synchronous Celery tasks responsible for simulating data collection from subcollectors, aggregating results, performing cleanup, and final processing.
   • The tasks_async.py module extends the functionality by introducing asynchronous Celery tasks to handle data processing in a distributed and scalable manner.
   • Both modules utilize Celery's capabilities for task management, parallel execution, and result aggregation, enhancing the project's efficiency and scalability.

2. Proposal of Test Cases:

   • The tests directory contains test files (test_tasks_collectors.py and test_tasks_async.py) to validate the functionality of the Celery tasks defined in the respective modules.
   • Test cases are designed to cover various scenarios, including normal operation, stress testing, and error conditions, ensuring the correctness and robustness of the task implementations.
   • Test cases in test_tasks_collectors.py validate synchronous Celery tasks, while test cases in test_tasks_async.py focus on asynchronous task behavior, leveraging asyncio for testing concurrency and scalability.
   • Detailed explanations and assertions within the test cases provide clarity on the expected behavior and help maintain code quality throughout development iterations.

3. Integration with Redis and RabbitMQ:

   • Redis and RabbitMQ are integrated into the Celery workflow as the result backend and message broker, respectively, enhancing task management and communication within the distributed system.
   • Redis serves as the result backend, storing task results and facilitating efficient retrieval and aggregation of data processed by Celery tasks.
   • RabbitMQ acts as the message broker, enabling communication between Celery workers and the task queue, ensuring reliable delivery and execution of asynchronous tasks across distributed environments.
   • The integration of Redis and RabbitMQ enhances the project's scalability, reliability, and performance, making it well-suited for handling complex asynchronous task workflows in production environments.

[xmnlab]

import should be in the first section of the python module