Rss Scraper Application

Rss Scraper app based on DRF.

Contents

• Stack Used
• Basic Commands
  • Build the docker image
  • Run flake8 checks
  • Run the unit tests
  • Generate the test coverage report
  • Run the application using docker compose
  • Create superuser to start checking the feeds details on the django admin panel
  • Also you can check the feeds and feed items at the customized django admin at
  • Now you can check and test the developed APIs docs
  • Now you can create a user for testing from the signup form at:
  • You can check the received emails using the local mail server
  • You can check the running background celery tasks and periodic tasks statuses using credentials defined at django env
  • Interact manually with the DB using the PostgreSQL shell
  • Stop the running application using docker compose
  • What's Next?
• License

Stack Used

• Docker v. 20.10.12
• Docker Compose v. 2.2.3
• PostgreSQL v. 13.5
• Redis v. 6
• Python v. 3.9
• Django Framework v. 3.2.11
• Django Rest Framework v. 3.13.1
• PyTest v. 6.2.5
• Celery v. 5.2.3
• Celery Beat v. 2.2.1
• Flower v. 1.0.0
• Mailhog v. 1.0.0 (Locally)

Basic Commands

Build the docker image

docker-compose -f local.yml build

Run flake8 checks

docker-compose -f local.yml run --rm django flake8

Run the unit tests

docker-compose -f local.yml run --rm django pytest

Generate the test coverage report

docker-compose -f local.yml run --rm django coverage run -m pytest
docker-compose -f local.yml run --rm django coverage report

Run the application using docker compose

docker-compose -f local.yml up  # use -d to run the services in the background.

Create superuser to start checking the feeds details on the django admin panel

docker-compose -f local.yml run --rm django python manage.py createsuperuser [email protected] --username=superuser

Also you can check the feeds and feed items at the customized django admin at

Django admin panel

Now you can check and test the developed APIs docs

APIs Swagger docs

Now you can create a user for testing from the signup form at:

Create new user Signup

You can check the received emails using the local mail server

Mailhog

You can check the running background celery tasks and periodic tasks statuses using credentials defined at django env

Flower

Interact manually with the DB using the PostgreSQL shell

docker-compose -f local.yml exec postgres psql --username=${POSTGRES_USERNAME} --dbname=${POSTGRES_DB_NAME}

Stop the running application using docker compose

docker-compose -f local.yml down -v

What's Next?

• Update database modeling to prevent saving & scraping duplicate feeds/items for different users.

• Using caching on the GET APIs Views and ORM.

  1. Cache on the ORM level using 3rd party package like: Django Cacheops.
  2. Cache on the View level using either the base DRF decorators.
  3. Most importantly invalidate the cached results after any feed/item related updates.

• Integrating APM solution.
• Implementing an exponential backoff celery retry mechanism on the failed tasks.

Commits icons are from: Gitmoji

License

Open source licensed under the MIT license (see LICENSE file for details).