This is a tutorial on how to run Django with Docker

(also in PT-PT: https://github.com/tcarreira/django-docker/blob/master/README-PT.md)

This is a tutorial on how to run Django with Docker

Includes:

django
celery
docker
- best practices for development
- best practices for Dockerfile
- volumes for fast development
debugging
some production best practices
- do not run as root
- use gunicorn and serve static content
- small docker image
- environemnt variables

Previous notes

This tutorial comes along with this presentation:
- http://tcarreira.github.io/presentations/django-docker
- (also in portuguese - PT)

Development best practices

What do you need to develop? (not an exaustive list)

tool	what for	example
a good IDE	auto-completion, debugging	vscode
good IDE plugins	framework specifics (django)	`ms-python.python`, `batisteo.vscode-django`
linter	you need to have a real-time feedback about what is wrong	mypy
formatter	it's great for code sharing	black
unit tests	you should really not test your code. Make your computer do it	pytest
code-to-execution	low latency after writing your code until it gets executed	manage.py runserver

Workshop

Start a Django project (from scratch)

Setup VirtualEnv and install Django

virtualenv -p $(which python3) venv
. ./venv/bin/activate
pip install "Django>=3.0.3,<4"

Create django project and initial setup

django-admin startproject django_demo
python django_demo/manage.py makemigrations
python django_demo/manage.py migrate
python django_demo/manage.py createsuperuser --username admin --email ""

You may test it with python django_demo/manage.py runserver 0.0.0.0:8000 and open the browser at http://127.0.0.1:8000

Setup Docker
- Install Docker
  - Do NOT install from apt-get: very outdated (almost useless)
  - Windows - Docker Desktop
    - https://docs.docker.com/docker-for-windows/install/
  - Mac - Docker Desktop
    - https://docs.docker.com/docker-for-mac/install/
  - Linux
    - Brain dead easy way: wget -qO- https://get.docker.com | sh
    - Other way: https://docs.docker.com/install/
- Setup a first draft of a Dockerfile, so we can see what is bad in this draft
```
FROM python
COPY django_demo/ /app/
RUN pip install --no-cache-dir "Django>=3.0.3,<4"
ENV PYTHONUNBUFFERED=1
CMD python /app/manage.py runserver 0.0.0.0:8000
```
  and test it
```
docker build -t django-docker-demo .
docker run -it --rm -p8000:8000 django-docker-demo
```
  build for the first time: 2:18
  build after minor change: 0:22
  
  problems:
  - COPY files before pip install
  - not using docker images tags (use alpine if possible)
  - beware of context (.dockerignore)
- Identify some problems in Dockerfile
```
FROM python:3.7-alpine
RUN pip install --no-cache-dir "Django>=3.0.3,<4"
COPY django_demo/ /app/
ENV PYTHONUNBUFFERED=1
CMD python /app/manage.py runserver 0.0.0.0:8000
```
  and .dockerignore
```
venv/
__pycache__/
db.sqlite3
```
  From Sending build context to Docker daemon 42.42MB to ...156.7kB
  
  build for the first time: 1:37
  build after minor change: 0:03

Testing Django (with Docker)

Let's create our own Django content

create django_demo/django_demo/views.py

import os
from django.http import HttpResponse

def hello_world(request):
    output_string = "<h1>Hello World from {}".format(os.environ.get("HOSTNAME", "no_host"))
    return HttpResponse(output_string)

change django_demo/django_demo/urls.py

from django.contrib import admin
from django.urls import path
from . import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', views.hello_world),
]

Docker volumes for development

Dispite the fast building time, I don't want to rebuild+restart every time I change something

add volume on docker run
```
docker run -it --rm -p8000:8000 -v "$(pwd)/django_demo/:/app/" django-docker-demo
```
now, every time you change some file, django wil reload itself. Very useful for developmemt.

Docker-compose - a simple multi-container orchestration tool

Let's add some more dependencies: Celery + Redis (This is a major step)

Celery depends on a message broker, and we are going to use Redis, for simplicity

Install Celery and Redis client

(as we are getting more dependencies, let's keep a django_demo/requirements.txt)

Django>=3.0.3,<4
Celery>=4.3.0,<4.4
redis>=3.3<3.4

and update Dockerfile (now we don't need to update this with every dependency change)

FROM python:3.7-alpine

COPY django_demo/requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
COPY django_demo/ /app/

ENV PYTHONUNBUFFERED=1
WORKDIR /app

CMD python /app/manage.py runserver 0.0.0.0:8000

Create a docker-compose.yml so we can keep 3 running services: django + redis + celery_worker

version: "3.4"
services:
    django:
        image: django-docker-demo
        ports:
            - 8000:8000
        volumes:
            - ./django_demo/:/app/
    celery-worker:
        image: django-docker-demo
        volumes:
            - ./django_demo/:/app/
        command: "celery -A django_demo.tasks worker --loglevel=info"
    redis:
        image: redis:5.0-alpine

Update python code with a test

create django_demo/django_demo/tasks.py

from celery import Celery
import os
app = Celery('tasks', broker='redis://redis:6379', backend='redis://redis:6379')

@app.task
def hello(caller_host):
    return "<h1>Hi {}! This is {}.</h1>".format(caller_host, os.environ.get("HOSTNAME", 'celery_worker_hostname'))

add to django_demo/django_demo/views.py

from . import tasks
def test_task(request):
    task = tasks.hello.delay(os.environ.get("HOSTNAME", "no_host"))
    output_string = task.get(timeout=5)
    return HttpResponse(output_string)

update django_demo/django_demo/urls.py

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', views.hello_world),
    path('task', views.test_task),
]

Finally, we can test our setup
- build docker image: docker build -t django-docker-demo .
- run docker-compose up (you can see the logs and terminate with ctrl+c. to run in background, add -d)
- open http://127.0.0.1:8000/task

Now that things got a little confusing, it gets worse.

do you remeber some development best practices?

Debugging a Django application

example with vscode, which uses ptvsd for debugging, .vscode/launch.json

{ 
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
        }, 
        {
            "name": "Python: Debug Django",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/django_demo/manage.py",
            "args": [
                "runserver",
                "--nothreading"
            ],
            "subProcess": true,
        }
    ]
}

The first one is good enough for python scripts, but not so nice for django applications. The second one is very good, but it runs locally (no docker)

or...

You could be running a debugger with docker

create a django_demo/requirements-dev.txt which has every dev package (result from pip freeze) (note: this example has too many packages)

amqp==2.5.2
appdirs==1.4.3
asgiref==3.2.3
astroid==2.3.3
attrs==19.3.0
billiard==3.6.1.0
black==19.10b0
celery==4.3.0
Click==7.0
Django==3.0.3
importlib-metadata==1.2.0
isort==4.3.21
kombu==4.6.7
lazy-object-proxy==1.4.3
mccabe==0.6.1
more-itertools==8.0.2
mypy==0.750
mypy-extensions==0.4.3
pathspec==0.6.0
ptvsd==4.3.2
pylint==2.4.4
pytz==2019.3
redis==3.3.11
regex==2019.11.1
six==1.13.0
sqlparse==0.3.0
toml==0.10.0
typed-ast==1.4.0
typing-extensions==3.7.4.1
vine==1.3.0
wrapt==1.11.2
zipp==0.6.0

change the Dockerfile in order to include requirements-dev.txt instead of requirements.txt (we need development tools for debugging) and some other dependencies

FROM python:3.7-alpine

RUN apk add --update --no-cache \
        bash \
        build-base

COPY django_demo/requirements-dev.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
COPY django_demo/ /app/

ENV PYTHONUNBUFFERED=1
WORKDIR /app

CMD python /app/manage.py runserver 0.0.0.0:8000

and build it again

docker build -t django-docker-demo .

expose port 5678 on django service inside docker-compose.yml

version: "3.4"
services:
    django:
        image: django-docker-demo:latest
        ports:
            - "8000:8000"
            - "5678:5678"
        volumes:
            - "./django_demo/:/app/"
    celery-worker:
        image: django-docker-demo:latest
        volumes:
            - "./django_demo/:/app/"
        command: "celery -A django_demo.tasks worker --loglevel=info"
    redis:
        image: redis:5.0-alpine

modify django_demo/manage.py (add it inside main, before execute_from_command_line(sys.argv))

from django.conf import settings
if settings.DEBUG:
    if (  # as reload relauches itself, workaround for it
        "--noreload" not in sys.argv
        and os.environ.get("PTVSD_RELOAD", "no") == "no"
    ):
        os.environ["PTVSD_RELOAD"] = "yes"
    else:
        import ptvsd
        ptvsd.enable_attach()

add a remote debugger on your IDE. For vscode add a configuration to .vscode/launch.json

{
    "name": "Python: Debug Django attach Docker",
    "type": "python",
    "request": "attach",
    "localRoot": "${workspaceFolder}/django_demo",
    "remoteRoot": "/app",
    "host": "127.0.0.1",
    "port": 5678,
},

and test it
```
docker-compose up
```
After adding a breakpoint inside django_demo.views.hello_world() reload your browser.

Improve Dockerfile and other thing...

You may clone the code with everying with git clone https://github.com/tcarreira/django-docker.git

do not run as root.
```
RUN adduser -D user
USER user
```
remove unnecessary dependencies. Keep different images for different uses (use multi-stage builds)
```
FROM python:3.7-alpine AS base
...
FROM base AS dev
...
FROM base AS final
...
```

clean your logs

services:
    app:
        ...
        logging:
            options:
                max-size: "10m"
                max-file: "3"

or shorter:

        logging: { options: { max-size: "10m", max-file: "3" } }

next level caching (with Buildkit DOCKER_BUILDKIT=1) - https://github.com/moby/buildkit
```
# syntax=docker/dockerfile:experimental
...
ENV HOME /app
WORKDIR /app
RUN --mount=type=cache,uid=0,target=/app/.cache/pip,from=base \
    pip install -r requirements.txt
```
note: # syntax=docker/dockerfile:experimental on the first line of your Dockerfile is mandatory for using BUILDKIT new features

quality as part of the pipeline

FROM dev AS qa
RUN black --target-version=py37 --check --diff . 
RUN mypy --python-version=3.7 --pretty --show-error-context .
RUN coverage run django_demo/manage.py test
RUN django_demo/manage.py check --deploy --fail-level WARNING

Prepare Django for production

Prepare Django for production - https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/ (out of the scope for this)
Use a decent webserver

from: https://docs.djangoproject.com/en/3.0/ref/django-admin/#runserver
DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone through security audits or performance tests. (And that’s how it’s gonna stay. We’re in the business of making Web frameworks, not Web servers, so improving this server to be able to handle a production environment is outside the scope of Django.)

and build static files (you will need it for the new webserver)
```
FROM dev AS staticfiles
RUN python manage.py collectstatic --noinput

FROM nginx:1.17-alpine
COPY conf/nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=staticfiles /app/static /staticfiles/static
```
and create the corresponding conf/nginx.conf (just a minimal example. Don't use in production)
```
server {
    listen 80 default_server;
    server_name _;

    location /static {
        root /staticfiles;
    }

    location / {
        proxy_pass http://django:8000;
    }
}
```

you probably want to use a different database

add to your docker-compose.yml

    db:
        image: mariadb:10.4
        restart: always
        environment:
            MYSQL_DATABASE: "django"
            MYSQL_USER: "user"
            MYSQL_PASSWORD: "pass"
            MYSQL_RANDOM_ROOT_PASSWORD: "yes"
        volumes:
            - django-db:/var/lib/mysql

and configure your django_demo/django_demo/settings.py

# If database is defined, overrides the default
if (
    os.environ.get("DJANGO_DB_HOST")
    and os.environ.get("DJANGO_DB_DATABASE")
    and os.environ.get("DJANGO_DB_USER")
    and os.environ.get("DJANGO_DB_PASSWORD")
):
    DATABASES["default"] = {
        "ENGINE": "django.db.backends.mysql",
        "NAME": os.environ.get("DJANGO_DB_DATABASE", ""),
        "USER": os.environ.get("DJANGO_DB_USER", ""),
        "PASSWORD": os.environ.get("DJANGO_DB_PASSWORD", ""),
        "HOST": os.environ.get("DJANGO_DB_HOST", ""),
        "PORT": os.environ.get("DJANGO_DB_PORT", "3306"),
    }

build a common entrypoint (so you don't have to change dockerfile later)

#!/bin/bash

# wait until database is ready
if [[ ${DJANGO_DB_HOST} != "" ]]; then
    while !</dev/tcp/${DJANGO_DB_HOST}/${DJANGO_DB_PORT:-3306} ; do 
        sleep 1;
        [[ $((counter++)) -gt 60 ]] && break
    done
fi

python manage.py migrate

case $DJANGO_DEBUG in
    true|True|TRUE|1|YES|Yes|yes|y|Y)
        echo "================= Starting debugger =================="
        python manage.py runserver 0.0.0.0:8000
        ;;
    *)
        gunicorn --worker-class gevent -b 0.0.0.0:8000 django_demo.wsgi
        ;;
esac

Test it all together

note: now it's a good time for git clone https://github.com/tcarreira/django-docker.git

Update your docker-compose docker-compose.yml

version: "3.4"
services:
    webserver:
        image: django-docker-demo:webserver
        build:
            dockerfile: Dockerfile
            target: webserver
            context: .
        ports:
            - 8080:80
        logging: { options: { max-size: "10m", max-file: "3" } }
    django:
        image: django-docker-demo:dev
        build:
            dockerfile: Dockerfile
            target: dev
            context: .
        ports:
            - 8000:8000
            - 5678:5678
        volumes:
            - ./django_demo/:/app/
        environment:
            DJANGO_DEBUG: "y"
            DJANGO_DB_HOST: "db"
            DJANGO_DB_DATABASE: "django"
            DJANGO_DB_USER: "djangouser"
            DJANGO_DB_PASSWORD: "djangouserpassword"
        logging: { options: { max-size: "10m", max-file: "3" } }

    celery-worker:
        image: django-docker-demo:latest
        build:
            dockerfile: Dockerfile
            context: .
        volumes:
            - ./django_demo/:/app/
        command: "celery -A django_demo.tasks worker --loglevel=info"
        logging: { options: { max-size: "10m", max-file: "3" } }

    redis:
        image: redis:5.0-alpine
        logging: { options: { max-size: "10m", max-file: "3" } }
    
    db:
        image: mariadb:10.4
        restart: always
        environment:
            MYSQL_DATABASE: "django"
            MYSQL_USER: "djangouser"
            MYSQL_PASSWORD: "djangouserpassword"
            MYSQL_RANDOM_ROOT_PASSWORD: "yes"
        volumes:
            - django-db:/var/lib/mysql
        logging: { options: { max-size: "10m", max-file: "3" } }

volumes:
    django-db:

and run

DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose up --build

note: you need docker-compose >= 1.25 in order to use builkit directly. If you don't have it, build images first, then run docker-compose up

Last notes

keep separate docker-compose.qa.yml for testing/qa with final images

version: "3.4"
services:
    webserver:
        image: django-docker-demo:webserver
        ports:
            - 80:80
        logging: { options: { max-size: "10m", max-file: "3" } }
            
    django:
        image: django-docker-demo:latest
        environment:
            DJANGO_DEBUG: "false"
            DJANGO_SECRET_KEY: "AVhqJxkBn5cSS7Zp4jqWAMMAOXRoKfuOHduKVFUo"
            DJANGO_DB_HOST: "db"
            DJANGO_DB_DATABASE: "djangoqa"
            DJANGO_DB_USER: "djangouser"
            DJANGO_DB_PASSWORD: "djangouserpassword"
        logging: { options: { max-size: "10m", max-file: "3" } }

    celery-worker:
        image: django-docker-demo:latest
        command: "celery -A django_demo.tasks worker --loglevel=info"
        logging: { options: { max-size: "10m", max-file: "3" } }

    redis:
        image: redis:5.0-alpine
        logging: { options: { max-size: "10m", max-file: "3" } }
        
    db:
        image: mariadb:10.4
        restart: always
        environment:
            MYSQL_DATABASE: "djangoqa"
            MYSQL_USER: "djangouser"
            MYSQL_PASSWORD: "djangouserpassword"
            MYSQL_RANDOM_ROOT_PASSWORD: "yes"
        volumes:
            - django-db-qa:/var/lib/mysql
        logging: { options: { max-size: "10m", max-file: "3" } }

volumes:
    django-db-qa:

And build your continuous integration process (this is a simple example)
- Code + commit + push
- auto-start CI/CD process
- DOCKER_BUILDKIT=1 docker build -t django-docker-demo:qa --target=qa . (this is the test stage)
- DOCKER_BUILDKIT=1 docker build -t django-docker-demo:dev --target=dev . + push docker image dev internally (optional)
- DOCKER_BUILDKIT=1 docker build -t django-docker-demo:latest --target . + push official docker image
- update running containers (outside the scope of this tutorial)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

This is a tutorial on how to run Django with Docker

TOC

Includes:

Previous notes

Workshop

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
.vscode		.vscode
conf		conf
django_demo		django_demo
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
README-PT.md		README-PT.md
README.md		README.md
docker-compose.qa.yml		docker-compose.qa.yml
docker-compose.yml		docker-compose.yml

JLira/django-docker

Folders and files

Latest commit

History

Repository files navigation

This is a tutorial on how to run Django with Docker

TOC

Includes:

Previous notes

Workshop

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages