Nginx Container Becomes Unresponsive After Esmerald Deployment: Debugging SWAG + Esmerald Deployment Issue

[sanket-parte]

Hi Everyone,

We have deployed the Esmerald application using Docker, and it's being served through nginx configured in the linuxserver/swag docker container. We’re encountering an issue where, after starting the Esmerald application, the server becomes unresponsive after some time. Specifically, the swag (nginx) container stops responding, leading to request timeout errors when accessing APIs.

Here are the current configurations:
Uvicorn Script
We are using the following uvicorn script to start the Esmerald application:

import uvicorn
from esmerald.conf import settings

if __name__ == "__main__":
    """
    Uses the `ESMERALD_SETTINGS_MODULE` settings configuration and loads \
    accordingly.
    """
    uvicorn.run(
        "main:app",
        port=8000,
        host="0.0.0.0",
        reload=True,
        lifespan="on",
        log_level="debug",
    )

Nginx Configuration
Here is the nginx configuration:

location /project/api/ {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-CSS-Protection "1; mode=block";
    proxy_set_header X-Content-Type-Options "nosniff";
    proxy_set_header Cache-Control "public,max-age=120,must-revalidate,s-maxage=120";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Content-Security-Policy "default-src 'self';" always;
    add_header X-Frame-Options "SAMEORIGIN";
    proxy_redirect off;
    proxy_buffering off;
    proxy_pass http://project_service:8000/project/api/;
}

Can anyone suggest how to debug this issue?

[tarsil]

@sanket-parte It seems like you might have some resources issues or some issues with configurations. You example is not enough to understand the cause but have a look at this
https://github.com/dymmond/esmerald-tortoise-nginx-gunicorn-supervisor

It's public archive but there also has some nginx, Gunicorn, docker and uvicorn configurations that you can probably check and use?

[tarsil]

Ohhhh wait. @sanket-parte that script to start uvicorn it won't work properly in n production. They even said that you should use that for development only (https://github.com/dymmond/esmerald/blob/main/esmerald/conf/directives/project_template/project_name/serve.py-tpl#L5).

The URL I sent before has an example of configurations how to start a server for production using also supervisor.

https://github.com/dymmond/esmerald-tortoise-nginx-gunicorn-supervisor

Have a look and see how we did.

Also in the docs of Esmerald we have an example as well

[sanket-parte]

Hey @tarsil, Thank you very much! for your quick response. I will try this configurations.

[sanket-parte]

Hey @tarsil ,

I’ve tried configuring Gunicorn using the setup provided in this URL that you shared, but the same issue persists.

Here are the configurations we're using:

Gunicorn Configuration Script::

import json
import multiprocessing
import os

proc_name = os.getenv("GUNIICORN_PROCESS_NAME", "gunicorn")
project_dir = os.getenv("PROJECT_DIR", "/app")
chdir = os.path.join(project_dir, "/project")
pidfile = os.getenv("GUNIICORN_PIDFILE", None)
host = os.getenv("GUNIICORN_HOST", "0.0.0.0")
port = os.getenv("GUNIICORN_PORT", "8000")
keepalive = int(os.getenv("GUNIICORN_KEEP_ALIVE", "5"))
timeout = int(os.getenv("GUNIICORN_TIMEOUT", "120"))
graceful_timeout = int(os.getenv("GUNIICORN_GRACEFUL_TIMEOUT", "120"))
forwarded_allow_ips = os.getenv("GUNIICORN_FORWARD_ALLOW_IPS", "*")
worker_class = os.getenv("GUNIICORN_WORKER_CLASS", "uvicorn.workers.UvicornWorker")
user = os.getenv("USER", "project_user")
group = os.getenv("GROUP", "project_user")
workers_per_core = float(os.getenv("GUNIICORN_WORKERS_PER_CORE", "1"))

bind = os.getenv("BIND", f"{host}:{port}")

# Logs configurations
loglevel = os.getenv("GUNIICORN_LOGLEVEL", "info")
logger_path = os.getenv("LOGS_DIR")
accesslog = "-"
errorlog = os.path.join(logger_path, "gunicorn-error.log")

web_concurrency = None
max_workers = os.getenv("GUNIICORN_MAX_WORKERS")
use_max_workers = None

if max_workers:
    use_max_workers = int(max_workers)

concurr = os.getenv("GUNIICORN_WEB_CONCURRENCY")
cores = multiprocessing.cpu_count()
default_web_concurrency = float(workers_per_core) * cores

if concurr:
    web_concurrency = int(concurr)
    assert web_concurrency > 0
else:
    web_concurrency = max(int(default_web_concurrency), 2)
    if use_max_workers:
        web_concurrency = min(web_concurrency, use_max_workers)

Supervisord Configuration FIle:

[program:gunicorn]
command = gunicorn main:app -c /app/conf/gunicorn-conf.py
user = project_user
numprocs = 1
priority = 14
startsecs = 10
autostart = true
autorestart = true
process_name = gunicorn-%(process_num)d
stdout_logfile = /dev/stdout
stdout_logfile_maxbytes = 0
redirect_stderr = true

Command to Start Supervisord:

/usr/bin/supervisord

Could you please assist us in debugging this issue?

[tarsil]

@sanket-parte first things first.

1. You call your env vars GUNIICORN and not GUNICORN, is that intentional?
2. In your docker did you place the config inside the "/app/conf" and called it gunivorn-conf.py?
3. Your user used in the supervisor has enough permissions?
4. Did you copy the supervisor.conf to the proper location to be used?

Show me the log from when you start the container to see the errors, please

[sanket-parte]

Yes, all these configurations are correct. I am able to run the application, and it is accessible from the browser. However, after some time, my Nginx container becomes unresponsive, leading to a gateway timeout error in the browser.

Following are the logs after running the application using above configurations:

Access logs:

[tarsil]

This is very odd. Do you use any "uvicorn.run()"? You should not do that

[sanket-parte]

Regarding resources, we are having 12 vCPU plus 48GB ram in the system

[tarsil]

This is very odd. This looks like a normal Esmerald application and it should work. As far as Esmerald itself it looks ok, what I'm not 100% sure is if its something else being used by Esmerald that is becoming irresponsive like the MongoDB access from asyncz (we updated it recently to a newer and better version).

Can you add logging to your application to understand better where it might hang? After how long it becomes irresponsive? I suspect this might be caused by asyncz (which would mean we might need to run an update if you don't have the latest of course or something else that might be causing it to do that).

We can try a couple of things.

1. Add more logging to the who e application to understand where it hangs because like this its hard to understand.
2. Disable temporarily asyncz (which I also think you are actually not using it, just declaring the configuration).

Does this happen in your local when fully testing it? I doubt but still asking.

Out of curiosity, I know Azure webapps with docker, they don't like supervisor so much as per my last experience with it, if you are also using it.

[tarsil]

Somehow this is boiling down to my understand of the access to the external resources like redis or the DB and not Esmerald itself.

[sanket-parte]

We will add additional logs to debug this.
Currently we are using global redis connection pool, which is a sync client as follows:

[tarsil]

Yes I understand but maybe something in the connections is not working so well and can be a myriad of reasons. We have Esmerald applications live running and this is a first but each project is of course, different so the logging is important to try to trace the root cause.

[Kokoserver]

Can you try to replicate the application using local redis or mongodb on you dev environment.

And add logs to check if the redis or mongodb is connecting then we will know when to do next

[tarsil]

Yeah, the API server is working fine now. However, when we use Nginx as a reverse proxy, it’s not working as expected. You mentioned that @Kokoserver successfully deployed with Nginx, but we should check a few things—like deploying an Esmerald application that uses MongoDB and monoz. And configuring with Nginx—since that's our use case.

Actually soon I will be doing that (not date specified) so I can share my inputs with you later on

[sanket-parte]

Thank you very much for considering it.

[sanket-parte]

Hey @tarsil , did you find time to check this?

[tarsil]

@sanket-parte we are trying to replicate this and so far we are having no issues but we will let you know once we have more news.

This looks like more a nginx configuration somehow than anything else. Looks like nginx receives some calls and then dies out. Reason? Well, only if we could debug nginx

[sanket-parte]

OK, thanks for the update