issue with waldur-queue and waldur-db #13

tyagisanjeev1211 · 2024-10-24T14:40:35Z

I have recently build solution using composure and getting error for Postgres database is not reachable

2024-10-24 14:33:41,196 WARNING django-csp could not be found, not updating Content-Security-Policy. Please make sure CSP is configured. This can be done by your reverse proxy, django-csp or a custom CSP handler via SAML_CSP_HANDLER. See https://djangosaml2.readthedocs.io/contents/security.html#content-security-policy for more information. This warning can be disabled by setting SAML_CSP_HANDLER=''in your settings. Checking Waldur MasterMind services... Database PostgreSQL connection [**ERROR**] Task runners (Celery workers) [OK] Queue and cache server (Redis) connection [OK]
and
File "/usr/local/lib/python3.11/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors raise ConnectionError(str(exc)) from exc kombu.exceptions.OperationalError: Error -2 connecting to **_waldur-queue:6379_**. Name or service not known.

and while creating user
File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.OperationalError: **_could not translate host name "waldur-db" to address_**: Name or service not known

The text was updated successfully, but these errors were encountered:

livenson · 2024-10-24T15:03:35Z

It seems that the PSQL container is not up. Can you please check the logs of the container?

tyagisanjeev1211 · 2024-10-24T16:07:49Z

DB container is working fine (showing healthy as well). however waldur-queue is not sable and showing exiting frequently.

livenson · 2024-10-24T16:22:20Z

ok, then what is the error in waldur-queue?

tyagisanjeev1211 · 2024-10-25T20:09:54Z

it is working now.

tyagisanjeev1211 · 2024-10-30T13:35:12Z

I am facing same issue again,
Please find logs of queue container and minstermind-worker container as below:

Logs of waldur-queue

redis 11:14:08.48 INFO  ==>
redis 11:14:08.48 INFO  ==> Welcome to the Bitnami redis container
redis 11:14:08.48 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis 11:14:08.48 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis 11:14:08.48 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis 11:14:08.49 INFO  ==>
redis 11:14:08.49 INFO  ==> ** Starting Redis setup **
redis 11:14:08.51 INFO  ==> Initializing Redis
redis 11:14:08.52 INFO  ==> Setting Redis config file
redis 11:14:08.55 INFO  ==> ** Redis setup finished! **

redis 11:14:08.56 INFO  ==> ** Starting Redis **
1:C 30 Oct 2024 11:14:08.577 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:C 30 Oct 2024 11:14:08.577 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 30 Oct 2024 11:14:08.577 * Redis version=7.4.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 30 Oct 2024 11:14:08.577 * Configuration loaded
1:M 30 Oct 2024 11:14:08.578 * monotonic clock: POSIX clock_gettime
1:M 30 Oct 2024 11:14:08.578 * Running mode=standalone, port=6379.
1:M 30 Oct 2024 11:14:08.579 * Server initialized
1:M 30 Oct 2024 11:14:08.580 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 30 Oct 2024 11:14:08.582 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 30 Oct 2024 11:14:08.582 * Ready to accept connections tcp

Logs from waldur-mastermind-worker

[2024-10-30 13:30:47,222: ERROR/MainProcess] consumer: Cannot connect to redis://:**@waldur-queue:6379/1: Error -2 connecting to waldur-queue:6379. Name or service not known..
Trying again in 32.00 seconds... (16/100)

[2024-10-30 13:31:19,312: ERROR/MainProcess] consumer: Cannot connect to redis://:**@waldur-queue:6379/1: Error -2 connecting to waldur-queue:6379. Name or service not known..
Trying again in 32.00 seconds... (16/100)

_CONTAINER ID  IMAGE                                       COMMAND               CREATED      STATUS                     PORTS                                     NAMES_
d0ae859c5b38  docker.io/library/postgres:12               postgres              2 hours ago  Up 2 hours (healthy)                                                 waldur-db
b7ddf2a1a7e8  docker.io/bitnami/redis:latest              /opt/bitnami/scri...  2 hours ago  Up 2 hours                                                           waldur-queue
6b5656d44eed  docker.io/library/postgres:10               postgres              2 hours ago  Up 2 hours (healthy)                                                 keycloak-db
095029fccf06  docker.io/umputun/docker-logger:latest      /srv/docker-logge...  2 hours ago  Exited (0) 1 second ago                                              logger
68e8573346b2  docker.io/opennode/waldur-mastermind:7.0.6  init-whitelabelin...  2 hours ago  Exited (1) 2 hours ago                                               waldur-mastermind-whitelabeling-init
097cf9e26fcf  quay.io/keycloak/keycloak:25.0              start --import-re...  2 hours ago  Up 5 seconds                                                         keycloak
beba9b7a03b4  docker.io/opennode/waldur-mastermind:7.0.6  initdb                2 hours ago  Exited (1) 2 hours ago                                               waldur-mastermind-db-migration
**9a71cbacb3d4  docker.io/opennode/waldur-mastermind:7.0.6  worker                2 hours ago  Up 29 minutes (unhealthy)                                            waldur-mastermind-worker**
d147eb28c2fe  docker.io/opennode/waldur-mastermind:7.0.6  beat                  2 hours ago  Up 33 minutes                                                        waldur-mastermind-beat
eba18fc6e19b  docker.io/opennode/waldur-mastermind:7.0.6  mastermind            2 hours ago  Up 2 hours                                                           waldur-mastermind-api
f43a764ad194  docker.io/opennode/waldur-homeport:7.0.6    /entrypoint.sh        2 hours ago  Up 2 hours (healthy)                                                 waldur-homeport
ea15ac314b8f  docker.io/library/caddy:2                   caddy run --confi...  2 hours ago  Up 2 hours                 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  caddy
exit code: 0

livenson · 2024-10-30T17:53:13Z

Can you please show logs of initdb container? docker logs initdb

tyagisanjeev1211 · 2024-10-30T20:31:13Z

logs for waldur-mastermind-db-migration

INFO: Welcome to Waldur Mastermind!
INFO: Creating user waldur 984:984
INFO: Create logging directory
'/etc/waldur/id_rsa' -> '/var/lib/waldur/id_rsa'
INFO: Spawning initdb
INFO: Initializing database...
/usr/local/lib/python3.11/site-packages/glanceclient/common/http.py:171: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if kwargs.get('cacert', None) is not '':
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not translate host name "waldur-db" to address: Name or service not known


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/waldur", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/src/waldur/src/waldur_core/server/manage.py", line 10, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: could not translate host name "waldur-db" to address: Name or service not known

and containers:

CONTAINER ID  IMAGE                                       COMMAND               CREATED             STATUS                         PORTS                                     NAMES
44e3d34871fe  docker.io/library/postgres:12               postgres              2 minutes ago       Up 2 minutes (healthy)                                                   waldur-db
29b58286b43c  docker.io/bitnami/redis:latest              /opt/bitnami/scri...  2 minutes ago       Up 2 minutes                                                             waldur-queue
22095297b90a  docker.io/library/postgres:10               postgres              2 minutes ago       Up 2 minutes (healthy)                                                   keycloak-db
731f5812c85f  docker.io/umputun/docker-logger:latest      /srv/docker-logge...  2 minutes ago       Up Less than a second                                                    logger
64b039788205  docker.io/opennode/waldur-mastermind:7.0.7  init-whitelabelin...  2 minutes ago       Exited (1) 28 seconds ago                                                waldur-mastermind-whitelabeling-init
c7b307f93ccc  quay.io/keycloak/keycloak:25.0              start --import-re...  2 minutes ago       Up 4 seconds                                                             keycloak
d2b2148c42a0  docker.io/opennode/waldur-mastermind:7.0.7  initdb                2 minutes ago       Exited (1) About a minute ago                                            waldur-mastermind-db-migration
ced48f8f1dcd  docker.io/opennode/waldur-mastermind:7.0.7  worker                2 minutes ago       Up 2 minutes (unhealthy)                                                 waldur-mastermind-worker
9abe31c23be7  docker.io/opennode/waldur-mastermind:7.0.7  beat                  2 minutes ago       Up 2 minutes                                                             waldur-mastermind-beat
8bb17520642b  docker.io/opennode/waldur-mastermind:7.0.7  mastermind            2 minutes ago       Up 2 minutes                                                             waldur-mastermind-api
1ba66d745cf8  docker.io/opennode/waldur-homeport:7.0.7    /entrypoint.sh        About a minute ago  Up About a minute (healthy)                                              waldur-homeport
2086dbef45c2  docker.io/library/caddy:2                   caddy run --confi...  About a minute ago  Up About a minute              0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  caddy

livenson · 2024-10-31T11:29:42Z

Symptoms are very similar to the issue described in docker/compose#9513 .
Basically for some reason service names are not resolvable.
Could you please consider upgrading your docker version?

tyagisanjeev1211 closed this as completed Oct 25, 2024

tyagisanjeev1211 reopened this Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

issue with waldur-queue and waldur-db #13

issue with waldur-queue and waldur-db #13

tyagisanjeev1211 commented Oct 24, 2024

livenson commented Oct 24, 2024

tyagisanjeev1211 commented Oct 24, 2024

livenson commented Oct 24, 2024

tyagisanjeev1211 commented Oct 25, 2024

tyagisanjeev1211 commented Oct 30, 2024

livenson commented Oct 30, 2024

tyagisanjeev1211 commented Oct 30, 2024

livenson commented Oct 31, 2024

issue with waldur-queue and waldur-db #13

issue with waldur-queue and waldur-db #13

Comments

tyagisanjeev1211 commented Oct 24, 2024

livenson commented Oct 24, 2024

tyagisanjeev1211 commented Oct 24, 2024

livenson commented Oct 24, 2024

tyagisanjeev1211 commented Oct 25, 2024

tyagisanjeev1211 commented Oct 30, 2024

Logs of waldur-queue

Logs from waldur-mastermind-worker

livenson commented Oct 30, 2024

tyagisanjeev1211 commented Oct 30, 2024

logs for waldur-mastermind-db-migration

and containers:

livenson commented Oct 31, 2024