network conflict

[QThans]

To Reproduce

using two same docker-compose to create services.

Current vs. Expected behavior

if deploy 2 compose has same configuration, the second deploy will using the first. for example, if there are all have pg, the second service will connect to the pg from the first service.

Provide environment information

Operating System:
    OS: Ubuntu 22.04

Which area(s) are affected? (Select all that apply)

Docker Compose

Are you deploying the applications where Dokploy is installed or on a remote server?

Same server where Dokploy is installed

Additional context

No response

Will you send a PR to fix it?

Yes

[Siumauricio]

Hi, yes you are right, this is a bug that I am seeing for some time you can not deploy 2 equal templates because it is possible to mix the information and cause bad gateway errors or something like that and it is because in the dokploy-network the service names must be unique, currently I have not found a way to solve it that is elegant, I know it is something related to networks but I have no idea to solve it, I have tried different methods but none with the success I want without having to do a lot of magic behind compose.

If you have any idea how to solve it in an elegant and direct way that is not something so manual would be great

I believe is related to this #821

[QThans]

#821

maybe, You can add a random variable which can into docker-compose?

[QThans]

how about insert service-name into configuration.
if the original name is pg, we can let it be like: pg-{service-name}. then we can use pg-{service-name} to connect. also can consider insert the random string.
@Siumauricio

[QThans]

I think this can solve the problem of official templates at least.

[Siumauricio]

Yes, I was thinking about some of that, I just didn't like it because it was something that had to be done manually and I don't know how comfortable people would feel doing that.

I assumed that adding --project-name would add the prefix to all containers, and it does but in the internal way of communication it directly uses the name of the service, so if you have a service like db in multiple compose, there might be problems.

https://docs.docker.com/reference/cli/docker/compose/#options

yes a temporary solution is to add a prefix to every container

[TheOnlyWayUp]

@Siumauricio is prefixing names a solution to #821 as well?

[Siumauricio]

Yes @TheOnlyWayUp , that really should be the only solution, add a prefix to each container, to prevent containers from colliding with names.

[vishalkadam47]

Randomize Compose works but each time you create a service specially from the templates we need to randomize and update a domain, which is a manual work again.

but i was thinking if there is a way to reuse the Randomize Compose component for each deployment even that would work instead of rewriting the entire code for prefix

[TheOnlyWayUp]

@Siumauricio, I believe this solution from @theboringhumane fixes this issue as well - #821 (comment).

[Siumauricio]

@TheOnlyWayUp Can you add steps to fix the issue? Or if there's anything from dokploy codebase change should I make

[nktnet1]

@Siumauricio the way that Coolify handles this is by assigning a UUID to each compose resource, and ensuring that they run on a private network named after the resource UUID (instead of automatically being on the global dokploy-network).
I previously wrote about these suggestions here:

• Fix metabase app template's compose file (networking setup) #1129 (comment)
• Warn users in the UI about needing to add "dokploy-network" to all services when configuring a domain #1067 (reply in thread)

This means that common service hostnames like db and redis will not conflict across services in different compose resources by default (as they are not on the same, global network).

This will be the case for the majority of resources.

---

Additionally, similar to what @QThans mentioned, Coolify implicitly adds the resource UUID suffix to the services' container name, volumes, etc - this is visible when you click on "Show Deployable Compose", the equivalent of Dokploy's "Preview Compose" feature.
This is to prepare for the case where services in the resource "opt-in" to be exposed on the global network in order to use (or be used by) external services in other compose resources.
Examples are API services like gotenberg, and metabase/superset (or any service) that makes use of external databases outside their own compose resource.

When "opt-in", these services are exposed on the global network with a suffixed hostname, i.e. service-{UUID}, and will now need to use this new hostname format to talk to each other (or for external services to talk to them).

For example, suppose we "opt in" to expose the superset compose resource, which contains the superset, db, and redis services, to the global dokploy-network, and the generated resource UUID is abcdefg.
We will now get unique hostnames like superset-abcefg, db-abcefg, redis-abcdefg, etc on the global dokploy-network.

Following Coolify's example, environment variables such as DOKPLOY_RESOURCE_UUID and DOKPLOY_CONTAINER_NAME can be exposed to each service in case they need to make used of it at runtime
I described a scenario where this was necessary, e.g. for superset, in my PR to Coolify here.

---

This is not perfect and not saying Dokploy needs to mimic it completely, although personally, I think it is a flexible design that was easy to use and understand. More docs here:

• https://coolify.io/docs/knowledge-base/docker/compose/#connect-to-predefined-networks

[Siumauricio]

Thanks @nktnet1 for the explanation and Yes, the truth is that I have thought about that solution, however it seems to me quite a lot of work to be naming for example an environment variable with the prefix of the UUID, which is probably the most optimal solution to prevent any kind of conflict, although I think that coolify internally after deploying the docker compose makes some kind of connection between networks manually.

[nktnet1]

it seems to me quite a lot of work to be naming for example an environment variable with the prefix of the UUID

Do you mean a lot of work for users, or for this to be implemented in Dokploy?

I think the majority of the time, the services that you would "opt-in" to be on the global dokploy-network are those that wants to use a database - the database would already expose an "internal URL" which can be copy/paste as an environment variable.

In the less common scenario where you want to reference an API service (like gotenberg), perhaps there could be some dropdown selections to automatically construct an internal URL for these too?

We can infer from things like the docker-compose service name, protocol, port, and resource UUID, i.e. generating something like

# Internal URL inputs:
# - protocol: http
# - service name: gotenberg
# - resource UUID:  `abcefg`
# - port: 3000

# Internal URL output:
http://gotenberg-abcdefg:3000

to mimic the behaviour of a database's internal URL.

[Siumauricio]

So you mean that before creating the template, in runtime we add a prefix to all services, and expose an environment variable that is DOKPLOY_SERVICE_UUID for example, so that users can manipulate the connection to the services?

I think adding documentation would make it easier, I will try to implement a solution for this, only we would have to change the templates.

Example if we have this wordpress template:

version: "3.8"
services:
  wordpress:
    image: wordpress:6.7.1
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress_data:/var/www/html

  db:
    image: mysql:5.7.34
    networks:
      - dokploy-network
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_ROOT_PASSWORD: rootpass
    volumes:
      - db_data:/var/lib/mysql

volumes:
  wordpress_data:
  db_data:

So we might need to convert to like this
DOKPLOY_SERVICE_UUID = abcdef

version: "3.8"
services:
  wordpress-${DOKPLOY_SERVICE_UUID}:
    image: wordpress:6.7.1
    environment:
      WORDPRESS_DB_HOST: db-${DOKPLOY_SERVICE_UUID}
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress_data:/var/www/html

  db-${DOKPLOY_SERVICE_UUID}:
    image: mysql:5.7.34
    networks:
      - dokploy-network
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_ROOT_PASSWORD: rootpass
    volumes:
      - db_data:/var/lib/mysql

volumes:
  wordpress_data:
  db_data:

Then in runtime, we will apply the changes so that the template automatically has the assigned values and becomes this

version: "3.8"
services:
  wordpress-abcdef:
    image: wordpress:6.7.1
    environment:
      WORDPRESS_DB_HOST: db-abcdef
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress_data:/var/www/html

  db-abcdef:
    image: mysql:5.7.34
    networks:
      - dokploy-network
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_ROOT_PASSWORD: rootpass
    volumes:
      - db_data:/var/lib/mysql

volumes:
  wordpress_data:
  db_data:

Correct me @nktnet1 If I'm wrong, what do you think? this will be apply only to templates

[nktnet1]

Thanks for responding with a concrete example @Siumauricio.
I do think this is another way to go about it, although it's a bit more explicit (arguable whether that's good or bad).

---

The way Coolify does it is a bit more implicit - the edtiable (source) template stays the same from a user's perspective.

While the resource UUID is generated when they initially create a resource (e.g. docker compose), it is used under the hood and is only visible when they click "Show Source Compose", the equivalent of this button in Dokploy:

Users don't have to worry about the existence of the UUID - it's like "magic" behind the scene.
Hopefully this video makes it a bit clearer - things to look out in the "Preview Compose" are:

1. the container_name is inserted, e.g. wordpress-abcdef
2. A resource network is created with the UUID as the name, e.g. abcdef
3. volumes are also prepended, e.g. abcdef_wordpress_data (not sure why prepend instead of append)

coolify-preview-compose-demo.webm

Now, only when users "opt-in" to be on the global network,

that's when wordpress-abcdef hostname gets exposed (as an alias) on the global network - users can technically still use wordpress hostname for services within this resource, there's just no guarantee of it not conflicting with another wordpress that may exist on the global network, hence why it is advised to use the wordpress-abcdef hostname instead.

---

I guess the main point I'm getting at is, ideally, everything regarding naming should still work the same as it currently does in Dokploy without the existence of UUID, when within a single compose.
However, when connecting with services in other compose stacks, users will need to be mindful of the extra -{UUID} suffix in the hostname, which is used to avoid conflicts.

Hope this makes sense ^^.

---

I think your suggested solution also has its benefit, with everything being explicitly viewable and editable by the user (less magic). I'm also happy with that solution, as it ultimately solves the problem at hand.

[Siumauricio]

I think the randomize feature is aligning with that behaviour, it would take little change to adapt the logic.

Here is the PR, I tried it and it works, at the end I reused some methods of the randomizeCompose method, and in runtime I add the prefixes to the container_names and the nets.

Implemented in #1276

deployable

and then we internally create the network and asociate the dokploy-traefik to the network

btw activating this option doesn't add the dokploy-network to any of the service

thanks @nktnet1

https://github.com/Dokploy/dokploy/pull/1276/files#diff-f9cc68879adf21866cfd9c3f794ccc2c914313263081bab55b29c6156f6faf05R37-R82

[nktnet1]

Thanks @Siumauricio for the speedy pull request 🚀🚀!

I'll take a look at it, and will move future discussions to #1276.