Docker port mapping across several IPs on same NIC results in error

Question

I have docker stack running on a Raspberry Pi with docker-compose. It’s been going nicely for about a year now, but now I want to bring it to the next level.

The issue is, the stack contains a few services with website, which I access from the network. Obviously, that means I always have to remember the ports of each of those services, which has become tiresome after a while. And I do this from several devices in the network and sometimes several browsers on each device (I have my reasons :) ) so creating bookmarks isn’t an option.

So I thought I would solve this via DNS, as I already have pi-hole running in a container nicely as both a DNS and a DHCP server. For this to work I need, among others, the containers to listen to port 80.

Unfortunately, this is where the trouble shows.

The first thing I tried was to use ipvlan and macvlan networks. And it kind of works. But they still don’t solve the issue of remembering the ports, as the containers keep their current one. Port mapping doesn’t help here because of the reason below.

Another issue with this is that, as the containers also communicate to each other and there is a lot of existing configuration there, with IPs and ports, I have to connect them to 2 networks, a bridge for the inter-container communication, and an ipvlan for the outside communication. However, the port mapping is only one, and I don’t understand how it works when a container is connected to 2 networks. I thought this should have worked, but it didn’t.

So the next idea I had was to create several IPs on the host, then use the syntax “host_IP:port:container_port” to map as necessary.

So, for example, I have on the host

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether dc:a6:32:07:f8:85 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.10/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 10.0.0.50/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet 10.0.0.51/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet 10.0.0.52/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::185a:8f0e:78b0:a110/64 scope link
valid_lft forever preferred_lft forever

and then in docker-compose.yml I have (only relevant part):

grafana:
ports:
- “3000:3000”
- “10.0.0.50:80:3000”

nodered:
ports:
- “1880:1880”
- “10.0.0.51:80:1880”

Unfortunately, this throws a bunch of errors when I run docker-compose up -d

ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint grafana (b70ca36d6f0953ca16e046c7af561a638ce2fe537417600e6c7f7796fbf228c0): Error starting userland proxy: listen tcp4 10.0.0.50Recreating 922d1912d260_nodered … error

ERROR: for 922d1912d260_nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (6676c921fc83a1a521db50bce6ef128ec7811b4dae4d69cf0597b9f118199bfc): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use

ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint grafana (b70ca36d6f0953ca16e046c7af561a638ce2fe537417600e6c7f7796fbf228c0): Error starting userland proxy: listen tcp4 10.0.0.50:80: bind: address already in use

ERROR: for nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (6676c921fc83a1a521db50bce6ef128ec7811b4dae4d69cf0597b9f118199bfc): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use
ERROR: Encountered errors while bringing up the project.

It sounds a bit like docker can not share the same port even when it belongs to several IPs on the host. I have a limited understanding of this, but from what I read, they shouldn’t conflict. A NIC with several IPs assigned can handle all ports on each IP, as the port belongs to TCP and UDP. I googled this a lot and haven't been able to find any solution

I will appreciate any suggestions.

Answer 1

There are several ways to address this particular problem.

Using a dynamic proxy

Using a dynamic proxy is nice because you only need a single ip address and port. Incoming requests are directed to the appropriate container based on metadata about the request (such as the hostname, or particular headers, or the request path, etc).

This is actually what I'm doing at home: I have a number of web services running on my local system and I want to access them by name. I'm handling this by using Traefik, a dynamic proxy designed to work well with systems like Docker. Configuration is primarily via Docker labels.

So for example, if I start the Traefik container like this:

docker run -d --name traefik \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 80:80 \
    -p 127.0.0.1:8080:8080 \
    docker.io/traefik:v2.8 \
    --api.insecure=true \
    --providers.docker

(That -p 127.0.0.1:8080:8080 is exposing the Traefik dashboard on http://localhost:8080.)

Then I can make a web service available at grafana.local by starting it like this:

docker run -d --name grafana \
    -l 'traefik.http.routers.grafana.rule=Host(`grafana.local`)' \
  docker.io/nginx:mainline

Now any host on my network can access that service at http://grafana.local (assuming I have arranged for the hostname "grafana.local" to map to the address of the host running the traefik proxy).

I've used docker run commands here, but you can obviously set this all up using docker-compose instead.

Using multiple ip addresses

It sounds a bit like docker can not share the same port even when it belongs to several IPs on the host.

That is in fact not true. Consider this; on my host I have the following addresses assigned on eth0:

$ ip -4 addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    altname eno2
    altname enp0s31f6
    inet 192.168.1.175/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 83583sec preferred_lft 83583sec
    inet 192.168.1.180/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.181/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.182/24 scope global secondary eth0
       valid_lft forever preferred_lft forever

Let's start up a few containers:

docker run -d --name darkhttpd  -p 192.168.1.180:80:8080 docker.io/alpinelinux/darkhttpd
docker run -d --name nginx -p 192.168.1.181:80:80 docker.io/nginx:mainline
docker run -d --name httpd -p 192.168.1.182:80:80 docker.io/httpd:2.4

We can see that the containers are all running successfully:

$ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                        NAMES
283c8e061ebc   httpd:2.4               "httpd-foreground"       16 seconds ago   Up 15 seconds   192.168.1.182:80->80/tcp     httpd
0d2960c07212   nginx:mainline          "/docker-entrypoint.…"   18 seconds ago   Up 18 seconds   192.168.1.181:80->80/tcp     nginx
68e56ed8c180   alpinelinux/darkhttpd   "darkhttpd /var/www/…"   18 seconds ago   Up 18 seconds   192.168.1.180:80->8080/tcp   darkhttpd

And I get different services depending on which IP address I use:

$ curl 192.168.1.180
<html>
<head>
 <title>/</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
...
$ curl 192.168.1.181
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...
$ curl 192.168.1.182
<html><body><h1>It works!</h1></body></html>

---

Both options work. I like the proxy model, because it gives me a lot of flexibility (if instead of separate hostnames I want to mount services on different paths -- e.g., I want monitoring.local/grafana to go to grafana and monitoring.local/prometheus to go to prometheus -- I can do that). It also means I don't need to allocate a new ip address every time I have a new service.

Using multiple IP addresses is arguably simpler to set up, and as long as your configuration isn't changing all that often it doesn't really introduce much overhead.

---

It's hard to tell exactly what's going on with your specific configuration because you haven't really provided enough information to reproduce the behavior you're asking about. If the information in this answer isn't enough to solve your problem, I'd be happy to take a closer look if you can update your question to include an [mcve]: ideally a simplified docker-compose.yaml file that we can copy and run locally to reproduce the problem.