Question on NGINX reverse proxy when there are services on the host and on containers

Question

I have the following architecture, which consists of an nginx server on the host machine, and 3 urls, app1.com, app2.com, app3.com:

app1 and app2 are both in the host machine, lets say they are an HTML file on the host, on the container, there is app3 and app2.com/example url.

My question is, how can I reference the container in the nginx of the host? Since I cannot expose port 80 and 443 of the container, because the host is already using these ports, I am not sure what is the best approach to this.

I though about putting the container on a not used port, like 8999 for example, and proxying the request to this port, and blocking these ports on the firewall, so no one outside can access them. If I go with this option, does the nginx configuration inside of the containers will need the server_name still? In the case of the app3.com/example

The other option I thougth about is using the docker socket, or even the docker0 network, but I don't know if this is viable.

An example of the nginx configuration file of the host:

server {
  server_name app1.com;
  location / { 
    root /var/www/app1/index.html;
  }
}

server {
  server_name app2.com;
  location / { 
    root /var/www/app2/index.html
  }

  location /example {
    proxy_pass http://localhost:8999;
  }
}

server {
  server_name app3.com;
  location / {
    proxy_pass http://localhost:8999;
  }
}

Example of the nginx configuration file of the container:

server {
  server_name app2.com
  location /example {
    root /var/www/app2/index.html;
  }
}

server {
  server_name app3.com;
  location / {
    root /var/www/app3/index.html;
  }
}

Answer 1

I though about putting the container on a not used port, like 8999 for example, and proxying the request to this port, and blocking these ports on the firewall, so no one outside can access them. If I go with this option, does the nginx configuration inside of the containers will need the server_name still?

You're thinking along the right lines. You do need the server_name directives in the container, since it can be accessed both by app2.com/example and app3.com.

Here's an example setup. I'll emulate an nginx on the host machine with a docker container running on the host network.

Scroll down past the code and console log for some explanations.

Here is the listing of the files I used:

lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ tail -n +1 *

==> app1.html <==

I am app1!

==> app2_example.html <==

I am app2 in example subpath!

==> app2.html <==

I am app2!

==> app3.html <==

I am app3!

==> container_nginx.conf <==

server {
  server_name app2.com;
  location /example {
    root /var/www/app2/;
  }
}

server {
  server_name app3.com;
  location / {
    root /var/www/app3/;
  }
}

==> host_nginx.conf <==

server {
  server_name app1.com;
  location / { 
    root /var/www/app1/;
  }
}

server {
  server_name app2.com;
  location / { 
    root /var/www/app2/;
  }

  location /example {
    proxy_pass http://localhost:8999;
    proxy_set_header Host $http_host;
  }
}

server {
  server_name app3.com;
  location / {
    proxy_pass http://localhost:8999;
    proxy_set_header Host $http_host;
  }
}

==> nginx.conf <==

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" "$http_host" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

==> run.sh <==

docker run --name host_nginx --network=host --rm -d \
  -v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
  -v `pwd`/host_nginx.conf:/etc/nginx/conf.d/default.conf \
  -v `pwd`/app1.html:/var/www/app1/index.html \
  -v `pwd`/app2.html:/var/www/app2/index.html \
  nginx

docker run --name container_nginx -p 8999:80 --rm -d \
  -v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
  -v `pwd`/container_nginx.conf:/etc/nginx/conf.d/default.conf \
  -v `pwd`/app2_example.html:/var/www/app2/example/index.html \
  -v `pwd`/app3.html:/var/www/app3/index.html \
  nginx

Here is the log from my console, where I am testing things out.

lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo ./run.sh 
73ebc99b8fae1d18bcbc571adbab6b65eaa49f861d66f11a7490d61b4e1beb0f
9f3761ab0619255d9bde08eee70e1fb9b686b645f38d976de7c39d4c8f0ad70d
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app1.com
I am app1!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com
I am app2!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app3.com
I am app3!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com/example
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com/example/
I am app2 in example subpath!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo docker logs host_nginx
127.0.0.1 - - [25/Apr/2020:15:11:15 +0000] "GET / HTTP/1.1" "app1.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:17 +0000] "GET / HTTP/1.1" "app2.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:20 +0000] "GET / HTTP/1.1" "app3.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:25 +0000] "GET /example HTTP/1.1" "app2.com" 301 170 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:29 +0000] "GET /example/ HTTP/1.1" "app2.com" 200 30 "-" "curl/7.68.0" "-"
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo docker logs container_nginx
172.17.0.1 - - [25/Apr/2020:15:11:20 +0000] "GET / HTTP/1.0" "app3.com" 200 11 "-" "curl/7.68.0" "-"
172.17.0.1 - - [25/Apr/2020:15:11:25 +0000] "GET /example HTTP/1.0" "app2.com" 301 170 "-" "curl/7.68.0" "-"
172.17.0.1 - - [25/Apr/2020:15:11:29 +0000] "GET /example/ HTTP/1.0" "app2.com" 200 30 "-" "curl/7.68.0" "-"

I corrected some errors in your config (missing semicolon here and there), changed the server roots to directories, rather than index.html and also the request for app2.com/example would request /var/www/app2/example/index.html so pay close attention to the way in which I mounted the files when starting up the containers in ./run.sh. Finally, it was necessary to do proxy_set_header Host $http_host; in the host nginx config, so that the requested Host header gets correctly relayed to the containerized nginx and both app2 and app3 work correctly.