Why apt command stuck with Waiting for headers in docker?

Question

I have an Ubuntu 19.04 based docker container:

docker run -d -v /home/adr:/root/adr -p 880:80 --name ubuntu-19.04-nginx ubuntu:19.04 tail -F /var/log/alternatives.log

on which I compiled an nginx and installed these packages:

apt-get update
apt-get install apt-file net-tools htop iputils-ping git curl wget build-essential libcanberra-gtk-module jq zlib1g-dev libssl-dev luajit luarocks libgeoip-dev libperl-dev libxslt1-dev libgd-dev libluajit-5.1-dev lua-cjson nano mlocate mc psmisc apt-utils telnet dnsutils iproute2

I then committed the image and created another container:

docker run --net albums-webapp -d -v /home/adr:/root/adrx -p 880:80 -h nginx --name nginx-lua --network-alias=nginx adrhc/ubuntu-19.04-nginx-lua tail -F /var/log/alternatives.log

on which I tried to install:

apt install mysql-client

which yields:

root@nginx:~# apt install mysql-client
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libaio1 libnuma1 mysql-client-5.7 mysql-client-core-5.7 mysql-common
The following NEW packages will be installed:
  libaio1 libnuma1 mysql-client mysql-client-5.7 mysql-client-core-5.7 mysql-common
0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.
Need to get 9459 kB of archives.
After this operation, 65.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Err:1 http://archive.ubuntu.com/ubuntu disco/main amd64 libnuma1 amd64 2.0.12-1
  Connection failed [IP: 91.189.88.162 80]
Err:2 http://archive.ubuntu.com/ubuntu disco/main amd64 libaio1 amd64 0.3.111-1
  Connection failed [IP: 91.189.88.31 80]
Ign:3 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-core-5.7 amd64 5.7.27-0ubuntu0.19.04.1
Err:4 http://archive.ubuntu.com/ubuntu disco/main amd64 mysql-common all 5.8+1.0.5
  Connection failed [IP: 91.189.88.162 80]
Err:3 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-core-5.7 amd64 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.174 80]
Ign:5 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-5.7 amd64 5.7.27-0ubuntu0.19.04.1
Ign:6 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client all 5.7.27-0ubuntu0.19.04.1
Err:5 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-5.7 amd64 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.31 80]
Err:6 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client all 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.174 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/n/numactl/libnuma1_2.0.12-1_amd64.deb  Connection failed [IP: 91.189.88.162 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/liba/libaio/libaio1_0.3.111-1_amd64.deb  Connection failed [IP: 91.189.88.31 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client-core-5.7_5.7.27-0ubuntu0.19.04.1_amd64.deb  Connection failed [IP: 91.189.88.174 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mysql-defaults/mysql-common_5.8+1.0.5_all.deb  Connection failed [IP: 91.189.88.162 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client-5.7_5.7.27-0ubuntu0.19.04.1_amd64.deb  Connection failed [IP: 91.189.88.31 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client_5.7.27-0ubuntu0.19.04.1_all.deb  Connection failed [IP: 91.189.88.174 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

but despite this issue the telnet command works on those ip & port:

root@nginx:~# telnet 91.189.88.174 80
Trying 91.189.88.174...
Connected to 91.189.88.174.
Escape character is '^]'.
^]
telnet> Connection closed.
root@nginx:~# telnet 91.189.88.162 80
Trying 91.189.88.162...
Connected to 91.189.88.162.
Escape character is '^]'.
^]
telnet> Connection closed.

Also ping to various servers (e.g. google, ubuntu) works fine.

Why apt install mysql-client doesn't work and what should I do in order to make it work?

Answer 1

Ok, it seems that the default docker MTU (1500) is not acceptable for me because I use a PPPoE Internet connection on the host on which I run the docker service:

ip addr show eth0
48: eth0@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

The solution is to create the network with an appropriate MTU:

docker network create --opt com.docker.network.driver.mtu=1492 albums-webapp
docker network inspect albums-webapp | grep -i mtu

Be aware that while no container is running the command below will still show the 1500 MTU:

ip link show | grep 4d725814c237 | grep mtu

Be also aware that newly created network must be allowed to go outside - so check it also with the firewall.

Also see the related issue: https://github.com/moby/moby/issues/28314