Sending HTTP requests to private EC2 DNS from other EC2 host works, but not from container on EC2 host

Question

I want to make a container hosted on one EC2 host communicate with another container hosted on another EC2 host.

Currently, if I send an HTTP request from host A to host B using host B's private DNS and the exposed container port, I receive a response correctly. However, if I try the same thing within the container on host B, I get a "Host is unreachable" error. To me, this would indicate it is not a host networking issue, which is configured in AWS, so I won't enter into detail on it.

What puzzles me is that I have done the exact same thing in our pre-production environment and it works perfectly.

What could be the issue and where do you think I should look to troubleshoot this?

Answer 1

There are a couple of things that you can rule out, thanks to the testing that you have already done. You know your container port has an active network socket, that is listening for and accepting connections. You also know that your EC2 instance has working networking.

I can think of three obvious problems:

1. Your host is using local DNS resolution to identify your hostname as 127.0.0.1 and your container port is not listening on the loopback adapter. you can test this on the server by pinging your DNS name and see which IP address it pings. You can also use telnet to connect to the port on 127.0.0.1 and see if that port accepts connections. To fix this, you may choose to make listening on the local loopback work, but you could also fix this by using a DNS entry other than the hosts AWS generated name.

2. You are resolving the hostname to its private IP address, which then is filtered by the AWS Security Group. On your security group you have enabled incoming connections to this port, but you have specified the source either as a cidr that doesn't include its own IP address, or as security group that is not attached to this EC2 instance. This is easy to fix in the security group editor. To prove that this is working successfully and allowing the connection, you can set up an AWS Flow log which will allow you to see all TCP connections on the Network interface.

3. The Container hosting which provides the port mapping also has a firewall element that restricts incoming connections to only some IP addresses, and may also provide some diagnostics to assist in understanding whether it receives the connection, and whether it decides to route or reject the connection to the container.

If it is less obvious, i think you may be in the world of using IP Tables to generate log messages in the syslog as your connection routes though the IP stack. This will at least allow you to confirm that at a network level you are generating a TCP connection and receiving the connection within the OS.