1

Kubernetes 1.5 cluster running in AWS. Node1 (worker) has an additional network interface assigned to it with a static IP, so if you run "ip addr show" on the node you see eth0, eth1, and the gang of virtual devices.

There is a webapp which is exposed using a NodePort service. The pod for the webapp is running on Node1. When I ssh into Node1 I can curl the webapp using the IP of eth0 or eth1, no problem. When I ssh into another worker node in the cluster, I can curl the webapp using eth0 but not eth1.

Can someone please advise on what I need to do to get eth1 to respond to requests to the webapp pod?

Here are the results of ss -laptn | grep kube:

LISTEN     0      128    127.0.0.1:10248                    *:*                   users:(("kubelet",pid=1270,fd=14))
LISTEN     0      128    127.0.0.1:10249                    *:*                   users:(("hyperkube",pid=2512,fd=5))
ESTAB      0      0      172.30.13.74:39240              172.30.10.12:8080                users:(("kubelet",pid=1270,fd=13))
ESTAB      0      0      172.30.13.74:45184              172.30.10.10:8080                users:(("kubelet",pid=1270,fd=30))
ESTAB      0      0      172.30.13.74:48968              172.30.10.11:443                 users:(("hyperkube",pid=2512,fd=26))
ESTAB      0      0      172.30.13.74:38854              172.30.10.10:8080                users:(("kubelet",pid=1270,fd=10))
ESTAB      0      0      172.30.13.74:42890              172.30.10.11:8080                users:(("kubelet",pid=1270,fd=20))
ESTAB      0      0      172.30.13.74:56880              172.30.10.10:8080                users:(("kubelet",pid=1270,fd=31))

It only lists the IP of eth0. The port being used for the NodePort service is listening on all interfaces though:

LISTEN     0      128         :::31102                   :::*                   users:(("hyperkube",pid=2512,fd=29))

And here is the iptables output, included because someone suggested that might be the problem:

sudo iptables -L -v
Chain INPUT (policy ACCEPT 39 packets, 9060 bytes)
 pkts bytes target     prot opt in     out     source               destination
 194M   44G KUBE-FIREWALL  all  --  any    any     anywhere             anywhere

Chain FORWARD (policy ACCEPT 6 packets, 328 bytes)
 pkts bytes target     prot opt in     out     source               destination
 368M   67G DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere
 181M   28G DOCKER     all  --  any    docker0  anywhere             anywhere
 179M   28G ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
 177M   38G ACCEPT     all  --  docker0 !docker0  anywhere             anywhere
19176 1151K ACCEPT     all  --  docker0 docker0  anywhere             anywhere

Chain OUTPUT (policy ACCEPT 43 packets, 7339 bytes)
 pkts bytes target     prot opt in     out     source               destination
 198M   73G KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */
 198M   73G KUBE-FIREWALL  all  --  any    any     anywhere             anywhere

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination
 368M   67G RETURN     all  --  any    any     anywhere             anywhere

Chain KUBE-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  any    any     anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Route table on sample node in cluster that I'm trying to curl from:

default via 172.30.11.1 dev eth0  proto dhcp  src 172.30.11.119  metric 1024
10.2.0.0/16 dev flannel.1
10.2.13.0/24 dev docker0  proto kernel  scope link  src 10.2.13.1
172.30.11.0/24 dev eth0  proto kernel  scope link  src 172.30.11.119
172.30.11.1 dev eth0  proto dhcp  scope link  src 172.30.11.119  metric 1024

Route table from the multi-homed node, the one I'm trying to curl to:

default via 172.30.13.1 dev eth0  proto dhcp  src 172.30.13.74  metric 1024
default via 172.30.13.1 dev eth1  proto dhcp  src 172.30.13.250  metric 1024
10.2.0.0/16 dev flannel.1
10.2.75.0/24 dev docker0  proto kernel  scope link  src 10.2.75.1
172.30.13.0/24 dev eth0  proto kernel  scope link  src 172.30.13.74
172.30.13.0/24 dev eth1  proto kernel  scope link  src 172.30.13.250
172.30.13.1 dev eth0  proto dhcp  scope link  src 172.30.13.74  metric 1024
172.30.13.1 dev eth1  proto dhcp  scope link  src 172.30.13.250  metric 1024
  • Is you AWS VPC configuration correct ? What you're routing table on the other node, what are the IPs of eth0 and eth1 ? – Tensibai Jun 19 '17 at 7:45
  • The VPC functions properly. It was an already established cluster, and inter-cluster traffic flows as expected. All nodes are in the same VPC, and the route table for it specifies that all traffic in the 172.30.0.0/16 subnet be routed locally. eth0 is 172.30.13.74, eth1 is 172.30.13.250. I added the route tables to the original question due to character limits. – user3177 Jun 19 '17 at 13:27
  • So you set two interfaces on the same subnet and wonder why you never get an answer from the second interface ? right ? – Tensibai Jun 19 '17 at 13:43
  • Yes. I get the feeling you're about to tell me something really obvious about this that I should have seen from the beginning... – user3177 Jun 19 '17 at 13:50
  • Well, statefull firewalls, when your node answers, it search its route table where to send the packet, the first match is the eth0 route, so the packet goes out from here with eth0 IP, but as it is an answer it doesn't match the destination the client did address, the firewall just drop the packet and you never get the answer – Tensibai Jun 19 '17 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.