connection refused from webserver in container

Question

I've got a simple python http server running on port 8080.

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        print(self.path)
        self.wfile.write(b'\n')
        return

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
    """Handle requests in a separate thread."""

if __name__ == '__main__':
    server = ThreadedHTTPServer(('0.0.0.0', 8080), Handler)
    server.serve_forever()

fire up 'python3 balancer.py' and it works fine.

curl -X GET http://10.203.32.90:8080/?path=/project-2

works every time. from the same machine and others.

Now after creating a docker image from this script and deploy it in k8s cluster on port 32003 it does fire up. But when calling the service with

curl -X GET http://10.203.32.90:32003/?path=/project-2

a connection refused error sometimes is returned if it's called from another machine. and when the curl is done from the same machine more often than not a connection refused is returned. sometimes (1 out of 10) it works though.

laurijssen@ub:~/pablo$ curl -X GET http://ub:32003/
curl: (7) Failed to connect to ub port 32003: Connection refused
laurijssen@ub:~/pablo$ curl -X GET http://ub:32003/
{"success": "true", "code": 0}

laurijssen@ub:~/pablo$ curl -X GET http://ub:32003/
curl: (7) Failed to connect to ub port 32003: Connection refused

The docker file of the webserver:

FROM python:3.8-slim-buster
RUN pip install requests
ADD balancer.py /
CMD ["python3", "balancer.py"]

The yaml of the k8s service/deployment:

apiVersion: v1
kind: Service
metadata:
 name: loadbalance-svc
spec:
 selector:
   role: app
 ports:
   - protocol: TCP
     port: 8080
     targetPort: 8080
     nodePort: 32003
 type: NodePort
 externalName: lb.nl

---

apiVersion: apps/v1
kind: Deployment
metadata:
 name: loadbalancer-deployment 
 labels:
   role: app
spec:
 replicas: 1
 selector:
   matchLabels:
     role: app
 template:
   metadata:
     labels:
       role: app
   spec: 
     containers:
     - name: web
       image: "10.203.32.90:5000/loadbalancer"
       env:
       imagePullPolicy: Always

Dont know how to fix this. What logs to check? What could cause this sometimes it works, sometimes it doesnt behaviour?

Answer 1

After diving into the rabbit hole of netfilter and flannel routing / firewall rules it turns out to be something really simple.

There were multiple deployments with the same label!

When different pods have the same selector labels k8s selects the wrong one sometimes and then gives connection refused. You have to have unique names for the selectors. Weird that this is not logged anywhere which pod is selected or why it is refused. So the following is wrong because 'app' exists over multiple deployments.

apiVersion: v1
kind: Service
metadata:
 name: revproxy-svc
spec:
 selector:
   role: app
 type: NodePort

---

apiVersion: v1
kind: Service
metadata:
 name: loadbalance-svc
spec:
 selector:
   role: app
 type: NodePort

---

apiVersion: apps/v1
kind: Deployment
metadata:
 name: loadbalancer-python
spec:
 replicas: 1
 selector:
   matchLabels:
     role: app
 template:
   metadata:
     labels:
       role: app

---

apiVersion: apps/v1
kind: Deployment
metadata:
 name: revproxy
spec:
 replicas: 2
 selector:
   matchLabels:
     role: app
 template:
   metadata:
     labels:
       role: app