HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)

Question

I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.

Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:

apiVersion: v1
kind: Service
metadata:
  name: snginx
  labels:
    app: nginx
spec:
  type: NodePort
  externalTrafficPolicy: Local    #Cluster or Local
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
    nodePort: 30080

From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:

localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"

The haproxy admin log says:

Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"

However, I've disabled the firewall with

sudo systemctl disable --now firewalld

and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.

"load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).

Also, if I use static NAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.

What gives? What's the difference between the proxied request and curl?

Thank you.

Answer 1

I figured it out, and SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):

"SELinux only allows the web server to make outbound connections to a limited set of ports"

That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.

sudo semanage port --add --type http_port_t --proto tcp 30080

More specifically, I guess, the security context that HAProxy runs under is different than curl.