I have a k8s cluster (three vms on my own hardware; no aws, google cloud, ...) that uses traefik (https://traefik.io/) as a reverse proxy to address services/deployments in the background.
For this I use the deployment-variant from this part of the documentation: https://docs.traefik.io/user-guide/kubernetes/#deploy-trfik-using-a-deployment-or-daemonset
Now I have multiple applications deployed in the cluster, which all have some ingresses assigned, which are read by the traefik-ingress-controller. Some of those applications are internal ones, like kibana or the traefik-web-ui, and some others are external, like the applications themselves. I distinguish those two by having different dns entries (like
https://app1.external.mycoolapp.com) and the internal dns is not resolved from the outside world (=the internet) whereas the external is (like from google dns).
That's for the setup. Now, let's come to the problem:
A couple of days ago, I thought: What happens, if I create a wildcard dns entry for
*.internal.mycoolapp.com on a machine, that is outside my network, and just resolve it to the same ip(s) the external dns entry resolves to. Et voila, my internal services are accessible from the outside!
So this is, of course, a state which is not acceptable. So I'm searching for solutions on this.
What first came to mind was to block all incoming requests on rules for internal services, if the host ip of the requester is outside of our network:
... kind: Ingress metadata: name: app1 namespace: default annotations: traefik.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8" ingress.kubernetes.io/whitelist-x-forwarded-for: "true" ...
Theoretically this should work. But as I found out later on, before reaching the traefik-ingress-controller, all requests are handled by kube-proxy and their host addresses are translated to local addresses ((S)NAT), so every request has an internal host address set.
So this is the point I am currently looking for a solution.
One solution supposedly is to deploy the traefik-ingress-controller not as a deployment, but as a daemon set and bind it to the ports on the host directly (as said here https://docs.traefik.io/user-guide/kubernetes/#deploy-trfik-using-a-deployment-or-daemonset). I tried that yesterday by just changing my traefik configuration to a daemon set and add the
NET_BIND_SERVICE capability to it, but it didn't really change anything there. So has someone any idea what I might have done wrong there? Or has someone a good how-to/tutorial/... on how to pass the actual requester host through to the ingress controller?
Here's my current configuration file for traefik:
--- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: - image: traefik:v1.6.4 name: traefik-ingress-lb volumeMounts: - mountPath: /ssl/external name: ssl-external - mountPath: /ssl/internal name: ssl-internal - name: traefik-toml subPath: traefik.toml mountPath: /config/traefik.toml ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: admin containerPort: 8080 args: - --configfile=/config/traefik.toml - --api - --kubernetes - --logLevel=INFO volumes: - name: ssl-external secret: secretName: external.mycoolapp.com.cert - name: ssl-internal secret: secretName: internal.mycoolapp.com.cert - name: traefik-toml configMap: name: traefik-toml --- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system spec: selector: k8s-app: traefik-ingress-lb ports: - protocol: TCP port: 80 name: web - protocol: TCP port: 443 name: sweb externalIPs: - 10.2.3.1 - 10.2.3.2 - 10.2.3.3
The toml-file just contains http-to-https redirect and the paths to the certificate files.
I have some other solutions in mind, but they do all have the downside that they need more things deployed and maintained, like a second ingress controller on another port, that handles only specific ingress objects, or a new entrypoint for external requests in front of the whole cluster, that deletes host headers from requests, when it is not my external address (does this even work with ssl?).
So I really need some help on this. Thanks in advance!