SSL issue while reaching kubernetes dashboard

Question

I'm configuring a kubernetes cluster (using microk8s) and cert-manager. Certificates work fine to traefik dashboard and to other sites but I have an issue with kubernates dashboard as it's already server via SSL (service is on port 443) and I currently expose a Let'sEncrypt certificate.

When I try to reach the dashboard the dashboard's pod logs:

2021/06/17 07:43:20 http: TLS handshake error from 213.215.191.83:39484: remote error: tls: bad certificate
2021/06/17 07:43:23 http: TLS handshake error from 213.215.191.83:39500: remote error: tls: bad certificate

The IngressRoute I'm using is:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard-k8s
  namespace: kube-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: "Host(`k.example.com`)"
      kind: Rule
      services:
      - name: kubernetes-dashboard
        port: 443
  tls:
    secretName: k8s-dashboard-http-cert

I have no clear idea of how should SSL work in this scenario. Would it be safe to have dashboard served on 80 and SSL terminated on traefik?

Answer 1

There are multiple ways to expose Dashboard through traefik.

1. Traefik terminates HTTPS from client and as client opens new HTTPS request to Dashboard.
   

     +-------------+       HTTPS       +-------------+        HTTPS      +-------------+ 
     |  Client     -----(session 1)-----  Traefik    ------(session 2)---- Dashboard   | 
     +-------------+                   +-------------+                   +-------------+

This is what you are using. The problem with your configuration is, on second session, Traefik will verify Dashboard's TLS certificate.
When Dashboard certificate is not signed by any of CA in Traefik list, it closes connection, then Dashboard raises error message.
If you want to use this scenario, you need to skip Dashboard certificate verification, or add Dashboard certificate to Traefik list of trusted CA.
Below configuration works by skipping Dashboard certificate verification.

    apiVersion: traefik.containo.us/v1alpha1
    kind: ServersTransport
    metadata:
      name: mytransport
      namespace: kube-system
    spec:
      serverName: "k.example.com"
      insecureSkipVerify: true
    ---
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: dashboard-k8s
      namespace: kube-system
    spec:
      entryPoints:
        - websecure
      routes:
        - match: "Host(`k.example.com`)"
          kind: Rule
          services:
          - name: kubernetes-dashboard
            port: 443
            serversTransport: mytransport
      tls:
        secretName: k8s-dashboard-http-cert

2. Traefik passes the TLS traffic to Dashboard without any processing.

     +-------------+                   +-------------+                   +-------------+ 
     |  Client     -------HTTPS-------Traefik(passthrough)---------------- Dashboard   | 
     +-------------+                   +-------------+                   +-------------+ 

In this scenario, Traefik does not do any TLS termination, the traffic passes as it is to Dashboard.
This works for me.

    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRouteTCP
    metadata:
      name: dashboard-k8s
      namespace: kube-system
    spec:
      entryPoints:
        - websecure
      routes:
        - match: HostSNI(`k.example.com`)
          services:
          - name: kubernetes-dashboard
            namespace: kubernetes-dashboard
            port: 443
      tls:
        passthrough: true

3. Traefik terminates HTTPS from client and open new plain HTTP request to Dashboard.
   

     +-------------+                   +-------------+                   +-------------+ 
     |  Client     -------HTTPS---------  Traefik    ---------HTTP-------- Dashboard   | 
     +-------------+                   +-------------+                   +-------------+ 

In this scenario, Dashboard need to provide plain HTTP termination.
I am not sure whether Dashboard support it.

Answer 2

There is 4th way:

• Instruct Kube dashboard to use your CA cert (not self-generated ones): Delete kube dashboard own certs:

kubectl -n kubernetes-dashboard delete secret kubernetes-dashboard-certs  

• Add cert from your CA: (note that the one of the dns name must be kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local )

kubectl -n kubernetes-dashboard create secret tls kubernetes-dashboard-certs --cert=<your cert open key pem> --key=<your cert private key>

• Edit kube dashboard deployment file for using your key:

kubectl edit -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml 

• add tls-cert-file/tls-key-file to container spec as shown below:

      spec:
        containers:
        - args:
          - --auto-generate-certificates
          - --namespace=kubernetes-dashboard
          - --tls-cert-file=/tls.crt
          - --tls-key-file=/tls.key

• Instruct Traefik to use your CA to check Kube dashboard certs (should go to kube-system namespace with key name ca.crt).

kubectl -n kube-system create secret generic <yourca> --from-file=ca.crt=<your ca open pem key>

• Modify ServerTransport

---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: mytransport
  namespace: kube-system
spec:
  serverName: kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local
  rootCAsSecrets:
  - <yourca>

• Use that transport in IngressRoute

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kube-dashboard
  namespace: kubernetes-dashboard
spec:
...
routes:
...
    services:
    - name: kubernetes-dashboard
      namespace: kubernetes-dashboard
      port: 443
      serversTransport: mytransport

Answer 3

If you want to use PathPrefix to distinguish services, beware that you may need to strip the prefix.

apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: skip-verify
  namespace: kube-system
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: kubernetes-dashboard-strip-prefix
  namespace: kube-system
spec:
  stripPrefix:
    prefixes:
      - /kubernetes-dashboard
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: PathPrefix(`/kubernetes-dashboard`)
      kind: Rule
      services:
        - name: kubernetes-dashboard
          port: 443
          serversTransport: skip-verify
      middlewares:
        - name: kubernetes-dashboard-strip-prefix