New Kubernetes Cluster: remote error: tls: bad certificate

Question

This is my first attempt at setting up a Kubernetes cluster in my test environment. In preperation, I created 3 instances running Fedora Atomic:

10.11.184.8: master/etcd
10.11.184.5: node01
10.11.184.6: node02

Then using contrib/ansible playbooks for Ansible, I deployed kubernetes to my instances. It completed with "0" failures for each host.

I then connect to my master and begin to check that status of it:

[root@kubemaster ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}  

However, I go to check my nodes and it returns nothing:

[root@kubemaster ~]# kubectl get nodes
No resources found.

I begin to check logs, and I find the below repeating over and over from all three of the IPs listed above:

http: TLS handshake error from 10.11.184.5:32788: remote error: tls: bad certificate

It appears that the ansible playbook did generate some certificates:

[root@kubemaster ~]# ll /etc/kubernetes/certs/
total 40
-r--r-----. 1 kube kube-cert 1220 Aug 15 19:11 ca.crt
-r--r-----. 1 kube kube-cert 4417 Aug 15 19:11 kubecfg.crt
-r--r-----. 1 kube kube-cert 1704 Aug 15 19:11 kubecfg.key
-rw-rw----. 1 root kube-cert 4417 Aug 15 19:11 kubelet.crt
-rw-rw----. 1 root kube-cert 1704 Aug 15 19:11 kubelet.key
-r--r-----. 1 kube kube-cert 4917 Aug 15 19:11 server.crt
-r--r-----. 1 kube kube-cert 1704 Aug 15 19:11 server.key

And the kube-apiserver binary is being passed these as a parameter:

/usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://10.11.184.8:2379 --insecure-bind-address=127.0.0.1 --secure-port=443 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --service-account-key-file=/etc/kubernetes/certs/server.crt --bind-address=0.0.0.0 --apiserver-count=1

If I view the logs on one of the nodes, I see a lot of messages like so:

Failed to list *v1.Node: Get https://10.11.184.8:443/api/v1/nodes?fieldSelector=metadata.name%3D10.11.184.8:443/api/v1/nodes?fieldSelector=metadata.name%3D10.11.184.5&resourceVersion=0: x509: certificate is valid for 10.254.0.1, 192.168.0.11, 172.16.63.0, 172.16.63.1, not 10.11.184.8

Is it likely that my certificates were not generated correctly? I am looking at this documentation. Perhaps I should attempt to manually create the certificates using openssl or easyrsa?

Answer 1

From the documentation you linked, it states,

The referenced file must contain one or more certificates authorities to use to validate client certificates presented to the API server.

This means your client's certificate is invalid because either:

1. The server doesn't trust the client's signing certificate authority
2. The client doesn't trust the server's signing certificate authority
3. The certificate's DN doesn't match the hostname

The most likely scenario is that 1. The server doesn't trust the client's signing certificate authority since the server doesn't verify DNS for the client certificate and the error indicates this is a remote error not on the client. Most likely this means the client's certificate was signed on the client itself, not on the server.

You will need to make sure to generate the certificate on the client, have the server sign it and then transfer the certificate back to the client. Otherwise, you need to add the server's CA certificate to the Client's keystore.. You can ensure you do not have a DNS/DN mismatch by setting hosts file entries.