0

I had set up a Kubernetes cluster on four Raspberry Pis for toy projects awhile back, and I originally set things up using Docker. But ever since Kubernetes decided to drop support for Docker I decided to re-do the cluster using ContainerD instead. On top of that, I wiped the SD cards and installed the x64 version of Raspberry Pi OS (instead of the 32-bit version which I was using previously). I've slowly been piecing together a runbook of how to install things but have encountered a snag: five minutes after calling kubeadm init on the master node, the cluster stops working. For reference, the internal IP of my master node is 192.168.1.194. Here's the full command I use to initialize things:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
    --token-ttl=0 --apiserver-advertise-address=192.168.1.194

After running that command, and subsequently copying the /etc/kubernetes/admin.conf file to ~/.kube/config, I can run the following command:

$ kubectl get nodes

NAME           STATUS     ROLES                  AGE     VERSION
k8s-master-1   NotReady   control-plane,master   3m36s   v1.23.4

And it will continue to show a NotReady status for about 5 minutes, after which point the same command yields a very different result:

$ kubectl get nodes

The connection to the server 192.168.1.194:6443 was refused - did you specify the right host or port?

I'm not sure why this is happening, but it is very consistent. I have tried a few times now to kubeadm reset and then kubeadm init again, and the connection timeout always happens at the 5-minute mark. So the last time I tried to do this, I decided to tail all the log files under /var/log/containers/. After the 5-minute mark, it is repeatedly logging some variation of a connection error to 127.0.0.1:2379. For example:

2022-03-09T19:30:29.307156643-06:00 stderr F W0310 01:30:29.306871 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

But prior to the 5-minute mark, it looks like a bunch of services start shutting down right at the end, followed by what looks like maybe etcd shuts down as well. I've uploaded the full logs from the time kubeadm init runs, up until the dreaded 5-minute mark, as a Gist.

Why won't Kubernetes start on my Pi? What am I missing?

Improve this question
4
  • does all the ports are open? could you do test all ports like telnet localhost 2379. Ref docs for all the ports: kubernetes.io/docs/reference/ports-and-protocols
    – Vinod
    Commented Mar 16, 2022 at 16:28
  • Yes, all ports are open. The telnet command succeeds during that first 5 minute window of initialization, while etcd is active.
    – soapergem
    Commented Mar 17, 2022 at 2:41
  • if all steps are correct and it is failing then what i can think of is kubeadm join is not able to bring this node in cluster. there could be multiple reason for that. may be you could try to specify the --control-plane-endpoint while configuring the cluster which is basically your cluster endpoint. Ref: kubernetes.io/docs/setup/production-environment/tools/kubeadm/…
    – Vinod
    Commented Mar 17, 2022 at 7:02
  • Again, this is not one of the nodes but the master, so kubeadm join is not used.
    – soapergem
    Commented Mar 21, 2022 at 2:52

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.