My company is running an IoT project with a distributed K3s cluster running on OpenVPN. Some K3s nodes are installed at customer premises, usually in a SOHO LAN with NAT, some are running in the cloud (currently AWS). Therefore, all K3s nodes have OpenVPN installed first so that we can manage all nodes using SSH/Ansible and the K3s nodes can communicate with each other despite NAT boundaries. This involves setting a separate OpenVPN server in the cloud. All of this is running fine for over two years now.
Recent developments in overlay networks like Netmaker, Netbird and Tailscale etc have got me interested in order to overcome some of the constraints with the current OpenVPN solution like bandwidth, network topology, certificate management etc. I have tried all of these overlay network but I'm always running into the same problem with all of these overlay network types:
After installing and setting up the client software, I can verify that the basic network topology is working fine: All nodes (including my local machine) can ping all other hosts in the mesh. I can run iperf between any of these nodes showing me some decent performance - more than a magnitude faster than OpenVPN on the same hardware.
I can run kubectl get nodes -o wide showing me all nodes are Ready. I can run kubectl get pods -A -o wide and it's showing me all pods are running just fine. However, I'm unable to obtain any logs from any pod. It's always running into a timeout. Here's an example:
$ kubectl logs -n kube-system -l k8s-app=kube-dns
Error from server: Get "https://100.90.63.179:10250/containerLogs/kube-system/coredns-576bfc4dc7-6gb27/coredns?tailLines=10": Service Unavailable
However, I can curl the exact same URL from any node just fine. Using Ansible for example, the following works:
$ ansible k3s -a 'curl --include --insecure --silent https://100.90.63.179:10250/containerLogs/kube-system/coredns-576bfc4dc7-6gb27/coredns?tailLines=10'
[...]
Unauthorized
Note that k3s refers to all K3s nodes in the cluster. Of course, I'm getting an Unauthorized error message because I did not include the certificates for the MTLS in the request. But as you can see, basic connectivity is working except kubectl logs refuses to work.
I was also trying to look at the iptables output using ansible k3s -b -a 'iptables --list', but the output is puzzling me. I tried flushing the iptables using iptables -F on all hosts, but that didn't help, too.
And now after this very long text (sorry for that), my question: What am I missing? What could I do to troubleshoot the connectivity?
PS: Netbird and Tailscale both have documentation on how to setup their overlay network in an existing K8s cluster, e.g. using sidecar containers. However, I have to install the overlay network first or otherwise I would not be able to talk to all nodes, so these instructions do not apply to my use case.
PPS: I'm running K3s v1.30.1+k3s1. I've also tried v1.26.15+k3s1 - it made no difference.
P3S: The K3s service shows the following log entry:
Jun 18 20:09:31 gamma-d1 k3s[15144]: E0618 20:09:31.111091 15144 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://100.90.193.169:10250/containerLogs/kube-system/coredns-576bfc4dc7-tr4gq/coredns?tailLines=10", Err:(*errors.errorString)(0x400d66fef0)}: Get "https://100.90.193.169:10250/containerLogs/kube-system/coredns-576bfc4dc7-tr4gq/coredns?tailLines=10": Service Unavailable
P4S: I discovered that k3s/k8s seems to be biased against some CIDRs: My OpenVPN network has the CIDR 10.19/16 and the netbird network has the CIDR 100.90/16. I cannot change the netbird CIDR easily, but if I change the OpenVPN CIDR to 100.19/16, then the cluster shows the same (broken) symptoms than on the netbird network.
