kubeadm init fails when using cri-o

Question

I've been following this doc to deploy kubernetes https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

After launching the "kubeadm init" command and during its excecution, this is the output:

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker. Here is one example how you may list all Kubernetes containers running in docker: - 'docker ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'docker logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster To see the stack trace of this error execute with --v=5 or higher

It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".

I've tried several things, but this is the content of my last try:

cat etc/sysconfig/kubelet

KUBELET_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster_domain=topota.madre"

systemctl cat kubelet

# /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=https://kubernetes.io/docs/

[Service] ExecStart=/usr/bin/kubelet Restart=always StartLimitInterval=0 RestartSec=10

[Install] WantedBy=multi-user.target

# /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/ Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS var EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sour EnvironmentFile=-/etc/sysconfig/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

cat /var/lib/kubelet/kubeadm-flags.env

is empty right now

Cri-O service ir running ok and Docker is not installed.

• kubelet version --short -> Client Version: v1.17.2
• kubeadm version -o short -> v1.17.2
• cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)
• crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]
• crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}
• grep -i cgroup_manager /etc/crio/crio.conf -> cgroup_manager = "systemd"
• lsmod | grep -i "overlay\|br_netfilter" -> ["br_netfilter","overlay"] (both ok).
• Other "extra args tested" (/etc/sysconfig/kubelet)

KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock

• Full output of some commands: -> "kube admin init --upload-certs" -> https://pastebin.com/isbAynEg -> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> https://pastebin.com/9pgtDMNu -> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"] -> "journalctl -xeu kubelet" -> https://pastebin.com/qreJyhsu (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).

Any help please?

Answer 1

I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7. One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).

I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:

cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external

My kubeadm init line is (I'm using an env variable to set the control plain endpoint):

kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs >> $HOME/log/init-cluster.log

I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.

cgroup_manager = "systemd"

Also make sure you are enabling the modules / sysconfig options specified on this page.

modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system