1

I've been following this doc to deploy kubernetes https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

After launching the "kubeadm init" command and during its excecution, this is the output:

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker. Here is one example how you may list all Kubernetes containers running in docker: - 'docker ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'docker logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster To see the stack trace of this error execute with --v=5 or higher

It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".

I've tried several things, but this is the content of my last try:

cat etc/sysconfig/kubelet

KUBELET_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster_domain=topota.madre"

systemctl cat kubelet

# /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=https://kubernetes.io/docs/

[Service] ExecStart=/usr/bin/kubelet Restart=always StartLimitInterval=0 RestartSec=10

[Install] WantedBy=multi-user.target

# /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/ Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS var EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sour EnvironmentFile=-/etc/sysconfig/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

cat /var/lib/kubelet/kubeadm-flags.env

is empty right now

Cri-O service ir running ok and Docker is not installed.

  • kubelet version --short -> Client Version: v1.17.2
  • kubeadm version -o short -> v1.17.2
  • cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)
  • crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]
  • crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}
  • grep -i cgroup_manager /etc/crio/crio.conf -> cgroup_manager = "systemd"
  • lsmod | grep -i "overlay\|br_netfilter" -> ["br_netfilter","overlay"] (both ok).
  • Other "extra args tested" (/etc/sysconfig/kubelet)

KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock

  • Full output of some commands: -> "kube admin init --upload-certs" -> https://pastebin.com/isbAynEg -> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> https://pastebin.com/9pgtDMNu -> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"] -> "journalctl -xeu kubelet" -> https://pastebin.com/qreJyhsu (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).

Any help please?

  • What's the output when you follow instructions bellow: The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' – Weyder Ferreira Feb 10 at 19:28
  • I just added the links to pastebin with those outputs. I did not see anything conclusive in them, but there are some interesting lines that I may be overseeing. – Sebastián Greco Feb 11 at 11:45
  • What distribution are you using? – Matt Feb 11 at 21:59
  • I've just added the info to the original question. Thx for your help! – Sebastián Greco Feb 12 at 11:47
2

I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7. One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).

I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:

cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external

My kubeadm init line is (I'm using an env variable to set the control plain endpoint):

kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs >> $HOME/log/init-cluster.log

I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.

cgroup_manager = "systemd"

Also make sure you are enabling the modules / sysconfig options specified on this page.

modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system
| improve this answer | |
  • Thanks for your suggestions. Please review the question as I've updated it with new info and also with the outputs of running kubeadm init with your parameters. Still no luck :/ – Sebastián Greco Feb 11 at 11:46
  • What's your cri-o version? Can you show output from crictl version and crictl info? – LArntz Feb 11 at 12:34
  • Hi LArntz. I've just added the info to the original question. Thank you! – Sebastián Greco Feb 12 at 11:48
  • 1
    OK. I think this might be your problem.... crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"] See the compatibilty matrix on the cri-o github. You need to be running a version of cri-o that matches the kubernetes version. If there's not binary version available for your OS it's pretty easy to build (but I've heard of issues with redhat/centos 8). – LArntz Feb 12 at 12:05
  • Makes sense and matches your original answer. I overviewed this point as it is not mentioned in the "getting started" kubernetes doc. I'll try to build the package and comment here the output. Thank you! – Sebastián Greco Feb 12 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.