Kubernetes: Custom second scheduler container unable to start

Question

I tried to add a new scheduler to the master node as a static POD. I create the yaml file and save it. The pod as been created and his state is running but the container is unable to start.

the kubectl output:

NAME                                   READY   STATUS    RESTARTS   AGE
my-scheduler                           0/1     Running   5          28m

I tried to see reason in the events and saw the reason cannot join network of a non running container and the last event with error:

Startup probe failed: Get "https://127.0.0.1:10251/healthz": http: server gave HTTP response to HTTPS client

I tried to send GET request to http URL and got OK answer but https answer was a error:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

also I tried to get the scheduler log and got:

    I1016 18:47:15.779350       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.779753       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791161       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791191       1 registry.go:173] Registering SelectorSpread plugin
    W1016 18:47:15.794300       1 authorization.go:47] Authorization is disabled
    W1016 18:47:15.794518       1 authentication.go:40] Authentication is disabled
    I1016 18:47:15.794695       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251

scheduler file:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: my-scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --scheduler-name=my-scheduler
    - --port=10251
    - --secure-port=0
    image: k8s.gcr.io/kube-scheduler:v1.19.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: my-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

what am I doing wrong? What is the right way to config the scheduler?

Answer 1

In your manifest, you should set the spec.containers[0].livenessProbe.httpGet.scheme to HTTP not HTTPS.

This is because you use the option --port=10251. As the documentation said, you should not use the deprecated --port option anymore, because it expose the API insecurely.

You should set the following options: --port=0 --secure-port=10251 and then you would be able to set the spec.containers[0].livenessProbe.httpGet.scheme to HTTPS.