0

Installation Process

I have been busy deploying a small high-available Kubernetes cluster with a Linux and Windows worker node. These two nodes are already deployed and working like a charm, but I also wanted to monitor these nodes and this is where the problem starts. I already successfully added the Linux node using the Fluent-Bit log shipper, but the Windows node won't work. I just wanted to monitor the workloads of the Node and already concluded that the following instructions don't work:

https://docs.fluentbit.io/manual/installation/windows

After some research I found the following link with some recently updated instructions:

https://docs.pivotal.io/tkgi/1-9/windows-logging.html

I followed the instruction, except I did not create the output plugin and deleted the output in the config-file like so:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit-win
  namespace: pks-system
--- 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit-win
  namespace: pks-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-win
  labels:
    app: fluent-bit
  namespace: pks-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     debug
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              C:\var\log\containers\*.log
        Parser            docker
        DB                /var/log/flb_kube1.db
        Skip_Long_Lines   On
        Refresh_Interval  60

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        DNS_Retries         10
        Kube_Tag_Prefix     kube.c.var.log.containers.
        K8S-Logging.Parser  On

  parsers.conf: |
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: fluent-bit
  name: fluent-bit-windows
  namespace: pks-system
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      nodeSelector:
        beta.kubernetes.io/os: windows
      tolerations:
      - key: "windows"
        operator: "Equal"
        value: "2019"
        effect: "NoSchedule"
      containers:
      - image: docker.io/<MY_REPO_NAME>/fluent-bit:windows
        imagePullPolicy: Always
        name: fluent-bit
        command: [fluent-bit]
        args:
        - --plugin
        - /syslog-plugin/out_syslog.so
        - --config
        - /fluent-bit/etc/fluent-bit.conf
        workingDir: /fluent-bit/etc
        volumeMounts:
        - mountPath: /fluent-bit/etc
          name: fluent-bit-config
        - mountPath: /var/log
          name: varlog
          readOnly: false
        - mountPath: /ProgramData/docker/containers
          name: dockercontainers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainers
        hostPath:
          path: /ProgramData/docker/containers
      - configMap:
          defaultMode: 420
          name: fluent-bit-win                                        
        name: fluent-bit-config
      serviceAccountName: fluent-bit-win
  updateStrategy:
    type: RollingUpdate

Expected

An successful deployment of the Fluent-Bit shipper onto the Windows worker node, which did not export any logs to an output. So the Fluent-Bit gathers the correct logs, but does not export the logs.

Environment

Master Nodes

OS: Ubuntu 18.04 LTS
Container Runtime: Docker 20.10.5
Kubernetes version: 1.21.0
Flannel-image version: 0.14.0
Kube-proxy version: 1.21.0

Windows Worker Node

OS: Windows Server 2019 Datacenter Core
Container Runtime: Docker 20.10.4
Kubernetes version: 1.21.0
Flannel-image version: 0.13.0-nanoserver
Kube-proxy version: 1.20.0-nanoserver #used because of following problem

Results

After the deployment I got the following error:

azureuser@Kube-M-001:~/fluent-bit/windows$ kubectl get pod -n pks-system
NAME                       READY   STATUS   RESTARTS   AGE
fluent-bit-windows-2mgps   0/1     Error    0          11s

Which eventually resulted into the following error:

azureuser@Kube-M-001:~/fluent-bit/windows$ kubectl get pod -n pks-system
NAME                       READY   STATUS             RESTARTS   AGE
fluent-bit-windows-2mgps   0/1     CrashLoopBackOff   7          11m

Also both kubectl logs fluent-bit-windows-2mgps -n pks-system and kubectl logs fluent-bit-windows-2mgps -p -n pks-system commands did not respond with any logs.

The kubectl get nodes -o wide command responded with the following results:

NAME         STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
kube-m-001   Ready    control-plane,master   30d   v1.21.0   10.0.10.4     <none>        Ubuntu 18.04.5 LTS               5.4.0-1047-azure   docker://20.10.6
kube-m-002   Ready    control-plane,master   29d   v1.21.0   10.0.10.5     <none>        Ubuntu 18.04.5 LTS               5.4.0-1046-azure   docker://20.10.6
kube-m-003   Ready    control-plane,master   29d   v1.21.0   10.0.10.6     <none>        Ubuntu 18.04.5 LTS               5.4.0-1046-azure   docker://20.10.6
kube-w-001   Ready    <none>                 29d   v1.21.0   10.0.64.5     <none>        Ubuntu 18.04.5 LTS               5.4.0-1046-azure   docker://20.10.6
kube-w-002   Ready    <none>                 42h   v1.21.0   10.0.64.4     <none>        Windows Server 2019 Datacenter   10.0.17763.1935    docker://20.10.5

The kubectl get events -n pks-system command results into the following response:

16m         Normal    Scheduled          pod/fluent-bit-windows-2mgps   Successfully assigned pks-system/fluent-bit-windows-2mgps to kube-w-002
14m         Normal    Pulling            pod/fluent-bit-windows-2mgps   Pulling image "docker.io/<MY_REPO_NAME>/fluent-bit:windows"
15m         Normal    Pulled             pod/fluent-bit-windows-2mgps   Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 145.4858ms
14m         Normal    Created            pod/fluent-bit-windows-2mgps   Created container fluent-bit
15m         Normal    Started            pod/fluent-bit-windows-2mgps   Started container fluent-bit
15m         Normal    Pulled             pod/fluent-bit-windows-2mgps   Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 121.3659ms
57s         Warning   BackOff            pod/fluent-bit-windows-2mgps   Back-off restarting failed container
15m         Normal    Pulled             pod/fluent-bit-windows-2mgps   Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 111.3591ms
15m         Normal    Pulled             pod/fluent-bit-windows-2mgps   Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 127.0707ms
14m         Normal    Pulled             pod/fluent-bit-windows-2mgps   Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 114.4943ms
16m         Normal    SuccessfulCreate   daemonset/fluent-bit-windows   Created pod: fluent-bit-windows-2mgps

And the kubectl describe pod fluent-bit-windows-2mgps -n pks-system command resulted into the following response:

Name:         fluent-bit-windows-2mgps
Namespace:    pks-system
Priority:     0
Node:         kube-w-002/10.0.64.4
Start Time:   Wed, 02 Jun 2021 09:14:37 +0000
Labels:       app=fluent-bit
              controller-revision-hash=679dfd8cbf
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           10.244.4.20
IPs:
  IP:           10.244.4.20
Controlled By:  DaemonSet/fluent-bit-windows
Containers:
  fluent-bit:
    Container ID:  docker://6b061fab2969048810a5fd8c1a6de7720c767026898e47f559d61b797932c62c
    Image:         docker.io/<MY_REPO_NAME>/fluent-bit:windows
    Image ID:      docker-pullable://<MY_REPO_NAME>/fluent-bit@sha256:a4f85570ec84668d44d6be9b15563761f9ada38b59e26ef6d7ecfb95a1e0c61b
    Port:          <none>
    Host Port:     <none>
    Command:
      fluent-bit
    Args:
      --plugin
      /syslog-plugin/out_syslog.so
      --config
      /fluent-bit/etc/fluent-bit.conf
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    -1073741515
      Started:      Wed, 02 Jun 2021 09:35:51 +0000
      Finished:     Wed, 02 Jun 2021 09:35:51 +0000
    Ready:          False
    Restart Count:  9
    Environment:    <none>
    Mounts:
      /ProgramData/docker/containers from dockercontainers (ro)
      /fluent-bit/etc from fluent-bit-config (rw)
      /var/log from varlog (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8gt4m (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
  dockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /ProgramData/docker/containers
    HostPathType:  
  fluent-bit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      fluent-bit-win
    Optional:  false
  kube-api-access-8gt4m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              beta.kubernetes.io/os=windows
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             windows=2019:NoSchedule
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  22m                   default-scheduler  Successfully assigned pks-system/fluent-bit-windows-2mgps to kube-w-002
  Normal   Pulled     22m                   kubelet            Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 145
.4858ms
  Normal   Pulled     22m                   kubelet            Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 121
.3659ms
  Normal   Pulled     22m                   kubelet            Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 111
.3591ms
  Normal   Pulled     21m                   kubelet            Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 127
.0707ms
  Normal   Started    21m (x4 over 22m)     kubelet            Started container fluent-bit
  Normal   Created    20m (x5 over 22m)     kubelet            Created container fluent-bit
  Normal   Pulling    20m (x5 over 22m)     kubelet            Pulling image "docker.io/<MY_REPO_NAME>/fluent-bit:windows"
  Normal   Pulled     20m                   kubelet            Successfully pulled image "docker.io/<MY_REPO_NAME>/fluent-bit:windows" in 114
.4943ms
  Warning  BackOff    2m18s (x91 over 22m)  kubelet            Back-off restarting failed container

What do I miss? Or is it to early to use Fluent-Bit as an Windows worker node log shipper?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.