Running builds requiring docker daemon in Jenkins installed using helm and running as containerd container

Question

Jenkins is installed using official helm chart on Kubernetes cluster that uses containerd as container runtime.

How to configure containerized agents which will run docker client commands e.g. for building docker images.

There's no docker daemon on Kubernetes nodes. Is there an option to use containerd client which will interact with containerd server on the nodes? The idea is similar to case where docker client inside the container interacts with docker daemon running on the node.

I'm also thinking about below options for Jenkins agents:

• docker daemon in docker
• sysbox containers having docker daemon

Is there a best practice?

Answer 1

There are ways to do it but I've not been able to manage it. I would recommend sidestepping the problem by using kaniko. You can add a purpose built kaniko container into your 'build' agent k8s podTemplate and let it handle the container build (and push) part of the workload with no docker daemons or sockets necessary.

It's working well for me and I am also using jenkins via helm on k8s. Here is a snippet of my helm chart values.yaml agent.podTemplates section.

Note that the 'builder' agent contains 2 containers (actually 3 including the default 'jnlp' container which handles communicating with the jenkins controller) and they share a common workspace:

agent:
  namespace: jenkins
  podTemplates:
    builder: |
    - name: 'builder'
      label: 'builder'
      serviceAccount: jenkins
      nodeUsageMode: EXCLUSIVE
      yamlMergeStrategy: "merge"
      yaml: |-
        apiVersion: v1
        kind: Pod
        metadata:
          name: builder
          namespace: jenkins
        spec:
          serviceAccountName: jenkins
          containers:
          - name: builder
            image: builder:latest
            imagePullPolicy: IfNotPresent
            command: ["sleep"]
            args: ["99d"]
            tty: true
            resources:
              requests:
                cpu: 1
                memory: 1Gi
              limits:
                cpu: 2
                memory: 2Gi
            workingDir: /home/jenkins/agent
          - name: kaniko
            image: gcr.io/kaniko-project/executor:debug
            imagePullPolicy: IfNotPresent
            command: ["sleep"]
            args: ["99d"]
            tty: true
            volumeMounts:
            - name: docker-config
              mountPath: /kaniko/.docker
            workingDir: /home/jenkins/agent
            env:
            - name: DOCKER_REPOSITORY
              value: "private_docker_repository"
          volumes:
          - name: docker-config
            configMap:
              name: docker-config

Note that I used the yaml: |- field because it was the only way I could find to mount a volume containing the configMap needed by kaniko for docker authorization, configuration reference. So I also kubectl applied a simple docker-config configMap in the jenkins namespace first before applying this helm chart values.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-config
  namespace: jenkins
data:
  config.json: |-
    {
       "credsStore": "ecr-login"
    }

Find more details on what that docker-config needs to contain for you here.

And then here is an example of using the different containers in a declarative Jenkinsfile script:

pipeline {
  
  agent {
    label 'builder' // Refers to the custom k8s pod template defined in the jenkins helm chart values.yaml
  }

  environment {
    RELEASE_TAG = "1.0.0"
  }
  
  stages {
    stage('Build') {
      steps {
        container('builder') { // refers to the 'builder' container within the 'builder' agent pod
          echo "do a maven build or whatever other steps you need to take here"
        }
        container('kaniko') { // refers to the 'kaniko' container within the 'builder' agent pod sharing the same workspace
          /kaniko/executor --dockerfile `pwd`/dockerfile --context `pwd` --destination=${DOCKER_REPOSITORY}:${RELEASE_TAG}
        }
      }
    }
  }
}

Answer 2

Eventually, I chose to use docker daemon in docker as a long running pod and Jenkins agents that have docker client inside triggered on demand. They interact with containerized Docker daemon.

I summarized my choice of Jenkins Docker in Docker Agent.