Jenkins agent fails to connect to docker socket in latest helm chart

Question

Preface

I have been using older version of Jenkins on Kubernetes with the Kubernetes plug spinning up Jenkins Agents deployed by the official Helm charts. I've had no issues with them. In fact I currently have many clusters running Jenkins with K8s plugin spinning up jenkins agents all deployed via official helm charts working just fine. However they are slightly older versions. By slightly I might very slightly, like only 1 or 2 releases behind (roughly no more than a month). I plan on upgrading them if I had this one working correctly. Point is, this is nothing new to me. Just these particular versions. Now that you have the background.

Context

When a jenkins agent spins up and runs a docker command. Let's say docker build. If I configure in the kubernetes plugin settings to mount the /var/run/docker.sock I get permissions denied. If I remove it, I get Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. These are the agents. I already have the /var/run/docker.sock mounted on the "Controller" (in the new Jenkins Helm Chart v3.1.1), it use to be "Master". And I can run docker ps and see all of my pods running on that kubernetes node. So I know this is working just fine in master/controller.

I have built my own Jenkins images. But only to add tools such as kubectl, docker and aws cli, etc in. They are all from the official Jenkins Master and inbound-agent images. I have not changed anything with the system or permissions wise, except usermod -a -G docker jenkins. I am aware a little about how the inbound agent has changed how it connects to the docker socket. Though I'm still not fully clear on this.

I come here today, because I don't know if I should post this to Jenkins dockerhub repos, or the kubernetes plugin, or the jenkins helm chart repo? The kubernetes plugin settings certainly look different. And the chart changed a lot. They are now using statefulsets where as 2.x.x was not.

Question

Why since the latest change to the official Jenkins Helm Chart of version 3.x.x can my jenkins agents no longer are able to use the mounted docker socket like I am able to with older official helm chart version of 2.x.x.? Is anyone else running into this issue? Or have discovered new configuration changes that must be made in order to get the agents once again able to mount and use the docker socket successfully?

Here is my Helm values.yaml

controller:
  image: "<my-image>"
  tag: "<my-tag>"
agent:
  jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"
  jenkinsTunnel: "jenkins-agent.jenkins.svc.cluster.local:50000"
  namespace: "jenkins"
  image: "<my-image>"
  tag: "<my-tag>"
  imagePullSecretName: "registrykey"
  podName: "jenkins-agent"
persistence:
  size: "250Gi"

I added to the templates/jenkins-controller-statefuleset.yaml. I don't know if this is the best way. But I couldn't seem to find anyone else doing it any other way.

volumeMounts:
  - name: docker-socket
    mountPath: /var/run/docker.sock
volumes:
  - name: docker-socket
    hostPath:
      path: /var/run/docker.sock
      type: Socket

I checked off to have the raw yaml output in the console for debugging and this is what I get

Agent jenkins-agent-brp6s is provisioned from template jenkins-agent
---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    jenkins/jenkins-jenkins-agent: "true"
    jenkins/label-digest: "3756805d9a07b9b4e6e92045b5d40394dcc52845"
    jenkins/label: "jenkins-jenkins-agent"
  name: "jenkins-agent-brp6s"
spec:
  containers:
  - args:
    - "********"
    - "jenkins-agent-brp6s"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-agent.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "jenkins-agent-brp6s"
    - name: "JENKINS_NAME"
      value: "jenkins-agent-brp6s"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins"
    - name: "JENKINS_URL"
      value: "http://jenkins.jenkins.svc.cluster.local:8080"
    image: "123456789012.dkr.ecr.us-east-1.amazonaws.com/jenkins:inbound-agent-4.6-1"
    imagePullPolicy: "Always"
    name: "jnlp"
    resources:
      limits:
        memory: "512Mi"
        cpu: "512m"
      requests:
        memory: "512Mi"
        cpu: "512m"
    securityContext:
      privileged: true
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins"
  hostNetwork: false
  imagePullSecrets:
  - name: "registrykey"
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  serviceAccount: "jenkins-agent"
  volumes:
  - emptyDir:
      medium: ""
    name: "workspace-volume"

A short example of my Jenkinsfile

#!/usr/bin/env groovy

pipeline {
  agent any

  stages {
    stage('Build and Push Docker Image') {
      steps {
        script {
          sh "docker build -t <my-image:tag> ."
        }
      }
    }
  }
}

Everything inside the agent before the docker command works just fine. Everything else I have been trying to change and toggle to get this to work has been in the kubernetes plugin settings which is at jenkins-url.example.com/configureClouds/. The "Test Connection" says connected. I've tried changing the "Jenkins URL", and the "Jenkins tunnel" values to many things. They usually don't work. I've tried running it as "Privileged" and that did not work. Any idea what changed and why it would cause this issue, and any idea of how to get this connecting to the Jenkins master/controller docker socket again? Maybe I should start using the official docker command in the Jenkinsfile's rather than writing it inside a sh "..."?

• Jenkins Helm Chart Version: 3.1.1
• Jenkins Master Docker Image: jenkins/jenkins:2.263.1
• Jenkins Inbound-agent Docker Image Tag: jenkins/inbound-agent:4.6-1
• Kubernetes Plugin: 1.28.6
• Helm: 3.4.2
• Kubernetes: 1.18.8