1

So Jenkins is installed inside the cluster with this official helm chart. And this is my installed plugins as per helm release values:

  installPlugins:
  - kubernetes:1.18.1
  - workflow-job:2.33
  - workflow-aggregator:2.6
  - credentials-binding:1.19
  - git:3.11.0
  - blueocean:1.19.0

my Jenkinsfile relies on the following pod template to spin up slaves:

kind: Pod
spec:
  # dnsConfig:
  #   options:
  #     - name: ndots
  #       value: "1"
  containers:
  - name: dind
    image: docker:19-dind
    command:
    - cat
    tty: true
    volumeMounts:
    - name: dockersock
      readOnly: true
      mountPath: /var/run/docker.sock
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
  volumes:
  - name: dockersock
    hostPath: 
      path: /var/run/docker.sock

Slaves (pod /dind container) starts nicely as expected whenever there is new Build.

However, it broke at the step of "docker build" in ( Jenkinsfile pipeline docker build -t ... ) and it breaks there :

Step 16/24 : RUN      ../gradlew clean bootJar

 ---> Running in f14b6418b3dd

Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip


Exception in thread "main" java.net.UnknownHostException: services.gradle.org

    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)

    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)

    at java.base/java.net.Socket.connect(Socket.java:591)

    at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)

    at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)

    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)

    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)

    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)

    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)

    at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)

    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)

    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)

    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)

    at org.gradle.wrapper.Download.downloadInternal(Download.java:67)

    at org.gradle.wrapper.Download.download(Download.java:52)

    at org.gradle.wrapper.Install$1.call(Install.java:62)

    at org.gradle.wrapper.Install$1.call(Install.java:48)

    at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)

    at org.gradle.wrapper.Install.createDist(Install.java:48)

    at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)

    at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)

The command '/bin/sh -c ../gradlew clean bootJar' returned a non-zero code:

At the first galance, I thought it's DNS resolution issue with the Slave container (docker:19-dind) since it is alpine. That's why I debug its /etc/resolv.conf by adding sh "cat /etc/resolv.conf" in the Jenkinsfile.

I got :

nameserver 172.20.0.10
search cicd.svc.cluster.local svc.cluster.local cluster.local ap-southeast-1.compute.internal
options ndots:5

I removed the last line options ndots:5 as per recommendation of many thread on the internet.

But it does not fix the issue. 😔

I thought again and again and I realized that the container responsible for this error is not the Slave (docker:19-dind), instead, it is the intermediate containers that are opened to satisfy docker build.

As consequence, I added RUN cat /etc/resolv.conf as another layer in the Dockerfile (which starts by FROM gradle:5.5-jdk11).

Now, the resolv.conf is different :

Step 15/24 : RUN cat /etc/resolv.conf

 ---> Running in 91377c9dd519

; generated by /usr/sbin/dhclient-script

search ap-southeast-1.compute.internal

options timeout:2 attempts:5

nameserver 10.0.0.2

Removing intermediate container 91377c9dd519

 ---> abf33839df9a

Step 16/24 : RUN      ../gradlew clean bootJar

 ---> Running in f14b6418b3dd

Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip

Exception in thread "main" java.net.UnknownHostException: services.gradle.org

Basically, it is a different nameserver 10.0.0.2 than the nameserver of the slave container 172.20.0.10. There is NO ndots:5 in resolv.conf this intermediate container.

I was confused after all these debugging steps and a lot of attempts.

Architecture

Jenkins Server (Container )
     ||
(spin up slaves)
     ||__ SlaveA (Container, image: docker:19-dind)
             ||
       ( run "docker build" )
             ||
             ||_ intermediate (container, image: gradle:5.5-jdk11 )
1

Found the answer here

Just add --network=host to docker build or docker run.

 docker build --network=host foo/bar:latest .
| improve this answer | |
0

Have you considered not using DinD? I would look to tools like Kaniko, Buildah, Img, Umochi, FTL, or Bazel.

Here's a good example from the Jenkins Kubernetes plugin that builds containers without using Docker or DinD in Kubernetes.

https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/kaniko.groovy

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.