0

I've narrowed my problem down to a very specific cause but am not sure how exactly how to fix it.

A Jenkins server set up using Jenkins-configuration-as-code in a docker container runs a pipelineJob script that sets up a listener to a github repo and whenever there's a new commit pulls a jenkinsfile from the repo and uses that to run a job using the declarative pipeline plugin. At the end of the job there's a post-build action using the AWS CodeDeploy plugin to push the built app out to an autoscaling group.

Previously there were two separate jobs for the frontend and backend of the app that deployed each piece separately and everything was fine. The whole system worked end-to-end without any human interaction required.

Now though, after merging the two repos and their declarative pipelines into just one using multiple docker agents, the job still builds and if I check the Jenkins server it says it was successful but if I check an ec2 instance CodeDeploy pushed the update to all I find is the original source files, no built files at all.

After a bunch of digging and head banging I've found that under the hood Jenkins is running git clone and the codedeploy plugin in /var/jenkins_home/workspace/FOLDER but building the backend using a golang docker agent and the frontend using a node docker agent in /var/jenkins_home/workspace/FOLDER@2. That means when CodeDeploy zips it all up it's missing everything that was actually built.

My question is how do I stop/fix that?

Pipeline Jenkinsfile:

pipeline {
    agent any
    
    stages {
        stage('Clone git') {
            steps { git 'https://github.com/REPO' }
        }

        stage('Backend') {
            agent { docker { 
                image 'golang:latest' 
                args '-v "$PWD"/backend:/test -w /test'
            } }

            environment { GOCACHE = "${WORKSPACE}" }

            steps { 
                dir('backend') {
                    sh 'make install' 
                    sh 'make build'
                }
            }
        }

        stage('Frontend') {
            agent { docker { image 'node:latest' } }

            steps { 
                dir('frontend') {
                    sh 'make install'
                    sh 'make build' 
                }
            }
        }
        
    }
    
    post ('Deploy') {
        success {
            step([
                $class: 'AWSCodeDeployPublisher', 
                applicationName: 'server_app', 
                awsAccessKey: '', 
                awsSecretKey: '', 
                deploymentGroupAppspec: false, 
                deploymentGroupName: 'server_group', 
                deploymentMethod: 'deploy', 
                excludes: '', 
                iamRoleArn: '', 
                includes: '**', 
                proxyHost: '', 
                proxyPort: 0,
                region: 'us-west-2', 
                s3bucket: 'server-bucket', 
                s3prefix: '', 
                subdirectory: '', 
                versionFileName: '', 
                waitForCompletion: false
            ])
        }
    }
}

Job console output (note where it switches from FOLDER to `FOLDER@2):

Checking out git https://github.com/REPO into /var/jenkins_home/workspace/FOLDER@script to read pipeline.jenkinsfile
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
 > git rev-list --no-walk 263260386f964b2d7bc8e9921f3b3bb7214ea09a # timeout=10
[Checks API] No suitable checks publisher found.
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Clone git)
[Pipeline] git
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D master # timeout=10
 > git checkout -b master 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Backend)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER@2
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO+refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . golang:latest
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9
$ docker run -t -d -u 1000:1000 -v $PWD/backend:/test -w /test -w /var/jenkins_home/workspace/FOLDER@2 --volumes-from 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** golang:latest cat
$ docker top 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d -eo pid,comm
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
Running in /var/jenkins_home/workspace/FOLDER@2/backend
[Pipeline] {
[Pipeline] sh
+ make install
go get "github.com/julienschmidt/httprouter"
[Pipeline] sh
+ make build
go build -o ../server main.go
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
$ docker stop --time=1 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d
$ docker rm -f 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Frontend)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER@2
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . node:latest
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/FOLDER@2 --volumes-from 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:latest cat
$ docker top ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7 -eo pid,comm
[Pipeline] {
[Pipeline] dir
Running in /var/jenkins_home/workspace/FOLDER@2/frontend
[Pipeline] {
[Pipeline] sh
+ make install
yarn install
yarn install v1.22.5
warning package.json: No license field
warning [email protected]: No license field
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.84s.
[Pipeline] sh
+ make build
make: 'build' is up to date.
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
$ docker stop --time=1 ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7
$ docker rm -f ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
Zipping files into /tmp/#2-8971329807001316273.zip
// deleting some aws account info that displays
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[Checks API] No suitable checks publisher found.
Finished: SUCCESS
Improve this question

1 Answer 1

Reset to default
1

Got it! reuseNode needs to be set true for each docker agent:

       agent { docker { 
            image 'golang:latest' 
            reuseNode true
        } }

Wish that didn't take a couple hours to learn but que sera sera

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.