How do I replicate a Jenkins setup via automation?

Question

I have a Jenkins setup running in production. I want to automate the Jenkins setup (installation) along with all the jobs that are set up in Jenkins.

One crude way I can think of is to copy the whole jobs directory to the new Jenkins setup.

How do I deal with this problem?

Answer 1

Jenkins itself can be configured in many ways. The same applies for Jenkins jobs.

Probably the most easy way is to configure Jenkins manually and replicate its setup by backing up and restoring. Unfortunately, this means dealing with copying XML files.

But fortunately there are other ways.

init.groovy.d directory

When Jenkins starts, it checks init.groovy.d directory and executes Groovy scripts there. To enforce locale I'm using a snippet.

// Copied from /var/lib/jenkins/init.groovy.d/set-locale.groovy
def pluginWrapper = jenkins.model.Jenkins.instance.getPluginManager().getPlugin('locale')
def plugin = pluginWrapper.getPlugin()

plugin.setSystemLocale('en_US')

Another example of setting up Jenkins by Groovy scripts is located on https://gist.github.com/hayderimran7/50cb1244cc1e856873a4 - I'm using that snippet to setup local Jenkins users on test machines.

Generating jobs

Jenkins Job builder

One approach of generating jobs is Jenkins Job Builder, which creates XML definitions jobs from YAML configurations. I don't like this approach personally as I consider it to be a bit hackish - looks like "generation of XML files without knowing what it means". But some big players are using this approach - JJB originates from Openstack.

Job DSL plugin

Another approach is Job DSL plugin (plugin page). There is special job type called seed job which creates other jobs from Groovy DSL. The seed job can pull the others job description in from Git, so you can easily version them, reuse them, etc.

I'm using personally this approach, as it was available when I started with Jenkins. I've even found some Jenkins tool to create DSL sources from existing jobs, but I'm not able to find it again.

To provide a seed job, you can still use the init.groovy.d approach.

Pipelines

The currently recommended approach is probably related to Jenkins Pipelines. You can store your building pipelines in the source VCS repository next to your code.

I haven't explored this very much, but I'm planning to do.

Answer 2

The simple way to deal with this is as you have described. However, some drawbacks of it are:

• manual Jenkins setup.
• No guarantee that your new Jenkins instance is one-on-one identical to the previous in terms of version, plugins, etc.

Here's one way I have achieved this in the past. This creates disposable Jenkins, that can come up with the same plugins and tools every time.

Note: my setup assumes

• Jenkins job history is backed up. Artifacts are not stored within Jenkins.
• Your Jenkins infrastructure is on the cloud, e.g. AWS
• Ansible used for automated provisioning of infrastructure
• Jenkins 2.x in use with full use of Pipeline-as-Code
• A defined set of Jenkins pipelines.

Machine/Instance Creation

• Ansible provisioner creates AWS Linux instance
• Adds it to Ansible dynamic inventory

Jenkins Install

• Ansible installs Jenkins on the instance
• Along with any ancillary tools, e.g. git, maven etc. required for Jenkins jobs to execute.
• Start Jenkins service, wait for it to start.

Jenkins Configuration

Ansible tasks for:

• Wait for Jenkins CLI to be available.
• Download Jenkins CLI JAR file
• Execute Jenkins CLI to create users, install plugins, restart Jenkins etc and create jobs.
• Restore job history from backup.

Caveat

• Works best for greenfield and simpler Jenkins setups. Is harder (not impossible) to achieve with brownfield and more complex projects.
• A large amount of training in Infra-as-Code mindset is required with this approach, such as for example "Don't make plugin installations manually".