DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.
Sign up to join this community
I have Jenkins 2.89 installed with many outdated plugins as well. What is the best path for upgrading both the plugins and Jenkins itself? Should one proceed the other?
I am leaning toward upgrading the plugins first and in batches to limit the risk. The system is being backed up in its entirety in case things go sideways. I would like to avoid that if at all possible though.
Update: I was able to upgrade to 2.138 without it breaking anything.
I can only speak from personal experience rather than best practices, but in our company we'll make a duplicate jenkins instance, then upgrade that. We do main upgrade first, then plugins second, then run testing to make sure our core jobs still work. If all is good, we'll then upgrade the real jenkins during off-hours.
We have 90+ jobs so testing them all is unrealistic - we made an effort to identify 8-10 jobs that span 90% of our use cases and run those on the standup jenkins to affirm everything is healthy first.
I'd highly advise doing something similar if you're using your jenkins in production scenarios. You never know what might break - we had one upgrade where a bug spun up 150+ aws ec2 instances in ~10mins instead of the 3 jenkins actually needed... another time all our unit and e2e testing of our code base broke because it turned out one of the output parser plugins had upgraded to become incompatible with our output format. We've even had our Bitbucket integration go sour and stop kicking off jobs on every push.
Sometimes upgrading all plugins at once can cause confusion - we had one scenario where something broke and finding the cause was difficult because it could have come from one of two different plugins. We ended up spinning up two additional jenkins and upgrading either plugin to identify which was the cause. In general though, upgrading everything at once will pretty quickly show its flaws with a little testing.
No matter what you decide, take a backup first!
Not to take away from @vaporwave_sailor or @Alex, answers, but to add to..
The basic answer is you MUST upgrade all the compatible plugins you can, then the war, then the newer plugins. All plugins depend on a minimum Jenkins version and the Update Centre will give you different information back.
A further clarification, especially if migrating from < 2.263.4 to > 2.277.1 and many breaking UI changes and tables-divs-regaression (Dashboard): You should ensure your plugins are at the latest version, compatible with the existing Jenkins core version before starting. So, it's plugins, war, plugins.
If you don't upgrade often, use the LTS version. READ the Upgrade Guide. READ the Changelog. Re-READ the Upgrade Guide. Do the same for all your core plugins. There are significant changes, especially around security which may require modifications on your part. Many previously bundled are no longer bundled. Long unsupported plugins may have better alternatives or contain security issues you should be aware of. Behaviors and settings may change.
The following are NOT exact steps, rather things to do in the process. Descriptive, not prescriptive.
Consider installing the Jenkins Configuration as Code (JCasC) plugin to your existing installation first. That will generally let you export almost your entire configuration and simply reload into a new instance. Much easier, but does not handle plugins. For now rip a solution from docker/install_plugins.sh for local use; see Preinstalling Plugins.
Take a backup of your existing Jenkins and install install a "Dev" location. Validate your upgrade there before applying to Prod. ALWAYS. (Also, regularly backup your Prod anyway).
Split the backup in two parts, the system configuration and the jobs. Jobs are in ${JENKINS_HOME}/jobs. The system configuration includes (in ${JENKINS_HOME}), ./*.xml ./secret.key* ./secrets ./security-tokens ./nodes , possibly others.
Back to the plugins for a minute. I agree w/adding them in segments:
All the above are "Global plugins, with Configurations"
If using the install_plugins.sh approach, you only need worry about the highest level plugin as it will pull the dependencies. If you install blueocean, it installs some 25 plugins. Just specify the first and you get the rest. Bear in mind, install_plugins.sh is dynamic, in that it gets the available dependencies at the time. So, we decide on the top list, manage that, run once to get the complete list, then read the validated explicit list into the Prod instance (manage the short one, use the complete one, save both). We manage 55 top-level plugins, but there are 160 installed including all dependencies in the full list.
This groovy script will get you your existing plugins:
Jenkins.instance.pluginManager.plugins.sort(false) { a, b -> a.getShortName().toLowerCase() <=> b.getShortName().toLowerCase()}.each { plugin ->
println "${plugin.getShortName()}:${plugin.getVersion()} | ${plugin.getDisplayName()} "
}
// the following may also be useful
// println "+++ ${plugin.getDependants()}"
// println "... ${plugin.getDependencies()"
In the Dev instance, install the war, install the plugins, create a file ${JENKINS_HOME}/init.groovy, add the line:
Hudson.instance.doQuietDown();. That lets your system start without letting any jobs start. That way you can examine the system logs w/o panicking or getting confused. Look at everything before INFO: Jenkins is fully up and running.
Now, you can install the system configurations, but not the ./jobs and not the ./nodes. Start it up and check the logs for information, warning alerts and errors.
You might even want to start with your existing war version and existing plugins and basic config, fire that up and save the logs as references. Then, using the UI, upgrade the war, then restart, then the plugins using the UI, then restart, then check and save the logs as references.
If using JCasC, you may go w/a slim install of the configurations,then import your JCasC configurations. JCasC usage here.
*NB: You will notice one major change is more plugins are moving to their own xml configuration instead of adding to the master config.xml. There are many plugins especially that say this configuration is NOT backward compatible. You would have to restore the old config when "downgrading" (correct language - start again from step 1)
Only once your core system starts fine, then I would add the jobs (with QuietDown() in place ), no nodes and in QuiteDown. Start up again, examine logs for errors, upgrade warnings, etc. Deal w/those. Many plugins will change their data format, hence the many warnings you see, and the "Manage Old Data". IF you're truly paranoid, you may even write a little groovy script that finds all your jobs and re-saves them. While Jenkins knows of the configuration changes, it does not actually change and update the plugin references in the jobs until the job is run or saved (if you need a script, ask a separate question).
Shutdown, delete the jobs directory (this is your DEV instance, so you are OK).
Startup up again and create a small set of validation jobs. These validate all the steps of your real jobs but without impacting them. They are the "test cases" @Alex refers to. We use "sampleApps" of different flavors we have and jobs that cover all the steps. We have test jobs that just get on a node, dump the environment, jobs that clone a repo and build (in the flavors), one that does the whole cycle: clone, build analyze, test, deploy, as well as pipeline jobs (phased). You can now add the nodes configuration and startup and execute those. They are your validation. If they work, then you are as prepared as you can be. You can now export the JCasC configuration as well.
Now, delete the entire Dev instance and REDO using only your finalized configuration and steps as you documented. Did everything work as expected? Run JCasC again, did they match?
Now you can apply the same finalized configuration to your Prod. Shutdown Production and TAKE ANOTHER BACKUP. I'd add the QuietDown() first to Prod.
Finally, consider taking advantage of some new "System Properties". You can now set locations outside of ${JENKINS_HOME} for jenkins.model.Jenkins.buildsDir (ps: sym-links in builds are now optional) and jenkins.model.Jenkins.workspacesDir, removing useless data from the ${JENKINS_HOME}. Your backups will be smaller and simpler and your system performacne may improve! You'll also want to set jenkins.install.runSetupWizard to false.
Remember, this is descriptive, not preservative. Tailor to your own needs and risk tolerance.
I've gained some experience from f-n up Jenkins a lot of times, here is order I kept sticking to and it saved me a lot of time:
I had 'after-upgrade' pipeline that would start build of few most important projects, it was running after each step.
What helped me a lot as well is taking snapshots in between of upgrades so I could do rollback easily and do not start from the beginning which could be time consuming. Obviously I do not recommend doing in-place upgrades, I always go for 1:1 copy and working on copy and then I just change DNS record.
