What challenges did you face once you gave devs access to Jenkins jobs?

Question

I am a DevOps engineer at a medium-sized company. Our team is super busy and we have a lot on our plate. Recently we decided to see where we can improve our productivity. One of our main responsibilities is to provide dev teams with different services they require (usually they open a ticket, sending a slack msg, etc…). As part of our efforts to improve productivity, we recently decided to assemble the most common popular “services” provided to devs and automate them using Jenkins. In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs.

On one hand, it does allow devs to act on their own, on the other hand, I get a lot of backfires from different personas...

Devs

• Why did this job fail?

• Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created"

• Devs make mistakes and deploy to the wrong environment.

• Why did my dev environment get terminated?

• "Update me if you modify parameters"

• etc...

Team lead

• Let’s add a manual approval for these jobs

• Make sure we have an audit log for every dev operation

• Make sure we devs are not spinning to many machines

• The developer team leader asked BI about these Jenkins Jobs because it became critical for developers' day-to-day.

• etc…

Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins :)

Do you have any suggestions? How do you manage it within your company?

Answer 1

In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs

This would be the first thing that should be changed. The pipelines should only be started automatically instead of manual.

DevOps is a culture and implies that all repetitive work should be automated.

Why did this job fail?

You should ask them whether they read the log. If the issue in the pipeline is NOT related to their code then the creator of the pipeline should investigate it and make a solid CI.

Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created"

This indicates that the devs see unrelated pipelines. If they create microservices then there would be a dedicated pipeline. If the pipeline is broken then one knows that the issue is related to their code. I had a lot of discussions with devs back in the day. Most of the time they said that it felt superfluous to create another repository for their code, but when I referred to the 12factor app and demonstrated the effect of running a pipeline for every app, they were convinced.

Devs make mistakes and deploy to the wrong environment.

The one that created the pipelines decides what permissions the user will get. A best practice is to apply the least privilege. In terms of CI this means that the developer will only be capable of reading the log. Once the log is read one should be capable of solving the issue. Sometimes, devs do not run their code locally on their laptop first.

Update me if you modify parameters

Once the pipeline have been implemented, you should be aware that you should NOT introduce breaking changes. If this is inevitable then you should communicate it well, but you should prevent it at all costs to prevent that the devs will decide not to use the pipelines anymore. It is better to spend some more time to create a solid pipeline, before let the devs use it.

Make sure we devs are not spinning to many machines

This should be automated as well. You could decided to stop the instances automatically if they are running for more than half an hour. You could introduce this and inform the devs and teamlead about it.

Make sure we have an audit log for every dev operation

Prevent that the devs login to Jenkins. The pipeline should run automatically and the devs should only be capable of reading the log, see also Devs make mistakes and deploy to the wrong environment

Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins :) Do you have any suggestions? How do you manage it within your company?

Gitlab with a gitlab-ci.yml for each microservice, Gitlab templates, helm and runners on k8s.