Jenkins pushing to branch overwrites commits made in interim

Question

A while ago I had written about how commits were being merged (successfully) into our develop branch in Git, but would later go missing, with no evidence of the missing code other than all the Bitbucket logs proving the merge happened.

We just figured out yesterday that what's happening is Jenkins clones our repo, then holds it for up to 40+ minutes while jobs run, then pushes back with git push origin +develop. This is essentially overwriting any commits we've made in that 40+ minutes as Jenkins pushes back the branch exactly as it saw it on the clone, and minus our additions.

We thought the solution would be to change to

git pull --rebase origin
git push origin develop

but that's returning

+ git pull --rebase origin
error: Cannot pull with rebase: You have unstaged changes.

To clarify, Jenkins' "branch" has files on it that we do need merged into the repo. The ideal situation is to find the right way to do a fresh pull of develop into Jenkins' copy before pushing back.

What's the correct git procedure to avoid this issue?

Note: My original post was deleted for being too vague, and rightfully so since at the time I had zero idea why things weren't working. In this case what happens is Jenkins checks out the repo, runs a script that makes parameter files needed for a second job, then checks it back in so the files (and associated logs) exist in the repo for use/auditing. Not having Jenkins do the process isn't an option.

Answer 1

It really depends on your situation, but I'll go with the generic/naive approach and say do exactly what your developers do in the same situation: I pulled an hour ago, I made some changes, now I want to push my changes, but other changes have been pushed in the meantime. I commit, pull/rebase, then push. Assuming that whatever Jenkins is committing isn't tied to the commit it built from, this is the same process it should follow to get the same result.

Answer 2

From a high level, why do you have both jenkins and developers pushing to the branch? If you have enough CI testing for jenkins to force-push to the branch, then you have enough CI testing to revoke push access from the developers. At first it sounds scary to only have robots pushing to branches (and you should also have a small core team with push access which is only used in the direst of situations), but once you take the plunge into branches gated by CI -- meaning CI must pass before merges are allowed -- operational confidence in the code will quickly increase.