We've got a ton of jobs in our Jenkins that all do git pushes, sometimes simultaneously. Unsurprisingly, if two jobs try to push at the same exact time, one will succeed while the other fails. My knee-jerk reaction is to build some kind of exponential backoff and retry for these git pushes, but I'm pretty unfamiliar with git. Does something like that exist already?
Our commit/push code generally looks like this:
git config --global user.name "username"
git config --global user.email email@example.com
git add --ignore-errors -- dir/
git commit -m "some message"
git pull --rebase origin
git push origin master
As you can see, the bandaid originally applied to the problem was to put a pull before the push to fix other jobs that pushed while the current one was running. That works for ~80 of cases, but if the two pushes fall within the same <1sec window, there are still failures.
Here's an example of a failure for the last two lines above: (Anything in <> is edited for privacy)
+ git pull --rebase origin
Warning: Permanently added '<bitbucket.org ip>' (RSA) to the list of known hosts.
From <bitbucket.org:dir>
<commit#>..<commit#2> master -> origin/master
First, rewinding head to replay your work on top of it...
Applying: <commit name>
+ git push origin master
Warning: Permanently added '<bitbucket.org, ip>' (RSA) to the list of known hosts.
remote: error: cannot lock ref 'refs/heads/master': is at <commit sha> but expected <different commit sha>
To <bitbucket.org:dir>
! [remote rejected] master -> master (failed to update ref)
error: failed to push some refs to 'git@bitbucket.org:<team/dir>'
So my question boils down to:
- Is there some form of backoff-and-retry built into git?
- If not, how can I build that (assume a shell block in a jenkins freestyle job)?