exponential backoff for git pushes?

Question

We've got a ton of jobs in our Jenkins that all do git pushes, sometimes simultaneously. Unsurprisingly, if two jobs try to push at the same exact time, one will succeed while the other fails. My knee-jerk reaction is to build some kind of exponential backoff and retry for these git pushes, but I'm pretty unfamiliar with git. Does something like that exist already?

Our commit/push code generally looks like this:

git config --global user.name "username"
git config --global user.email email@example.com
git add --ignore-errors -- dir/
git commit -m "some message"
git pull --rebase origin
git push origin master

As you can see, the bandaid originally applied to the problem was to put a pull before the push to fix other jobs that pushed while the current one was running. That works for ~80 of cases, but if the two pushes fall within the same <1sec window, there are still failures.

Here's an example of a failure for the last two lines above: (Anything in <> is edited for privacy)

+ git pull --rebase origin
Warning: Permanently added '<bitbucket.org ip>' (RSA) to the list of known hosts.
From <bitbucket.org:dir>
   <commit#>..<commit#2>  master     -> origin/master
First, rewinding head to replay your work on top of it...
Applying: <commit name>
+ git push origin master
Warning: Permanently added '<bitbucket.org, ip>' (RSA) to the list of known hosts.
remote: error: cannot lock ref 'refs/heads/master': is at <commit sha> but expected <different commit sha>        
To <bitbucket.org:dir>
 ! [remote rejected] master -> master (failed to update ref)
error: failed to push some refs to 'git@bitbucket.org:<team/dir>'

So my question boils down to:

1. Is there some form of backoff-and-retry built into git?
2. If not, how can I build that (assume a shell block in a jenkins freestyle job)?

Answer 1

Instead of trying to find a solution in git, I would recommending using the Lockable Resources Plugin, which is an official plugin developed by the Jenkins developers. This plugin implements locks in Jenkins, which are basically the same idea as a semaphore or mutex. Using this plugin, each of your jobs that needs to push to the same branch of the same repo will request a lock when performing the push. If the lock is available, the job will acquire the lock, push right away, and then release the lock when it is finished pushing. If the lock is not available, the job will wait until the lock is available (i.e., all other jobs are done pushing) before grabbing it, pushing, and releasing the lock.