Initially asked here: https://stackoverflow.com/questions/60674502/how-to-keep-many-servers-5000-up-to-date-with-git-and-its-rate-limits
The initial post:
We're making a php service that will run on many servers, think 5000+. We host our code on git (bitbucket). We wondered what the best way to keep the servers up-to-date would be.
We figured either post-commit hooks (but what would happen then if a few servers didn't receive the update notification?) or git fetching every minute with cron. We want to go with the cron way of doing things, since it isn't possible it would fail, even if a server would be offline (either turned off or disconnected from the network), it would still resolve itself eventually.
We're doing a fetch every minute, and then compare to see if it needs to pull, if so it pulls and runs the migration code.
We would like to run this every minute so that the servers will be synchronized with each other as soon as possible.
Now we wonder, what about rate limits? We're using bitbucket, and the rate limits are 60 000 requests per hour (so a 1000 per minute), which would limit us to 1000 servers max before we will get problems then?
But it also says, if we make a public repo, we can make unauthenticated calls, the limits of which go by IP rather than per user, so we won't run into any limits then no matter how many servers we will have. Downside is, we will have to encrypt the repo then, then on pull, decrypt it and copy over the decrypted files.
Is this the best way of handling this? It seems very unconventional. What is the standard or recommended way of handling this (if there is any)?
After having read the answers, we now think that having a group of servers that pull in any changes (from git). For readability, we're calling these the "gitpull-servers", and the 5000+ servers the "webservers".
The plan is to make the gitpull-servers fetch (and possibly pull) from git.
The webservers periodically query the gitpull-servers to see if there is an update, and if there is, the webservers will pull the update in either by downloading a tar/zip (requires more network) or by using rsync (requires more iops on the gitpull-servers).
We think this is a good solution, but none of us have any experience with deployments at this scale, so we aren't sure if we've overseen anything or if this could be done in any other (potentially better) way.
It would be really helpful to get feedback on this, as well as to know about the conventional or the recommended way of handling a problem like this at this scale.
