5

Initially asked here: https://stackoverflow.com/questions/60674502/how-to-keep-many-servers-5000-up-to-date-with-git-and-its-rate-limits

The initial post:

We're making a php service that will run on many servers, think 5000+. We host our code on git (bitbucket). We wondered what the best way to keep the servers up-to-date would be.

We figured either post-commit hooks (but what would happen then if a few servers didn't receive the update notification?) or git fetching every minute with cron. We want to go with the cron way of doing things, since it isn't possible it would fail, even if a server would be offline (either turned off or disconnected from the network), it would still resolve itself eventually.

We're doing a fetch every minute, and then compare to see if it needs to pull, if so it pulls and runs the migration code.

We would like to run this every minute so that the servers will be synchronized with each other as soon as possible.

Now we wonder, what about rate limits? We're using bitbucket, and the rate limits are 60 000 requests per hour (so a 1000 per minute), which would limit us to 1000 servers max before we will get problems then?

But it also says, if we make a public repo, we can make unauthenticated calls, the limits of which go by IP rather than per user, so we won't run into any limits then no matter how many servers we will have. Downside is, we will have to encrypt the repo then, then on pull, decrypt it and copy over the decrypted files.

Is this the best way of handling this? It seems very unconventional. What is the standard or recommended way of handling this (if there is any)?

After having read the answers, we now think that having a group of servers that pull in any changes (from git). For readability, we're calling these the "gitpull-servers", and the 5000+ servers the "webservers".

The plan is to make the gitpull-servers fetch (and possibly pull) from git.

The webservers periodically query the gitpull-servers to see if there is an update, and if there is, the webservers will pull the update in either by downloading a tar/zip (requires more network) or by using rsync (requires more iops on the gitpull-servers).

We think this is a good solution, but none of us have any experience with deployments at this scale, so we aren't sure if we've overseen anything or if this could be done in any other (potentially better) way.

It would be really helpful to get feedback on this, as well as to know about the conventional or the recommended way of handling a problem like this at this scale.

Improve this question
1
  • git pull from 5000 places every minute, oh my you must hate your sundays. – Jiri Klouda Mar 15 '20 at 19:30
4

I don't think the proposed plan here makes much sense. Tools like Ansible are free and designed to do this with good, central management and logging.

  • Put code in BitBucket.
  • Make a Jenkins server (open source / free / low on resources / well documented and easy to use).
  • Make BitBucket call Jenkins with a hook on change.
    • Or, if you feel like it, Jenkins can poll bit-bucket, but this is less efficient.
  • Have Jenkins call ansible (also open source / free / low on resources / well documented and easy to use) in its pipeline.
  • Ansible can target all 5000 servers with a dynamic inventory based on querying a database/file/API/whatever you have.
  • This way, bit-bucket has almost no load, ansible does everything from one location, runs are recorded in Jenkins with info on each host.

You won't have any API / rate limits here and you'll easily be able to target more hosts, with no effort on your end, with the dynamic inventory.

This is industry standard tooling. You can also use Ansible to keep the servers up to date patch wise and all kinds of other stuff while you're at it, so it's a re-usable solution.

Improve this answer
4
  • 1
    Thanks for the description, it will be useful to us as we now know what we should be looking into. – patrick_vane Mar 14 '20 at 5:51
  • While this is a great ideal. It is not going to work at the scale of 5000 due to infrastructure issues. Our of 5000 servers at any point you will have some being replaced for being obsolete, 3% has some kind of HW issue with either memory, disk, controller or networking. There is a lot that will go randomly wrong. At this scale it is about error correction and inventory management where you can bring servers gradually to current state as they are rejoining system. That ansible code sounds like pice of cake, but it might give you a lot of challenge. – Jiri Klouda Mar 15 '20 at 19:28
  • @Jiri Klouda - Mostly agree - You can add another answer if you have something better for on premise (I’m guessing). With a cloud host and AKS/GKE/EKS, ECS, or something it would be much easier but the original question is a far cry from those levels of tech. So, I tried to make something that sounded achievable. I’m sure there are better large scale systems management solutions though for data centers that I don’t know about. – John Humphreys - w00te Mar 15 '20 at 19:54
  • Yeah, I like the answer, it is just not going all the way to 5000. It might if Amazon handles the infrastructure, but you still rotate certificates, they reboot instances, etc. Based on the original question there is a db migration. I think there needs to be some component to bring a server up to date when it becomes available or joins the cluster. – Jiri Klouda Mar 15 '20 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.