How to create staging servers for multiple git branches?

Question

I have to create a new staging process for our development and testing.

At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may be database evolution scripts (straight SQL) that must be run, as well as evolution scripts from the backend for heavier processing (essentially these are HTTP routes that must be called within the app using admin credentials that perform database migrations and other changes that would be too difficult/not possible to script in the aforementioned plain SQL evolution scripts).

Our live DB is moderately sized ~ 4.2 GB. We have a brand new Dell PowerEdge server that is ready to be set up and is at my disposable.

I would love advice on the following questions, as well as to know how more experienced DevOps would approach this:

1. How can I run several different branches on the staging server? These branches pop up and disappear frequently as they pass QA and are merged into master and are released.

2. How would I set up the DB evolution system to make sure it always has appropriate DB for each branch? Each branch may make modifications to the DB in different ways that are not necessarily compatible with each other until they're merged.

3. How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

Would love any more input as I'm a bit lost on how to set this all up. Current workflow is difficult for everyone involved: Devs have a completely isolated local copy of the app running locally and QA has 3-4 laptops in rotation that act as their staging "servers"

Answer 1

1) How can I run several different branches on the staging server?

Docker

2) How would I set up the DB evolution system to make sure it always has appropriate DB for each branch?

This depends on how much you expect your DB to scale. You can get pretty crazy with methods to clone database data but typically you will want a master copy that you do not alter until your code is released into production - and keep good backups when you do. While you infrastructure can be immutable and disposable, your data is not. You can only emulate disposability with you data. 4.2 GB really isn't that much to copy. You could make a script to spawn a new database instance for each build to operate on, then discard it once testing is complete.

3) How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

You could consider using something like git hooks to trigger a build, force a code checkout or begin spawning a container and cloning your database. You could make an API call to a build automation system like Jenkins and/or use it to kick off a configuration management system like Puppet, Chef, Salt Stack, Ansible or something else. This would be more like an auto-pull than an auto push.

From the wording of your question, it's clear you are thinking of a mutable infrastructure, but consider using immutable testing deployments instead.

Answer 2

I'd argue that this isn't actually about staging servers. A staging server closely mimics the production environment, and is where a release goes immediately prior to going to production. A feature branch that hasn't been merged into master is not going to be released directly to production, so it should not go on a staging server.

If we reframe the question as being about shared development servers, then you'll find more resources as you search. As you've noticed, sharing development resources like this causes a number of problems, so it may perhaps be better to focus on solving the underlying issue instead: why is it hard for everyone involved in this process to run a dev server on their local machine?

---

Now, sometimes the situation is that you do indeed need to have these shared servers. If you've gone through and determined that you really do, then there are a number of techniques you can take to simplify the process.

How can I run several different branches on the staging server? These branches pop up and disappear frequently as they pass QA and are merged into master and are released.

One of the easiest ways is to use part of the url to switch which code branch gets executed (this change is part of why this is not an appropriate staging environment). I've done this in the past with a wildcard DNS (so any foo.our-dev-domain.com would redirect to the same server) and routing code that loaded up /our/release/directory/foo in the include path.

You can also have a command (using something like Ansible) that adds the necessary configuration for a branch as necessary. This would probably be part of the deploy process, which I'll talk about in a minute.

How would I set up the DB evolution system to make sure it always has appropriate DB for each branch? Each branch may make modifications to the DB in different ways that are not necessarily compatible with each other until they're merged.

The easiest thing to do is just live with it, and encourage people to have short-lived branches.

Another approach is to use dynamic routing code to switch to separate copies of the database (in RDBMS terms, a "database") based on the name of the branch.

How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

You can either push or pull. For instance:

• pull - set up a cron job that does a git pull (or probably, a git fetch and a git reset --hard) every few minutes
• push - set up a webhook that triggers upon push, and a little web service on the machine that listens to that and updates the repos as necessary
• push - have developers explicitly run a deploy command to update their branches on the server

Talk to your users first and gather their requirements to see what they want (they may not like having it update automatically while they're in the middle of testing something), and that will inform your decision.

---

But again:

Devs have a completely isolated local copy of the app running locally and QA has 3-4 laptops in rotation that act as their staging "servers"

The problem seems to be that QA rotates shared laptops instead of just checking out a particular branch the same way devs do.

Answer 3

That is what gitlab does best, consider to move to gitlab, setup a kube cluster and auto deploy. This will deploy each branch to an unique URL for testing.