What's the best practice for managing / deploying multi-environments in production app?

Question

I'm spinning up a production site and wanted to get advice on the latest best practices to manage and deploy multiple environments. At a high level, I'd like to have a good balance between keeping prod from being broken vs dev speed.

This is my best high-level guess at what I should strive for:

1. Have devs try to ship small changes + small branches into trunk branches to avoid overhead.

2. Have multiple environments aside from prod, say a "dev" environment.

3. Have the "dev" environment use CI/CD so that whenever branches merge into trunk, the "dev" environment will build and deploy the latest trunk

4. Dogfood / QA "dev" environment frequently, and whenever we feel good about it and want to ship to prod new features that made into "dev", we manually deploy the current trunk to "prod" environment.

Thoughts on this approach? If this seems reasonable, I was wondering about tips on the details:

• To prevent bad trunk merges from breaking prod, does this imply deploying to "prod" environments is entirely manual w/o any CI / CD? Is this a common practice?
• We're using github. Would it make sense for both "prod" and "dev" environments to deploy from the master branch (aka trunk branch), one automatically vs one manually? Or would it make more sense for a separate prod release branches like "prod-release-v1", "prod-release-v2", etc. where we just cut from from the right commit in trunk branch whenever we want to deploy to prod?

Other context:

• We're a lean startup, 10 people atm, and looking to scale a bit more.
• We have prod website up and running and enough customers that it's worth investing in this to avoid too much breakage.
• AWS is most of our stack, but also comfortable with adding any other devops tech that's well recommended in the industry.

Appreciate any help or pointers I can get as a noob devops eng! Thank you!

Answer 1

You've picked the right time to ask these questions (well, I can see you asked a long time ago, but I figured people will find this question who are experiencing the same situation - sorry the answer comes a bit late for you!)

Sorting out your deployment pipeline before you scale up makes it far easier to co-ordinate the required changes. Later on, you'd have to deal with different teams who grown their own deployment pipelines and might not want to change them.

The research

The research is on your side for most of your plans... it has found a strong predictive relationship between the following techniques and improved software delivery performance and stability.

1. Work in small batches
2. Integrate continuously into trunk

You hare already identified that automated checks are vital to this. You need a five-minute build process that runs tests that catch problems early. If you have a longer set of tests that need to run on an environment, you should deploy to your "edge" environment automatically and kick off those long-running tests.

The goal is to ensure no human even looks at a software version unless it passes these checks.

After the automation

If you have manual review of software versions, you may want to allow the people using that environment to pull new versions, so they can ensure an update doesn't occur in the middle of a test session.

This should be a push-button operation that deploys the software to their test environment. Ideally, they should also have a push-button operation to reset their test data to a known state (it sounds like hard work, but the investment really pays off!)

When they are done, one more push-button to say it's good to go.

Production

It is possible to set up a continuous deployment pipeline that runs off the automated checks and deploys to production after the checks complete successfully.

Your organization may have policy constraints that mean a human has to approve the release. This is fine - you'll be in continuous delivery mode. Continuous Delivery really has all the same automation as Continuous Deployment, but it also has additional "manual intervension" steps, such as sign-offs and approvals.

In some rare cases, you may even want to schedule deployments to occur at specific times, so another manual intervention is needed - but the deployment itself should still be automated. You want the automated deployment process to be re-used for all environments as this makes deployments reliable and repeatable.

You can use a deployment automation tool to manage environments, approvals, deployments, and the different varaibles you need to use for dev, test, and production. You should also be able to set up things like self-service deployments and operations runbooks in your deployment automation tool, so you could make it easy to reset test data, clear out caches, or empty log file folders.

Agile, Lean, and DevOps (oh my!)

DevOps as defined in the Accelerate State of DevOps report is the best way humanity has come up with to deliver software (up to now). It's worth keeping tabs on the annual report to see what comes up. You can use the DevOps capability model to get ideas for things that may improve your process (but don't try and adopt them all at once!)

Hope this helps some folks!