Given a blank slate, tool recommendations? [closed]

Question

so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.

As a very high level overview

So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.

The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.

The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment.

Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.

Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.

Answer 1

This question runs two risks:

1. of being marked as "primarily opinion-based", and
2. its answers might age poorly.

However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.

I would split the problem up into four sections:

1. Infrastructure
2. Application Building, testing and integration
3. Deployment
4. Monitoring, logging and analytics

Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure etc, as well as on-prem OpenStack, bare metal cloud (e.g. Packet, etc).

Infrastructure

Infrastructure components consist of

1. Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.

   1. Each module in its own repo
   2. Consider carefully separation of state, so that you can easily create and destroy components and keep costs down

2. "DevOps" infrastructure. This includes:

   1. code repos
   2. CI/CD instances (controller, agent pools, etc) - e.g. Jenkins master with cloud agent pools.
   3. Deployment runners (see later).
   4. Monitoring, logging and analytics. Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.

3. Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.

4. secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.

Building, testing, integration.

These happen in Jenkins. Each application should contain a subdirectory .pipeline with the parts necessary to build it. The part which concerns us here is everything before prod.

I would suggest a pipeline as code approach with a Jenkinsfile in the repo declaring the steps for

1. building the application:
2. Testing the application
   1. Unit tests
   2. Functional tests
   3. Integration tests
   4. (Regression tests)
3. Building artifacts
   1. building and signing artifacts
   2. Shipping them to the artifactory, with new version info and other metadata
4. Building instance images (if your application is "installed" in an environment, and not serverless).
5. Enforcing changes to the Infrastructure (terraform plan/apply)
6. Enforcing changes to the deployment configuration (capacity, etc)

At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.

Deployment

I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.

Monitoring, logging and analytics

As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.

The SRE book gives good advice for instrumenting and monitoring the applications you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.

---

So - the toolkit looks like:

1. Vault for secrets
2. GitHub for code, projects, tasks and issues
3. Slack for communication
4. Terraform for infrastructure
5. Ansible for configuration management
6. Packer for image baking
7. Quay for (docker) image registry
8. Inspec for compliance and correctness of state
9. Cucumber for Acceptance Testing
10. Jenkins for pipeline execution
11. Spinnaker for Deployment
12. Prometheus for metrics Grafana for visualisation and dashboards
13. Taurus (Blazetool) for performance testing

Is this a lot of tools? Yes.

Will you be able to deploy anywhere? Also yes.

Will you need all of them? probably not. Also, I've chosen tools which are mostly free to use and open source.

Also note: in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, etc.

Answer 2

I am a big fan of saltstack, mainly because it provides a comprehensive solution regarding the whole server life cycle, unlike ansible which need another tool (such as terraform) to provision vms.

salt + salt-cloud is one very good suite from my opinion.

I do not see any differences regarding the amount of work required by either tool (chef, puppet, ansible, salt) to get a working IAC system in place, this is, of course, according to your language preference. If you're in python then go for salt or ansible, otherwise go check puppet or chef. Do not forget to plug your iac code base to a VCS.