This question runs two risks:
- of being marked as "primarily opinion-based", and
- its answers might age poorly.
However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.
I would split the problem up into four sections:
- Application Building, testing and integration
- Monitoring, logging and analytics
Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure etc, as well as on-prem OpenStack, bare metal cloud (e.g. Packet, etc).
Infrastructure components consist of
Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.
- Each module in its own repo
- Consider carefully separation of state, so that you can easily create and destroy components and keep costs down
"DevOps" infrastructure. This includes:
- code repos
- CI/CD instances (controller, agent pools, etc) - e.g. Jenkins master with cloud agent pools.
- Deployment runners (see later).
- Monitoring, logging and analytics.
Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.
Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.
- secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.
Building, testing, integration.
These happen in Jenkins. Each application should contain a subdirectory
.pipeline with the parts necessary to build it. The part which concerns us here is everything before prod.
I would suggest a pipeline as code approach with a
Jenkinsfile in the repo declaring the steps for
- building the application:
- Testing the application
- Unit tests
- Functional tests
- Integration tests
- (Regression tests)
- Building artifacts
- building and signing artifacts
- Shipping them to the artifactory, with new version info and other metadata
- Building instance images (if your application is "installed" in an environment, and not serverless).
- Enforcing changes to the Infrastructure (terraform plan/apply)
- Enforcing changes to the deployment configuration (capacity, etc)
At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.
I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.
Monitoring, logging and analytics
As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.
The SRE book gives good advice for instrumenting and monitoring the applications you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.
So - the toolkit looks like:
- Vault for secrets
- GitHub for code, projects, tasks and issues
- Slack for communication
- Terraform for infrastructure
- Ansible for configuration management
- Packer for image baking
- Quay for (docker) image registry
- Inspec for compliance and correctness of state
- Cucumber for Acceptance Testing
- Jenkins for pipeline execution
- Spinnaker for Deployment
- Prometheus for metrics Grafana for visualisation and dashboards
- Taurus (Blazetool) for performance testing
Is this a lot of tools? Yes.
Will you be able to deploy anywhere? Also yes.
Will you need all of them? probably not. Also, I've chosen tools which are mostly free to use and open source.
Also note: in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, etc.