1

so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.

As a very high level overview

So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.

The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.

The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment.

Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.

Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.

  • 1
    Could you make this question more generic? Please process the feedback as provided in one of the answers by BruceBecker. Once done, please leave a comment so this question can be evaluated again. – 030 Nov 7 '19 at 8:38
5

This question runs two risks:

  1. of being marked as "primarily opinion-based", and
  2. its answers might age poorly.

However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.

I would split the problem up into four sections:

  1. Infrastructure
  2. Application Building, testing and integration
  3. Deployment
  4. Monitoring, logging and analytics

Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure etc, as well as on-prem OpenStack, bare metal cloud (e.g. Packet, etc).

Infrastructure

Infrastructure components consist of

  1. Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.

    1. Each module in its own repo
    2. Consider carefully separation of state, so that you can easily create and destroy components and keep costs down
  2. "DevOps" infrastructure. This includes:

    1. code repos
    2. CI/CD instances (controller, agent pools, etc) - e.g. Jenkins master with cloud agent pools.
    3. Deployment runners (see later).
    4. Monitoring, logging and analytics. Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.
  3. Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.

  4. secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.

Building, testing, integration.

These happen in Jenkins. Each application should contain a subdirectory .pipeline with the parts necessary to build it. The part which concerns us here is everything before prod.

I would suggest a pipeline as code approach with a Jenkinsfile in the repo declaring the steps for

  1. building the application:
  2. Testing the application
    1. Unit tests
    2. Functional tests
    3. Integration tests
    4. (Regression tests)
  3. Building artifacts
    1. building and signing artifacts
    2. Shipping them to the artifactory, with new version info and other metadata
  4. Building instance images (if your application is "installed" in an environment, and not serverless).
  5. Enforcing changes to the Infrastructure (terraform plan/apply)
  6. Enforcing changes to the deployment configuration (capacity, etc)

At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.

Deployment

I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.

Monitoring, logging and analytics

As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.

The SRE book gives good advice for instrumenting and monitoring the applications you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.


So - the toolkit looks like:

  1. Vault for secrets
  2. GitHub for code, projects, tasks and issues
  3. Slack for communication
  4. Terraform for infrastructure
  5. Ansible for configuration management
  6. Packer for image baking
  7. Quay for (docker) image registry
  8. Inspec for compliance and correctness of state
  9. Cucumber for Acceptance Testing
  10. Jenkins for pipeline execution
  11. Spinnaker for Deployment
  12. Prometheus for metrics Grafana for visualisation and dashboards
  13. Taurus (Blazetool) for performance testing

Is this a lot of tools? Yes.

Will you be able to deploy anywhere? Also yes.

Will you need all of them? probably not. Also, I've chosen tools which are mostly free to use and open source.

Also note: in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, etc.

| improve this answer | |
  • 1
    Thank you for the highly detailed answer. I think the packet tool sounds quite promising. The main brain picker really is how development is going to stay consistant as possible between projects and in line with UAT. The plan as mentioned is to keep UAT and PROD the same and only change with deployments to ensure no issues can come about. I can see myself using packer to deploy set images based on requirements so if issues were to come about that could help replicate a bug along side the dev instances if they are developing on the same packet image they are pulling from if that makes sense? – Nabil Aziz Nov 6 '19 at 15:16
  • We follow the pattern of re-using the Ansible roles, and setting environment-specific variables, to change configuration. The playbooks can then largely be re-used and the Ansible roles independently tested. In packer, we typically build all environments at the same time, so that they are linked back to a git commit. – Bruce Becker Nov 6 '19 at 15:18
  • Not sure i understand your wording here, are you saying you use ansible to build the consistent enviorments and packet is used to build them at the same time? – Nabil Aziz Nov 6 '19 at 15:23
  • (note : Packer is a tool from Hashicorp to build images. Packet.net is a bare-metal cloud provider -- these are different :) ). I'm afraid the mods will shut us down here for using the comments to chat, so perhaps it would be best to ask a different question to clarify this. – Bruce Becker Nov 6 '19 at 15:25
  • Its okay i understand it now, so as a high level overview i could use terraform to provision the servers with packer which basically installs things like nginx and stuff before the server is actually deployed then can use ansible actually setup the specifics. – Nabil Aziz Nov 6 '19 at 15:42
1

I am a big fan of saltstack, mainly because it provides a comprehensive solution regarding the whole server life cycle, unlike ansible which need another tool (such as terraform) to provision vms.

salt + salt-cloud is one very good suite from my opinion.

I do not see any differences regarding the amount of work required by either tool (chef, puppet, ansible, salt) to get a working IAC system in place, this is, of course, according to your language preference. If you're in python then go for salt or ansible, otherwise go check puppet or chef. Do not forget to plug your iac code base to a VCS.

| improve this answer | |
  • 1
    The push (salt/ansible) vs pull (chef/puppet) model difference may play a role in the choice too (for DMZ systems for example) – Tensibai Nov 6 '19 at 22:16
  • salt has an interesting ability for both model, pull (salt-call executed from the minion) and push (salt command executed on the master), this helps a lot for minion with an unstable connection. – Pier Nov 6 '19 at 22:23
  • Missed that model in salt, I didn't check it for a while that said, it still use the system python (as ansible) if I get it right, which make it quite complicated when running legacy apps incompatible with its requirements (Chef has a point here coming with its own ruby independent of the system one at the cost of a larger client of course) – Tensibai Nov 6 '19 at 22:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.