Are there automated quality checking tools for Ops code

Question

In the dev world, we have several tools available to us for automated quality checks. We have unit tests that run on each build, Sonar to check for code quality, metrics being sent to Grafana and Okta with automated alarms, etc.

I'm not very familiar with the ops world; are there similar automated tools available for our ops code?

For Terraform scripts, Dockerfiles, Helm/K8 scripts, Cloudbees configurations and other ops code, the concerns I would like to see automation for are:

• Syntax checking
• Code smells / anti patterns
• Routing / Connection checking for accuracy
• Security best practice checking (don't expose these ports or include these credentials,or grant these permissions kind-of-a-thing)
• Appropriate provisioning (too many or too few resources allocated for the size of the task)
• High cost warnings
• Unit test-ish tools to ensure we aren't breaking the build in more subtle ways than "did it deploy successfully or not"

There's likely a lot more that I don't even know to look for given my lack of experience. Are there support tools that will analyze our ops code that could provide automated support in this fashion?

Answer 1

It's difficult to provide a definitive answer given the scope of the question, but the goal of testing infrastructure is both laudable and seldom done at all.

To my knowledge, you are looking for something akin to Open Policy Agent which provides a general policy language in which to express arbitrary policy as code. I would suggest investigating that at length.

At the risk of not providing a complete answer, I will give my best shot for some concrete examples as well.

Compliance as Code

In general, you need something which is often referred to as a "compliance framework", and in the DevOps world more often referred to as "Policy as Code". This allows you to define policies as code, which really means "machine-readable and executable". You specify what the policy is in the DSL of the framework, it is executed at some stage of your pipeline, and fails the change if it violates the policy.

To illustrate the point, consider someone submitting a pull request to a terraform module that inadvertently opens a port on a security group to the world which the policy says should be closed. This may be difficult to determine when you're writing the code, but the compliance test will catch it.

There are a few ways to do this, but I will mention two. A notable exception is Pulumi's Crossguard, but I don't have enough experience to speak to that.

Hashicorp Sentinel

In the Hashicorp world, there is the Sentinel Framework. This is an Enterprise product, so may not be relevant to your situation, but it is one of the best implementations I've seen. Importantly, it has a built-in testing framework which aids in writing good policy code, which is also very expressive. I have used this as inspiration for how to write policy in Chef Inspec, which is the other example I will cite.

Chef Inspec

Chef Inspec is a framework for writing policy as code using rspec-style assertions. It uses a DSL based on ruby and supports major clouds (AWS, GCP, Azure) as well as OS-level resources. It is open source, so you can use it anywhere. You use it to make assertions in individual resources which are defined in the inspec DSL, e.g. "user on an OS" or "AWS EC2 instance". Writing the tests consists of assertions against the actual state of those resources, which makes it quite like a unit test. However, Inspec does not have a built-in testing framework for itself like Sentinel does - although you could use the ruby ecosystem for doing this.

---

Some notes:

1. Both of these tools allow you to express your policy as code, and to start extending the test coverage of that policy over your existing infrastructure.
2. Sentinel will catch violations before they are applied, but Inspec needs to be executed against an already-provisioned state.
3. Both will do what you required, but depend on your tech stack as to whether they will be appropriate for you
4. There is no magic bullet
   1. for both of them you will need to write your policy as code. Usually this means taking some form of human-readable documentation and writing it in the DSL of the framework.
   2. On the other hand, this provides an excellent opportunity to collaborate with the security team and do what folks are calling DevSecOps these days.

Answer 2

The tools that we have include in our Terraform pipelines to check code quality and best practises are;

1. Tflint
2. TFSec
3. Checkov

This any issues get highlighted early at the MR stage.