10

We've been growing as a company, our products are expanding and our DevOps related activities and efforts are growing as well - we've switched from Bamboo to a more flexible and configurable Jenkins, using deployment pipelines and other plugins; switched to Ansible and starting to use Docker here and there internally.

All of these things require some level of coding or configuration - Ansible scripts and configs, Jenkins groovy scripts, Dockerfiles and YAML configs.

For now, we've created a separate "ops" repository with high-level directories for jenkins, ansible, docker and other (which is a terrible name, but for now all "other" DevOps automation things are there).

Our approach does not feel right and might not scale, but what are the best practices and recommendations to keep DevOps-related code in a code repository or repositories?

  • 6
    I go with the "each part is an app, one repo per app" method, in chef that means 1 repo per cookbook. – Tensibai Aug 24 '17 at 8:24
  • @Tensibai right, I was afraid that a single "ops" repo would quickly become impractical. Thanks. – alecxe Aug 24 '17 at 16:15
  • 1
    That has been the legacy form of cookbook management in chef, 1 repo with all cookbook and proven a footgun in most cases hence the change, but I'm less comfortable with ansible to say it would fit also, Jenkins pipelines (if v2) and docker files should live with the project they handle IMO, and I've no idea of what you put under other so I can't really give any advice here – Tensibai Aug 24 '17 at 19:11
  • @Tensibai got it! Other consists of mostly bash and python utilities or periodically executed scripts for multiple internal tools..they don't really fit anywhere and we could not think of a better place than "other"..I'll see if I can post the contents of the directories into the question as well. Thanks. – alecxe Aug 24 '17 at 20:07
  • 1
    I'd split them in multiple repo by 'affinity' of work, scripts working on app X together, you may have a script used on two apps, but if app A change in a way the script has to handle which app it talks to, it's better to have two separated versions, so ATEOTD I'd store them with the application they relate to or if they span multiples application in a specific repository per task, so you always have a version in line with the deployed applications and you don't need to tag an unrelated script at the same time. – Tensibai Aug 29 '17 at 13:32
3
+50

The current organisation of the code and configuration you describe is structured by the technical solutions involved. This is a bad design that will add a lot of overhead in our maintenance activities and will add a lot of traps in our way as well. Instead, that organisation should be structured around the artefacts we are deploying.

The reason for this is that we want to consider artefacts (e.g. a docker image or a software package) as the objects of the following verbs:

  • build
  • test
  • deploy

to consider a minimal set of automated tasks we want to perform. If we want to change something about how the test verb is implemented, it is easy to visit the folder corresponding to that artefact in the appropriate repository and then discover the jenkins-specific automation items that needs to be updated. Instead, if the automation recipes are structured around technical solutions, then we need to figure out of the blue that jenkins is involved in the test procedures and find there the artefact related automation items. In complex situations, the organisation around technical solutions makes updates very hard, because we have to know a priori all the technical solutions involved in some service to update them accordingly.

For instance a repository containing the code for a website and a micro-service “a” could have the following sub-directories dedicated to operations:

./ops/website
./ops/micro-service-a

each having three scripts called build, test and deploy. Now that the organisation of automation items has somehow been clarified, let's turn our attention to configuration.

The main conditions and requirements about the configuration organisation are set by the deploy verb when applied on a service-like artefact. The deploy verb should have the following parameters:

  • the version of the artefact to deploy,
  • the deployment target of the artefact, which describes the concrete environment where the deployed artefact will run (e.g. a cluster and endpoints it should talk to)
  • the credentials it should use to connect to other endpoints (e.g. databases)
  • the runtime configuration of (like how long cache entries should live, etc.)

From the operational perspective, this breakdown of the parametrisation matches the natural degrees of freedom of the deployment problem – aside from the credentials that could be bundled with the runtime configuration, but it is better to separate them to avoid spreading them carelessly.

5

I can answer bout docker, one of the best practice to use docker is to keep the docker file and the compose files in the same repository of the project, so wherever you clone the project you can build the docker image, and it is good to keep multiple versions of docker compose files for example (prod, staging, dev) so you can build the image and run the container with specific option for each env for example for dev machine you can use specific network and run more dependencies container or whatever.

5

Each tool's code goes into its own repo. for e.g

  1. Jenkins Groovy template into a Jenkins repo
  2. Ansible YAML playbooks in its own repo (with the roles , tasks , inventory sub directories
  3. Cloudformation / Terrform templates in its own repo
  4. Docker files in its own 5.. And So on

This would help you scale better in terms of process orchestration and maintaining various branches for each environment

This would give you more granular control and offload all your versioning overhead to the version control systems. Also create separate branches for each environment and tag the code for every production release (as we do for application code base). Think Infra and process in terms of code. (Any change in process has to be codified and sent to QA, SIT , UAT and then to PROD) similar to application.

For e.g you may have V2.1 of Ansible running in Production ( master branch) but V2.0 of docker containers running in Prod (master branch)

Similarly keep your DB scripts / bash scripts in their own repositories and maybe you can have a healthcheck file (JSON/YAML) configured to show the versions of all the tools / parts in each deployed URL for tracking and automating purposes. (So that your webhooks reads the URL and automates deployments)

  • 2
    The pitfall of this approach is, v2.1 is in qa and not validated and you have to patch production urgently, you can't modify v2.0 and if you make a v2.2 for this patch there's a high risk of it being lost or overwritten when v2.1 goes to production, multiply by the amount of separate code in a repo and you soon have a nightmare of backports (it works, but I had to add this disclaimer :)) – Tensibai Aug 25 '17 at 5:02
  • 3
    Using branches to keep track of environment/deployment-specific information looks like an ant-pattern to me: if we have 20 environments, this means that we have 20 branches to keep in sync… a likely source of errors and confusion. Could you explain why you do not use configuration files to keep track of environment/deployment-specific information and what is your workflow working with these branches? These are not trivial problems! – Michael Le Barbier Grünewald Aug 29 '17 at 13:03
3

Making a distinction between Ops, Dev and DevOps promotes isolation and enforces a "throw it over the wall" mindset. To increase cooperation between teams one should put everything in a repository that is required to build and deploy the project.

Having said that, the answer to the question:

How to structure DevOps related code and configs in a code repository?

is that if config is required to run the project then one should put it in the same directory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.