10

My organisation is experiencing an explosion of microservices. We currently have no formalised way of bootstrapping new projects. I'm finding that a team will come to me with a bug in their deployment or build process, and I will spend time on it only to realise I've already resolved it in another project. There's also a lot of inconsistency between projects that I'd like to see standardised.

The changes often involve a single file (e.g. serverless.yml or a Makefile) so a solution involving shared libraries e.g. git submodules doesn't seem viable. Each project will have its own set of configuration that needs to be maintained, e.g. Dockerfiles or serverless.yml, so centralised configuration management solutions for VMs aren't really applicable.

How can I ensure that new microservices conform to organisation standards and include bugfixes/features from existing projects in a way that's easy and intuitive for developers who want to start new projects? What are some best practises around resolving these issues?

The current workflow we have is to ask the person next to you "what project should I clone from to use as a template?" and then delete all the stuff that's not necessary for that project.

Improve this question

3 Answers 3

Reset to default
5

I'll add an answer of what my solution is so far, but I'm still really interested in hearing how other organisations are solving this problem and the best practices they have.

To resolve the issue of not having a consistent base to create projects for, my idea is to create a repository (repositories?) of boilerplates/templates and use cookiecutter as a tool to scaffold out new microservices. This way, each project is created from a standard base with all the lessons we've learned as an organisation integrated in to it. Any changes we make can be integrated upstream in to the boilerplate repository. I imagine we'll have templates for Nodejs Docker images, Serverless SPAs, Python lambdas, etc.

To resolve the problem of changes made to the templates being propogated downstream to every project, my solution is to implement a process where owners of microservices are made aware of changes to the template and are then responsible for propagating those changes to their microservice.

Improve this answer
1
  • This is what we do, in combination with a simple hello world app that illustrates the best practices as a concrete example.
    – Xiong Chiamiov
    Commented Oct 20, 2017 at 6:38
4

Use a configuration management/automated deployment system. This is what these were designed for. Things like Kubernetes, Puppet, Chef, Ansible and Salt Stack are designed for just this purpose and can be utilized with Golden Master templates, kickstart scripts, Amazon AMIs or just a Docker container. This allows you to use default base templates and then layer on additional roles. This will ensure builds are documented completely (as code) and it will be fast and easy to deploy to production, and will be deployed exactly identically to what was designed or deploy additional instances when the need for scalability or redundancy arises or something breaks. Changes/updates can also be integrated this way. Just as you release software updates, you can release updates to your configuration and the configuration code can be managed just as your software code is managed - in the same repos and with the same workflows. And when upgrade time comes, there is no mystery how the thing is built, just look at the script.

One way that configuration management systems do this is through heavy use of templating for your configuration files. For example, there are generally many lines that will be the same or similar in your environment. SaltStack utilizes jinja templates and puppet uses Embedded Ruby templates. Using AWS as an example, you may need to set a, api key, IAM role, region (or randomly select a region from a list of regions), a VPC, etc that is all the same across all instances. But then you need to have your functions and outputs unique. Or better yet you could write a puppet module or salt formula which defines "contracts" and use those contracts (api definitions) for both inputs and outputs saving you from having to configure your microservices two or three places.

SaltStack for example recently had a meetup to discuss this particular scenario. Furthermore, SaltStack is also able to manage and deploy docker containers natively. (Puppet also has a module for Docker) Likewise Ansible has playbooks and docs for working with serverless deployments and Docker containers.

Also, if you wish to continue with your serverless theme/paradigm, Puppet, Ansible and saltstack all are masterless or support a masterless mode, if you wish to continue this theme.

Improve this answer
4
  • I've added some examples and clarification to my question. Configuration management doesn't really help because each project is self-contained in its configuration. There's no issues with migrating to configuration as code, it's about maintaining the configuration as code and the configuration sprawl you could end up with if you hadd 100 microservices each with differing configuration. We currently use CI/CD with consistent builds so that's not a concern either.
    – user2640621
    Commented Oct 18, 2017 at 22:23
  • 1
    @user2640621 - have you ever used a configuration management system? Centralizing the "configuration sprawl" helps you manage it more easily and from one place (instead of 100 different places). While each project may be self contained, clearly there is some overlap as you are asking about templating deployments. It might be worth trying a couple out in a sanbox to get a feel for how they work before you write them off... This doesn't just automate the management of your config files - it does much more than that.
    – James Shewey
    Commented Oct 19, 2017 at 5:17
  • 1
    I've used SaltStack, Chef and Puppet, but only ever for managing VMs. Thanks for your answer, I'm definitely seeing how configuration management can be used outside of managing VMs now.
    – user2640621
    Commented Oct 19, 2017 at 6:08
  • 2
    @user2640621: If they are all different: "You code it, you run it". Let the teams manage the ops of their services. Let them feel your pain.
    – Martin Schröder
    Commented Oct 20, 2017 at 20:47
3

This question is broad so if my answer is a little off-base feel free to add context and specific examples so that I have a better understanding.

Using a machine image such as AWS' AMI would allow you to create a base or golden image, which you could then maintain and distribute or implement in your continuous delivery. Using this architecture you are ensuring that every microservice is deployed on consistent hardware with identical configuration so any problems you face are related to the microservice/application configuration.

You can further this immutability with the addition of configuration tools such as Ansible and Packer. Using configuration management you could provision the machine image with whatever you want on it (including the application). Packer would then allow you to take a snapshot of that machine image and each deploy would be identical.

Using this example you could 'bake' a base AMI with the correct packages, updates, and configuration installed with the help of Ansible and Packer. Additionally, you could look at 'Ansible-Pull' to complete the deploy by pulling the application code, making any changes, and deploying the microservice on top of that base image.

However the most important advice I can give is to just come up with a solution that the entire organization can support and maintain. It's worth trying to establish an SDLC that solves your particular set of problems, matches the culture and attitude of leadership, and embraces modern architecture practices.

I've been with three organizations and we've taken three very different approaches.

Good luck!

Improve this answer
2
  • We're not using any VM-based solutions (mostly Serverless and a bit of Docker), but I'll try apply my problem to your example. When someone wants to create a new packer image, where would they start? If each project is self-contained and there's no central repository for packer configuration, what do they use as their base for creating images? Perhaps one of the differences is that the projects I'm working with try to be as self-contained as possible, without any dependencies on centralised services such as Ansible where you can update your configuration for all projects at once.
    – user2640621
    Commented Oct 18, 2017 at 22:08
  • With serverless and Docker based architecture you can still apply these fundamentals. One strategy my be to maintain a base docker file. You can build a centOS-based docker file that includes some of configuration you expect on each microservice, then each team can pull that docker file in and build their own microservice specific docker file on top of that. To help with container management and continuous deployment you can use a tool like Kubernetes.
    – Chad
    Commented Oct 20, 2017 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.