1

Our build system at work is a series of Go pipelines with go agents running in docker containers on a docker cluster. It compiles java and runs tests. This is in a financial services institution.

We're using consul and registrator for service discovery and vault for secret storage.

I've experienced some pushback in my workplace about the perceived complexities of using docker for running compiles and tests. To counter this I've written lots of documentation, and run training courses.

There are some comments online that docker is complex.

In the Adrian Cockcroft videos, he mentions that in some cases they had to hire Google-level engineers, to wrap their heads around the concepts of distributed computing. In another video he mentions that it took them seven years to achieve their vision of containerisation.

My question is: Is there evidence to suggest that the Docker/Hashicorp ecosystem requires a higher calibre of software development engineer?

  • 1
    Was following until it became "Docker/Hashicorp" - how is HashiCorp involved in any of this? – Adrian May 19 '17 at 19:16
  • The "google level" is less about their skillsets, and more about their vision. It takes a certain type of personality that is in the minority to envision an ecosystem that is outside the norm of their experience. Heck, a couple years ago, I had a hard time getting people to understand the concept that your phone may one day be your desktop computer, tablet, and phone. Now that Nintendo has the "Switch" people are finally starting to come around that this is possible. Similarly, there are people that still see phones and tablets as "consumption" devices and not "creation" devices. – Erik Funkenbusch May 20 '17 at 3:05
3

I apologize for the novel length response, but I want the industry to start having these discussions as the potential advantages are huge.

The problem is not that it is necessarily more complicated but that it requires experience that is difficult to find today and which goes against traditional models.

This is not an uncommon occurrence in the computing industry, and I personally remember a time when TCP/IP was considered too complicated. The talent base will grow over time though.

Here are the main barriers I see today:

Tools:

On the platform side most Engineers are just really starting to leverage automation. Unfortunately they are leveraging configuration management tools which were designed to solve a different problem. As an example Puppet and Chef assume idempotency and long lived systems. A large portion of the industry is just starting to get good at those tools but they are inappropriate for a model where dynamic istanciation is the rule and long lived systems are rare.

Most of the monitoring tools or management tools in use also assume long lived systems do not work well in a dynamic environment.

Then there is the whole we must have a single source of truth problem that I will ignore, however it is a significant barrier.

Virtualization:

This is mostly an issue with vendor lock-in and market forces. But many SE's have been separated from the simple mechanics of virtualization that their is a knowledge gap. These individuals are capable of the work but have spent most of their time managing vendors so understanding concepts like linux namespaces or c-groups is almost foreign.

Networking:

Docker and Kubernetes really need network segmentation, and tenant isolation. While this happens at a google scale most companies are using fragile and expensive enterprise solutions. These tend to either have no automation or massive centralized management. Really each swarm or cluster needs it's own segment and the technologies like vxlan or sdn that enable this are just popping up.

Security:

Related to the lack of overlay networks, or shared VPCs between swarms and clusters across security domains this is a very hard nut for companies to crack right now. Most security teams have no way to figure out how to audit these systems. Most teams I see are trying to fix it with solutions that don't directly address the problem. Segmentation and limited super user access will improve security in the long run, but like most humans they can't really parse json policy files with ease.

Related to the knowledge gap in virtualization, most team do not understand the attack surface involved here. Mix this in with a tenancy to give into the need for super user access this is an unsolved problem for organizations. As an example the default AWS VPC limit per region pretty much means anyone running kubernetes in AWS today most likely has serious security issues and any breach will be massive.

This is less of a problem on GCE but it is a very real concern today. This is far from an unsolvable, but it is a real problem.

In closing:

With all of those negatives I still think it is one of the most promising directions looking forward. But our industry today is full of companies that struggle to even implement OpenStack, which is really just a a bunch of python wrappers, a message bus and a database.

It will be a hard sell for a while, but the advantages of true loose coupling will probably win out in the end.

0

Your probably looking more at the calibre of your platform engineers. You don't want to make developers jobs hardware nor do you want your software engineers to need to handle more complexity, it's simply contrary to your goal.

Instead it sounds like you are looking for a platform engineer who is able to wrangle the relevant technologies together so that they become transparent to the user's (devs) as the devs becomes more comfortable using them they will be able to stretch out and use some of the more unique features, but until then, providing some abstraction from the underlying complexity is probably a good idea.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.