On-Prem and Cloud Deployments

Question

If I have a mix of cloud and "on-prem" (across different networks) systems to deploy to is there a common tool, technique, or software available? I've used Kubernetes for cloud deployments, but am unsure of what approach to take when On-Prem installs/upgrades are mixed in.

Is is considered best practice to attempt to standardize the deployment or work with the ops team to develop two separate deployment strategies?

Answer 1

If you can keep one common deployment strategy across both that would be simpler and less effort from the ops side of things. Kubernetes seems to be a great choice for the cloud orchestration layer at this point in time. Does Kubernetes make sense for the on-prem side though? If you're not planning on each on-prem install including 5-10 servers to spread Kubernetes out over then it probably doesn't make sense to use it there. All hope is not lost. :) You could take the same Docker containers that you ran under Kubernetes and run them yourself for on-prem. You're going to be stuck dealing with service discovery, but if you're only installing on one server that can be solved in some old school way like picking and assigning ports to specific services.

Answer 2

(Kublr CTO) If you can use Kubernetes as the deployment target for your applications, it is a great tool to standardize delivery and operations. The only problem is, as you correctly pointed, a consistent approach to deployment and operations of Kubernetes clusters themselves.

The best practice in my opinion is definitely to centralize and unify Kubernetes management; but whether it is actually worth to do depends on the expected scale of operations (how many clusters you want to manage, and how often changes are implemented), and on availability of Kubernetes management tools that fit your specific needs.

Generally speaking here are some links where you can start gathering information on various tools in this area:

• A place to start when selecting a Kubernetes management solution that suites your needs: https://kubernetes.io/docs/setup/ - it includes a list of products and tools suitable for managing cluster fleets in production.
• Another good curated list of tools and solutions related to Kubernetes, including deployment and management tools, can be found here: https://github.com/ramitsurana/awesome-kubernetes
• One more reference point for research is the Kubernetes conformance testing results maintained by CNCF (Cloud Native Computing Foundation) on their GitHub account: https://github.com/cncf/k8s-conformance ; you may find conformance testing results there from different Kubernetes management tools and see which of them maintain conformance with relatively current versions of Kubernetes, e.g. here is the K8s 1.15 conformance testing results: https://github.com/cncf/k8s-conformance/tree/master/v1.15

Although the full list of tools and vendors looks huge, if you filter it though the "production readiness requirements" list, there may actually be very few solutions left that fit the bill.

Here is the list, in no particular order, of considerations that may be important in case like yours:

• the tool is mature enough for production use, and deploys production quality (secure, reliable, self-healing) clusters,
• deployed clusters compatibility and conformance - whether upstream Kubernetes is used or a custom build,
• configurability - whether and how the tool limits deployed Kubernetes clusters configuration options; e.g. can you adjusts Kubernetes API server, controller manager, or kubelet configuration options
• whether support is available, for the tool itself, for the clusters deployed through the tool, and for the "borderline" problems - problems for which you cannot be sure whether it is your application running on the cluster, or the cluster itself, or a specific configuration of this cluster,
• cost,
• whether the tool is SaaS or can be installed in your environment, cloud or DC;
• whether the deployment and management are centralized, or each cluster is separate; and whether a "single pane of glass" for your clusters and a single API for cluster management are available;
• management and "day 2 operations" of clusters - updates, upgrades, recovery etc
• if the tool provides centralized features, then how access control management for cluster operations looks like (this is different from Kubernetes RBAC, which is implemented by Kubernetes itself),
• integrations with enterprise IDM for SSO and user federation,
• support for different and heterogeneous target environments - e.g. AWS, GCP, Azure, on-prem, vSphere; and support for different target OS
• uniformity of Kubernetes clusters deployed into different environments: e.g. if your Kubernetes clusters in AWS are deployed using AWS EKS, and Google cloud clusters are GKE, then you may run into a situation when these clusters cannot achieve feature parity due to different set of supported versions, addons etc,
• availability of operations tools like log and metrics collection, visualization, alert and notification management etc.

In cases where you know for sure that you will stay in one specific cloud, the corresponding cloud provider's managed Kubernetes solution would probably be the first candidate - such as AWS EKS, GCP GKE, and Azure AKS; although it should still be run through the list of your specific requirements.

In other cases, cloud-agnostic tools should be considered.

Another consideration to keep in mind is how entangled or integrated your infrastructure management, Kuberletes cluster management, and application operations are going to be with different tools. Different tools may put different emphasis on different parts of the stack; e.g. some Kubernetes distributions may also provide integrated and/or opinionated application lifecycle management components, or integrated infrastructure/cost management features.

While integrated solutions may be better in terms of time to market and learning curve; at the same time they may limit your ability to benefit from the latest developments in DevOps/CI/CD technologies.

I personally lean towards treating Kubernetes as a convenient infrastructure abstraction layer, on top of which not only your applications, but also your DevOps/CI/CD tools, and various managed services (e.g. cloud native storage solution, databases, etc) can run. We have some presentations published on our web site about that: https://kublr.com/register/all-things-kubernetes-online-meetup/

Talking about specific vendors, I must put a disclaimer here - I am a CTO at Kublr - https://kublr.com/ - an enterprise Kubernetes management solution built around the requirements listed above. It can be found in the lists of Kubernetes tools mentioned above; you can also schedule a demo on the web site, or get an unlimited non-production version at https://kublr.com/deploy/