0

Problem in a nutshell

What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.

Our configuration

  • Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture
  • So far the cluster have no persistent storage configured (other than the host storage)
  • We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a distributed and Highly Available storage.

What do i want

I've seen Ceph&Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.

What's the problem with that

If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.

Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.

The above example makes no sense to me, especially when i have an HDFS next to this cluster.

My question is what is my best option to use the HDFS as a dynamic storage provisioner?

Improve this question

1 Answer 1

Reset to default
1

A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option

VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

architecture overview of vSphere Cloud Provider

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

Disclaimer: I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph & Rook. Only storage nodes will have restrictions on them. If you are going with Ceph & Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.

Improve this answer
4
  • Hello @haroon, thanks a lot for your answer. I'll take a look at VCP, it might indeed be a good option. i'm not going to run any classic RDBMS into kubernetes. But i may have some elasticsearch, mongo or these kind of thinks running in the cluster. I may end up using two storage NFS for regular volumes that does not requires any specific performances and Ceph or VCP for fast storage then...
    – Anthony Raymond
    Commented Feb 5, 2020 at 13:40
  • Also about the NFS, did you used an external NFS server or an in-cluster NFS provider? I would like to use the HDFS over NFS so i'd just need the NFS-client-provisioner, but the only one i see seems outdated and no longer maintained. Which one did you used?
    – Anthony Raymond
    Commented Feb 5, 2020 at 15:38
  • I am using external NFS server. For NFS provisioning I am using quay.io/external_storage/nfs-client-provisioner, you can deploy it using helm chart nfs-client-provisioner. If you are using HDFS over NFS, one thing to keep in mind that it does not support random writes so Mongo, Elastic may have to go to "fast storage".
    – haroon
    Commented Feb 6, 2020 at 3:23
  • Ok thanks :) What does that means "it does not support random writes" and how does that affects mongo, elastic etc?
    – Anthony Raymond
    Commented Feb 6, 2020 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.