Problem in a nutshell
What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.
- Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture
- So far the cluster have no persistent storage configured (other than the host storage)
- We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a distributed and Highly Available storage.
What do i want
I've seen Ceph&Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.
What's the problem with that
If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.
Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.
The above example makes no sense to me, especially when i have an HDFS next to this cluster.
My question is what is my best option to use the HDFS as a dynamic storage provisioner?