Persistent Volume with Kubernetes

Question

I have an application which consists of 2 containers which are running on an on-premise system. The responsibility of the application is to accept data from other on-premise systems and upload it to cloud. First container exposes a store service which accepts data and writes the data to a volume mounted on the on-premise system. There is another upload service which gets triggered whenever a new file is available in this volume and uploads the data to S3 in the background.

The problem I am trying to solve is to host these containers in a K3S cluster to provide HA capabilities. I do not want to combine store and upload into a single POD as failure detection and recovery becomes complex. I want to host each container in a separate POD. In such a scenario how do I ensure that data is not lost whenever one container fails and new POD is started? For example, lets say store service received 100 files and written it to the volume. Upload started uploading but crashed while uploading 50th file. Now K3S will ensure that new upload POD is started, but how do I ensure that it mounts the same volume which had 50 pending files?

Answer 1

To ensure that the new upload pod mounts the same volume as the previous pod and continues uploading the remaining files, you can use a persistent volume claim (PVC) to claim the volume and mount it to the new pod.

A PVC is a resource in Kubernetes that allows a pod to request and consume a persistent volume (PV) in the cluster. When you create a PVC, you can specify the size of the volume and the access mode (e.g. read-write). The PVC is then bound to a PV that meets the specified criteria.

To use a PVC to claim the volume in your scenario, you can do the following:

1. Create a PV to represent the volume on the on-premise system. The PV should specify the path to the volume on the on-premise system, as well as the access mode (e.g. read-write).

2. Create a PVC that references the PV. The PVC should specify the same size and access mode as the PV.

3. Mount the PVC to the upload pod using a volume mount in the pod's spec.

Here is an example of how you can create a PV, PVC, and pod that mounts the PVC:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteMany
  hostPath:
    path: /path/to/volume

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: my-pv

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: upload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: upload
  template:
    metadata:
      labels:
        app: upload
    spec:
      containers:
      - name: upload
        image: upload-image
        volumeMounts:
        - name: my-volume
          mountPath: /path/to/volume
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-pvc

This configuration will create a PV and PVC that represent the volume on the on-premise system and claim it for the upload pod. The pod will then mount the volume to the specified path using the PVC.

If the upload pod fails and is restarted, the PVC will still be bound to the PV and the new pod will be able to mount the same volume and continue uploading the remaining files.

I hope this helps!

Answer 2

I'm picturing a use-case like a log-exporter on network with a very flaky external internet connection. I take it you want to ensure that even if there are considerable delays accessing S3, your client applications are not blocked, and you also do not risk data loss from local failures (of cluster nodes or storage drives).

You will need a Persistent Volume Claim, which both of your pods can then mount.

Since you are using k3s you may be limited in what underlying types of persistent volume are supported, unless you install additional components.