I have an application which consists of 2 containers which are running on an on-premise system. The responsibility of the application is to accept data from other on-premise systems and upload it to cloud. First container exposes a
store service which accepts data and writes the data to a volume mounted on the on-premise system. There is another
upload service which gets triggered whenever a new file is available in this volume and uploads the data to S3 in the background.
The problem I am trying to solve is to host these containers in a
K3S cluster to provide HA capabilities. I do not want to combine
upload into a single POD as failure detection and recovery becomes complex. I want to host each container in a separate POD. In such a scenario how do I ensure that data is not lost whenever one container fails and new POD is started? For example, lets say
store service received 100 files and written it to the volume.
Upload started uploading but crashed while uploading 50th file. Now K3S will ensure that new
upload POD is started, but how do I ensure that it mounts the same volume which had 50 pending files?