2

After starting a long running kubernetes job we've found that the final result will fail to upload to its final location. We would like to force the pod to stay open after the main process fails so we can exec in and manually process the final results. If the main process fails and the pod exits before uploading the final result we will lose a large amount of time to re-process the job.

Is there a way to ensure the pod stays alive manually?

3

Keeping your pod alive after its main process exited is an anti-pattern and not possible in K8s.

The entire concept of a K8s Job is to run till completion. So what you are looking to do is neither ideal nor recommended.

It would be better to change the entrypoint command of your container to include a retry/error handling mechanism in case of failure.

With that said, you do have ways in which you can debug the failed pod.

If your Pod's container process exits due to error, the Pod will be marked with Status: Failed

But you can still debug the failed pod by attaching an Ephemeral container: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container

Note: Ephemeral containers are available as K8s alpha feature only from v1.16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.