After starting a long running kubernetes job we've found that the final result will fail to upload to its final location. We would like to force the pod to stay open after the main process fails so we can exec in and manually process the final results. If the main process fails and the pod exits before uploading the final result we will lose a large amount of time to re-process the job.
Is there a way to ensure the pod stays alive manually?
Keeping your pod alive after its main process exited is an anti-pattern and not possible in K8s.
The entire concept of a K8s Job is to run till completion. So what you are looking to do is neither ideal nor recommended.
It would be better to change the entrypoint command of your container to include a retry/error handling mechanism in case of failure.
With that said, you do have ways in which you can debug the failed pod.
If your Pod's container process exits due to error, the Pod will be marked with Status: Failed
But you can still debug the failed pod by attaching an Ephemeral container: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container
Note: Ephemeral containers are available as K8s alpha feature only from v1.16
I want to add a quick and dirty solution to this. I often add sleep 600 to the end of a command that's failing unexpectedly so that I can exec in and poke around to see what happened. Yaml example (note: - >- enables multi-line concatenation, just a convenience):
command: ["bash", "-c"]
args:
- >-
python myscript.py; sleep 600;
With that said, Karthick's answer is better, this is just a simple solution that doesn't follow best practices. I encourage you not to use sleep or sleep infinity, because it's very easy to forget that your pod is running and holding onto resources indefinitely.