2

I have setup a kubernetes node with a Nvidia Tesla k80 and followed this tutorial to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.

I have managed to install the nvidia daemonsets and I can now see the following pods:

nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84

The problem is that even while using the recommended image nvidia/cuda:10.0-runtime-ubuntu18.04 I still can't find the Nvidia drivers inside my pod:

root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src

But the tutorial mentions:

CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.

I have also tried to test if CUDA was working through torch.cuda.is_available() but I get False as a return value.

Many thanks in advance for your help.

1
1

Ok so i finally made it work.

It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..

Not the answer you're looking for? Browse other questions tagged or ask your own question.