GPU Support on AWS

[SNagappan]

I would like to request GPU support on KOPs in aws. I was able to get it working manually with the following:

• adding the experimental_nividia_gpu=1 flag to kubelet on each worker node
• adding SecurityPrivileged: true for each pod to map /dev/nivida0 to the container
• installing CUDA on each worker node

[jimfleming]

What would be the best, correct, reproducible way of adding the experimental-nvidia-gpus flag? (Ideally without ssh'ing in and adding it manually.)

FWIW, you don't need to add SecurityPrivileged: true or map /dev/nvidia0 as this is done for you when the experimental-nvidia-gpus flag is enabled: kubernetes/kubernetes@08440b5#diff-0db82891d463ba14dd59da9c77f4776eR585

[justinsb]

I'll map the flag right now, if someone tells me how to install CUDA ;-)

[jimfleming]

Hah, deal! Which OS?

[justinsb]

Sorry for delay - got sidetracked. I opened a PR to map the flag, so hopefully we can get that into a release.

For "what OS" - I think the answer is "whatever OS you would recommend". I'd like to test the flag working all the way through myself, and I'd like to have some procedure to install CUDA that we can point people to, even if we say "we have docs for Ubuntu 16.04, so you should probably do that" or something like that. If we can get CUDA into the base image that would be great, but I've been told licensing is problematic. I opened #1726 for that particular issue.

[jimfleming]

Awesome thank you. Last I checked (about 6 months ago) the recommended OS from NVIDIA was Ubuntu 14.04 but 16.04 also works and is better supported by Kubernetes—I'm using CUDA + Ubuntu 14.04 on a locally deployed cluster and would like to use Kops + AWS to replicate it.

These are the steps I used to install CUDA on the Ubuntu 14.04 nodes:

• Download and install the appropriate drivers for your GPU: http://www.nvidia.com/Download/index.aspx?lang=en-us
• wget cuda deb for Ubuntu: https://developer.nvidia.com/cuda-downloads
• sudo dpkg -i cuda.deb # from download above
• sudo apt-get update
• sudo apt-get install cuda
• Add nouveau to the blacklist on Ubuntu 14.04 (only if necessary, e.g. driver doesn't work):

sudo cat <<EOF >/etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
option nouveau modeset=0
EOF

You can verify the driver is installed by running nvidia-smi which should report the found GPU and correct driver version. To verify CUDA is more use-case specific. As I'm working with TensorFlow primarily, I just installed nvidia-docker and ran:

nvidia-docker run -it gcr.io/tensorflow/tensorflow:latest-gpu python -c 'import tensorflow as tf; print tf.VERSION; tf.Session()'

TensorFlow should report finding multiple CUDA so's and identifying the GPU.

Now to get CUDA inside the container is tricky because it needs the correct drivers mounted as well as CUDA. What I've done locally is replicate the functionality provided by nvidia-docker using a bash script to build a template spec with the appropriate volumes depending on which GPU I'm targeting. Kind of hacky but it works well.

[mzsanford]

I have a working container using a modified version of the AMI k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09 (ami-5f1afc49) with CUDA installed and a second kops instancegroup for my GPU servers. I tested my manually adding --experimental-nvidia-gpus=1 to my kubelet script on an instance and it worked. That's hardly a long term solution since it's in an auto-scaling group. I see the referenced pull request has been merged. What I can't figure out is where in my instance group I can add the new nvidiaGPUs flag. How do I apply that to nodes in an instancegroup?

[diwu1989]

im also interested in this, is it possible to have an official tutorial written for how to enable GPU on AWS?

[justinsb]

The challenge is that it's all changing in k8s 1.6 :-(

The good news is that it looks like we'll be able to have a pretty nice experience on 1.6 - do check out #2298 and offer feedback - the idea is that we'll have a docker container that can install the nvidia drivers, and currently you'll have to manually pass the (new) required feature flag.

If that goes well we can probably simplify this further as well...

[diwu1989]

I read over https://github.com/kubernetes/kops/blob/master/docs/gpu.md and see that there's some instructions.

Which ami should we be using to make that one work? Also, is there a single place that lists all the official AMIs that Kops recommends for each version of kubernetes?

[diwu1989]

I'm also getting error following that documentation when trying to add hooks to the cluster spec.

Getting error

# Found fields that are not recognized
# ...
#         name: us-west-2a
#       name: events
# +   hooks:
# +   - execContainer:
# +       image: kopeio/nvidia-bootstrap:1.6.0
#     kubernetesApiAccess:
#     - 0.0.0.0/0
# ...

[chrislovecnm]

You run master build with custom nodeup and protokube?

[tartavull]

Had the same error as @diwu1989 and building from master fixed. Maybe something to add to gpu.md