Provide a pathway for installing Device Support on kops

[benjigoldberg]

Hello Maintainers!

I would like to propose a new path for installing Device Plugins in Kubernetes in 1.8+. The reason this issue came up for me is a desire to expose GPUs in my AWS Cluster. While kops does support GPUs via hooks, it is using the soon-to-be-deprecated Accelerators API.

I would be happy to contribute to making this effort a success, but I would like some input from more experienced maintainers on where they best see this work being placed. Currently, we use user data in our kops GPU node to expose the NVIDIA GPUs via the instructions they outline.

Here's an excerpt from the prerequisites for installing the NVIDIA Daemonset that exposes support within the cluster:

NVIDIA drivers ~= 361.93
nvidia-docker version > 2.0 (see how to install and it's prerequisites)
docker configured with nvidia as the default runtime.
Kubernetes version = 1.9
The DevicePlugins feature gate enabled

As you can see, this is a nontrivial set of requirements. Given that, I don't think the current hooks implementation will support the new Device Plugin architecture and this set of requirements, as it requires such heavy modification to the host OS.

[byndcivilization]

Is there any work happening on this? We are currently trying to run gpus on debian-stretch. I've baked the nvidia driver into an AMI but can rework that into an nvidia-bootstrap update. I'm having trouble with the nvidia-docker piece though. Currently if I run this as a hook, nodeup will overwrite the changes made by the hook.

[brunsgaard]

For the drivers @mindprince has made some quite good work on an installer for ubuntu, the work is easily converted to debian if needed.

The problem as a see it is the nvidia-docker runtime. Either it has to be baked into the AMI, or we need need to install it and restart the docker daemon. Non of the solutions seems great to me.

@mindprince what are GKE doing here, my guess is that nvidia-docker runtime is preinstalled on the machine images that have accelerators attached?

[rohitagarwal003]

what are GKE doing here, my guess is that nvidia-docker runtime is preinstalled on the machine images that have accelerators attached?

GKE doesn't use nvidia-docker. The shared libraries that need to be injected into the container are injected by the device plugin: https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/cmd/nvidia_gpu/README.md

[brunsgaard]

@mindprince Is your guess that the same approach would work for kops?

[rohitagarwal003]

It should.

[brunsgaard]

@strin I am working on an installer for nvidia-docker that uses kops hooks.

A rough but working solution can be found here:
https://github.com/brunsgaard/kops-nvidia-docker-installer

Let me know if you give it a go, I am available in the slack channel for questions if you need help 👍

[bhack]

There is already a PR/Thread in #4971

[brunsgaard]

@bhack Okay, I did not know there already was people working on this. The work in the PR uses the GKE approach what I have done is using the Nvidia approach. Should I do a PR on my nvidia-docker-installer-hook or would you rather that I drop it in favour of #4971?

My biggest problem with #4971 is that it is using accelerators and is not promoting the DevicePlugin approach. Although, it might be able to work with the GKE DevicePlugin with some settings or minor modifications.

[bhack]

@brunsgaard As you know upstream has still not found a unified solution in kubernetes/kubernetes#54011. So it is really up to us as kops community to choiche what path we want to follow.

[kaazoo]

@brunsgaard Thanks for the hook script. I forked the repo at https://github.com/kaazoo/kops-nvidia-docker-installer and replaced the comands for CentOS 7. Perhaps both can be unified to support Debian/Ubuntu + CentOS.
I also had to update the Nvidia driver version, as building the driver failed when using stable upstream kernel versions (currently using kernel 4.16.x from 'elrepo' repository).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale