Bugfix: Limit nvidia-device-plugin to gpu instance types

[psnilesh]

Motivation

Nvidia device plugin (running as a DaemonSet) is currently set to run on all instances running linux os. Pods created by this DaemonSet in instances that don't have an Nvidia GPU perpetually stay in CrashLoopBackOff state due to init error (logs below). This change is for limiting pods to instances that we know for certain have an Nvidia GPU(s).

Logs from failing pods

I1124 03:32:15.826992       1 main.go:199] Starting FS watcher.
I1124 03:32:15.827061       1 main.go:206] Starting OS watcher.
I1124 03:32:15.827295       1 main.go:221] Starting Plugins.
I1124 03:32:15.827321       1 main.go:278] Loading configuration.
I1124 03:32:15.827891       1 main.go:303] Updating config with default resource matching patterns.
I1124 03:32:15.828083       1 main.go:314]
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "mpsRoot": "/run/nvidia/mps",
    "nvidiaDriverRoot": "/",
    "nvidiaDevRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": false,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/driver-root"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {}
  }
}
I1124 03:32:15.828096       1 main.go:317] Retrieving plugins.
E1124 03:32:15.828215       1 factory.go:87] Incompatible strategy detected auto
E1124 03:32:15.828225       1 factory.go:88] If this is a GPU node, did you configure the NVIDIA Container Toolkit?
E1124 03:32:15.828230       1 factory.go:89] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E1124 03:32:15.828237       1 factory.go:90] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E1124 03:32:15.828243       1 factory.go:91] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E1124 03:32:15.832475       1 main.go:149] error starting plugins: error creating plugin manager: unable to create plugin manager: invalid device discovery strategy

---

Instance types that use Nvidia GPUs were pulled directly from EC2 using a simple script pasted below,

import boto3
ec2 = boto3.client('ec2', region_name='us-east-1')
instance_types = ec2.get_paginator('describe_instance_types').paginate().build_full_result()['InstanceTypes']

gpu_instances = []
for instance in instance_types:
	if not 'GpuInfo' in instance: continue
	gpus = instance['GpuInfo'].get('Gpus', [])
	nvidia_gpus = [gpu for gpu in gpus if gpu['Manufacturer'] == 'NVIDIA']
	if nvidia_gpus:
		gpu_instances.append(instance['InstanceType'])

gpu_instances.sort()
for i in gpu_instances: print("- ml." + i) # Printing as yaml array for convenience

This may contain instance types that are not supported by Hyperpod today, it's better to keep them in the interest of future proofing.

Testing

Created a hyperpod cluster with 3 instance groups (2 cpu, 1 gpu) and deployed updated helm chart.
Verified NVD DaemonSet pods were only launched in gpu nodes.

# g5.2xlarge has GPUs, others don't. 
bash-3.2$ kubectl get nodes -o=custom-columns=NAME:.metadata.name,LABEL:.metadata.labels.'node\.kubernetes\.io/instance-type'
NAME                                         LABEL
hyperpod-i-030ad565a57fcced8                 ml.c5.2xlarge
hyperpod-i-05b04af8627a2e65c                 ml.g5.2xlarge
hyperpod-i-0d173f3bbea28b862                 ml.m5.xlarge
ip-10-2-249-242.us-west-2.compute.internal   c5.xlarge
ip-10-3-212-25.us-west-2.compute.internal    c5.xlarge

# Just one nvidia device plugin pod is running
bash-3.2$ kubectl get pods -A | grep nvidia
kube-system    hyperpod-dependencies-nvidia-device-plugin-ps7w4             1/1     Running   0          6m56s

# And that pod is running in g5.2xlarge instance
bash-3.2$ kubectl describe pod hyperpod-dependencies-nvidia-device-plugin-ps7w4 -n kube-system | grep -i node
Priority Class Name:  system-node-critical
Node:                 hyperpod-i-05b04af8627a2e65c/10.2.12.34
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             sagemaker.amazonaws.com/node-health-status=Unschedulable:NoSchedule

[adheshgarg]

LGTM

[xiaoxshe]

LGTM

[psnilesh]

Closing PR. Need to set up signed commits and raise another one.