CD2H gitForager

NVIDIA/k8s-device-plugin

Name: k8s-device-plugin

Owner: NVIDIA Corporation

Description: NVIDIA device plugin for Kubernetes

Created: 2017-10-10 21:31:02.0

Updated: 2018-04-01 17:19:59.0

Pushed: 2018-03-28 03:17:42.0

Homepage: null

Size: 1686

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

About

The NVIDIA device plugin for Kubernetes is a Daemonset that allows you to automatically:

Expose the number of GPUs on each nodes of your cluster
Keep track of the health of your GPUs
Run GPU enabled containers in your Kubernetes cluster.

This repository contains NVIDIA's official implementation of the Kubernetes device plugin.

Prerequisites

The list of prerequisites for running the NVIDIA device plugin is described below:

NVIDIA drivers ~= 361.93
nvidia-docker version > 2.0 (see how to install and it's prerequisites)
docker configured with nvidia as the default runtime.
Kubernetes version = 1.10
The DevicePlugins feature gate enabled

Quick Start

Preparing your GPU Nodes

The following steps need to be executed on all your GPU nodes. Additionally, this README assumes that the NVIDIA drivers and nvidia-docker has been installed.

First you will need to check and/or enable the nvidia runtime as your default runtime on your node. We will be editing the docker daemon config file which is usually present at /etc/docker/daemon.json:


"default-runtime": "nvidia",
"runtimes": {
    "nvidia": {
        "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
    }
}

if runtimes is not already present, head to the install page of nvidia-docker

The second step is to enable the DevicePlugins feature gate on all your GPU nodes.

If your Kubernetes cluster is deployed using kubeadm and your nodes are running systemd you will have to open the kubeadm systemd unit file at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add the following environment argument:

ronment="KUBELET_EXTRA_ARGS=--feature-gates=DevicePlugins=true"

If you spot the Accelerators feature gate you should remove it as it might interfere with the DevicePlugins feature gate

Reload and restart the kubelet to pick up the config change:

do systemctl daemon-reload
do systemctl restart kubelet

In this guide we used kubeadm and kubectl as the method for setting up and administering the Kubernetes cluster, but there are many ways to deploy a Kubernetes cluster. To enable the DevicePlugins feature gate if you are not using the kubeadm + systemd configuration, you will need to make sure that the arguments that are passed to Kubelet include the following --feature-gates=DevicePlugins=true.

Enabling GPU Support in Kubernetes

Once you have enabled this option on all the GPU nodes you wish to use, you can then enable GPU support in your cluster by deploying the following Daemonset:

bectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.10/nvidia-device-plugin.yml

Running GPU Jobs

NVIDIA GPUs can now be consumed via container level resource requirements using the resource name nvidia.com/gpu:

ersion: v1
: Pod
data:
me: gpu-pod
:
ntainers:
- name: cuda-container
  image: nvidia/cuda:9.0-devel
  resources:
    limits:
      nvidia.com/gpu: 2 # requesting 2 GPUs
- name: digits-container
  image: nvidia/digits:6.0
  resources:
    limits:
      nvidia.com/gpu: 2 # requesting 2 GPUs

WARNING: if you don't request GPUs when using the device plugin with NVIDIA images all the GPUs on the machine will be exposed inside your container.

Docs

Please note that:

the device plugin feature is still alpha which is why it requires the feature gate to be enabled.
the NVIDIA device plugin is still considered alpha and is missing
- Security features
- More comprehensive GPU health checking features
- GPU cleanup features
- …
support will only be provided for the official NVIDIA device plugin.

The next sections are focused on building the device plugin and running it.

With Docker

Build

Option 1, pull the prebuilt image from Docker Hub:

cker pull nvidia/k8s-device-plugin:1.10

Option 2, build without cloning the repository:

cker build -t nvidia/k8s-device-plugin:1.10 https://github.com/NVIDIA/k8s-device-plugin.git#v1.10

Option 3, if you want to modify the code:

t clone https://github.com/NVIDIA/k8s-device-plugin.git && cd k8s-device-plugin
cker build -t nvidia/k8s-device-plugin:1.10 .

Run locally

cker run --security-opt=no-new-privileges --cap-drop=ALL --network=none -it -v /var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins nvidia/k8s-device-plugin:1.10

Deploy as Daemon Set:

bectl create -f nvidia-device-plugin.yml

Without Docker