Why it matters

K8s is the standard for ML deployment. Understanding GPU scheduling unlocks production ML at scale.

Advertisement

The architecture

Device plugin: DaemonSet on each GPU node. Advertises nvidia.com/gpu resources.

Pod requests: resources.limits.nvidia.com/gpu: 1 requires one GPU.

K8s GPU schedulingDevice pluginadvertises GPUsPod requestresource limitSchedulerassigns to nodeSharing options: MIG (hardware isolation), MPS (concurrent), time-slicing
GPU scheduling model.
Advertisement

How it works end to end

Node labels: node-selector or affinity to schedule to specific GPU types.

Sharing: MIG partitions or time-slicing enable multiple pods per GPU.

NVIDIA GPU Operator: helm chart that installs driver, toolkit, device plugin, DCGM.