Why architecture matters here
The economic argument is blunt: datacenter GPUs are the most expensive compute in the fleet, and fleet-wide utilization for inference and interactive workloads is notoriously poor — single-digit percentages are common for notebook and dev-cluster GPUs. Packing seven small workloads onto one accelerator changes the cost per workload by nearly an order of magnitude. But packing only works if tenants cannot hurt each other; otherwise every latency SLO in the building inherits the behavior of the noisiest colocated job.
That is the architectural point of MIG: quality-of-service is a hardware property, not a scheduler promise. With time-slicing, a tenant's P99 latency depends on what else is running — a long kernel from a neighbor adds whole milliseconds of head-of-line blocking. With MPS, compute is shared spatially but the L2 and DRAM controllers are common property, so a bandwidth-bound neighbor degrades everyone, and a fatal fault in one client historically could take down the shared context. A MIG instance's SMs, L2 slice, and DRAM slice are its own: measured latency on a 1g.10gb instance is essentially flat whether the other six slices are idle or saturated. That flatness is the product.
Isolation also has a security dimension that pure scheduling cannot reach. Separate page tables and memory slices mean instances are acceptable tenant boundaries in multi-tenant clouds — this is how cloud providers sell fractional GPUs to mutually distrustful customers. Error containment follows the same lines: an uncorrectable ECC error or an XID fault is scoped to one instance, and the others keep serving. The trade is capacity — MIG's fixed partitions cannot lend idle resources to a busy neighbor the way MPS can — which is exactly the trade you want for latency-critical serving and exactly the one you do not want for throughput-oriented training.
The architecture: every piece explained
Slices: the units of partitioning. An A100 80GB exposes eight memory slices (10GB each) and seven usable compute slices, each compute slice being a fixed share of SMs plus a share of L2 and copy engines (the eighth compute slice's worth of resources is consumed by MIG's own overhead). H100 raises the slice sizes; the model is the same. Partitions are built from these quanta, which is why MIG profiles have fixed shapes rather than arbitrary percentages.
GPU Instances and profiles. A GPU Instance (GI) is a bundle of compute and memory slices with hardware-partitioned L2, DRAM bandwidth, and its own set of engines. Profiles name the shapes: 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb on A100 80GB. Valid combinations are constrained by slice geometry — seven 1g instances, or 3g+2g+2g, or 4g+2g+1g, but not arbitrary mixes — and the profile table is effectively the bin-packing vocabulary operators plan capacity in. Each GI gets its own memory page tables and its own fault domain; a reset of one GI does not disturb the rest.
Compute Instances within a GI. A second-level split: a 3g GI can host three 1c Compute Instances that share the GI's memory and L2 but have dedicated SM subsets. CIs are the right tool when workloads trust each other with memory but want compute separation — for example, three replicas of the same model sharing one weight copy. Most deployments stop at one CI per GI; the memory-sharing subtlety is a common operational surprise.
Control and scheduling surface. MIG mode is enabled per GPU (a mode flip that requires the GPU to be idle and, on many platforms, a reset). Instances are created and destroyed via nvidia-smi mig or NVML, and each appears with its own UUID, visible to CUDA via CUDA_VISIBLE_DEVICES. In Kubernetes, the GPU operator's device plugin advertises instances as distinct extended resources (nvidia.com/mig-3g.40gb), and MIG Manager applies declarative per-node partition layouts from a ConfigMap, handling the drain-reconfigure-restart dance. DCGM exports per-instance utilization, memory, and error telemetry so each tenant gets its own graphs.
End-to-end flow
Follow a fleet from bare metal to serving traffic. A node with eight H100s boots; the GPU operator's MIG Manager reads the node's label — say all-3g.40gb for a mid-size-model serving pool — and finds GPUs in whole-GPU mode. It cordons the node, enables MIG mode on each GPU, creates two 3g.40gb GPU Instances per card (plus a 1g.10gb from the remainder, if the layout says so), creates one Compute Instance per GI, and uncordons. The device plugin re-inventories and the node now advertises sixteen nvidia.com/mig-3g.40gb resources instead of eight whole GPUs.
A deployment requests one mig-3g.40gb per replica. The scheduler places pods exactly as it would any resource; the container runtime injects the instance's devices, and inside the container nvidia-smi shows what looks like a smaller GPU — 40GB of memory, a fraction of SMs, its own UUID. The model server initializes CUDA against that UUID with no code changes: MIG is transparent to CUDA programs, which is most of why adoption is tractable.
At steady state, each replica's requests flow through its own copy engines into its own DRAM slice; kernels occupy its own SMs; a neighbor replica hammering its slice moves the first replica's P99 by approximately nothing. DCGM scrapes per-instance metrics; per-tenant dashboards and alerts key on the GI UUID.
Reconfiguration closes the loop. A new large-model rollout needs 7g.80gb instances: the layout ConfigMap changes, MIG Manager drains the affected nodes pod by pod (this is the expensive part — every colocated tenant on the card must move, because destroying GIs requires them idle), destroys and recreates instances in the new geometry, and the scheduler backfills. Fleets that reshape often learn to dedicate node pools per layout precisely to contain this blast radius.