Multi-Instance GPU is the only GPU-sharing mechanism that does not depend on a scheduler behaving well. Rather than interleaving several tenants’ work across one shared set of SMs, MIG physically divides the chip: compute slices, L2 slices, and — the part that actually decides the outcome — the memory controllers and the bandwidth path sitting behind them. Each partition surfaces as a separate CUDA device with its own address space and its own fault domain. What follows: what gets divided, what that buys, and the capacity you hand over for it.
What MIG actually partitions
The instructive way to read MIG is as a list of resources that stop being shared. An instance receives a fixed group of SMs at a coarse hardware granularity — you cannot ask for an arbitrary SM count, only whole slices. It receives a share of the L2 cache, not a quota against a common cache but specific L2 banks. It receives a region of HBM together with the memory controllers in front of that region. It receives its own copy engines, so a large host-to-device transfer in one instance does not queue behind a neighbour’s.
Everything on the critical path of a running kernel — issue, cache, bandwidth, DMA — falls on the divided side of the line; only the host attach point and the driver’s own management state stay global. That is the entire design intent.
The memory path is the whole argument
Partitioning SMs alone would buy far less than people assume. Give two tenants disjoint SMs on a shared memory system and a bandwidth-hungry neighbour still saturates the HBM controllers, still thrashes L2, and still moves your latency. Most interesting GPU work is memory-bound rather than issue-bound — LLM decode is the canonical case, where each token re-reads the weights — so the shared resource that actually determines your throughput is bandwidth, not FLOPs.
MIG’s key move is that a memory slice is not a capacity quota. It is a set of physical memory channels and the L2 banks that front them, bound to the instance. A neighbour running flat out is issuing traffic through different controllers. There is no arbiter for it to win at your expense, no shared cache for it to evict you from. That is why MIG isolation is described as a hardware property: the interference channel was removed, not scheduled around.
Slices, GPU Instances, and Compute Instances
The partition vocabulary is two levels deep. A GPU Instance (GI) is the real isolation boundary: compute slices plus memory slices, with the L2 and DRAM path, its own page tables, and its own error domain. A Compute Instance (CI) is a second-level split inside a GI, dividing that GI’s SMs among several contexts while they continue to share the GI’s memory and L2.
CIs are the right tool when the workloads trust one another with memory — several replicas of one model sharing a single weight copy, for instance — because you get compute separation without paying for a second copy of the weights. They are the wrong tool for hostile tenants, since the memory isolation people believe they bought stops at the GI boundary. Most production layouts create exactly one CI per GI and treat the GI as the unit.
Profile geometry and the placement constraint
You do not request a percentage; you request a profile, named by compute slices and memory capacity in the familiar Ng.Mgb form — a 3g.40gb instance carries three compute slices and enough memory slices to reach that capacity. The catalogue is fixed per generation and memory configuration, so treat any specific list, including the figure above, as illustrative.
Two constraints bite in practice. First, compute and memory slices do not come in equal numbers on a given part, which is why a full split yields seven instances on A100-class hardware rather than a rounder eight. Second, instances occupy aligned positions in the slice array, so a shape being legal does not make it placeable next to what is already there. Destroy a middle instance and you can end up with enough free slices for a larger profile and nowhere contiguous to put it — classic fragmentation, in a space far too small to bin-pack your way out of.
What a slice looks like to CUDA
Adoption is tractable mainly because the abstraction holds. An instance gets its own UUID, appears through CUDA_VISIBLE_DEVICES like any device, and reports a smaller memory size and SM count. An unmodified model server initialises against it and runs, with no MIG-specific API to call.
The seams show at the edges. Peer-to-peer between instances is not available, so anything that assumes device-to-device copies or CUDA IPC across the partition has to stage through the host. Multi-GPU collectives are correspondingly awkward and MIG is best treated as a serving mechanism rather than a training one. MIG mode is compute-only, so graphics and display paths are off the table, and several whole-GPU profiling counters are unavailable or scoped differently inside an instance.
The predictability argument for multi-tenant serving
The reason to accept all of that is a latency distribution that stops depending on strangers. Under time-slicing, your P99 encodes your neighbours’ kernel durations, because a long kernel ahead of yours is head-of-line blocking you cannot see or control. Under MIG, an instance’s measured latency is close to identical whether the rest of the card is idle or saturated, and that flatness is the actual product being sold.
It compounds operationally. Capacity planning becomes arithmetic instead of queueing theory: benchmark one instance, multiply. An SLO can be written per tenant and defended, because nothing another tenant does can breach it. Error containment follows the same lines — a fault scoped to one instance leaves the others serving. And because instances have separate address spaces, they are a defensible boundary between mutually distrustful tenants, which is precisely how providers sell fractional GPUs.
MIG versus MPS — work-conserving versus predictable
MPS is the natural comparison, and the honest framing is a single trade. MPS runs several processes concurrently on the whole GPU without hardware partitioning: SMs, L2, and memory controllers stay common property. That makes it work-conserving — an idle client’s share is immediately usable by a busy one, so aggregate throughput on bursty, uncorrelated workloads beats anything a fixed partition can reach.
It also makes it unpredictable in exactly the dimension MIG fixes. Bandwidth and cache contention flow straight through, so a neighbour’s memory-bound phase shows up in your tail. Pick MPS when tenants are cooperative, load is spiky, and you are optimising aggregate throughput. Pick MIG when tenants are strangers, the SLO is a promise, and you would rather waste some silicon than explain a P99 excursion nobody in your team caused.
Where MIG wastes capacity
A fixed partition cannot lend. If your instance is idle, those SMs and that bandwidth are idle with it, no matter how starved the instance beside you is. On bursty traffic with a low duty cycle that is straightforwardly worse utilisation than sharing, and it is the honest cost of the guarantee.
Two subtler losses matter. L2 is partitioned along with everything else, so a working set that fits comfortably in the whole GPU’s cache can spill to HBM inside a slice, and per-slice efficiency stops scaling linearly downward. And capacity is a hard wall, not a soft one — memory is partitioned, so weights plus activations plus KV cache must fit the slice, and a model that spills is not slow, it fails to load. The catalogue is coarse too, so a workload needing slightly more than one profile offers pays for the next size up.
Reconfiguration is the expensive operation
MIG layouts are not elastic at runtime. Enabling or disabling MIG mode requires the GPU idle and, on many platforms, a reset. Creating and destroying instances happens through nvidia-smi mig or NVML, and destroying one requires nothing running against it. Because reshaping usually disturbs the slice array as a whole, changing the layout in practice means draining every tenant on that card, not just the one whose shape is changing.
That blast radius shapes fleet design more than any other MIG property. In Kubernetes the device plugin advertises instances as distinct extended resources and an operator applies declarative per-node layouts, but underneath it is still cordon, drain, destroy, recreate, uncordon. The standard response is to stop reshaping: dedicate node pools per layout and route workloads to the pool whose geometry already fits. Treat the partition layout as a slow-moving capacity decision, not a scheduling knob.
Which workloads fit a slice, which need the whole GPU
The shape that suits a slice is small, steady, and latency-sensitive. A modest model whose weights and KV cache sit comfortably inside the partition, serving reasonably constant traffic under a per-tenant SLO, is close to the ideal case: it wants isolation, it does not want elasticity, and its footprint is known. Notebooks, dev environments, embedding and reranking services, and small classifier endpoints sit here, and the utilisation win is dramatic — those workloads routinely leave a whole card in single digits.
The shape that needs a whole GPU is large, bursty, or collective. Training wants every SM and full bandwidth, and it wants peer-to-peer that MIG removes. Large-model serving needs the full memory and often more than one card. Bursty traffic wants a work-conserving mechanism that can absorb a spike. If your answer to how much GPU does this need is ‘it depends on the hour’, a fixed partition is the wrong instrument.