Why it matters

CPU offload extends effective GPU capacity. Understanding shapes long-context serving.

Advertisement

The architecture

Hot cache on GPU.

Cold cache on CPU RAM.

Fetch back on reactivation.

KV offload flowActive sessionsGPU HBMIdle sessionsCPU RAM offloadReactivateCPU -> GPU transferCPU RAM 5-10x GPU HBM; PCIe transfer adds latency on reactivation
KV offload.
Advertisement

How it works end to end

Session detection: LRU idle.

Transfer via PCIe.

Reactivation cost: milliseconds.

vLLM + SGLang support.