Why it matters

Memory pools shape GPU allocation cost. Understanding shapes performance.

Advertisement

The architecture

Pre-allocate large block.

Sub-allocate on demand.

Free back to pool.

Memory pool flowLarge blockreservedSub-allocateon demandReturn to poolon freePyTorch caching allocator is a pool; cudaMallocAsync in CUDA 11+ also pools
Memory pool.
Advertisement

How it works end to end

PyTorch: caching allocator.

cudaMallocAsync: CUDA 11+.

Custom pools for large runs.

Fragmentation risks.