Why it matters

Data loading bottlenecks GPU workloads. GPUDirect eliminates the bottleneck. Understanding shapes ML pipeline design.

Advertisement

The architecture

GPUDirect Storage: NVMe reads directly to GPU memory. Bypasses CPU-RAM bounce.

GPUDirect RDMA: InfiniBand/RoCE NIC reads/writes GPU memory directly.

GPUDirect familyGPUDirect StorageNVMe → GPUGPUDirect RDMANIC → GPUGPUDirect P2PGPU → GPUEnables 100+ GB/s effective bandwidth to GPU without CPU bottleneck
GPUDirect variants.
Advertisement

How it works end to end

Enablement: kernel module + hardware support (Mellanox NICs, NVMe SSDs). Application uses cuFile API for storage.

Impact: 5-10x lower CPU utilization; 2-3x higher throughput.

Use cases: dataloading for training, checkpointing, high-throughput inference input.