Why architecture matters here

HBM utilization failures are what limit LLM serving. Architecture matters because layout + coalescing + roofline analysis together explain throughput.

Advertisement

The architecture: every piece explained

The top strip is the primary tier. HBM stacks. Bandwidth. Memory banks. Coalescing.

The middle row is caches + model. L2 cache. L1 / shared memory. Roofline model. Async copy (TMA).

The lower rows are practice. Memory-bound tuning. Metrics. Ops — Nsight compute + roofline.

GPU HBM — bandwidth + banks + coalescing + roofline + memory-bound tuningthe memory tier that decides throughputHBM stacks80GB / 141GB per GPUBandwidth3.35 TB/s H100Memory banksconflict avoidanceCoalescingconsecutive accessL2 cache50MB sharedL1 / shared memoryper-SM 256KBRoofline modelcompute vs memory boundAsync copy (TMA)hide latencyMemory-bound tuninglayout + tilingMetricsachieved BW + occupancyOps — Nsight compute + kernel profiling + rooflinehitreusedecidepipelinelayoutwatchwatchoperateoperate
GPU HBM tier + caches + roofline.
Advertisement

End-to-end flow

End-to-end: attention kernel achieves 80% of HBM bandwidth by coalescing reads. Roofline shows kernel is memory-bound. TMA async copies hide latency by pipelining. L2 hit rate 60%. Total 65% of peak.