Why it matters
The physical constraints of GPU deployment increasingly bound how fast AI can scale. Understanding them explains why cloud GPU capacity is so tight and why new datacenter builds take years.
The architecture
Power: 700W per H100 × 8 GPUs per server = 5.6 kW for GPUs alone; add CPU, memory, NIC = 6.5-8 kW per server. Rack of 8 servers = 50-65 kW. Traditional datacenter racks handle 10-15 kW.
Cooling: air cooling limits reached around 40 kW/rack. Beyond that, direct-to-chip liquid cooling or immersion cooling required. Liquid cooling can extract 100+ kW/rack.
How it works end to end
Networking: training clusters need high-bandwidth low-latency fabric. InfiniBand NDR (400 Gb/s) or Spectrum-X Ethernet (400 Gb/s+) with RDMA. Every server needs 8+ 400G links for full bisection.
Topology: fat-tree or dragonfly designs give predictable bandwidth. Rail-optimized designs pair specific GPUs across nodes to accelerate certain collectives.
Physical density: modern GPU racks are 42U or 45U tall, filled with 8-GPU servers. Racks weight 1000+ kg fully loaded.