GPU Technology

100+ MW AI Clusters

Hyperscale AI training + inference sites.

RTX 4090 vs A100 Economics

4090 has better FP16 per dollar.

GPU Allocation Queues

How NVIDIA + cloud vendors allocate scarce capacity.

Amazon Inferentia2 + Inferentia3

AWS inference silicon.

Amazon Trainium 2 + UltraServers

AWS custom training silicon. Anthropic scale.

AMD MI300X

192GB HBM3. Chiplet design. ROCm software.

AMD MI325X + MI350

MI300X successors. Higher HBM capacity.

AMD ROCm Software Stack

CUDA alternative. HIP for porting. PyTorch supported.

Apple M-Series Neural Engine

On-device ML on iPhone + Mac.

ARM Neoverse for AI

ARM CPU cores with ML extensions.

AWS P5 + P4 GPU Instances

H100 (P5) + A100 (P4) instance types.

AWS SageMaker HyperPod

Persistent training cluster. Resiliency built-in.

AWS UltraCluster

P5 + Trainium clusters. Massive AI infrastructure.

Azure ML Compute

Managed training compute in Azure ML.

Azure ND-Series GPU VMs

NDv5 (H100) + NDv6 (H200) instances.

Bandwidth per GPU (Bisection)

Effective per-GPU network BW. Critical metric.

Shared Memory Bank Conflicts

Shared mem banks. Simultaneous access = conflict.

Bare Metal vs Cloud GPU Cost Analysis

Own hardware vs cloud rental. Break-even calc.

Batching Impact on Inference Cost

Batch 32 = 10x cheaper per query than batch 1.

Blackwell (B200) Pricing 2025

$60-80k per GPU. Significant premium.

Datacenter Busbars + Power Distribution

Rack-scale kW requires new PDU design.

Cerebras WSE-3

Single silicon wafer as one chip. 900k cores.

Chilled Water Systems for AI DCs

Central cooling. Adaptation for AI.

Chinchilla Scaling Laws Applied

Balance parameters + data. 20 tokens/param optimal.

Coalesced Memory Access

Consecutive threads read consecutive addresses = one transaction.

Coherent Optics for Long-Distance

400G ZR + 800G ZR for DCI.

Cold Plate Manufacturing

Copper + microchannel design. Custom per chip.

Colossal-AI Framework

Chinese open framework. Multi-parallel strategies.

Compute + Storage Co-Design

Where storage sits relative to GPUs.

Consumer vs Pro Cards for ML

4090 vs A100. Trade-offs.

Continuous Batching for GPU LLM Inference

Requests join/leave mid-generation. 10x throughput.

CUDA Cooperative Groups

Flexible thread grouping. Beyond fixed block.

Copper vs Optical for Intra-DC

Cost + reach + power trade-offs.

CoreWeave

Fastest H100 availability. IPO 2025.

Cost per Token Trained

Amortized compute cost per token.

Crusoe Energy

Datacenters at stranded energy sources.

cuBLAS

Matmul + BLAS on GPU. Foundation of ML.

CUDA Cores + Streaming Multiprocessors (SMs)

GPU parallelism unit. Thousands of cores organized in SMs.

CUDA Events for Timing

Precise GPU timing. Sync + async.

CUDA Graphs

Capture + replay op sequence. Saves microsecond-per-op.

CUDA Grid + Block + Thread Hierarchy

3-level thread hierarchy. Configure per kernel.

CUDA Streams + Concurrency

Async ops on different streams overlap. Hide latency.

CUDA Toolkit Deep Dive

Compiler + libraries + runtime + tools.

CUDA Unified Memory

Single address space CPU+GPU. Auto-migration.

cuDNN

Fast primitives for DL. Auto-selected by frameworks.

CUTLASS

Header-only. Efficient matmul templates.

Data Curation Cost for Foundation Models

Beyond compute: data pipeline investment.

Data Parallel Training

Replicate model, split batch. Foundation of scaling.

DGX H100 System Architecture

NVIDIA's flagship 8xH100 server.

Distillation for Cost Savings

Teacher-student model compression.

Distributed Checkpointing

Save + load sharded models. Async writes.

Dragonfly Network Topology

Groups of routers. Fewer optics + longer path.

CUDA Dynamic Parallelism

Kernel launches other kernels. Recursion + adaptive.

AWS EFA

AWS's HPC network. Bypasses TCP.

Fat-Tree Network Topology

Classic HPC topology. Full bisection.

Fine-Tuning vs From-Scratch Cost

Fine-tune is 100-10000x cheaper.

FlashAttention

Fused attention. Massive speedup + memory savings.

FLOPS Budget for LLM Training

How many FLOPS to train a frontier model.

FluidStack

Aggregates GPUs from many providers.

fp4_inference_cost

FP8 Training with Transformer Engine

H100+ FP8. 2x FP16 throughput. Careful scaling.

FSDP

PyTorch's ZeRO-3 equivalent. Shards params + grads + optimizer.

GB200

Grace CPU + 2 Blackwell GPUs. NVL72 rack.

GKE with TPU

Kubernetes + TPU. Standard Vertex AI alternative.

Google Cloud TPU Pods

Multi-thousand TPU interconnected.

Google TPU v4 + v5 (v5e + v5p)

TPU generations. v5e for inference cost, v5p for peak.

Google TPU v6

Latest TPU (2024). 4.7x v5e perf.

Ada Lovelace / Blackwell architecture

Deep-dive on NVIDIA Hopper→Blackwell arc: SM redesign, FP8/FP4, memory bandwidth, NVLink generations, GB200 Grace-Blackwell.

Admission Control for LLM Serving

Reject requests to protect SLOs.

AI Gateway Overview

AI gateway: central proxy for LLM traffic.

Alignment Data

How alignment data (preferences, demos) is collected + curated.

All-Reduce Algorithms

All-reduce algorithms in NCCL.

AMD Instinct GPUs

How AMD Instinct GPUs (MI300X) compete with NVIDIA for AI workloads.

Anthropic Workbench + Claude

Anthropic direct API + Workbench.

Apigee AI Gateway

Google Apigee: enterprise API gateway with AI.

Apple Silicon

How Apple M-series (M1/M2/M3/M4) with unified memory serves local AI workloads.

GPU Architecture Overview

How GPUs achieve massive parallelism through streaming multiprocessors, warps, and thousands of cores. The architecture that powers modern AI.

ASR (Speech-to-Text) Serving

Speech-to-text serving.

Audio Encoders

Audio encoders for speech + audio LLMs.

Axolotl

Axolotl: ergonomic LLM fine-tuning.

Azure AI Foundry

Azure AI Foundry: platform for AI apps.

Azure OpenAI Service

Azure OpenAI: enterprise OpenAI on Azure.

NVIDIA B100

B100: base Blackwell GPU for datacenter.

NVIDIA B200

What the B200 brings over H100: FP4, larger memory, denser packages.

GPU Batch Inference

How to run efficient batch inference for offline workloads.

Batch Inference Deep Dive

Batch inference: offline high-throughput processing.

GPU Batching Strategies Overview

Overview of batching approaches for LLM serving.

AWS Bedrock

AWS Bedrock: managed multi-model LLM service.

Arena-Hard

Arena-Hard: automatic Arena-style benchmark.

Custom LLM Benchmarks

Building custom benchmarks for your workload.

LLM Eval Frameworks

Evaluation frameworks for LLMs.

HELM Benchmark for Serving

HELM used for LLM benchmarking (broad).

LiveBench

LiveBench: contamination-resistant LLM benchmark.

LLMPerf Benchmark

LLMPerf: LLM-specific perf benchmark.

LMSYS Chatbot Arena

LMSYS Arena: human preference benchmark.

MLPerf Benchmark

MLPerf: standard ML training + inference benchmarks.

SWE-bench for Serving Model Choice

SWE-bench: real-world SWE tasks.

vLLM Benchmark Suite

vLLM's own benchmarking utilities.

BentoML for LLM Serving

BentoML: Python framework for ML serving.

Bricktree / Traefik AI

Emerging AI gateway options.

GPU Capacity Planning

Planning GPU capacity for training + serving.

GPU Datacenter Carbon Footprint

Measuring + reducing carbon of GPU DCs.

GPU Carbon Footprint

How to measure + reduce carbon footprint of GPU workloads.

GPU Checkpointing Deep Dive

Checkpointing best practices for reliable training.

Chunked Prefill

Split long prefills into chunks.

CLIP Training on GPU

How CLIP (contrastive language-image pretraining) is trained.

GPU Cluster Bandwidth

GPU cluster bandwidth analysis.

GPU Collective Operations Overview

All-reduce, all-gather, broadcast, reduce-scatter.

Collective Communication Overlap

Overlapping computation with collectives.

GPU-Enabled Containers

How to run GPU workloads in Docker containers via nvidia-container-toolkit.

Continuous Batching

Continuous batching: token-level scheduling.

GPU Datacenter Cooling Overview

Cooling GPU datacenters: air vs liquid.

Core ML for LLM

Apple Core ML for iOS + macOS LLM.

GPU Cost Optimization

How to reduce GPU costs: rightsizing, spot, reserved, MIG, off-peak scheduling.

GPU Cost Optimization for Serving

How to reduce GPU serving costs via batching, quantization, model choice, routing.

CUDA Graphs

CUDA graphs: capture + replay for low overhead.

GPU CUDA Programming

The CUDA programming model: how kernels launch threads organized into blocks and grids, and the __global__/__device__ function distinction.

CUDA streams and graphs -- overlap and launch-overhead elimination

Deep-dive on CUDA streams and graphs: streams as ordered work queues enabling concurrency and copy-compute overlap, events for cross-stream synchroniz…

CUTLASS

How NVIDIA CUTLASS provides composable templates for high-performance GEMM.

GPU Data Center Economics

How to model true cost of GPU infrastructure: capital + power + cooling + ops.

GPU Data Pipeline

How to build data pipelines that keep GPUs fed.

GPU Datacenter Deployment

The physical realities of GPU datacenters: 700W+ per GPU, liquid cooling, high-density networking, and the constraints that shape modern AI infrastruc…

GPU Datacenter Power Requirements

Powering GPU datacenters: MW scale.

GPU DataLoader Bottleneck

How to diagnose + fix data loading bottlenecks.

NVIDIA DCGM

How DCGM provides monitoring, diagnostics, and management for GPU clusters.

Decode Compute Math

How to compute decode throughput.

DeepInfra

DeepInfra: budget open-model inference.

DeepSpeed

How DeepSpeed enables training of massive models via ZeRO optimizations and pipeline parallelism.

DeepSpeed Deep Dive

DeepSpeed: Microsoft's training + inference library.

GPU Depreciation Timeline

Hardware ROI over 3-5 years.

Diffusion Model Serving

Serving diffusion image models.

Direct Liquid Cooling

Direct liquid cooling implementation details.

Disaggregated LLM Serving

Disaggregated serving: separate prefill and decode instances.

DPO Training on GPU

How DPO replaces RLHF with direct preference optimization.

Dropless MoE

Avoid dropping tokens on expert overflow.

Dynamic Batching

Dynamic batching: variable batch + max wait.

PyTorch Dynamo

Dynamo: PyTorch's Python-level graph capture.

Edge Inference for LLM

Running LLMs on edge devices.

Edge NPUs

How edge NPUs (Qualcomm Hexagon, Apple Neural Engine, ARM Ethos) enable on-device AI.

Embedding Model Serving

Embedding model serving.

GPU ECC + Reliability

Error Correcting Code on datacenter GPUs.

GPU Experiment Tracking

How to track ML experiments: MLflow, W&B, Neptune.

Expert Choice Routing

Experts choose tokens instead of tokens choosing experts.

Expert Parallelism

How expert parallelism distributes MoE experts across GPUs for efficient MoE training.

GPU Serving: Systems View

Systems view of GPU serving.

DoRA Fine-Tuning

DoRA: weight-decomposed LoRA fine-tuning.

DPO Fine-Tuning

DPO: direct preference optimization.

Full Fine-Tuning on GPU

Full-parameter fine-tuning: update all weights.

GRPO Fine-Tuning

GRPO: group-relative PPO used by DeepSeek R1.

LoRA Fine-Tuning

LoRA: rank-decomposed adapter fine-tuning.

Fine-Tuning Ops on GPU

Operational practices for LLM fine-tuning.

ORPO Fine-Tuning

ORPO: SFT + preference in one stage.

PPO Fine-Tuning for LLMs

PPO: original RLHF optimizer.

QLoRA Fine-Tuning

QLoRA: 4-bit quantized base + LoRA adapters.

SFT

SFT: instruction tuning on labeled demonstrations.

GPU Fine-Tuning Costs

How to estimate GPU fine-tuning costs based on model, data, method.

LLM Fine-Tuning Optimization

Techniques for efficient LLM fine-tuning.

Fireworks AI

Fireworks AI: fast open-model inference.

FlashInfer

How FlashInfer provides fast attention kernels specifically for LLM inference.

FLUX

FLUX by Black Forest Labs: modern image generation architecture.

PyTorch FSDP

How PyTorch FSDP shards model states across GPUs, enabling large model training natively.

NVIDIA GB200

GB200: Grace CPU + Blackwell GPU superchip.

NVIDIA GH200

GH200: Grace CPU + Hopper GPU superchip.

GPU Sharing Strategies Overview

Overview of GPU sharing approaches.

GPUDirect

How NVIDIA GPUDirect enables direct GPU access to storage and network without CPU involvement.

GPUDirect Deep Dive

GPUDirect for RDMA + storage.

GRPO

GRPO: PPO variant used by DeepSeek for R1 reasoning.

NVIDIA H100

What makes NVIDIA H100 the workhorse of modern AI training and inference.

H100 SM architecture

Deep-dive on H100 SM: warp scheduler, CUDA cores, 4th gen tensor cores, shared memory, TMA async copy, cluster launch, FP8/INT8.

NVIDIA H200

H200: H100 successor with more HBM3e memory.

GPU Hardware Faults

How to detect + recover from GPU hardware faults (ECC, hangs, thermal).

GPU HBM architecture

Deep-dive on GPU HBM tier: bandwidth, banks, coalescing, L2/L1 caches, roofline model, async copy (TMA), memory-bound tuning.

HCCL

HCCL: Huawei Ascend collective communication.

Hugging Face Ecosystem

Hugging Face: model hub + libraries.

GPU Hyperparameter Tuning

How to tune hyperparameters efficiently on GPUs.

Image Classification Serving

Image classification serving.

Immersion Cooling for GPU

Immersion cooling: single-phase and two-phase.

GPU Incremental Training

How to add training data without full model retrain.

PyTorch Inductor

Inductor: PyTorch's default torch.compile backend.

GPU Inference Latency

How LLM inference latency decomposes: time-to-first-token vs time-per-output-token.

LLM Inference Optimization Overview

Techniques for LLM inference optimization.

InfiniBand + NVLink architecture

Deep-dive on GPU cluster fabric: NVLink intra-node, InfiniBand inter-node, GPUDirect RDMA, NCCL + SHARP collectives, topology-aware placement.

InfiniBand Deep Dive

InfiniBand fabric deep dive.

GPU Infrastructure Planning

How to plan GPU cluster: node config, networking, storage, cooling.

GPU Infrastructure Cost

Total cost of ownership analysis for GPU infrastructure.

Intel Gaudi + GPU Max

How Intel Gaudi 3 and GPU Max compete for AI training and inference.

Iterative DPO

Iterative DPO: multiple rounds of preference training.

ITL

Compute + optimize ITL (per-token latency).

GPU kernel fusion -- fewer kernels, less memory traffic

Deep-dive on GPU kernel fusion: the memory-bound HBM-traffic problem, fusing operations to keep intermediates on-chip (registers/SRAM), elementwise/ep…

GPU Kernel Launch

How kernels are launched: grid/block configuration, occupancy calculation, and CUDA streams for concurrency.

Kong AI Gateway

Kong: API gateway with AI features.

KServe for LLM

KServe: Kubernetes-native model serving.

KTO Training

KTO: prospect-theory-inspired preference optimization.

Kubeflow

Kubeflow: K8s-native ML platform.

GPUs on Kubernetes

How Kubernetes schedules GPU workloads via device plugins, and how to enable sharing.

Kueue

Kueue: K8s-native job queue for batch + AI workloads.

KV Cache Compression

Compress KV cache for memory savings.

KV Cache Disk Offload

Offload KV cache to NVMe / SSD.

KV Cache Math

How to size KV cache memory.

KV Cache Offload to CPU

Offload KV cache to CPU RAM.

KV Cache Optimization

How to reduce KV cache memory: GQA/MQA architectures, KV quantization, compression.

KV Cache Sizing for Deployments

Right-sizing KV cache pool for LLM serving.

Liquid Cooling for GPU Datacenters

Liquid cooling approaches for GPU DCs.

LiteLLM

LiteLLM: universal LLM SDK + proxy.

llama.cpp Deep Dive

llama.cpp: C++ LLM inference on CPU + GPU.

LLM A/B Testing

A/B test LLM models in production.

LLM Active Learning

Active learning: prioritize high-value labels.

LLM Annotation Tools

Annotation tools for LLM data.

LLM Audit Logging

Audit logging for LLM services.

LLM Bill of Materials

LLM-BoM: catalog LLM stack components.

LLM Blue-Green Deployment

Blue-green: full switch with instant rollback.

LLM Bottleneck Analysis

Systematic bottleneck analysis for LLMs.

LLM Burst Testing

Burst testing LLM services.

LLM Canary Deployment

Canary: send small % to new LLM.

LLM Capacity Analysis

Capacity analysis for LLM serving.

LLM Capacity Forecasting

Forecast LLM capacity needs.

LLM Capacity Testing

Capacity testing for LLM planning.

LLM Change Management

Change management for LLM systems.

LLM Chaos Testing

Chaos testing for LLM services.

LLM Chargeback

Internal chargeback for LLM costs.

LLM CI/CD Pipeline

CI/CD for LLM applications.

LLM Committed Use Discounts

Committed use discounts for LLM.

LLM Content Moderation

Content moderation for LLM outputs.

LLM Cost Analysis

Analyze LLM app cost drivers.

LLM Cost Attribution

Attribute LLM costs to features + users.

LLM Cost Optimization

Reduce LLM costs.

LLM Data Curation Pipelines

Pipelines for curating LLM training data.

LLM Data Exfiltration Defense

Preventing data exfil via LLM.

LLM Data Flywheel

Continuous improvement via user feedback.

LLM Deployment Pattern Overview

Deployment patterns for LLM serving.

LLM Distillation Data

Distillation data from teacher LLMs.

LLM Disaster Recovery Drill

DR drills for LLM services.

LLM Endurance Testing

Endurance testing LLM services.

LLM Error Budgets

Error budgets for LLM services.

LLM Failover Testing

Testing LLM failover behavior.

LLM FinOps

FinOps discipline for LLM apps.

Flame Graphs for LLMs

Using flame graphs to visualize LLM hotspots.

LLM Gameday

Gameday exercises for LLM services.

GitOps for LLM Deployments

GitOps: Git as source of truth for LLM deployments.

LLM Guardrails

Guardrails for LLM production.

LLM System Health Scoring

Composite health score for LLM systems.

Helm for LLM Deployments

Helm charts for LLM engines + apps.

LLM Human-in-the-Loop

HITL: human review + intervention in LLM systems.

HITL Review Patterns

Patterns for HITL reviews.

LLM Incident Communication Channels

Communication channels during incidents.

LLM Incident Response

Incident response for LLM services.

LLM Jailbreak Defense

Defending LLMs from jailbreaks.

LLM KPI Dashboards

Dashboards for LLM KPIs.

LLM Individual Contributor KPIs

KPIs for ICs.

LLM Leadership KPIs

KPIs for leadership.

LLM North Star Metric

Define North Star metric for LLM.

Kustomize for LLM Deployments

Kustomize: overlay-based K8s customization.

LLM Data Labeling

Labeling data for LLM training + evaluation.

LLM Load Testing

Load-testing LLM serving stacks.

LLM Serving Math 101

Overview of LLM serving math + trade-offs.

LLM Memory Profiling

Profiling GPU memory for LLMs.

LLM Metrics Deep Dive

SLI + KPI metrics for LLM serving.

NVIDIA Nsight for LLMs

Using Nsight Systems and Compute for LLMs.

LLM On-Call Playbook

Playbook for on-call engineers.

LLM Output Validation

Validating LLM outputs.

LLM Ownership Matrix

RACI for LLM system components.

LLM On-Call Rotation

On-call rotation for LLM services.

LLM Penetration Testing

Pen testing for LLM services.

LLM Performance Analysis

How to analyze LLM inference performance.

LLM Performance Profiling

Profiling tools and workflow for LLMs.

LLM PII Detection

PII detection in LLM inputs/outputs.

LLM Pricing Models

Understanding LLM provider pricing.

LLM Health Probes

Health + readiness + liveness probes.

LLM Prompt Injection Defense

Defending LLMs from prompt injection.

LLM Python Frameworks

Python frameworks for LLM apps.

PyTorch Profiler for LLMs

Using PyTorch profiler for LLMs.

LLM Ranking + Comparison UI

UI for pairwise ranking + comparison of LLM outputs.

LLM Performance Regression Analysis

Detecting and analyzing LLM perf regressions.

LLM Reserved Capacity

Reserve LLM capacity for discounts.

LLM Rollback Strategies

Rollback: revert LLM to previous version.

LLM Routing Strategies

Route requests to models based on task / cost / SLO.

LLM Runbook Index

Central runbook index for LLM.

GPU Serving Architecture for LLMs: The Full Stack

A 2500-word walkthrough of a modern GPU serving stack for LLMs: ingress, mesh, router, engine, autoscaler, pod, fabric, object store, and observabilit…

LLM Serving Stacks

The main LLM serving stacks and their trade-offs.

LLM Shadow Deployment

Shadow deploy new LLM alongside existing.

LLM SLO Burn Rate Alerts

Multi-window burn rate for LLM SLOs.

LLM Soak Testing

Soak testing LLM services.

LLM Spend Alerts

Alert on LLM cost anomalies.

Full LLM Stack Overview

End-to-end LLM stack: hardware to application.

Python LLM Stack Overview

Python ecosystem for LLM development.

LLM Status Page

Public status page for LLM service.

LLM Stress Testing

Stress-testing LLM serving beyond limits.

Synthetic Data Generation for LLM

Generate training data with LLMs.

LLM Synthetic Monitoring

Synthetic monitoring for LLM.

LLM TCO

Total cost of ownership for LLM apps.

LLM Trace Analysis

Analyzing LLM traces for bottlenecks.

LLM Uptime Calculation

Compute + report LLM uptime.

LMDeploy

LMDeploy: OpenMMLab LLM serving stack.

Lookahead Decoding

How Lookahead Decoding speeds LLM generation without separate draft model.

LoRA Serving Architecture

How to serve many LoRA adapters efficiently on one base model.

LoRAX

LoRAX: serve many LoRA adapters efficiently.

Megatron-LM

NVIDIA Megatron: large-model training.

GPU Memory Hierarchy

How GPU memory is organized: registers, shared memory, L1/L2 cache, HBM. Bandwidth matters more than latency.

CUDA Memory Pool

Memory pool: efficient GPU memory allocation.

AMD MI300X

MI300X: AMD's H100 competitor.

AMD MI325X

MI325X: MI300X with more memory.

NVIDIA MIG architecture

Deep-dive on Multi-Instance GPU: compute and memory slices, GPU Instance and Compute Instance hierarchy, profile geometry on A100/H100, Kubernetes GPU…

MIG Deep Dive

MIG: NVIDIA Multi-Instance GPU.

Mixed Precision Training

How mixed precision (FP16/BF16 compute, FP32 master weights) speeds training.

ML Framework Comparison

The three major ML frameworks: PyTorch dominant, JAX for research, TensorFlow legacy.

GPU ML Ops

How ML Ops for GPU workloads differs: cluster management, checkpointing, monitoring.

ML Reproducibility on GPUs

How to achieve reproducible ML training on GPUs (or accept nondeterminism).

MLflow for GPU Training Ops

MLflow for tracking + registry in GPU training.

Multimodal LLM Training

Training pipelines for multimodal LLMs.

MLX

MLX: Apple's ML framework for Apple silicon.

MoE All-to-All Communication

All-to-all: token shuffle for MoE.

MoE Expert Parallelism Deployment

Deploying MoE with expert parallelism.

Grouped GEMM for MoE

Grouped GEMM: variable-size expert matmuls.

MoE Kernels

Custom kernels for MoE efficiency.

MoE Load Balancing

Balance token routing across experts.

MoE Routing Math

Router: pick top-K experts per token.

MoE Serving Overview

Serving Mixture-of-Experts models.

Mixture-of-Experts Serving Architecture in Depth

A 2500-word walkthrough of MoE serving architecture: router, dispatch, expert parallelism, combine, capacity, load balancing, and runtimes.

MSCCL

MSCCL: custom collective algorithms via XML.

Multi-GPU Communication

How multi-GPU systems communicate via NVLink and NVSwitch, and how NCCL implements collective operations.

Multi-Instance GPU Deployment

Deploying multi-instance workloads on shared GPUs.

Multi-LoRA Serving

How multi-LoRA serving lets one base model serve many fine-tuned personalities.

Multi-Provider LLM Strategy

Use multiple LLM providers.

Multi-Stream Execution

Using multiple CUDA streams for concurrency.

gpu_multi_tenant

Multi-Turn KV Cache

How to reuse KV cache across conversation turns for efficient chat.

Multimodal Model Serving

Serving multimodal AI models.

NCCL collectives

Deep-dive on NCCL collective communication: all-reduce and collective ops, bandwidth-optimal ring and latency-optimal tree algorithms, NVLink/NVSwitch…

NCCL Deep Dive

NCCL collectives library deep dive.

NVIDIA Nemotron

Nemotron: NVIDIA's open model + training data.

nvidia-smi

How nvidia-smi provides quick GPU status, and the key views for debugging.

NVLink and NVSwitch architecture

Deep-dive on NVLink/NVSwitch: link anatomy and memory semantics, the non-blocking crossbar, SHARP in-switch reductions, NCCL topology mapping, link-fl…

NVLink Deep Dive

NVLink interconnect deep dive.

NVLink Switch

NVLink switch: connects GPUs at high bandwidth.

NVSwitch Deep Dive

NVIDIA NVSwitch chip details.

Object Detection Serving

Object detection model serving.

Ollama Deep Dive

Ollama: local LLM runtime + management.

Online DPO

Online DPO: preference optimization with fresh preference data.

GPU Online Learning

How to continuously update models on GPUs.

ONNX Runtime

ONNX Runtime: cross-platform inference.

OpenAI Realtime API

OpenAI Realtime API: low-latency audio + text.

OpenRouter

OpenRouter: unified LLM API + routing.

OpenVINO for LLM

OpenVINO: Intel's inference stack.

Hugging Face Optimum

Optimum library for accelerated inference.

GPU Orchestration Overview

GPU orchestration platforms overview.

ORPO

ORPO: single-stage SFT + preference optimization.

P2P KV Cache Transfer

Peer-to-peer KV cache transfer between GPUs.

GPU P2P Memory Access

GPU peer-to-peer memory access.

Page-Locked (Pinned) Host Memory

Pinned host memory for fast H2D transfers.

Paged KV cache architecture

Deep-dive on PagedAttention-style KV cache management: physical block pools and block tables, copy-on-write prefix caching, continuous batching under …

PCIe and GPU

How GPUs connect to CPU via PCIe, and why PCIe bandwidth matters for GPU workloads.

PCIe host interconnect architecture

Deep-dive on PCIe + host GPU interconnect: lanes, generations, root complex, GPU P2P, host bounce, GPUDirect Storage, NUMA.

Prefill/Decode Disaggregation Architecture in Depth

A 2500-word walkthrough of prefill/decode disaggregation: router, prefill pool, KV transfer, decode pool, continuous batching, independent SLOs, autos…

Pipeline Parallelism

How pipeline parallelism splits model layers across GPUs, with micro-batches for utilization.

GPU Datacenter Placement Strategy

Where to place GPU datacenters: power, cooling, latency, cost.

GPU Pod Network

Network design for AI pod: NVLink, InfiniBand, RoCE.

Portkey AI Gateway

Portkey: enterprise AI gateway.

PPO for LLMs

PPO implementation specifics for LLM alignment.

GPU precision architecture

Deep-dive on GPU precisions: fp32/bf16/fp16/fp8/fp4, accumulator design, mixed precision training, numerics safety, kernel matrix.

GPU Preemption Handling

Handling GPU preemption in training + serving.

Prefill vs Decode Split

Prefill and decode phases have different profiles.

Prefill Compute Math

How to compute prefill FLOPs + time.

Prefix Caching

How prefix caching reuses KV cache for shared prompt prefixes, cutting compute.

GPU Priority Queue Scheduling

Priority queue for LLM request scheduling.

GPU Profiling

How to profile GPU workloads: Nsight Systems for timeline, Nsight Compute for kernel-level detail.

Prompt Caching

Provider prompt caching: OpenAI, Anthropic, Google.

LLM Provider Failover

Fail over between LLM providers.

PUE

PUE metric for datacenter efficiency.

PyTorch Lightning

Lightning: high-level PyTorch training.

QPS Math for LLM Serving

Compute queries/sec capacity.

Ray

Ray for distributed ML + GPU workloads.

Ray Serve for LLM

Ray Serve: distributed model serving.

RDMA for GPUs

How RDMA enables direct GPU-to-GPU communication across the network without CPU involvement.

GPU Regulatory Landscape

The regulatory landscape affecting GPU deployment: export controls, AI safety regulations.

Replicate

Replicate: run open models with simple API.

Response Caching for LLM APIs

Exact-match response caching for LLM.

GPU Pricing: Retail vs Cloud vs Enterprise

Different pricing tiers explained.

Reward Model Training

How reward models are trained from preference data.

Ring Attention

How ring attention enables million-token context by distributing sequence across GPUs.

RLHF Pipeline on GPU

Full RLHF pipeline: SFT → RM → PPO.

RoCE (RDMA over Ethernet)

RDMA over Converged Ethernet deep dive.

RoCE v2

RoCE v2: RDMA over Ethernet as InfiniBand alternative.

Run:ai

Run:ai commercial GPU orchestration.

GPU Scheduling

How multiple workloads share a GPU: time-slicing, MPS (Multi-Process Service), and MIG (Multi-Instance GPU).

GPU scheduling architecture

How to combine CUDA streams, priorities, MPS, and MIG into a scheduling plane that offers SLO tiers and predictable multi-tenant GPU sharing.

GPU Scheduling With Slurm

How Slurm schedules GPU jobs on HPC clusters via GRES (Generic RESources).

Segmentation Model Serving

Image segmentation model serving.

GPU Selection Guide

How to choose GPUs based on workload, budget, availability.

Semantic Cache for LLM

Semantic cache: match by meaning, not exact match.

Active-Active LLM Serving

Active-active multi-region.

Active-Passive LLM Serving

Active-passive multi-region.

Anycast for LLM Serving

Anycast routing for LLM APIs.

GPU Serving Capacity Estimation

Estimate LLM serving capacity per GPU.

DNS Failover for LLM

DNS-based failover for LLM.

LLM Serving DR

Disaster recovery for LLM serving.

Edge PoP LLM Serving

Edge Points of Presence for LLM.

Geographic Routing for LLM

Route LLM users to nearest region.

Multi-Region LLM Serving

Deploy LLM across regions.

LLM Serving Reliability

Reliability engineering for LLM serving.

LLM Serving SLOs

SLOs for LLM serving.

SGLang

SGLang: structured programming + fast LLM inference.

Shared Prefix KV Caching

Share KV cache for common prefixes across requests.

SLO-Aware Scheduling

Schedule LLM requests to meet SLOs.

S-LoRA

S-LoRA paper: multi-LoRA scheduling.

Slurm

Slurm for GPU + HPC workloads.

Speculative Decoding Architecture in Depth

A 2500-word walkthrough of speculative decoding: draft/target models, generate + verify, accept + reject sampling, correctness proof, acceptance rate,…

Speculative Batching

Batch multiple speculative decoding streams.

Speculative Decoding Deep Dive

How speculative decoding uses draft models to speed up LLM generation.

GPU Spot Instances

Using GPU spot / preemptible instances.

Stable Diffusion Training + Inference

Training + inference of Stable Diffusion image models.

Static Batching

Static batching: fixed batch size per iteration.

CUDA Stream Capture

Stream capture for building CUDA graphs.

StreamingLLM

How StreamingLLM enables efficient inference on very long conversations.

Streaming LLM Serving

Streaming LLM outputs to clients.

GPU Supply Chain

How GPU supply constraints affect AI infrastructure planning.

Switch Transformer Architecture

Google Switch Transformer: foundational MoE.

GPU Synthetic Data

How synthetic data generated by LLMs is used for training.

GPU Tail Latency Management

Reducing tail latency for LLM serving.

Tensor core architecture

Deep-dive on GPU tensor cores: WMMA/MMA, tile size, accumulator dtype, input dtype, sparsity, multi-buffering, kernel selection.

GPU Tensor Cores

How tensor cores accelerate matrix multiplication for deep learning, precision modes (FP16, BF16, FP8, INT8), and when they kick in.

Tensor Parallelism

How tensor parallelism splits weight matrices across GPUs, enabling per-layer parallelization.

TensorRT-LLM

How NVIDIA TensorRT-LLM provides highly-tuned LLM inference on NVIDIA GPUs.

TensorRT-LLM Deep Dive

TensorRT-LLM: NVIDIA's optimized LLM inference.

TensorFlow Lite for LLM

TensorFlow Lite: mobile + edge inference.

GPU Thermal Management

How thermal management affects GPU performance + reliability.

GPU Throughput Math for LLM

Compute tokens/sec throughput per GPU.

GPU Time Slicing

Software GPU time slicing.

Together AI

Together AI: LLM inference platform.

Topology-Aware GPU Scheduling

Topology-aware placement + collectives.

GPU Topology Awareness

Topology-aware GPU communication.

torch.compile

How torch.compile accelerates PyTorch code via JIT compilation to optimized GPU kernels.

Google TPU

How Google TPU compares to GPU for training and inference on Google Cloud.

Google TPU v5

Google TPU v5e + v5p for training + inference.

Google TPU v6 Trillium

TPU v6 Trillium: latest Google TPU.

LLM Training Checkpointing

Checkpointing during LLM training.

LLM Training Cluster Design

Design of GPU training clusters.

LLM Training Cost Optimization

Cost optimization for LLM training.

LLM Training Data Pipeline

Data pipeline for LLM training.

LLM Training Debugging

Debugging LLM training.

LLM Training Experiment Tracking

Tracking training experiments.

LLM Training Health Checks

Health checks during LLM training.

LLM Training Hyperparameter Search

Hyperparameter search for LLM training.

LLM Training Orchestration

Orchestrating LLM training jobs.

LLM Training Reproducibility

Reproducibility in LLM training.

AWS Trainium

AWS Trainium: purpose-built AI training chip.

Triton Inference Server for LLM

NVIDIA Triton Inference Server for LLM.

Triton

How OpenAI's Triton lets Python developers write GPU kernels without full CUDA C++.

Triton Kernels for LLM

Writing custom Triton kernels for LLM performance.

TRL Deep Dive

TRL: HF Transformer Reinforcement Learning.

TTFT

Compute + optimize TTFT.

TTS (Text-to-Speech) Serving

Text-to-speech serving.

UCX

UCX: unified transport abstraction for GPU comm.

Unsloth

Unsloth: memory-efficient LLM FT.

Vertex AI Gemini

Google Vertex AI: Gemini + other models.

NVIDIA vGPU

vGPU: virtualization + graphics + compute.

Video Diffusion Models

Video generation via diffusion: Sora, Kling, Runway, open models.

Video Generation Serving

Video generation model serving.

Video LLM

Video-capable LLMs: architecture and training approaches.

Vision Encoders

Vision encoders used in modern MLLMs.

Vision Transformer Serving

Serving vision transformers.

vLLM Deep Dive

vLLM: PagedAttention + continuous batching.

vLLM on GPU

How vLLM uses PagedAttention and continuous batching for state-of-art LLM serving throughput.

Vision-Language Model Architectures

Common VLM architectures: LLaVA, Flamingo, IDEFICS, Molmo.

WebAssembly for LLM Inference

WASM: portable inference beyond browser.

WebGPU for LLM

WebGPU: browser-based GPU compute for LLM.

GPU Workload Types

How different GPU workloads (training, inference, rendering, HPC) have different requirements.

ZeRO Optimizer

How ZeRO progressively shards optimizer state, gradients, and weights across GPUs to reduce per-GPU memory.

GPUDirect RDMA

Direct GPU-to-network. Skip host.

GPUDirect Storage

Direct file → GPU memory. Skip host.

Gradient Clipping

Cap gradient norm. Prevent explosion.

Gradient Accumulation

Accumulate grads over micro-batches before step.

Gradient Checkpointing (Activation Recomputation)

Don't store all activations. Recompute in backward.

Graphcore IPU

Bulk Synchronous Parallel. Startup that struggled.

Grid Connection Challenges for AI

Multi-hundred MW loads stress power grids.

Groq LPU

Custom silicon for LLM inference. Deterministic + fast.

H100 Availability History (2023-2025)

From acute shortage to abundance.

H100-Hours per Foundation Model

Compute cost per model in H100-hours.

H100 System Network Breakdown

How H100 systems talk internally + externally.

H100 Pricing History

$25k-40k. Cloud spot dropped 50%.

H200 vs H100 Economics

H200 pricier but more memory bandwidth.

HBM3 + HBM3e Memory

High Bandwidth Memory. Stacked DRAM. Datacenter GPUs.

HGX Platform

NVIDIA reference design for OEMs.

Immersion Cooling

Submerge servers in dielectric fluid. Extreme density.

Inference Cost per Query

Cents to dollars per LLM call.

InfiniBand for AI Clusters

Low-latency + high-BW network. Standard for training.

InfiniBand NDR (400G) + XDR (800G)

Latest InfiniBand generations. Bandwidth per port.

Intel Gaudi 2 + Gaudi 3

Habana Labs (Intel). BF16 focus. Competitive perf.

Kubernetes GPU Operator

NVIDIA device plugin + drivers on K8s.

GPU Kernel Fusion

Combine kernels to avoid HBM roundtrip.

Kubeflow

End-to-end ML platform on K8s.

Kueue

Job queue for K8s. Fair sharing + priorities.

KV Cache Reuse + Prefix Caching

Amortize prefill across requests. Massive cost savings.

Lambda Labs

Reserved + on-demand GPUs. Developer-friendly.

Latitude.sh

Bare metal cloud. Reserved + on-demand GPUs.

Direct-to-Chip Liquid Cooling

Cold plates on chips. Standard for H100+ SXM.

Megatron-LM Framework

NVIDIA's LLM training framework. Tensor + pipeline parallel.

Meta AI Research SuperCluster (RSC)

Meta's internal AI cluster.

Meta MTIA

Meta's inference accelerator. v1 + v2 deployed.

Azure AI Supercomputer for OpenAI

Microsoft's massive OpenAI infrastructure.

MIG

Partition A100/H100 into isolated slices.

MIG for Multi-Tenant GPU Sharing

Hardware isolation. Multi-tenant inference.

Mixed Precision Training (BF16 + FP32)

Storage in BF16, master weights in FP32. Standard.

Modal Labs

Python-native serverless GPU.

Model Lifetime Utility

Amortize training cost over inference lifetime.

Model Parallel Training

Split model across GPUs. Different layers on different GPUs.

Modular Datacenters for AI

Prefab modules. Faster deployment.

Small Modular Reactors (SMR) for AI DCs

Nuclear power for AI. Emerging trend.

MoE Model Inference Cost

Mixture of Experts activates subset. Cheaper.

Multi-GPU Peer-to-Peer Access

GPUs directly access each other's memory.

Multi-Year GPU Capacity Commitments

How hyperscalers lock capacity years ahead.

All-Reduce

Sum gradients across GPUs. Foundation of data parallelism.

NCCL

All-reduce + broadcast + all-gather across GPUs.

NCCL Topology-Aware Communication

NCCL picks best path per topology.

Nebius

Yandex spinoff. GPU cloud + AI Studio.

NCCL + NIXL

NCCL evolution + NIXL for LLM disaggregation.

nsight_compute

NVIDIA Nsight Systems

System-level profiling. See CPU + GPU + memory.

NVIDIA Ada Lovelace (RTX 4090 + L40S)

Consumer + workstation Ada. FP8 support.

NVIDIA Ampere A100

Previous gen. Still workhorse for many.

NVIDIA Blackwell B100 + B200

Next-gen (2024-2025). FP4 + 2 dies bridged.

NVIDIA GH200

Grace CPU + Hopper GPU. NVLink-C2C coherent.

NVIDIA H200

141GB HBM3e. Same compute. Bigger + faster memory.

NVIDIA Hopper H100 Architecture

80B params, 700W SXM. 4th-gen tensor cores + FP8.

NVIDIA Rubin (Next Gen)

Announced roadmap: 2026 successor to Blackwell.

NVIDIA Spectrum-X

NVIDIA's Ethernet answer to IB.

GB200 NVL72 Rack

72 GPUs unified via NVLink Switch.

NVLink

Direct high-bandwidth GPU communication. Multi-GPU foundation.

NVSwitch

Switch fabric for NVLink. All-to-all connectivity.

GPU Occupancy Calculator

Threads per SM ratio. Balance parallelism + resources.

OCI GPU SuperCluster

H100/H200 clusters. Highest scale non-hyperscaler.

Optical Transceivers for AI Networking

400G + 800G optics. Cost + reach.

Optimizer State Offloading to CPU/NVMe

Move optimizer state off GPU. Trade PCIe bandwidth.

PagedAttention

OS-inspired paging for KV cache. vLLM foundation.

PCIe Gen4 + Gen5 for GPUs

Alternative to NVLink. Host-to-GPU interconnect.

Pipeline Parallelism

Layer groups across GPUs. Micro-batch pipelining.

Power Backup for AI Datacenters

UPS + diesel + battery. Sub-second failover.

Power Density in AI Datacenters

kW per rack. Traditional 5-10, AI 30-120.

PUE for AI Datacenters

Power Usage Effectiveness. AI dc target <1.2.

Qualcomm Cloud AI 100

Datacenter inference chip. Meta partnership.

Rail-Aligned Topology for AI Clusters

Multiple network rails. GPUs use own rail.

Ray on GPU + Distributed Training

Ray Train + Serve + Data. Popular ML orchestration.

Rear-Door Heat Exchangers

Liquid-cooled rack doors. Retrofit-friendly.

GPU Register Pressure

Registers per thread. Limits threads per SM.

Reserved GPU Capacity vs On-Demand

Multi-year commitments for capacity + discount.

Ring All-Reduce Algorithm

Bandwidth-optimal collective. Baidu 2017.

RoCE v2

RDMA on standard Ethernet. Cheaper alternative.

runpod

SageMaker HyperPod on GCP Comparable

GCP's answer via managed Slurm.

SambaNova RDU

Reconfigurable Dataflow Unit. Alternative arch.

Second-Hand GPU Market

Enterprise churn feeds resale.

SGLang

Fast + structured LLM serving. RadixAttention.

SHARP

Aggregate on switch. Faster all-reduce.

SIMT Execution Model

Single Instruction Multiple Thread. Different from SIMD.

Slurm Cluster Software

HPC job scheduler. Foundation of ML training clusters.

Speculative Decoding on GPU

Draft model + target model. 2-3x throughput.

DGX SuperPod Topology

Multi-DGX cluster reference architecture.

SXM vs PCIe Form Factor

Datacenter GPU form factors. SXM = NVLink native.

Tensor Cores

Dedicated matmul units. Foundation of modern ML perf.

Tensor Parallelism (Megatron-style)

Split individual layers across GPUs. Column + row parallel.

TensorRT

Compile models for max GPU inference perf.

TensorRT-LLM Deep Dive

TensorRT specialized for LLMs. NVIDIA's LLM serving.

tenstorrent_wormhole

Together AI

GPU cloud + LLM API service.

torch.compile

PyTorch 2.0's compilation. Torch Dynamo + Inductor.

Foundation Model Project Cost Breakdown

Compute + data + team + evals + safety.

Training Cost Trajectory 2024-2027

Where costs are going.

Training on Thousands of GPUs

Practical challenges. Failures, comms, checkpointing.

Tree All-Reduce

Latency-optimal. Small messages + many workers.

NVIDIA Triton Inference Server

Model serving. Any framework. Ensemble models.

OpenAI Triton Language

Python DSL for GPU kernels. Simpler than CUDA.

Ultra Ethernet Consortium

Open alternative to InfiniBand.

Vast.ai

Peer-to-peer GPU rental.

vLLM

PagedAttention + continuous batching. Community standard.

Voltage Park

H100 for research + open source.

Warp Scheduling on GPU

32 threads = 1 warp. SIMT execution model.

Waste Heat Reuse from Datacenters

District heating + agriculture + desalination.

Water Usage in AI Datacenters

Evaporative + chilled water consumption.

xAI Colossus

100k+ H100 built in 122 days.

XLA

TensorFlow + JAX compiler. Fuses ops.

Zero-Copy Memory (Pinned Host)

GPU accesses host memory directly over PCIe.

DeepSpeed ZeRO Stages

Progressive sharding: optimizer, grads, params.