All 383 articles, sorted alphabetically
The Autoregressive Generation Loop
How an LLM produces text, one token at a time.
Read article →Backpropagation Through Transformers
How gradients flow back through the network — chain rule applied.
Read article →Beam Search
Sometimes greedy isn't enough.
Read article →The Complete Transformer Block
All the pieces assembled with pseudocode.
Read article →CPU Cache Hierarchy and Transformer Inference
Why memory bandwidth is the SLM CPU inference bottleneck.
Read article →CPU Inference Pipelines
The software stacks that make SLM CPU inference fast.
Read article →CPU Matmul Kernels and BLAS
Why your transformer is 10× faster than a naive Python loop.
Read article →CPU Memory Budget for SLM Training
Counting every byte: weights, gradients, optimizer, activations.
Read article →Cross-Entropy Loss for Next-Token Prediction
What loss to minimize and why the gradient is so clean.
Read article →Dataloader and Tokenization Pipeline
Streaming data efficiently to the training loop.
Read article →The Geometry of Dot Products
How attention measures similarity in embedding space.
Read article →End-to-End CPU SLM Recipe
From scratch: train and serve a small model on a workstation.
Read article →Feed-Forward Network
Two-thirds of model params live here. Worth understanding.
Read article →Fine-Tuning Math
How to adapt without retraining the whole model.
Read article →FlashAttention
Same math, very different memory pattern.
Read article →Gradient Accumulation and Microbatches
Effective large batches on small memory.
Read article →Gradient Clipping and Training Stability
Stopping spikes from blowing up training.
Read article →KV Cache
Why and how to cache keys and values.
Read article →KV Cache Quantization — int8 and int4
Compress the biggest memory hog at inference.
Read article →LayerNorm
How LayerNorm stabilizes activations.
Read article →Learning Rate Schedules
Why LR changes during training and the canonical curves.
Read article →Linear Algebra for Transformers
Vectors, matrices, and operations you actually need.
Read article →Long Context Strategies
How models handle 128K, 1M tokens.
Read article →Diagnosing Loss Curves
What healthy vs unhealthy training looks like.
Read article →Mixed Precision Training
Half-precision for speed; full precision for stability.
Read article →Mixture of Experts
Active params vs total params, and what MoE means for CPU inference.
Read article →Multi-Head Attention
Why split into heads and how the math reassembles.
Read article →Multi-Token Prediction (MTP)
Predict the next 2-4 tokens at once for free speed.
Read article →Output Projection and Logits
From final hidden state to next-token probabilities.
Read article →Perplexity and Evaluation Metrics
How to measure if your model is any good.
Read article →Positional Encoding
How transformers know which token is where.
Read article →Quantization Layouts in Memory
Block-wise scaling, packed INT4, and the layout-vs-quality trade.
Read article →Residual Connections and Gradient Flow
Why every transformer block adds the input back.
Read article →RLHF and DPO
From a pretrained model to a useful assistant.
Read article →RMSNorm
Why everyone moved away from LayerNorm.
Read article →Extending RoPE to Longer Contexts
YaRN, NTK-aware scaling, and LongRope.
Read article →Sampling Strategies
How to pick the next token from the model's distribution.
Read article →Scaled Dot-Product Attention
The math of the attention operation.
Read article →SGD vs Adam vs AdamW Optimizer Math
The update rules — and why Adam dominates LLM training.
Read article →SIMD Instructions for Transformer Math
AVX-512, AMX, and how they vectorize matmul.
Read article →SLM Architectures
Hyperparams of the leading sub-7B models.
Read article →Softmax Derivation
Why softmax, what it preserves, and how temperature changes it.
Read article →Speculative Decoding — The Math
Use a fast draft model; verify with the big model.
Read article →The Future of CPU SLM in 2026 and Beyond
Where this is headed.
Read article →Tied Embeddings and Parameter Sharing
Why SLMs share weights between input embedding and output projection.
Read article →3D Parallelism
Combine DP, TP, PP for scale.
Read article →Ablation Study Math
Ablation studies in LLM research.
Read article →Beyond ReLU
Comparison of activation functions in modern LLMs.
Read article →Activation Checkpointing Math
Trade compute for memory via activation recomputation.
Read article →Activation Memory Math
Calculating activation memory in transformers.
Read article →Activation Patching
Patch activations to test causal role.
Read article →Activation Quantization
Activation quantization challenges + techniques.
Read article →Activation Recomputation Math
Math of activation recomputation trade-off.
Read article →Activation Scaling
Scaling activation variance in transformers.
Read article →Adafactor
Adafactor: factored second moment for memory savings.
Read article →Adam and AdamW -- the optimizer that trains transformers
Deep-dive on Adam and AdamW: the single-learning-rate limitation, momentum (first moment, smoothed gradient), adaptive per-parameter scaling (second m…
Read article →AdamW Math Deep Dive
AdamW: Adam with decoupled weight decay.
Read article →Adapter Theory
Adapter layers: bottleneck insertions.
Read article →Adaptive RAG
Route queries to different RAG strategies.
Read article →Admission Control Math
Admission control math for LLM serving.
Read article →Agent Scaling Laws
Scaling laws for agent capability.
Read article →ALiBi
Math of ALiBi position encoding.
Read article →Alignment Tax Math
Alignment tax in RLHF.
Read article →Annoy
Annoy: tree-based ANN.
Read article →MongoDB Atlas Vector Search
MongoDB Atlas: vector search.
Read article →Attention compute and memory -- the quadratic cost of context
Deep-dive on attention compute and memory: the O(N^2) QK^T attention matrix, quadratic compute and memory, the attention-vs-FFN FLOPs breakdown, compu…
Read article →Attention Math
The mathematical foundation of attention: query-key similarity produces weights over values, which get summed into output.
Read article →Attention variants architecture
Architecture deep-dive on attention variants: the control-plane and eval harness that let you pick per layer against a serving budget.
Read article →AWQ Math
AWQ: Activation-aware Weight Quantization.
Read article →Backpressure Math
Backpressure math for LLM serving.
Read article →Backprop architecture
Deep-dive on backpropagation in transformer training: chain rule, gradient accumulation, activation checkpointing, fused kernels.
Read article →Bandwidth-Bound Operations
Identifying bandwidth-bound ops in LLM.
Read article →Batched GEMM Math
Batched matrix multiplication math.
Read article →Batch Size Math for LLM Training
How to choose LLM training batch size.
Read article →Beam Search Math
Beam search: track K best sequences.
Read article →BF16 Training
BF16 training: 8-bit exponent, 7-bit mantissa.
Read article →BGE Embeddings Math
BGE: BAAI general embeddings.
Read article →Bio Capability Evals
LLM biosecurity capability eval.
Read article →BitFit Theory
BitFit: fine-tune only biases.
Read article →bitsandbytes int8 Quantization
bitsandbytes: LLM.int8() implementation.
Read article →Bottleneck Theory of Deep Learning
Information bottleneck theory for deep nets.
Read article →Byte-Pair Encoding Deep Dive
BPE tokenization deep dive.
Read article →Blockwise Parallel Transformer
BPT: block-parallel transformer for long context.
Read article →Capability Evaluation Overview
Math + methodology of LLM capability evals.
Read article →Capability Transitions in Training
How capabilities transition during training.
Read article →Causal Tracing
Trace causal roles across layers.
Read article →Classifier-Free Guidance (CFG)
CFG: guidance without separate classifier.
Read article →Chinchilla Scaling Law
Chinchilla: compute-optimal ratio between params and data.
Read article →Chinchilla Scaling
Chinchilla scaling law math.
Read article →Chroma Math + Architecture
Chroma: developer-friendly vector DB.
Read article →Chunked Prefill
Chunked prefill deep dive.
Read article →Circuit Analysis
Reverse-engineer neural network circuits.
Read article →Classifier Guidance Math
Classifier-guided diffusion sampling.
Read article →Coding Capability Evals
LLM coding capability eval.
Read article →ColBERT Math
ColBERT: late-interaction retrieval.
Read article →Communication Math for Distributed Training
How to estimate communication volume + time.
Read article →Compute-Bound Operations
Compute-bound: dominated by FLOPs.
Read article →Compute-Optimal Training in Practice
Choosing model size + tokens under compute budget.
Read article →Compute-Optimal Training — The Math
Compute-optimal training math.
Read article →Constitutional AI Math
Constitutional AI (CAI) math.
Read article →Context Length Extension Math
Extending context length after pretraining.
Read article →Contextual RAG Math
Contextual retrieval: chunk with doc context.
Read article →Continuous Batching Math
Math of continuous batching.
Read article →Contrastive Search
Contrastive search: penalize repetition + degeneration.
Read article →Corrective RAG (CRAG)
CRAG: evaluate + correct retrieved documents.
Read article →CPU Offload Math
Math of CPU offload for LLM.
Read article →Cross-entropy loss architecture
Deep-dive on cross-entropy for language models: the maximum-likelihood reading and the softmax-minus-onehot gradient, log-sum-exp stability in half pr…
Read article →Cyber Capability Evals
LLM cybersecurity capability eval.
Read article →Dangerous Capability Evaluations
Evals for dangerous frontier capabilities.
Read article →Training Data Filtering
Filtering pretraining data.
Read article →Training Data Mixing
Data domain mixing for training.
Read article →Data Quality Math
Data quality effects on training.
Read article →Data Quality Scaling
How data quality affects scaling.
Read article →Data Scaling for LLMs
How training data scales with model + quality.
Read article →DDIM
DDIM: deterministic accelerated diffusion sampling.
Read article →DDP Math
DDP: data-parallel training math.
Read article →DDPM
DDPM: original diffusion model math.
Read article →Deception Capability Evals
Evaluate LLM deceptive capabilities.
Read article →DeepMind Scaling Law Series
DeepMind's scaling law contributions.
Read article →Dense Retrieval Math
Bi-encoder dense retrieval math.
Read article →Depth Scaling
Scaling model depth: layers.
Read article →Dictionary Learning for NN
Dictionary learning: sparse feature representation.
Read article →DoRA
DoRA: LoRA that decomposes weights into magnitude + direction.
Read article →DoRA Theory
Weight-Decomposed LoRA theoretical foundation.
Read article →Double Descent
Double descent: test error re-descends past interpolation threshold.
Read article →Double Descent Deep Dive
Double descent: test error re-descent past interpolation.
Read article →DPM-Solver Sampling
DPM-Solver: high-order diffusion ODE solver.
Read article →DPO Math
Direct Preference Optimization math.
Read article →DyT
Math of Dynamic Tanh (DyT) as attention normalization alternative.
Read article →E5 Embeddings Math
E5: Microsoft's embedding family.
Read article →EAGLE — Feature-Level Recurrence
EAGLE speculative decoding.
Read article →EAGLE in Context
Math of EAGLE speculative decoding variant.
Read article →EDM
Karras et al.: modular diffusion framework.
Read article →Effective Batch Size
Effective batch = per-device * world * grad-accum.
Read article →Elasticsearch Vector Search
Elasticsearch KNN + vector search.
Read article →Embeddings -- turning tokens into vectors
Deep-dive on transformer embeddings: token embeddings as a lookup into the embedding matrix (vocab x d_model), learned representations and embedding-s…
Read article →Emergent Capabilities Deep Dive
Emergent capabilities in LLMs.
Read article →Emergent Abilities in LLMs
How abilities emerge with scale.
Read article →Emergent Abilities Math
Emergent abilities in LLMs.
Read article →EP Math
Expert parallelism: distribute MoE experts across GPUs.
Read article →Expert Choice Routing
Expert choice routing.
Read article →Expert Parallel Math Deep
Expert parallel math for MoE.
Read article →Faiss Math
Faiss: Facebook's ANN library.
Read article →Feature Disentanglement
Separating polysemantic features.
Read article →Transformer Math: Big Picture
Big picture of transformer math.
Read article →Fine-Tuning Scaling
Fine-tuning scaling: data + compute laws.
Read article →Flash Attention Architecture
How FlashAttention rewrites attention to tile through GPU SRAM, avoiding O(N²) HBM traffic.
Read article →FlashAttention-2
FlashAttention-2 improvements over v1: better parallelism, fewer non-matmul ops.
Read article →Flash Attention Math
Flash Attention memory + compute math.
Read article →Flow Matching
Flow matching: learn ODEs instead of SDEs.
Read article →FP4 Quantization
FP4: 4-bit floating point quantization.
Read article →FP4 Training on Blackwell
FP4 training: 4-bit float on Blackwell GPUs.
Read article →FP8 Training on H100+ Hardware
FP8 training: E4M3 + E5M2 formats.
Read article →FSDP Math
FSDP: shard model across GPUs to reduce per-GPU memory.
Read article →Full FT vs PEFT Trade-offs
Full fine-tuning vs PEFT trade-offs.
Read article →Fusion RAG
Fuse multiple retrieval sources.
Read article →Game Theory for LLM Agents
Game theory applied to LLM agents.
Read article →Activation function architecture
Deep-dive on activation functions in transformers: ReLU family, GELU, SiLU, SwiGLU/GeGLU, approximations, numerics, kernel fusion.
Read article →GGUF Format Deep Dive
GGUF: llama.cpp quantization format.
Read article →Distillation Math
Math of knowledge distillation from teacher to student LLMs.
Read article →GLU FFN
Math of Gated Linear Units in feed-forward networks.
Read article →GPTQ Math
GPTQ: post-training quantization via error correction.
Read article →GQA Math
Deep math of grouped-query attention.
Read article →Gradient Accumulation — Linearity and 1/K
Gradient accumulation: virtual batch size.
Read article →Gradient Checkpointing Math
Math of gradient checkpointing.
Read article →Gradient Clipping for LLM Training
Gradient clipping strategies.
Read article →Gradient Memory Math
Gradient memory in training.
Read article →Graph RAG Math
Graph-augmented RAG.
Read article →Grokking
Grokking: model suddenly generalizes after long overfitting.
Read article →Grokking Deep Dive
Grokking: delayed generalization phenomenon.
Read article →Grokking Math
Grokking phenomenon in neural nets.
Read article →Grouped Query Attention (GQA)
How GQA and MQA reduce KV cache memory by sharing keys/values across query heads.
Read article →GTE Embeddings Math
GTE: Alibaba embeddings.
Read article →Guided Generation Math
Constrained decoding via grammar / regex.
Read article →HNSW Math
Hierarchical Navigable Small World graphs.
Read article →Hoffmann Chinchilla Paper
Chinchilla paper 2022 details.
Read article →Hybrid Architectures
Hybrid models combining attention + SSM layers.
Read article →Hybrid Parallelism Math
Combine multiple parallelism axes.
Read article →Hybrid Retrieval Math
Hybrid: fuse dense + sparse.
Read article →HyDE Math
Hypothetical Document Embeddings.
Read article →Hyena
Hyena: long convolutions replacing attention.
Read article →Hyena Hierarchy Math
Hyena hierarchy math.
Read article →IA3 Theory
IA3: multiplicative adapters.
Read article →Inference-Optimal Training
Inference-optimal training math.
Read article →Inference-Time Scaling Laws
Scaling capability via more inference compute.
Read article →In-Flight Batching
In-flight batching in TRT-LLM.
Read article →Information Bottleneck Method
IB method: trade compression vs prediction.
Read article →Initialization Scaling
Init variance scaling with model size.
Read article →Mechanistic Interpretability Overview
Math foundations of mechanistic interpretability.
Read article →Iteration-Level Scheduling
Iteration-level scheduling math.
Read article →Kaplan Scaling Laws (2020)
Original Kaplan scaling laws vs Chinchilla revisions.
Read article →Kaplan Scaling Laws Deep Dive
Kaplan 2020 scaling laws in detail.
Read article →KTO Math
Kahneman-Tversky Optimization.
Read article →KV Cache
How KV caching speeds up autoregressive generation by storing computed keys and values, and the memory cost.
Read article →KV Cache Deep Dive
Deep dive on KV cache math.
Read article →KV Cache Compression
Compressing KV cache for long context.
Read article →KV Cache Quantization
Math of quantized KV cache for long context serving.
Read article →KV Cache Quantization at Low Bits
Quantizing KV cache for memory savings.
Read article →Label smoothing architecture
Deep-dive on label smoothing: why one-hot cross-entropy causes runaway logits and overconfidence, the 1-ε soft target and its gradient, the calibratio…
Read article →Layer-wise LR Decay
Different LR per layer: layer-wise decay.
Read article →Layer Normalization
How layer norm normalizes across features per token, why it's used instead of batch norm, and pre-norm vs post-norm placement.
Read article →LayerNorm vs RMSNorm architecture
Deep-dive on LayerNorm vs RMSNorm: math, gain/bias, numerics, kernel fusion, pre-norm vs post-norm placement.
Read article →Learning Rate Math for LLM
How to choose LR for LLM training.
Read article →Learning-rate schedules
Deep-dive on learning-rate schedules for transformer training: why warmup stabilizes adaptive optimizers, cosine vs inverse-sqrt vs linear decay, how …
Read article →Linear Attention
Linear attention: kernel + feature-map approaches.
Read article →Linear Attention Math
Linear attention variants.
Read article →Lion Optimizer Math
Lion: sign-based optimizer discovered by search.
Read article →LLM Inference Architecture: The Full Pipeline in Depth
A 2500-word walkthrough of a modern LLM inference architecture: client, gateway, admission, scheduler, prefill, decode, paged KV cache, model shards, …
Read article →LLM.int8() Paper Math
LLM.int8() paper details.
Read article →Lookahead Decoding
Lookahead decoding for training-free speedup.
Read article →LoRA
Mathematics of Low-Rank Adaptation.
Read article →LoRA Theory
Low-Rank Adaptation theoretical foundation.
Read article →Loss Curves
How to read and diagnose LLM training loss curves.
Read article →Loss Scaling for fp16 Training
Loss scaling: prevent fp16 gradient underflow.
Read article →Lottery Ticket Hypothesis
LTH: dense networks contain sparse trainable subnetworks.
Read article →Lottery Ticket — Rewinding and Stability
LTH: sparse subnets train to full accuracy.
Read article →Lottery Ticket — Strong Tickets
Lottery ticket hypothesis math.
Read article →Learning Rate Schedules for LLM
LR schedules: warmup + decay strategies.
Read article →Learning Rate Transfer
Transferring LR across model sizes.
Read article →Mamba SSM Math
Mamba state space model math.
Read article →Math Capability Evals
LLM math capability eval.
Read article →Matmul Arithmetic Intensity
Arithmetic intensity of matmul.
Read article →Matryoshka Embeddings
Matryoshka Representation Learning.
Read article →Medusa Speculative Heads
Medusa multi-head speculative decoding.
Read article →Medusa
Math of Medusa multi-head speculative decoding.
Read article →LLM Memory Math
How to compute LLM training memory.
Read article →Mesa-Optimization
Mesa-optimizers: learned optimization within models.
Read article →Micro-Batch Size Selection
Choosing micro-batch size for memory + throughput.
Read article →Milvus Math + Architecture
Milvus: enterprise open-source vector DB.
Read article →Min-P Sampling
Min-p: filter tokens below min probability threshold.
Read article →Mirostat Sampling
Mirostat: control perplexity dynamically.
Read article →Mixed Precision Training
Mixed precision: fp16/bf16 compute + fp32 master weights.
Read article →Mixture of Experts Math
Mixture of experts architecture math.
Read article →MQA Math
Deep math of multi-query attention.
Read article →Multi-Agent Communication
Communication protocols between LLM agents.
Read article →Multi-Agent Coordination
Coordination mechanisms in multi-agent.
Read article →Multi-Agent Debate
Multi-agent debate for reasoning.
Read article →Multi-Agent Emergence
Emergent behavior in multi-agent systems.
Read article →Multi-Agent Negotiation
Negotiation between LLM agents.
Read article →Multi-Agent Planning
Planning in multi-agent systems.
Read article →Multi-Agent Reflection
Reflection in multi-agent systems.
Read article →Multi-Agent Shared Memory
Shared memory + blackboard patterns.
Read article →Multi-Agent System Theory
Multi-agent system theoretical foundation.
Read article →Multi-Hop RAG
Multi-hop retrieval + reasoning.
Read article →Multi-Head Attention
Why transformers use multiple attention heads instead of one big attention, and how heads specialize.
Read article →Multilingual Scaling
How languages scale together.
Read article →Multilingual Tokenization
Multilingual tokenization challenges.
Read article →Multimodal Scaling
How multimodal (vision + audio) scale.
Read article →Muon Optimizer
Muon: Newton-Schulz orthogonalization + momentum.
Read article →muP
muP: parameterize models so hyperparameters transfer across sizes.
Read article →Neural Scaling Law Theory
Theoretical foundations of neural scaling laws.
Read article →Neural Tangent Kernel
NTK: theory linking neural networks to kernel methods.
Read article →Neural Tangent Kernel
Neural tangent kernel theory.
Read article →NTK-Aware Interpolation
NTK-aware position scaling.
Read article →NVMe Offload Math
Math of NVMe offload.
Read article →Offload Math
Math of memory offload.
Read article →OpenAI Scaling Law Contributions
OpenAI's scaling law papers.
Read article →OpenSearch Vector Search
OpenSearch KNN plugin.
Read article →Optimized Product Quantization
OPQ: rotation-optimized PQ.
Read article →Optimizer State Offload Math
Math of optimizer state offload.
Read article →Optimizer State Memory Math
Optimizer state memory: Adam vs alternatives.
Read article →Optimizer Math for LLM Training
Overview of optimizer math used in LLM training.
Read article →PagedAttention architecture
Deep-dive on PagedAttention, the KV-cache management scheme behind vLLM that treats the cache like OS virtual memory. Covers why KV-cache fragmentatio…
Read article →Paged Attention Deep Dive
Paged Attention memory model.
Read article →PagedAttention
Math + memory model of PagedAttention (vLLM).
Read article →Paged KV Cache Deep Dive
Paged KV cache design.
Read article →PEFT Theory Overview
Parameter-Efficient Fine-Tuning theory.
Read article →Persuasion Capability Evals
Evaluate LLM persuasion capability.
Read article →pgvector Math + Architecture
pgvector: PostgreSQL vector extension.
Read article →Pinecone Math + Architecture
Pinecone: managed vector DB.
Read article →Pipeline Parallel Math Deep
Pipeline parallel math for LLM.
Read article →Polysemanticity
Neurons encoding multiple unrelated concepts.
Read article →Position Interpolation Math
Position Interpolation (PI) for context extension.
Read article →Positional Encoding
Why attention needs positional information and how sinusoidal and learned positional encodings work.
Read article →PP Math
Pipeline parallelism: split layers into stages across GPUs.
Read article →PPO for LLMs Math
PPO applied to LLMs.
Read article →PQ Math
Product quantization for compressed ANN.
Read article →Prefill Math
Prefill phase math.
Read article →Prefix Tuning Theory
Prefix tuning: learned prefixes for tasks.
Read article →Pre-norm vs post-norm transformers
Deep-dive on transformer normalization placement: post-norm on the residual path versus pre-norm inside the branch, gradient flow and attenuation, war…
Read article →Pretraining Scaling
Pretraining-specific scaling considerations.
Read article →Priority Scheduling
Priority scheduling for LLM requests.
Read article →Linear Probes
Probing internal representations with linear classifiers.
Read article →Prompt Prefill vs Decode
Prefill vs decode phase differences.
Read article →Prompt Lookup Decoding Math
Prompt lookup decoding math.
Read article →Prompt Tuning Theory
Prompt tuning: learn soft prompts at input.
Read article →Qdrant Math + Architecture
Qdrant: Rust vector DB.
Read article →QLoRA Theory
QLoRA: 4-bit quantized base + LoRA.
Read article →Quantization Quality Evaluation
How to evaluate quantization quality drop.
Read article →Advanced RAG Techniques Overview
Overview of advanced RAG methods.
Read article →Reasoning Capability Evals
LLM reasoning capability eval.
Read article →Reasoning Model Scaling
Scaling laws for reasoning models (o1 / R1 style).
Read article →Rectified Flow Math
Rectified flow: straight ODE paths.
Read article →Redis Vector Search Math
Redis vector search + RediSearch.
Read article →Representation Learning Theory
Theory of learned representations in DL.
Read article →Reranking Math
Cross-encoder reranking after retrieval.
Read article →RetNet
RetNet: parallel + recurrent + chunkwise forms.
Read article →RetNet Math
Retentive network math.
Read article →Retrieval Architectures Overview
Overview of retrieval architectures.
Read article →Reward Hacking Math
Reward hacking in RLHF.
Read article →Reward Model Math
Reward model math for RLHF.
Read article →Reward Model Scaling
Scaling reward models for RLHF.
Read article →Ring Attention
Ring attention: distribute attention computation across GPUs.
Read article →Ring Attention Math Deep
Ring Attention: distributed long-context attention.
Read article →RLHF Math Deep Dive
RLHF math deep dive.
Read article →RLHF Scaling Laws
Scaling laws for RLHF.
Read article →RLHF Scaling Math
Scaling RLHF to larger models.
Read article →RMSNorm architecture - root-mean-square normalization in modern transformers
Deep-dive on RMSNorm: the root-mean-square formula, contrast with LayerNorm's mean-centering and bias, forward and backward p…
Read article →RMSNorm Deep Dive
Deep math of RMSNorm and kernel implementation.
Read article →Roofline Model for LLM Ops
Roofline model: reasoning about performance.
Read article →Rotary Position Embedding (RoPE)
How RoPE encodes position by rotating query/key vectors, and why it extrapolates better than absolute PE.
Read article →Rotary Position Embedding Architecture in Depth
A 2500-word walkthrough of RoPE architecture: rotation math, base theta, NTK-aware scaling, YaRN/PI extension, kernel fusion, comparisons.
Read article →RWKV Math
RWKV RNN-like transformer math.
Read article →Sparse Autoencoders (SAE)
SAE: extract interpretable features.
Read article →Sampling mathematics
Deep-dive on LLM decoding math: logits through temperature and softmax, top-k vs nucleus vs min-p truncation, repetition/presence/frequency penalties,…
Read article →Scaled Dot Product Attention
The specific formulation used in transformers: scaled dot product attention, and why scaling by sqrt(d_k) matters.
Read article →Scaling law architecture
Deep-dive on scaling laws: Chinchilla ratio, compute-optimal N+D, overtraining for inference cost, data quality, emergent capabilities.
Read article →Scaling Laws — Power Law and Chinchilla
Mathematical formulation of LLM scaling laws.
Read article →Scaling Laws — Where They Break Down
Neural scaling laws deep dive.
Read article →Hyperparameter Scaling Transfer
Transferring HPs from small to large models.
Read article →ScaNN
ScaNN: Google's ANN library.
Read article →Score Matching
Score matching: learn gradient of log density.
Read article →Search Scaling in Reasoning
MCTS / beam search scaling for reasoning.
Read article →Selective Activation Recomputation
Recompute only expensive-to-store activations.
Read article →Self-Consistency Scaling
Self-consistency: majority vote over samples.
Read article →Self-RAG
Self-RAG: LLM decides when + what to retrieve.
Read article →SentencePiece Deep Dive
SentencePiece tokenizer library.
Read article →Sequence Length Math
Memory + compute impact of sequence length.
Read article →Sequence Parallel Math
Sequence parallelism math for LLM.
Read article →Shampoo
Shampoo: preconditioned second-order optimizer.
Read article →Shared KV Cache
Prefix sharing in KV cache.
Read article →Signal Propagation in Deep Networks
Theory: how signals travel through depth.
Read article →Sliding Window Attention
Math of sliding window attention.
Read article →SLO-Aware Scheduling Math
Scheduling math driven by SLOs.
Read article →SmoothQuant Math
SmoothQuant: shift activation outliers to weights.
Read article →Softmax in Attention
How softmax converts attention scores to probabilities, its computational cost, and numerically-stable implementation.
Read article →Sophia Optimizer
Sophia: Hessian-based optimizer for LLM.
Read article →SP Math
Sequence parallelism: split sequence across GPUs.
Read article →Sparse Retrieval Math
Sparse retrieval: BM25 + SPLADE.
Read article →Speculative decoding architecture
Deep-dive on speculative decoding: exploiting memory-bound decode with a cheap draft model whose K proposals the target verifies in one parallel pass,…
Read article →Speculative Decoding in Practice
Deep dive on speculative decoding.
Read article →Speculative Decoding — Acceptance Rate
Math of speculative decoding for LLM inference speedup.
Read article →SPLADE Math
SPLADE: learned sparse retrieval.
Read article →State Space Models Math
State space model math.
Read article →Selective SSM
How selective SSMs adapt dynamics to input.
Read article →Stable Diffusion Math
Math behind Stable Diffusion.
Read article →State Space Models
State-space model foundation for Mamba, S4, and successors.
Read article →Stochastic Rounding
Stochastic rounding for low-precision training.
Read article →Superposition Hypothesis
Superposition: features stored non-orthogonally in NN.
Read article →SwiGLU
SwiGLU: SiLU-gated GLU used in Llama, PaLM, most modern LLMs.
Read article →Switch Transformer Math
Switch Transformer math.
Read article →Synthetic Training Data
Synthetic data for training.
Read article →Temperature Sampling Math
Temperature: scale logits before softmax.
Read article →Temperature scaling in LLM sampling
Deep-dive on temperature scaling: dividing logits by a scalar T before softmax to rescale logit differences, the cold/neutral/hot regimes and their ef…
Read article →Tensor Parallel Math Deep
Tensor parallel math for LLM.
Read article →Tensor Programs (Yang's Framework)
Yang's Tensor Programs: theory unifying scaling.
Read article →Test-Time Compute Math
Test-time compute: scale reasoning at inference.
Read article →tiktoken Deep Dive
OpenAI's tiktoken tokenizer.
Read article →Top-K Sampling Math
Top-k: sample from k highest-prob tokens.
Read article →Top-P (Nucleus) Sampling Math
Top-p: sample from cumulative-mass p.
Read article →Transfer Learning Scaling
How transfer + fine-tune scale from pretrain.
Read article →Typical Sampling
Typical: sample from tokens near expected entropy.
Read article →Unigram LM Tokenizer
Unigram language model tokenizer.
Read article →Variance Maintenance in Deep Nets
Keeping activation variance stable across layers.
Read article →Vector DB Math
Foundational math of vector databases.
Read article →Verifier Scaling
Scaling verifier models for reasoning.
Read article →Vespa Math
Vespa: search + vector DB.
Read article →LR Warmup Deep Dive
Why warmup is essential for LLM training.
Read article →Weaviate Math + Architecture
Weaviate: open-source vector DB.
Read article →Width Scaling
Scaling model width: hidden dim + heads.
Read article →YaRN + NTK Scaling
How YaRN + NTK-aware scaling extend RoPE context length.
Read article →ZeRO Stage 1 Math
Math of ZeRO Stage 1.
Read article →ZeRO Stage 2 Math
Math of ZeRO Stage 2.
Read article →ZeRO Stage 3 Math
Math of ZeRO Stage 3.
Read article →Zero-bubble pipeline parallelism
Deep-dive on zero-bubble pipeline scheduling: the (p-1)/m bubble, GPipe vs 1F1B vs interleaved 1F1B, splitting backward into input-gradient B and weig…
Read article →ZeRO-Infinity
ZeRO-Infinity: scale to trillion params via GPU + CPU + NVMe.
Read article →ZeRO Math
ZeRO: DeepSpeed's memory optimization stages.
Read article →ZeRO-Offload
Offload optimizer states + gradients to CPU or NVMe.
Read article →Token Embedding Lookup
How discrete tokens become continuous vectors.
Read article →Tokenizer Math
How text becomes a token sequence.
Read article →Training Data for SLMs
What to train on when you can't use the whole internet.
Read article →Weight Initialization
Why all-zeros doesn't work and what to do instead.
Read article →How Weights Are Stored on Disk
From PyTorch tensor to GGUF/SafeTensors bytes.
Read article →