All 18 articles, sorted alphabetically
The Transformer Breakdown: A Deep Dive into Self-Attention, Key-Value Pairs, and Positional Encoding
Read article →The Vanishing Gradient Problem: How Transformers Solved What Killed Earlier RNNs
Read article →Hugging Face generate() architecture
Deep-dive on the Transformers generate() method: the GenerationConfig as single source of truth, the ordered logits-processor pipeline (temperature, t…
Read article →HuggingFace Transformers Library Architecture in Depth
A 2500-word walkthrough of the HuggingFace Transformers library architecture: Hub, auto classes, tokenizer, model, Trainer, TRL/PEFT, Accelerate, and …
Read article →HuggingFace Transformers Pipelines Architecture in Depth
A 2500-word walkthrough of HuggingFace pipelines: task registry, model selection, preprocessor, forward, postprocessor, batching, device, streaming, O…
Read article →Tokenizers
Deep-dive on tokenizers: the normalization/pre-tokenization/subword pipeline, BPE/WordPiece/Unigram algorithms, vocabulary and token IDs, special toke…
Read article →Hugging Face Trainer architecture
Deep-dive on the HF Trainer: how TrainingArguments configures a run, the Trainer orchestrates the step over an Accelerate backend, gradient accumulati…
Read article →ALiBi attention architecture
Deep-dive on ALiBi (Attention with Linear Biases): replacing input position embeddings with a static per-head linear penalty on attention scores, the …
Read article →Attention sinks -- why the first tokens matter for streaming LLMs
Deep-dive on attention sinks: the infinite-generation KV-cache problem, sliding-window KV and its collapse, attention sinks (initial tokens absorbing …
Read article →Cross-attention architecture
Deep-dive on cross-attention, the attention variant where queries come from a target sequence and keys/values from a separate source, powering transla…
Read article →FlashAttention -- IO-aware exact attention
Deep-dive on FlashAttention: the O(N^2) attention memory problem, the IO-awareness insight (attention is memory-IO-bound), tiling Q/K/V into SRAM bloc…
Read article →KV cache architecture
Deep-dive on the transformer KV cache: prefill vs decode regimes, per-token memory math, paged allocation and block tables, prefix caching with copy-o…
Read article →Multi-head latent attention (MLA) architecture
Deep-dive on MLA: down/up-projecting keys and values into a cached low-rank latent, decoupled RoPE for position, weight absorption so inference attend…
Read article →QK-Norm architecture
Deep-dive on query-key normalization: why QK^T/sqrt(d) logits grow without bound, how softmax saturation kills gradients, per-head RMSNorm placement b…
Read article →Ring Attention: Architecture Deep-Dive
How ring attention shards sequences across devices and rotates KV blocks to compute exact attention beyond single-device memory limits — architecture,…
Read article →RMSNorm
Deep-dive on RMSNorm: normalizing activations by their root-mean-square with a learned gain and no bias, why dropping LayerNorm&#x…
Read article →Speculative decoding architecture
Deep-dive on speculative decoding: why decoding is memory-bandwidth bound, a draft model proposing K tokens, single-pass parallel verification, the re…
Read article →SwiGLU architecture
Deep-dive on the SwiGLU FFN: the SiLU/Swish activation, the two up-projection gate multiplied elementwise and projected down, the two-thirds d_ff resc…
Read article →