Transformers

Attention Is All You Need

What aged and what didn't.

Attention Is All You Need

The 2017 paper that started everything, re-explained for 2026.

Attention Variants in 2026

MHA, MQA, GQA, MLA — what's deployed.

Audio-Native Transformers: Understanding Models That Process Raw Sound Instead of Text-to-Speech

Beyond Transformers: Sparse Architectures and State-Space Models for Infinite Context

Causal vs Bidirectional Attention

Decoder encoder prefix-LM uses.

FlashAttention Explained

How the attention kernel was rewritten to be 5-10x faster.

FlashAttention-3 and Beyond: How Hardware-Aware Algorithms are Making Models 10x Faster

KV Cache Explained

Why generation is fast: the attention KV cache and its memory implications.

Linear Transformers: Can We Finally Achieve O(n) Complexity for Infinite Context?

Long-Context Evaluation

Needle-in-haystack and what it misses.

Mixture of Depths

Routing tokens through different depths.

Multi-Token Prediction (MTP)

DeepSeek V3's MTP and the speed-quality trade.

Performance Analysis Mamba Vs Transformer

Rotary Position Embeddings (RoPE)

The position encoding that replaced sinusoidal and enabled long-context extrapolation.

The Transformer Breakdown: A Deep Dive into Self-Attention, Key-Value Pairs, and Positional Encoding

The Vanishing Gradient Problem: How Transformers Solved What Killed Earlier RNNs

Tokenization Choices Compared

BPE, SentencePiece, Tiktoken — and why it matters.

Transformer Block Anatomy

What's inside one of the N layers.

Transformer Training Loss Curves

Reading them and what each shape means.