Why architecture matters here
The architecture matters because the quantity that governs how fast a matrix parameter can be optimized is not the average magnitude of its gradient — which is what Adam normalizes — but the spread of its singular values, its condition number. When a weight matrix's gradient has a few very large singular values and many small ones, Adam's per-coordinate scaling does nothing about the imbalance: the update still moves mostly along the dominant directions, the loss landscape's narrow valleys stay narrow, and progress along the flat directions crawls. Orthogonalizing the update flattens that spectrum by construction — every direction the gradient identified gets a unit-magnitude push — so the optimizer makes balanced progress in all directions at once. Empirically this shows up as fewer steps to a target loss and a training curve that tolerates larger effective learning rates without instability.
The architecture also matters because it makes learning-rate transfer far more predictable, which is worth as much as the raw speedup in practice. Because the orthogonalized update has a controlled spectral norm regardless of the raw gradient's scale, the step size a matrix takes is decoupled from the accident of how large its gradients happen to be at a given layer or a given point in training. Combined with an RMS-to-RMS scaling that accounts for a matrix's fan-in and fan-out, this means one learning rate tends to work across matrices of different shapes, and — a result that has drawn intense interest — a learning rate tuned on a small model transfers to a much larger one. For teams whose compute budget is dominated by hyperparameter search at scale, that transfer is a first-order economic effect: you tune cheaply and inherit the setting expensively.
It matters, finally, because it is a genuinely different preconditioner than Adam, and understanding which parameters it suits prevents the most common failure. Muon's geometry is the geometry of matrices; applying it to a 1D vector is meaningless (there is no non-trivial orthogonalization of a vector), and applying it to embedding and output-head matrices — whose rows or columns correspond to individual tokens and whose gradients are extremely sparse and structured — tends to hurt rather than help. The design's insistence on routing those parameters to AdamW is not a hedge; it is the recognition that the orthogonalization argument only holds for the dense hidden matrices, and the hybrid is the correct object, not a fallback for the timid.
The cost side keeps the trade honest. Each Muon step runs a handful of extra matmuls per matrix (the Newton-Schulz iteration), so the per-step compute is higher than Adam's. The bet is that the reduction in step count — and the avoided hyperparameter sweeps — more than pays for the per-step overhead, and at scale the iteration can be sharded across data-parallel ranks so its cost is amortized. Whether the bet pays depends on the model and the budget, which is exactly why an engineer needs to understand the mechanism rather than treat it as a drop-in.
The architecture: every piece explained
Top row: the update pipeline for one weight matrix. Start with the gradient G for a 2D parameter. Feed it into a momentum buffer — an exponential moving average of past gradients (Muon uses standard heavy-ball momentum, typically around 0.95) — which smooths noise and gives the update a consistent direction to orthogonalize. The smoothed momentum M is then passed to the Newton-Schulz orthogonalizer, which produces an approximation of the orthogonal factor of M (the U Vᵀ from its SVD). Finally the orthogonalized update is turned into a scaled step and subtracted from the weight. That is the whole outer loop: momentum, orthogonalize, scale, apply.
Middle row: how the orthogonalization is actually computed. The SVD intuition is the target — replace Σ with the identity so M = U Σ Vᵀ becomes U Vᵀ — but a real SVD is far too slow to run every step for every matrix. Instead Muon uses a quintic Newton-Schulz iteration: a fixed polynomial recurrence in the matrix (each step is a few matmuls of the matrix with itself) whose coefficients are chosen so that, starting from a normalized M, it converges toward U Vᵀ in about five iterations. It uses only matrix multiplications, so it runs fast on tensor cores and, with carefully chosen coefficients, remains stable in bf16 — no high-precision decomposition required. The RMS-to-RMS scaling then multiplies the orthogonalized update by a factor derived from the matrix's shape (fan-in and fan-out), so that the update's effect on activations has a consistent scale across differently-shaped matrices — this is what makes a single learning rate transfer across layers and sizes. The AdamW fallback routes 1D parameters and the embedding and output-head matrices to a conventional AdamW update, because orthogonalization is undefined or harmful for them.
Bottom rows: the interpretations and the scaling story. The spectral view is the cleanest way to understand what Muon guarantees: the orthogonalized update has all singular values equal to one, so its spectral norm (largest singular value) is bounded by construction — the step can never explode along a single dominant direction the way a raw gradient step can. This is a form of implicit spectral regularization on the updates, which is why Muon runs tolerate aggressive learning rates that would make a raw-gradient optimizer diverge. Distributed Muon addresses the one real cost: the Newton-Schulz iteration is extra matmul work, so at scale it is sharded across data-parallel ranks — each rank orthogonalizes a slice, and the results are gathered — keeping the per-step overhead small relative to the forward/backward pass. The ops strip lists the knobs that actually matter in practice: how many Newton-Schulz iterations to run, how learning rate and weight decay transfer, and — the decision that most affects whether Muon helps or hurts — precisely which parameters use Muon versus AdamW.
End-to-end flow
Walk one optimizer step for a transformer training with Muon. The forward and backward passes have just produced gradients for every parameter. The optimizer iterates the parameter groups.
A hidden MLP weight matrix (Muon path): its gradient G, a fan_in × fan_out matrix, arrives. Muon updates its momentum buffer: M ← 0.95·M + G. It then normalizes M and runs the five-step Newton-Schulz iteration — five passes of a fixed quintic polynomial, each a small number of matmuls of M with itself — producing O, an approximation of the orthogonal factor of M. O has (approximately) unit singular values: it points in the same directions M did but with the magnitudes flattened. Muon scales O by the RMS-to-RMS factor for this matrix's shape and by the global learning rate, applies weight decay, and subtracts the result from the weight. Because O's spectral norm is bounded, this step cannot blow up along a single direction no matter how ill-conditioned the raw gradient was — the update is inherently well-behaved.
A LayerNorm gain (AdamW path): this 1D parameter has no matrix structure to orthogonalize. Muon routes it to the AdamW fallback: maintain first and second moment estimates, bias-correct, and apply the familiar per-coordinate adaptive step. The same happens for biases and for the embedding and output-head matrices — the latter routed to Adam not because they lack matrix shape but because their token-indexed, sparse-gradient structure makes orthogonalization counterproductive.
Why the training curve moves faster: across thousands of these steps, the hidden matrices make balanced progress in all directions their gradients identify rather than lurching along a few dominant singular directions and crawling elsewhere. Narrow valleys in the loss landscape are traversed as efficiently as flat plateaus, so the loss falls in fewer steps than an Adam run at a comparable per-parameter learning rate. Meanwhile the bounded update norm lets the run use a larger effective learning rate without the loss spiking, compounding the advantage.
At scale, the distributed step: with data parallelism across many accelerators, running the full Newton-Schulz iteration for every matrix on every rank would duplicate work. Distributed Muon shards the orthogonalization — each rank runs the NS iteration for a subset of matrices (or a slice of each) and the orthogonalized updates are communicated — so the extra matmul cost is spread out and stays a small fraction of the forward/backward compute. The net picture: a per-step cost modestly higher than Adam, a step count meaningfully lower, learning rates that transferred from a small tuning run, and — the payoff the whole design targets — the same or better final loss reached with less total compute.