Gradient descent · a field guide

The optimizer family tree, organized by the state it keeps

Every optimizer here is the same step — walk downhill — differing only in how it transforms the gradient and what extra memory it carries to do so. So the tree branches by state, not by publication date. The accent green always marks the one new thing a node adds over its parent.

θ ← θ − η · 𝚏(g, state) pick f and state → pick your optimizer

new state / idea added here + added line vs parent ~ changed line vs parent inherited state

MAP

The lineage at a glance

Branches are the three ways to spend memory: a first moment2 (velocity), a second moment3 (per-parameter scale), or a structured preconditioner5 (whole-matrix curvature). Adam is literally the merge of the momentum and RMSProp branches. The dashed node is the one deterministic / full-batch method (L-BFGS) — everything else is built for stochastic mini-batches. The dotted node up top is the gradient-free world (ES, MeZO, EGGROLL): it changes where g comes from, not what you do with it. Tap any node to jump to its card; numbered marks like these jump to short footnotes at the bottom.

Stochastic · mini-batch

SGD · Momentum · Nesterov · AdaGrad · RMSProp · Adam(W) · Lion · Adafactor · Muon · Shampoo · K-FAC · PSGD

Designed to make progress from one noisy mini-batch gradient at a time. The whole tree above lives here — noise tolerance is the point.

Deterministic · full-batch

Newton · BFGS · L-BFGS (secant / quasi-Newton)

Infer curvature from gradient differences (B·s = y, the secant condition8) and usually pair with a line search — both assume a stable objective. Mini-batch noise corrupts the (s, y) pairs, so these are rare in deep learning.

LAB

Race them on a landscape

The master equation, running live: same start, same landscape, same η dial — only f(g, state) differs. The structured branch appears through its honest 2-D stand-ins: Full-matrix keeps a real 2×2 accumulator G = Σ g·gᵀ (the full-matrix-AdaGrad / Shampoo idea), and Muon's Ortho(M) on a 1×2 “matrix” reduces to unit-norm momentum. + marks the minima. Reading the map: you're looking straight down at the loss surface. Each line joins points of equal loss — tightly packed lines mean steep, the shading darkens uphill, and the small arrows point downhill. Hover to read the loss under the cursor; click to move the start point. (Per-optimizer step scaling is applied so one η dial is fair to all.)

click to set a new start point

top-down view · lines join equal loss · darker = higher · arrows point downhill

Landscape

downhill arrows

Optimizers · tap to toggle

step size η = 0.020 gradient noise σ = 0.00

Ravine: Momentum piles up speed and swings wide at the bend; Adam and Muon hug it. That's a raw first moment vs a first moment with a scale.
Tilted bowl: Adam still zigzags — a diagonal scale can't rotate. Turn on Full-matrix and watch it cut straight along the tilted axis. That's the entire pitch of section 05 in one picture.
Slide σ up: SGD gets drunk while the EMA methods glide (footnotes 1–2, live). Then enable L-BFGS: untouchable at σ = 0, broken the moment noise corrupts its (s, y) pairs — exactly why it's the dashed node.
Lion & Muon take fixed-magnitude steps (sign / unit-norm), so they never fully sit still — the real ones lean on lr decay and weight decay to land. And AdamW makes decay visible: it parks slightly short of the +, dragged toward the origin.
Crank η to ≥0.1 on the ravine: SGD flies off (✗) while the adaptive family shrugs — per-parameter scaling as a stabilizer.
ZO / SPSA uses no gradient at all — two loss probes per step. Watch it wiggle even at σ = 0: its randomness is the estimator itself. In 2-D it keeps up with SGD; in a billion dimensions that wiggle is the whole battle (hence EGGROLL's population tricks).
Four minima: from the same start, different optimizers land in different basins — the optimizer isn't a neutral pipe to “the” solution, it's part of what gets learned.

Footnotes — the concepts, briefly

Every recurring term above, defined once. The numbered marks throughout the page land here.

EMA — exponential moving average
s ← ρ·s + (1−ρ)·x. A running average that forgets at rate ρ; its memory horizon is roughly 1/(1−ρ) steps (β=0.9 ≈ the last 10 gradients, β=0.999 ≈ the last 1000). Almost every buffer in this tree is an EMA of something.
First moment
The mean. In statistics the k-th moment of a random variable is E[xᵏ]; the first is E[x]. An EMA of the raw gradient estimates E[g] — the average direction — which is exactly what a velocity/momentum buffer is. It smooths mini-batch noise and builds speed along directions that keep agreeing with themselves.
Second moment
E[g²], per coordinate — the average squared gradient, i.e. the uncentered variance. Its square root is an RMS (root mean square — hence RMSProp). Dividing the step by √E[g²] gives every parameter its own learning rate: coordinates with habitually large gradients get reined in, quiet ones get amplified.
Bias correction
An EMA initialized at zero underestimates at first: after t steps it has only accumulated a fraction (1−βᵗ) of the true average. Adam divides by that fraction (m̂ = m/(1−β₁ᵗ), v̂ = v/(1−β₂ᵗ)) so early steps aren't artificially tiny. The correction fades to 1 as t grows.
Preconditioner
Any matrix P in θ ← θ − η·P·g that reshapes the gradient before stepping. The ideal is Newton's P = H⁻¹ (inverse Hessian) — perfect but unaffordable. Adam/RMSProp use a diagonal P (1/(√v+ε) per coordinate); Shampoo, K-FAC and PSGD spend more memory on structured full matrices that also capture correlations between parameters.
Orthogonalization — Ortho(M)
Write the SVD M = UΣVᵀ: U, V hold the directions, Σ the magnitudes. Then Ortho(M) := UVᵀ — same directions, every singular value set to 1. It's the nearest orthogonal(-ish) matrix to M, and equivalently (MMᵀ)^−½·M. The point: a raw momentum matrix is dominated by a few large singular directions; orthogonalizing makes the update push equally along all of them.
Newton–Schulz iteration
Computing UVᵀ by actual SVD is slow and GPU-unfriendly. Newton–Schulz instead iterates a fixed odd matrix polynomial, X ← a·X + b·(XXᵀ)X + c·(XXᵀ)²X, on the normalized momentum. Each pass squashes all singular values toward 1, so ~5 iterations of pure matmuls (bf16-safe) ≈ Ortho(M) — no decomposition ever computed. Muon uses tuned coefficients (a≈3.44, b≈−4.78, c≈2.03) that converge fast without needing high precision.
Secant / quasi-Newton
Don't compute the Hessian — infer it from how the gradient moved: demand B·s = y where s = Δθ and y = Δg (the secant condition, the multivariate cousin of a finite-difference slope). BFGS refines an estimate with each new (s, y) pair; L-BFGS keeps only the last m pairs and rebuilds H⁻¹g on the fly. Mini-batch noise corrupts y (the gradient changed because the batch changed, not θ), which is why this family stays full-batch.
Fisher information / natural gradient
The Fisher matrix F measures how much the model's output distribution shifts per unit change of each parameter. Stepping along F⁻¹·g — the natural gradient — takes uniform steps in output space rather than parameter space, making the update invariant to how the network happens to be parameterized. K-FAC makes F tractable per layer as a Kronecker product A ⊗ G (input-activation covariance ⊗ output-gradient covariance), so F⁻¹g ≈ A⁻¹·g·G⁻¹.
Zeroth-order / evolution strategies
Methods that never see a true gradient — they probe the loss directly. The two-point estimate ĝ = [ℒ(θ+εz) − ℒ(θ−εz)] ⁄ 2ε · z measures the slope along one random direction z: in expectation an unbiased (but very noisy) sketch of the gradient of a slightly smoothed loss. Evolution strategies average many such perturbations weighted by fitness; larger populations mean less noise. The appeal: only forward passes needed — tiny memory, and non-differentiable objectives are fine. The cost: each probe yields one scalar, so high dimensions demand many probes or structured noise (EGGROLL's low-rank perturbations).

The whole tree in five lines

State is the storyline. SGD keeps nothing; momentum adds one tensor (velocity), RMSProp adds one tensor (squared-grad EMA), Adam keeps both.
AdaGrad → RMSProp is a single edit: replace the growing sum s += g² with a decaying average s = ρs + (1−ρ)g², so the step size stops collapsing.
Adam = Momentum + RMSProp (plus bias correction). AdamW just pulls weight decay out of the gradient and applies it straight to θ. Lion drops the second moment entirely and takes the sign of the momentum.
Adafactor keeps Adam's idea but refuses to store a full v for matrices — it factors it into row + column stats and reconstructs a rank-1 approximation. It's compressed second moment, not compressed momentum.
PSGD, Shampoo, K-FAC leave the diagonal world: instead of one scalar per parameter they apply a matrix preconditioner (learned, Gram-based, or Fisher-based) to the whole gradient — all in the stochastic mini-batch regime. Shampoo in particular is best read as full-matrix AdaGrad made affordable: the ideal accumulator Σ g·gᵀ over all parameters is Kronecker-factored into a left and right matrix per layer.
L-BFGS sits apart. It's a secant / quasi-Newton method — curvature inferred from successive gradient differences (B·s = y), making it a sibling of Shampoo, not a descendant. It's also the lone deterministic / full-batch method here: mini-batch noise corrupts the (s, y) pairs and breaks the line search, which is why it's rare in deep learning.
Muon is the newest branch: keep only momentum, then orthogonalize that momentum matrix (Ortho(M) = UVᵀ) with a few cheap Newton–Schulz steps — no v, no accumulation. And the Shampoo link is exact, not a metaphor: plug the instantaneous L = MMᵀ, R = MᵀM into Shampoo's L^−¼·M·R^−¼ and the algebra collapses to precisely UVᵀ.
Zeroth-order sits outside the master equation entirely. MeZO probes the loss along one random direction (two forward passes, no backprop, inference-level memory); evolution strategies average a whole population of perturbations; EGGROLL makes that population affordable at LLM scale with rank-r noise. They trade gradient information for memory and generality — a different question, not a different answer.
One trick, two scales. sign(m) = m/√m² normalizes a vector per coordinate; Ortho(M) = UVᵀ normalizes a matrix per singular direction. Lion and Muon are the same instinct — keep the direction, flatten the magnitudes — applied to different geometry. And Adam sits between them: it divides by √v, a smoothed version of the same per-coordinate normalization.