Two 2026 papers claim that during RLVR training an LLM’s weights barely wander: they move along essentially one fixed direction in weight space, at a steady pace, so you can predict a far-future checkpoint from a handful of early ones. This page explains what that claim means, then shows — with plots from our own model (Qwen2.5-1.5B, GRPO on GSM8K) — how much of it holds here and how well we can actually project the weights forward. Every plot is labelled with which way is “better.”
Checkpoint delta Δθₕ — take the model weights at training step t and subtract the starting weights: Δθₕ = θₕ − θ₀. It is the total change so far, computed per weight tensor (each attention/MLP matrix separately). We saved 160 such checkpoints and work with one tensor at a time.
Rank-1 = “one direction” — stack a tensor’s deltas from many checkpoints as the rows of a matrix and run an SVD. If a single component explains almost all of it, the deltas are just (one shared unit direction v₁) × (a per-checkpoint number cₕ). So “the update is rank-1” literally means “there is basically ONE direction the weights move along; each checkpoint only differs in how far along it they’ve gone.” Yes — rank-1 ≡ a direction.
The projection — if that holds, predicting a future checkpoint is trivial: θ̂ᴛ = θ₀ + cᴛ·v₁, i.e. keep the direction, extend the number. That is RELEX.
Reading it: C2 (left) — for each tensor we take the fraction of its update “energy” captured by the single top direction (EVR₁ = λ₁/Σλ). Median 99.3%: one direction explains essentially the whole change — even more rank-1 than the paper’s ~81% (they measured a different quantity, a rank-5 window). F (right) shows this is uniform: every module type — attention q/k/v/o, MLP gate/up/down, norms, biases — is rank-1 to the same degree, and so is every depth we probed (layers 0/7/13/20/27). Higher = more one-directional; there is no special “non-rank-1” module.
We project each delta onto v₁ to get the scalar cₕ, then fit a straight line c(t)=a·t+b. Median R² = 0.979 across tensors (per-module medians 0.986–0.994, plot F used the same fit). That means the “how far along” number really does grow at a steady rate — the paper’s central claim. We land just under their clean “R²>0.98” bar (49% of tensors clear it here; 37% at early anchors), and we report that honestly rather than rounding up.
Higher = more perfectly linear (1.0 = a flawless line). The red dashed line marks the paper’s 0.98 threshold.
This is the key question: the paper says the direction barely changes — one shared v₁ for the whole run. We tested it directly by estimating v₁ from a sliding 16-checkpoint window and measuring the angle between v₁ taken at different points in training (|cos| = 1 means identical direction, 0 means fully rotated away).
Reading it — two curves, two timescales:
So your intuition is right and refined: the direction is very stable step-to-step (as in the paper), but it is not perfectly frozen — it rotates slowly over the whole run. That single fact explains everything downstream: a short recent window sees an almost-fixed direction, while the full-run window (§5) tries to fit one line to a gently curved path and pays for it.
Depth effect (plot I): the drift is mild everywhere but slightly larger in deeper layers — layer 27’s direction falls to |cos|≈0.59 vs the start, layer 0’s only to ≈0.72. Early/shallow structure settles into its final direction sooner; later layers keep re-aiming a bit longer. All curves are monotone and smooth — the rotation is a steady curve, not noise.
Sanity: v₁ unit-norm reconstructed from the Gram to max error 2e-15; probe is 11 windows spanning ticks 16–116.
Now the practical test our comm-eff work cares about: given a stale checkpoint at an anchor, predict where the weights will be h ticks later. We score weight_proj_ratio = ‖prediction − truth‖ / ‖stale-anchor − truth‖: below 1 beats doing nothing, above 1 is worse than just keeping the stale weights.
| method [window] | h=1 ↓ | h=2 ↓ | h=5 ↓ | h=10 ↓ | h=20 ↓ | h=40 ↓ |
|---|---|---|---|---|---|---|
| hold_stale[-] | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| naive_last2[-] | 0.463 | 0.534 | 0.686 | 0.893 | 1.193 | 1.599 |
| two_point_window[8] | 0.677 | 0.709 | 0.795 | 0.933 | 1.153 | 1.484 |
| rank1_traj[8] | 3.945 | 2.419 | 1.546 | 1.252 | 1.120 | 1.076 |
| rank1_traj[prefix] | 21.715 | 11.269 | 5.206 | 3.156 | 2.193 | 1.805 |
| rank1_anchored[8] | 0.991 | 0.992 | 0.993 | 0.995 | 1.005 | 1.026 |
| rank1_anchored[16] | 0.998 | 0.998 | 0.999 | 1.002 | 1.012 | 1.033 |
| rank2_anchored[8] | 0.699 | 0.730 | 0.817 | 0.953 | 1.172 | 1.506 |
| rank2_traj[8] | 1.245 | 0.973 | 0.899 | 0.986 | 1.184 | 1.511 |
Global pooled ratio, median over anchors 79/119. Cell tint: green < 0.98 = beats hold-stale · grey ≈ 1 = neutral · red > 1.05 = harmful. Lower is better in every cell.
What it says: the paper’s own form rank1_traj scores 3.9–22
at short h (way above 1 — clipped ↑) because it re-pays the accumulated drift of
§3. Our anchor-pinned variant rank1_anchored is the only arm that is
never harmful out to h=40 (0.98–1.06) — but it removes only ~1–2% of the
error. Plain recent-step momentum (naive_last2) is the real short-horizon
winner (0.43–0.46 at h=1) but turns harmful past h≈15.
Why so little skill (plot E): even anchored, the leftover error stays ≈97% “radial” — pointing straight along the true move — at every horizon. The fitted line’s next step is nearly orthogonal to where the weights actually go over the next ≤40 ticks. The direction is great at describing accumulated position (§1–3) but poor at predicting the next increment, which is dominated by a fast component the window-averaged line can’t see.
Window size (how many checkpoints feed the SVD): B2 measures how much of the accumulated delta lies off the single fitted direction — 4.5% for a recent 8-checkpoint window, rising to 33% for the full-run window. That is §3’s slow rotation showing up as cost: the longer the window, the more curve you force onto one line, so bigger windows predict worse here (B1). Lower is better in both.
Training stage: maybe the line only fails late, once training has converged? No — anchors at ticks 39/59 (early, weights still moving fast) give the same neutral profile as 79/119. If anything the full-run window is worse early because it averages over the fast-rotating warm-up. The rank-1 line is a checkpoint-scale object at every stage — good for “where are we,” not “where next.” Lower = better.
naive_last2 0.43–0.46 at h=1,
rank2_anchored 0.69 at h=1 (its 2nd component is the local dynamics,
though harmful by h=20), with rank1_anchored as the never-harmful fallback
(0.98–1.06 across all h ≤ 40, both training regimes).python3 scripts/rank1_scorecard.py --self-test python3 scripts/rank1_scorecard.py \ --trace-root /workspace/trace/EXP-57 \ --manifest runs/EXP-57/regimeA/weights/full_manifest.jsonl \ --out runs/RANK1-ANALYSIS/scorecard --scope panel # late anchors (79,119) # + --anchors 39,59 --out .../scorecard-early # early anchors python3 scripts/rank1_direction.py # v1 direction-stability probe (§3) python3 scripts/rank1_report.py # this page
Implementation:
scripts/weight_proj/rank1_traj.py + scripts/rank1_scorecard.py
(new lane; moat_scorecard.py and the online comm-eff path untouched).
Canonical findings: runs/RANK1-ANALYSIS/verdict.md · independent
verification: runs/RANK1-ANALYSIS/verify_verdict.md · papers: Paper A
arXiv:2601.04537 (linear dynamics / weight extrapolation), Paper B
arXiv:2605.21468 (RELEX). Trace: EXP-57 regimeA fp32, 160 ticks (2 optimizer ticks
per GRPO step), 61-matrix panel = layers {0,7,13,20,27} + final norm.