Weight projection for two-circuit-network RLVR stability at longer K

Weekly update · self-contained for other teams · every term defined on this page · all numbers from full-precision (fp32) weight trajectories.

✓  Verified by Shamane

1. This week, in one line

2. Background & the goal

The method: the two-circuit-network RLVR setup

We train the model split into two circuits (two partitions of the network) that must stay coordinated while the model learns from reward (RLVR). Exchanging everything between the circuits on every step is expensive; the entire point of the method is to cut that boundary traffic. So between coordination points, each circuit does its work against a stale copy of the other circuit’s weights.

The problem: staying stable at longer K

K is how many steps we let pass between coordinations (refreshes). Larger K means less communication, which is cheaper, but the stale copy drifts further from the truth, and that drift is what threatens training stability. Staying stable as K grows is the real target; everything below is in service of it.

The initial idea: go deep on weight projection

The bet: rather than wait for a fresh copy (or refresh more often), the circuit holding the stale copy could locally extrapolate the partner’s weights forward along their recent trajectory, a cheap prediction that keeps it close even at larger K. If a stale copy can predict where the partner is heading, staleness stops mattering. That is the bet we went deep on.

What we did

A focused, offline weight-projection study on real saved training trajectories: does the bet hold, how far ahead does it hold, on which parts of the network, and on which datasets, answered before committing anything to the live system. Everything below is a pure replay of saved weights; no training was run to produce these numbers.

3. Setup

The two datasets

Same model (a 1.5B instruct model), same RL recipe (vanilla GRPO), two training datasets: an easier one (grade-school math word problems, “GSM8K”) and a harder one (filtered competition-style math, “MATH”). We look at both on purpose, a trick that only works in the easy regime is not one we can ship.

How we collected the checkpoints

During each run we saved the complete set of weight matrices (all 338 of them) at regular snapshots, 160 across the easier run, 50 across the harder run (spaced ~20 steps apart, 1000 steps total). All analysis is on the fp32 trajectories; we deliberately excluded lower-precision traces so the numbers reflect real weight motion, not quantization noise.

4. How to read the numbers

The projection ratio: “did extrapolating help?”

For a stale copy at time t, doing nothing means keeping θ_t; its error is the distance to the true future weights, ‖θ_t − θ_future‖. A prediction θ̂ has error ‖θ̂ − θ_future‖. The projection ratio is simply

ratio = predicted error ÷ do‑nothing error

• Below 1 → the prediction beats doing nothing (real skill).
• Exactly 1 → no better than doing nothing.
• Above 1 → actively worse than doing nothing.

Median, and the p10-p90 band

We never score a single checkpoint pair. For each setting we slide the prediction window across the whole trajectory and report the median ratio over all windows, with the 10th-90th-percentile band as the spread. Median (not mean) so a few pathological windows can’t dominate; the band shows how consistent the effect is.

5. The projection methods, and which one wins

Each method is a rule for turning a few recent snapshots into a guess for a future one, graded by the ratio above and always out-of-sample (any tuning is chosen only from windows strictly earlier than the one being scored, no peeking at the answer). This is the definitional reference for the rest of the doc.

The methods as equations

Let θt be the weights at step t, and predict h steps past a stale anchor at t. With reach κ = h/Δ:

So λ=0 recovers hold-stale, λ=1 recovers naive linear, and μ=0 drops the curvature term. The whole ladder is one family with two knobs (λ for the slope, μ for the curvature); “which method wins” is really “which knobs to keep, and whether to fix them or learn them.”

How the adaptive rule learns its step size (rolling least-squares / MLS)

The self-correcting arms do not fix λ; they re-estimate it online from the most recent real checkpoints. Once the true θs+h for an earlier window s has arrived, it becomes a label: we fit the λ that would have predicted it best, then apply that λ to the current window. Over the last K completed windows this is a one-parameter least-squares fit with a closed form (the projection of the realized move onto the extrapolation step):

We tried a family of these: bounded rolling least-squares over the last K checkpoints (rolling-LS / moving least-squares, the best arm at K=5), an expanding-window LS, EWMA and momentum smoothing of λ, a two-parameter version that also fits the curvature μ, and warm-up / clip schedules, all fit strictly out-of-sample. On the frozen trace λ genuinely keeps adapting (it drifts 0.19→0.36 over training, and tracks higher, ~0.5, at short spacings) and never collapses. It just does not buy enough here to beat a single constant (next).

Reading the chart: why the fixed rule wins even though it is not the lowest bar

A fair question from Figure 1: the fixed damped-linear bar (0.940) is a hair longer than the adaptive bars (0.935-0.938), so why is it the winner? Because the winner is the simplest method that captures essentially all the available skill, not the numerically lowest one. The best adaptive arm beats the fixed rule by at most 0.0045 in ratio, which is below the 0.01 margin we require before paying for extra machinery, while it carries online state (re-fitting λ every step) and gives no better prediction accuracy. In skill-over-do-nothing terms (how far below 1.0 the ratio sits) the fixed rule recovers about 93% of the best arm’s advantage with a single constant λ≈0.3 and zero online fitting, so prefer-simplicity selects it. This is a verdict about a frozen trace; the live system is a different regime (see the box below).

Every arm clears ratio < 1; the adaptive arms are lower by ≤ 0.0045 (sub-margin) at the cost of online state. GSM8K operating point (per-step Δ=10, h=10, global row). Curvature and undamped variants sit above 1.0 and are omitted here (see Figure 1).

What the ladder says

• First-order beats second-order, decisively. The extra curvature parameter fits noise and is driven to zero out-of-sample; both second-order variants are worse than doing nothing.
• Damping is essential. A single global λ≈0.3 converts the harmful naive line into the winner (ratio 0.94).
• Adaptive is real but marginal here. The self-correcting arms are lower by at most 0.0045 (sub-margin) and only by paying for online state, so prefer-simplicity keeps the fixed one-scalar rule (see the table and “Reading the chart” above). This is a frozen-trace verdict, not a claim that learnable correction is useless live (box below).
• The whole network cooperates. On the easier dataset a single global scalar is enough, every layer family (attention Q/K/V/O, MLP gate/up/down, norms, biases, embedding) projects safely and none breaks the win; embedding, biases and value-projections are the most predictable, the attention query/key projections the least.

6. The cross-dataset verdict (fp32): easier vs harder

The headline: the exact same projector that wins on the easier dataset falls back to “do nothing” on the harder one: because on the harder task the per-step learning is very noisy, so the weight path is far less coherent (more on why below).

Why MATH looks random, and why it may still be workable

“Looks random.” The 0.15 coherence is uniform: the same ~0.15 across every anchor spacing, every layer family, and every phase of training. At the operational staleness (checkpoints ~20 steps apart, horizons of ~100-200 steps) there is nothing for a projector to gain over do-nothing, and the biases family is actively harmed by any projection.

Why the harder path is more random, and why this is NOT just “projection fails.” Three things to hold together. (1) Scope: the 0.15 is a step-to-step (consecutive-update) number. The whole-trajectory (cumulative) motion stays far more coherent than that: its per-matrix linearity R² is ≈0.62 on MATH vs ≈0.68 on GSM8K: barely different across datasets, whereas the step-to-step number drops sharply from 0.86 to 0.15 (the per-step learning on MATH is very noisy; Figure 4). On the easier set the SVD section (below) goes further and shows the cumulative motion is ~99% rank-1 with a near-linear coefficient (R² ≈ 0.98); we ran that rank-1 decomposition only on the easier trace, so on MATH we lean on the trajectory-R² comparison, not the 99% figure. Either way, the coarse direction of travel stays coherent on both datasets; it is the fine, per-tick increments that go noisy on the hard one. (2) Prior work already shows projection improves the metric that matters (downstream performance) on essentially this setup (a small model, GRPO/RLVR, math): Wang et al. 2026 (“Linear Dynamics in the RLVR Training of LLMs”) find weight and output trajectories are strongly linear (R² > 0.7) and that simple linear extrapolation matches RL at 6.1× less compute in weight space and beats it by +4.2% in output space; Wei et al. 2026 (“You Only Need Minimal RLVR Training”, RELEX) show a rank-1 SVD extrapolation from as little as ~15% of the steps matches or exceeds full RLVR on MATH (including on Qwen2.5-Math-1.5B) and generalizes out-of-domain, training-free, with the SVD acting as a spectral denoiser. (3) Capability: RELEX used a math-specialized base (Qwen2.5-Math-1.5B); we used a general 1.5B-Instruct on the harder Big-Math set. A general small model at the edge of hard competition math gets a sparser reward signal and noisier, less directional updates, consistent with both papers’ premise that RLVR mostly elicits and amplifies capabilities already latent in the base model. Read together: step-to-step coherence (hence short-horizon projectability) tracks how well the model can actually learn the task and should rise with a stronger / task-aligned base or an easier curriculum, so “do-nothing wins on the harder set” is as much about a general small model on hard math as about the method.

Why it could still work (from prior work and our own data):

• There is structure at the freshest anchor. When the stale copy is only one checkpoint old (~20 steps) and we predict a step or two ahead, projection does beat do-nothing (ratio ~0.99, +2% skill, statistically clean). The coherence is short-lived, not absent, and the prior work above lives in exactly this early-prefix / coarse-direction regime (RELEX observes ~15% of the run, then extrapolates far ahead). Shorter refresh + short-horizon projection is the version that transfers.
• Weight-space accuracy ≠ task performance. Our ratio measures raw weight-space fidelity (L2); Wang et al. and Wei et al. validate on downstream task accuracy and win there on this exact kind of setup, and the two metrics can diverge. A weight-space ratio near 1.0 does not prove there is no performance-preserving correction, only that a naive raw-weight predictor has no room at this short-horizon staleness. The downstream-performance test is the one that decides, and it is still open on our runs.
• The coherent part lives at a coarser scale. As the next section shows, the cumulative direction of travel is far more coherent than the step-to-step increments; even when 20-step increments look orthogonal, a coarse-grained / low-rank correction can still hold. This is exactly the door RELEX walks through (rank-1 projection as a spectral denoiser), and it is still open on the harder set.

Takeaway: projection is coherence-gated, not universal. Measure coherence online and switch, where it is high, project with the cheap fixed rule; where it is low, fall back to shorter refresh / hold-stale and drop the fragile families (biases).

7. The SVD / rank-1 direction work, and what it opens up for correction

Separately, we asked a sharper structural question: does each weight matrix really move along one direction? For every matrix we stack its changes-from-start across all saved checkpoints and run a singular value decomposition (SVD) on that little stack, done cheaply in a tiny checkpoint×checkpoint domain, never forming a full-size vector. The top singular direction v₁ is the one direction the matrix has mostly moved along; the coefficient c(t) says how far along it each checkpoint sits. This trajectory-SVD construction follows Wei et al. 2026 (RELEX); we re-implement it and test it on our own runs.

What we found

• The motion is almost one-dimensional. One direction captures ~99% of the energy of each matrix’s accumulated change (~92% over the full run, near 100% over short spans), uniformly across attention, MLP, norms and biases. The coefficient advances at a near-constant rate (~99% steady, R²≈0.98).
• But the direction slowly rotates. Consecutive checkpoints are ~0.99 aligned, yet the cumulative alignment to the first direction falls to ~0.64 over the full run. So v₁ is a slowly moving target over long spans.
• Consequence for staleness. The textbook far-ahead projection along a fixed early v₁ (an absolute end-checkpoint reconstruction objective) is the wrong tool here, the direction has rotated away by the time you extrapolate, so it is 4-22× worse than doing nothing at a one-step horizon. It predicts accumulated position, not the next increment. Short-horizon staleness is better served by a momentum-like signal, with the anchor-pinned rank-1 line as a safe fallback.

8. Caveats & open questions (the catches)

• Coherence-gated, never automatic. Projection helps only where the path is locally straight and consecutive steps keep pointing the same way. Conditional, not a fixed on-switch.
• Dataset-dependent. A clear winner on one dataset was worthless on another. Any deployment must carry the online gate.
• Small-model caveat. This is a 1.5B model; linearity and coherence may look different at larger scale. Validate before trusting broadly.
• Weight-accuracy ≠ model performance. We measured weight-space fidelity; the ratio is a proxy. Wang et al. and Wei et al. validate on downstream task performance (and show real gains on this kind of setup), and the two can diverge, so our weight-space verdict is necessary but not sufficient; the downstream test is still open and is the one that ultimately decides whether the trick buys stability at longer K.

9. Next steps

• Wire the online coherence gate (consecutive-update alignment, or the recent-trajectory R²) into the anchor: fixed damped-linear (λ≈0.3, freshest anchor) where coherence is high; shorter-refresh / hold-stale where it is low; exclude biases on low-coherence regimes.
• Stand up the rank-1 trajectory state (v₁ + two scalars per tensor) as the spectral-correction / compression basis, and test a learnable low-rank correction against the fixed rule live.
• Move from weight-space ratio to the real test: does gated projection let us push K larger without losing training stability or final task accuracy? Measure on downstream performance, on both datasets, then at larger scale.

References

1. Wang, T., Liu, J., Wu, Z., Jin, S., Chen, W., Xu, H., Miao, N. (2026). Linear Dynamics in the RLVR Training of Large Language Models. arXiv:2601.04537. RLVR weight and output trajectories are strongly linear (R² > 0.7); linear weight extrapolation matches RL at 6.1× less compute, and output-space extrapolation improves downstream accuracy by +4.2%.
2. Wei, Z., Zhu, X., Chen, W.-L., Huang, C., Huang, J., Meng, Y. (2026). You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (RELEX). arXiv:2605.21468. RLVR weight deltas are extremely low-rank; the rank-1 SVD coefficient is near-linear (R² > 0.98); extrapolation from as few as ~15% of steps matches or exceeds full RLVR on MATH (incl. Qwen2.5-Math-1.5B) and out-of-domain, training-free, with the SVD acting as a spectral denoiser.