Two-circuit RLVR on long runs: it learns to full strength, then collapses

Weekly update · week of 2026-07-13 · self-contained for other teams · base Qwen2.5-Math-1.5B on MATH · follows the correct-base-models readout.

Draft · fix runs still in flight

1. This week, in one line

We took the two-circuit compression method to long (8 to 10 hour, 200-step) runs on a real base math model, the standard setting the efficient-RLVR literature uses. Two results:

The good news. The compressed system genuinely learns the task to the same strength as an uncompressed run: validation accuracy 0.667 versus the uncompressed 0.674. (see §3)
The problem. Past roughly 150 steps it collapses, and the short-horizon tuning we did earlier did not prevent it. (see §4)
Why. The evidence points to a train/inference mismatch that compression makes about 300 times larger than normal (not to reconstruction error or generic over-optimization), though whether that mismatch drives the collapse or is a symptom of it is still open. (see §5, §6)
Now running. A one-knob-per-run sweep testing the two obvious handles: correct more often, and leash the drift harder. (see §7)

2. The setup, for anyone new to this

We train a language model with reinforcement learning on math problems that have checkable answers. This is RLVR (reinforcement learning from verifiable rewards), and the RL algorithm is GRPO (Group Relative Policy Optimization). To cut the communication cost of distributed training, we split the work into two cooperating circuits:

Fast circuit. Compresses the weight updates and activations (a low-rank PowerSGD scheme plus a rank-1 weight projection) so that far less data crosses the network.
Anchor circuit. Every so often it replays one dense, uncompressed update, pulling the fast circuit back toward where a normal uncompressed run would be. It exists to correct the drift that compression introduces.

We evaluate on Qwen2.5-Math-1.5B and the MATH dataset on purpose: this exact model and dataset are what the efficient-RLVR literature reports on, so our numbers line up with published baselines and we can measure drift precisely. Over the past weeks we established that the anchor and the weight projection genuinely help, and settled on the best, most stable settings for the knobs they carry (anchor freshness and sign-strength, the projection window, the fire cadence). This week pushed those settings to a longer, harder horizon.

Figure 1. The two-circuit system (schematic). The sampler runs the true weights and generates the answers; the trainer computes the GRPO gradient through the compressed weights. GRPO therefore improves one policy using data drawn from a slightly different one. The anchor periodically replays a dense update to pull the two back together. The rest of this note is about what happens when the gap between them grows.

3. It works: the compressed run reaches full strength

Through the first ~150 steps the compressed two-circuit run sits right on top of the uncompressed reference. Training reward peaks at 0.76 (uncompressed reference 0.72), and validation accuracy on MATH reaches 0.667, level with the uncompressed ceiling of 0.674. So the method learns the task at full strength on a real base model. This is the result we most needed to see: the anchor-plus-projection design does its job through the productive part of training.

Reward and validation: compressed tracks uncompressed to the peak, then val drops to 0.561 — **Figure 2. It learns, then it breaks.** Left, training reward (batch accuracy): the compressed two-circuit run (blue) tracks the uncompressed reference (green) and peaks at 0.76, then craters in the shaded window. Right, validation accuracy on MATH: the compressed run reaches 0.667 at step 150, matching the uncompressed ceiling of 0.674, then falls to 0.561 by step 200.

4. But on long runs it collapses

Keep the same run going and past step ~150 it falls off a cliff. Validation drops from 0.667 to 0.561. In a sharp window around steps 162 to 173 the training reward craters from 0.71 to 0.08, then recovers partially into a degenerate state: reward back near 0.58, but with very low-entropy, repetitive outputs. In the long runs we have taken to this horizon so far this has recurred, and the short-horizon tuning we did earlier delayed it but did not remove it. Whether any single setting removes it is exactly what the sweep in §7 is running to find out.

The crash coincides with an anchor fire, but we do not over-read that

The collapse lands next to the step-160 anchor fire. The anchor fires every 10 steps, though, so any crash is bound to fall near a fire, and two details keep this a hypothesis rather than a conclusion: instability was already building a few steps earlier (gradient norm climbing over steps 157 to 159), and earlier fires, including the one at the step-150 validation peak, caused no crash. So a late, stale anchor correction landing on an already-drifting policy is a plausible trigger we have not yet isolated. The one-knob cadence run in §7 tests it directly.

What the crash window does show clearly: the trainer's codec-view KL from the base model jumps from about 6.5 to 9.8 nats (a nat is the natural-log unit for KL; larger means further from the starting model), gradient norms rise into the high tens (and keep spiking through the degenerate phase, peaking near 190 around step 185), and mean output length balloons transiently from about 550 to over 900 tokens.

Four panels: reward, codec-view KL, gradient norm, response length break after step 160 — **Figure 3. Anatomy of the collapse.** Compressed base-Math, steps 120 to 200; grey lines mark anchor fires, the orange line is the step-160 fire the crash coincides with. Reward craters, the codec-view KL from base climbs, gradients spike (worst later, around step 185, in the degenerate phase), and outputs balloon transiently.

5. What it is not

Not reconstruction error (how much the compressed weights differ from the true ones). A second model whose activations compress about 6 times cleaner (reconstruction error flat around 0.05, versus 0.2 to 0.55 for the math model) collapses too, in fact harder (validation 0.545 to 0.11). Low compression error does not buy stability.
Not generic over-optimization. The uncompressed run reaches the same reward as the compressed one while climbing steadily, with tiny gradients and staying essentially on top of the base policy, so the task does not require the model to drift far from base. One honest limit: our uncompressed reference only runs to step 100, so it rules out over-optimization through the productive phase, not through the crash zone itself. The in-window uncompressed baseline now running (§7) is what will settle that.

Honest caveat

The cleaner-compression model is instruction-tuned, so part of its validation drop may be plain instruction-forgetting, and we do not yet have an uncompressed control for it. The point it makes, that clean reconstruction does not prevent collapse, still holds; the exact size of its drop is not the claim.

Reconstruction error 6x apart, both models collapse in validation — **Figure 4. It is not a reconstruction problem.** Left, weight reconstruction error: the base-Math model (blue) is about 6x higher and ratchets upward, while the cleaner-compression model (orange) stays flat near 0.05. Right, validation accuracy: both collapse anyway, the cleaner model more steeply.

6. What we think is going on: a train/inference mismatch that compression amplifies

We generate answers with a fast inference engine (vLLM) running the true weights, then compute the GRPO training signal through the compressed model. Those are two slightly different policies. We measure the gap directly, at the exact tokens the model sampled:

Uncompressed. The probability the trainer assigns to a sampled token differs from the sampler by 0.002 on average; the two are 0.999 correlated; their KL is about 0.0003 nats. Train and inference are effectively the same policy.
Compressed. That gap is about 0.77 at the very first step and stays around 0.7 throughout (a structural offset from the compression, present before any drift accumulates), the correlation is only about 0.14, and the rollout-versus-trainer KL sits at 5 to 13 nats.

So compression turns a 0.002 mismatch into a roughly 0.7 mismatch, about 300 times larger. Here is the honest nuance, and the part we find most interesting: that large standing mismatch is compatible with learning. It stays around 0.7 through the entire productive phase, and is actually highest (about 0.83) right at the reward peak. So the level of the mismatch is not what trips the collapse. What coincides with the collapse is a late escalation: the sampler-trainer probability gap climbs to 0.93, their correlation falls to zero and briefly negative, and the rollout-versus-trainer KL explodes to about 31 nats. That is the training policy tearing loose from the data it is trained on.

Our working picture: compression installs a large, permanent gap between the policy we sample from and the policy we train; the KL leash we run (coefficient 0.001) is far too weak to bound it; and the anchor corrects it only every 20 optimizer ticks, too rarely and too stale. Past some point the gap escalates and the run collapses. We are explicit that we have not yet established the causal direction of that late escalation: it may be driving the collapse, or it may be reading out a policy that is already breaking for another reason. The fix runs are built to separate those.

Probability gap 300x larger under compression, correlation collapses to zero — **Figure 5. The mismatch, measured.** Left, the probability gap between sampler and trainer at sampled tokens (log scale): the uncompressed run (green) sits flat at 0.002 while both compressed runs (blue base-Math, orange cleaner model) sit near 0.7, about 300x larger, rising toward 0.9 at the crash. Right, the rollout-versus-trainer correlation: near-perfect (0.999) uncompressed, but only ~0.14 under compression and falling to zero at the collapse.

A note on the metrics

The trainer-side probability and KL numbers, including the codec-view KL from base in §4, are read through the compression path, so we treat them as relative signals across runs rather than absolute nats. The direction of the effect and the ~300x gap are robust; the exact magnitudes are view-dependent. Reward and validation are measured the same way in every run and are directly comparable.

7. What we are testing right now

On the operator box, one knob per run, 200 steps each, to separate the levers:

Run	What it changes	Question it answers
Uncompressed reference (Math, 200 steps)	compression off	a clean in-window baseline through the crash zone
Anchor twice as often	fire every 10 ticks instead of 20, half the staleness	does correcting more often and less stale beat the drift? (primary fix)
Same, projection off	as above, minus the weight projection	is the weight projection protecting against drift, or causing it?
Stronger KL leash	KL coefficient 0.01, ten times higher	does binding the drift directly hold it?
Wider clipping	raise the GRPO clip ceiling (the importance-ratio clipping bound) to 0.28	does more exploration room prevent the degenerate lock-in?

From the RL side these are the standard handles: a stronger KL penalty, wider clipping, and more frequent correction. The open question is whether any single one is enough given how large the mismatch is, or whether we have to shrink the mismatch itself, for example with a higher compression rank or by splitting the pipeline so that less of the forward pass runs through the compression path.

8. Bottom line

The compression method works where it counts: on a real base-model math task it learns to the same strength as uncompressed training, which is the thing we most needed to show. The open problem is horizon. Past roughly 150 steps the run collapses, and it does so in the presence of a train/inference mismatch that compression makes about 300 times larger than normal, only weakly leashed and under-corrected by the anchor. Whether that mismatch is the cause or a symptom of the collapse is the question the runs in flight are built to answer: they test the two obvious handles, correct more often and leash harder. If neither holds, the next move is to shrink the mismatch at its source.

Base model Qwen2.5-Math-1.5B, MATH dataset, GRPO, 200-step horizon. Figures generated from the logged training metrics of the runs described above (uncompressed reference, compressed base-Math, compressed cleaner-compression model). Draft; fix-run results to be added as they land.