Anchor signal reuse builds the collapsing bias, and controlling it removes the collapse without an optimizer-state swap

The week in short

• What we explored. A weight-level autopsy of the collapsed run, and a family of designs that vary one thing only, how much of the anchor signal is consumed between its 20-step refreshes.
• What we found. The collapse is a small, directional shift at about 0.1 percent of weight movement, driven by a correction held frozen between refreshes. Using the anchor signal once per refresh prevents the collapse with no optimizer surgery, and the residual form used once is the best result of the program.
• What we think. Training under this mismatch tolerates large unbiased noise but not small persistent biases. The damage scales with how long an error persists, not with how large it is.
• The default we adopted. Use the anchor signal once per refresh. Every reuse design paid for its extra early speed later, either as collapse or as slow drift.

Where last week left off

Last week we reported that swapping the trainer's optimizer state from the anchor kept a 500-step run alive where the plain compressed run collapsed. We promised to measure the weight-space distortion itself and to explain why the swap helps. This week does both, and the explanation led to a cheaper fix.

The protocol is unchanged. Qwen3-4B base model, RL fine-tuning with GRPO on MATH, 4096-token context, 500 steps, one optimizer update per step. Boundary activations on the training path are compressed to 5 percent of coordinates. The anchor circuit fires every 20 steps, replaying the same batch densely at weights delayed by 20 steps to produce one uncompressed gradient, which we call the anchor signal M. The refresh period and the delay are set by the slow network path and are not ours to tune. Validation is greedy decoding on held-out MATH, and the single-run noise floor is about plus or minus 0.8 points, so differences under about 2 points are not meaningful.

Weight-space archaeology

Before changing anything, we asked what the collapse physically is. We compared how the weights move between healthy checkpoints against how they move across the collapse window.

The full checkpoint-level autopsy is ported into the hub as a specialist appendix, with the causal surgery that pins the damage down and every intermediate measurement. It keeps its internal terminology and is written for readers who want all the numbers: the weight archaeology report.

Why the optimizer-state swap prevents the collapse

In that first test the anchor circuit kept its own copy of the optimizer moments, built only from the uncompressed refresh gradients and updated once per refresh.

From time to time we overwrite the trainer's live moments with the anchor's copy, rescaled to the magnitude the live optimizer is running at. So the anchor holds a moving average refreshed at every fire, not a snapshot taken once. It helps because a correction that stays the same between refreshes accumulates coherently in the optimizer moments as a directional bias, and the swap wipes that bias out before it grows into damage. The weight archaeology above confirms this is the right picture, since the damage it found is exactly a small coherent drift.

The swap works, but it repairs the bias after the fact rather than preventing it, the overwrite replaces everything the optimizer had learned and not only the unwanted part, and it is surgery on the most delicate state in the trainer. That motivated this week's question. Can we prevent the collapse by changing how the current design consumes the anchor signal, without touching the optimizer at all?

Where the bias comes from

The swap treats the bias after it has accumulated. That points at the more direct question of what puts it there, and the answer is in how we consume the anchor signal between refreshes.

M refreshes every 20 steps. In the design that collapsed we applied sign(M) on every one of the 19 steps in between, so those 19 updates all carried the same correction direction. That correction is exact on the refresh step and increasingly wrong as the policy moves away from the weights it was measured at. This is the bias the optimizer integrates, and it suggests a cheaper fix than repairing the optimizer afterwards. Apply the correction once, on the refresh step, and let the other 19 steps run on the compressed gradient alone.

Two anchor signals, one pattern

If reuse is what causes the collapse, the way the signal is merged into the gradient should not matter much. We tested that with two different corrections built from the same anchor signal.

The sign form transplants the anchor's update signs onto the compressed gradient and sends only signs across the slow link. The residual form measures how wrong the compressed gradient was at the refresh step and adds that difference back, which on the refresh step returns the uncompressed gradient exactly. They are different operations on the same signal, and under reuse they behave the same way.

Both corrections collapse when the signal is reused, and both survive when it is used once

Greedy validation accuracy on held-out MATH, plotted every 100 steps. The left panel is the sign form, the right panel is the residual form, and the shaded bands mark where the reused designs collapsed.

The figure lines up with the hypothesis. Used once per refresh, the sign correction lands on par with the optimizer-state swap, so the cheap fix and the expensive one are equivalent. The residual correction used once is the best result of the program. Neither used-once run shows any sign of the drift that killed its reused twin, and neither needed the optimizer touched.

A third correction consumed the same reused signal as a bounded convex mixture and confirmed the pattern from another angle, degrading quietly rather than collapsing. Its training vitals raised no alarm at all, since response length stayed normal, clipping stayed near zero and the rollout log-perplexity actually fell, so only greedy validation caught it.

Controlling how much of the signal is used

Since the collapse does not depend on which correction we use, the question that matters is how to use the anchor signal between refreshes. Use once and reuse every step are the two ends of one dial, so we tried the middle.

The annealed design keeps applying the correction between refreshes, but decays its weight with the number of steps since the last refresh.

We ran the annealed design with the residual correction, and by the argument above the choice of correction should not matter much. It is the first design that reuses on every step and still finishes all 500 steps. Its early speed sits between the two ends, and it pays for that afterwards with a slow drift in validation. We also ran a smaller dose, a single extra application ten steps after each refresh, and it finishes as well with the same shape. Decaying the dose stretches the timeline of the damage without removing it.

Middle doses of the reused signal all finish 500 steps

Greedy validation accuracy on held-out MATH for the two middle-dose designs, with the used-once design as the reference.

Reading this figure with the one above, the dose buys the first hundred steps and not the last four hundred. Every design that reused more of the signal arrived earlier and then either collapsed or drifted, and by step 500 all the finished designs sit within the noise floor of each other.

Here C(M, t) is whichever correction is in use, so both designs we ran are special cases of one expression and the only change is that w becomes a small set of parameters trained with the policy. That lets the schedule differ by layer and change as training proceeds, instead of being one number we picked. The same weight applies to the sign form directly, and that is where we want it to work first, because sending signs is the lightest traffic the slow link has to carry.

Current observations

Training under this train-inference mismatch tolerates large unbiased noise and does not tolerate small persistent biases.

The compression inflates gradient variance heavily, and every design that limits reuse survives it, because independent errors average out across steps. A reused correction is different. Its direction stays the same for every step of the interval, so those errors are perfectly correlated and add up in the optimizer state instead of cancelling. The damage scales with how long an error persists rather than with how large it is.

Related evidence for the same model family appears in arXiv 2602.01826, where the mismatch grows with response length and annealing the update stabilizes training. Our learning rate is fixed by the protocol, so we annealed the correction weight instead.

• Use the anchor signal once per refresh as the default.
• Learn the per-step weight instead of fixing the decay by hand, starting with the sign form.
• Repeat the reuse ladder on bigger models and harder tasks.

Related reading

• Last week's report introduced the optimizer-state swap and left the promises this week picked up.
• The weight archaeology report is the ported specialist appendix behind the autopsy, with the full causal surgery.
• The two-circuit long-horizon weekly explains why the two-circuit method survives long runs in the first place.
• The earlier long-run collapse weekly is the first anatomy of a collapse in this program.
• The system status page is the one-page description of the two-circuit design and its codec.
• The longer-context collapse report covers the length feedback loop that reappears in this week's reuse collapse.