The week in short
- What we explored. The scale ladder: the agentic multi-turn task on the small model, a compressed-against-dense pair one size up, and the largest model at long context.
- What we found. The system holds where generations stay short and single-turn, and breaks wherever the per-token error has room to accumulate, the dense controls staying healthy throughout.
- What we think the cause is. The trainer scores tokens through the compressed pipeline, the sampler through a clean one, and the compounding disagreement slowly distorts the weights.
- The idea we discussed. Correct the weight-space distortion itself. As a quick first probe we swapped the trainer's optimizer state from the anchor, preliminary, and untested at larger scale.
Up the scale ladder
Three rungs, each giving the same per-token error more room to accumulate.
1.5B, agentic, up to fifty interactions per trajectory
Every interaction trains as its own row and feeds the next turn's prompt. The compressed run scored well for a few steps, fell through zero by step six, and sat pinned to the formatting-penalty floor from step seventeen, where the gradient is zero. The dense control stayed positive throughout. Full analysis in the agentic training report.
4B at 4k context
The controlled version: Qwen3-4B on MATH, same base checkpoint and data order, one arm compressed and one dense. The dense control climbs faster over the first fifty steps. The compressed arm catches up, and the two curves track each other until about step 150. Then the compressed arm breaks in two phases, responses growing until they truncate at the generation cap, then turning short and wrong as the score falls away. The dense control trained on, healthy, for nearly three hundred steps.
8B at 16k context
An 8B model with a 16k generation budget climbed to a high plateau, held it deep into the run, then gave everything back in the last twenty steps. The spark was a length feedback loop at the generation cap, the fuel the same mismatch over far more tokens per response. Details in the longer-context collapse report.
One instrument note. The reference KL rose smoothly through every collapse and also rises in healthy runs, so it is not a health gauge. Policy entropy separates the arms within the first ten steps.
What we think is going on
In our two-circuit networks a log-probability difference between rollout generation and the trainer is inevitable. The rollouts are generated dense, the trainer computes its per-token log-probabilities through the compressed pipeline, and because the compression is lossy by design a lossless boundary codec is not an option.
Inside a band that difference is tolerable. Bigger models and longer contexts let the error accumulate over more tokens and more boundaries, and the distortion corrupts the weights even though each weight update is almost minimal. So we discussed that reducing the weight-space distortion is the key, and ran an initial test of one idea in that direction.
A starting point for correcting the weight space
The quick idea from our discussion: at a fixed cadence, overwrite the trainer's AdamW moments, a memory of every distorted gradient, with moments the anchor maintains from clean dense replay gradients, norm-matched to the compressed scale. The weights are never touched.
On the same 4B setup we ran a first 500-step test with this swap and the collapse did not appear. This could be a starting point for correcting the weight space. It is one run at one scale, and it needs proper testing, including at 8B and 16k.
Training reward with and without the swap
The actual logged training-reward curves at 4B. Same base checkpoint, same data order.
At the matched step 200 checkpoint the swap arm sits near the dense control on the ten-benchmark suite, 0.360 against 0.389 from a base of 0.281.
Capability at step 200
In-domain left of the divider, out-of-domain right. The collapsed arm is absent by design.
Next steps
- Test the optimizer-state swap at 8B with 16k generations, then on the agentic task.
- Sweep the cadence and try partial blends of the anchor moments.
- Measure the weight-space distortion itself, against a dense shadow.
- Watch policy entropy and the response-length slope as the live health gauges.