What happened
Through the first 150 steps the run was healthy. Training score rose from 0.57 to 0.85, and held-out MATH accuracy rose 9.0 points, from 0.711 to 0.802, roughly 4.7 sigma on 499 problems. Between steps 170 and 189 the training score fell from 0.85 to 0.39, below where it started. The run was stopped at step 189 of a planned 1000, with logging continuing through step 191.
A · Training score per step
Held-out evaluations marked in green. Faint vertical ticks mark anchor replay steps.
The run, measured
Every logged step, no smoothing. The rose band marks the collapse window, steps 170 to 189, and the faint ticks mark the slow dense circuit's replay steps in every panel.
B · Divergence from the reference policy
Nats per token, rising steadily from 0.09 to 2.75 across the run.
C · Response length, batch mean and batch maximum
The mechanism panel. Length grows into the ceiling, then falls away with the score.
D · Truncated fraction
Share of samples that hit the generation limit.
E · Gradient norm
Falls from about 5 to about 1.2, then spikes late.
The spark: a truncation cliff
Generations lengthened as the policy learned, until up to 8 percent of samples hit the generation limit. A truncated sample scores zero even when its reasoning was sound. Advantages are computed within each prompt's group of samples, so this penalised length itself.
The correction overshot. The longest sample in a batch fell from the 15360 limit to under 4000, the long chains that had been earning reward disappeared, and accuracy went with them. Entropy stayed flat throughout, so this is lost capability rather than a policy going diffuse.
The fuel: two views of one policy
Samples are generated from dense weights. The training forward pass, the recomputed old log-probabilities and the reference pass all run under the sparse boundary mask. The two views differ by construction.
The trainer could not see the drift it was causing. Its own ratio divergence read exactly zero at every step, while divergence from the reference reached 2.7 nats, roughly two orders of magnitude beyond a dense run at comparable steps.
Per-token mismatch compounds along the sequence, so a 16k-token generation accumulates several times what a 3k generation does. Longer context both amplified the mismatch and created the truncation cliff in the first place.
The same gap at a safe length: a controlled comparison
The 16k run is not the only evidence on the sampler-trainer gap. The smaller 1.5B model was trained for 600 steps at a 2048-token generation limit twice under the same recipe, including the anchor replay on the slow dense circuit, once dense and once with the same 95 percent activation sparsity. The compressed run carried a gap near 14 nats, nearly five orders of magnitude above its dense control, for the entire run. It still matched dense on training score, 0.746 against 0.747 over the final 20 steps, and finished within 1.6 points on held-out accuracy, 0.661 against 0.677. The 16k run's gap was only about 25 percent higher and it collapsed. The gap is the enabling condition, not the trigger.
F · The sampler-trainer gap, three runs
Train-inference mismatch in nats per step, log scale.
Reference divergence is what separates the two compressed runs, and it does so long before any score moves. Over the shared window of steps 1 to 189 it grows at 0.0147 nats per step at the 16k limit against 0.0016 at the 2048 limit, nine times faster at only a quarter more mismatch. The dense control accumulates 0.016 nats in 600 steps.
G · Divergence from the reference policy
Nats per step on a shared axis. The 16k line ends where the run was stopped.
What separates the length regimes is where each policy starts relative to its own limit. At 2048 the policy starts at about 38 percent of the limit with 11 to 12 percent of samples already truncated, and training shrinks generations away from the cap, so truncation decays to a stable 1 to 2 percent. At 15360 the policy starts at 5 percent of the limit with almost no truncation, then grows mean length 4.9 times into the cap. Truncation passes 1 percent at step 65 and peaks at 8.2 percent at step 159, about ten steps before the collapse begins. The cliff is a rising signal that arrives only once the policy has learned to write long. It cannot form when the tail is pressed against the limit from the first step and the pressure only falls.
H · Length against each run's own limit
Batch mean generation length as a share of the limit.
I · Truncated fraction
Share of samples cut at the generation limit.
J · Three runs at a glance
| Run | Generation limit | Sampler-trainer gap | Training score | Held-out accuracy | Outcome |
|---|---|---|---|---|---|
| 1.5B dense, 2k limit | 2048 tokens | 0.0002 nats | 0.747 at step 600 | 0.677 at step 600 | Stable for 600 steps |
| 1.5B compressed, 2k limit | 2048 tokens | 14.4 nats | 0.746 at step 600 | 0.661 at step 600 | Stable for 600 steps |
| 8B compressed, 16k limit | 15360 tokens | 17.9 nats | 0.872 peak, 0.453 at stop | 0.802 at step 150 | Collapsed near step 170 |
One honest caveat. Model size, model family and generation limit all changed together between the small runs and the 16k run, so the collapse cannot be pinned on any one of them from this data alone. The 16k run's reference divergence also grew steadily from the opening steps, long before the truncation spike, so truncation was at most the trigger and not the sole cause. And the short-limit runs truncated 11 to 12 percent of samples early and still 1 to 2 percent at the end without collapsing, so the level of truncation by itself does not predict failure. The single-variable run that resolves this is the larger model at the 2048 limit, with the smaller model at the 16k limit as its mirror.
What this does not blame
The slow dense circuit. All nine replays used the correctly delayed batch with bounded retained state, and the decline began between replays rather than at one.
Model scale as a defence. When earlier variants of the compressed recipe failed on the smaller model they turned over near step 200, and this larger run turned over near step 180, so size did not buy stability and longer generations plausibly cost some. The comparison above shows the current recipe holding for 600 steps at the smaller scale.
What holds up
Under 95 percent activation sparsity the method still produced a 9-point held-out gain at step 150. Compression did not prevent learning.
The failure is an interaction between sequence length and the sampler-trainer gap, not evidence that the compressed path cannot train.
What a rerun changes
- Token-level importance weighting, to close the ratio blindness. Earlier smaller-scale tests cut reference divergence by 40 to 78 percent.
- A generation limit set with headroom above where length naturally settles, so the truncation cliff is never reached.
- Frequent checkpoints, because on this run the best policy was never saved and is unrecoverable.