Back to the hub · longer context · 2026-08-05

Compressed reinforcement learning at 16k

Larger model, longer context: where compressed RLVR collapses

The same training circuit that holds steady on a small model at short generation length was moved to a larger base model and a 16k generation budget. It learned well for 150 steps, gaining nine points of held-out accuracy, and then gave all of it back in twenty. The trigger was not the compression budget on its own. It was a feedback loop on generation length that the compressed training path had no way to see.

What happened

Through the first 150 steps the run was healthy. Training score rose from 0.57 to 0.85, and held-out MATH accuracy rose 9.0 points, from 0.711 to 0.802, roughly 4.7 sigma on 499 problems. Between steps 170 and 189 the training score fell from 0.85 to 0.39, below where it started. The run was stopped at step 189 of a planned 1000, with logging continuing through step 191.

A · Training score per step

Held-out evaluations marked in green. Faint vertical ticks mark anchor replay steps.

Training score per step with two held-out validation points Training score rises from 0.54 at step 1 to about 0.85 by step 150, then falls to 0.37 by step 189. Held-out accuracy is 0.7114 at step 0 and 0.8016 at step 150. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 collapse starting level 0.565 step 0 held out 0.711 step 150 held out 0.802, up 9.0 points faint ticks: anchor replay, every 20 steps 0 25 50 75 100 125 150 175 step
Nothing in the first 150 steps signals what is coming. The decline begins between anchor replays, not at one.

The run, measured

Every logged step, no smoothing. The rose band marks the collapse window, steps 170 to 189, and the faint ticks mark the slow dense circuit's replay steps in every panel.

B · Divergence from the reference policy

Nats per token, rising steadily from 0.09 to 2.75 across the run.

Divergence from the reference policy per step Divergence from the reference policy climbs smoothly from 0.09 nats early in the run to 2.75 nats at step 189, while the trainer's own ratio divergence reads exactly zero at every step. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 collapse the trainer's own ratio divergence read exactly 0.0000 at every one of these steps 2.74 nats at step 189 0 25 50 75 100 125 150 175 step
The trainer's own ratio divergence stayed at exactly zero for every step of the run. Inside its masked view of the policy, nothing had moved.

C · Response length, batch mean and batch maximum

The mechanism panel. Length grows into the ceiling, then falls away with the score.

batch maximum batch mean
Response length per step, batch mean and batch maximum Batch mean response length climbs from about 800 tokens to about 3900 tokens by step 160. The batch maximum sits at the 15360-token generation limit for most of the run. After step 176 both fall away, the maximum to about 5000 tokens and the mean to about 1500. 0 4k 8k 12k 16k collapse generation limit 15360 batch maximum batch mean 0 25 50 75 100 125 150 175 step
Mean length climbs from about 800 tokens to about 3900 while the longest sample in each batch sits pinned at the limit. Both fall away exactly as the score collapses.

D · Truncated fraction

Share of samples that hit the generation limit.

Fraction of samples that hit the generation limit The truncated fraction stays under 0.01 for the first 100 steps, rises through the 140s and peaks at 0.082 near steps 157 to 160, then falls to exactly zero from step 176 onward. 0.00 0.02 0.04 0.06 0.08 collapse peak 8 percent 0 50 100 150 step
Peaks at 0.082 near steps 157 to 160, then reads exactly zero from step 176 onward.

E · Gradient norm

Falls from about 5 to about 1.2, then spikes late.

Gradient norm per step Gradient norm falls from about 5 early in the run to about 1.2 by step 150, then spikes to 3.76 at step 187 inside the collapse window. 0 1 2 3 4 5 6 collapse 3.76 at step 187 0 50 100 150 step
Policy entropy stayed flat across the whole run, 5.654 to 5.627, so the late spike is not a policy going diffuse.

The spark: a truncation cliff

Generations lengthened as the policy learned, until up to 8 percent of samples hit the generation limit. A truncated sample scores zero even when its reasoning was sound. Advantages are computed within each prompt's group of samples, so this penalised length itself.

The correction overshot. The longest sample in a batch fell from the 15360 limit to under 4000, the long chains that had been earning reward disappeared, and accuracy went with them. Entropy stayed flat throughout, so this is lost capability rather than a policy going diffuse.

The fuel: two views of one policy

Samples are generated from dense weights. The training forward pass, the recomputed old log-probabilities and the reference pass all run under the sparse boundary mask. The two views differ by construction.

The trainer could not see the drift it was causing. Its own ratio divergence read exactly zero at every step, while divergence from the reference reached 2.7 nats, roughly two orders of magnitude beyond a dense run at comparable steps.

Per-token mismatch compounds along the sequence, so a 16k-token generation accumulates several times what a 3k generation does. Longer context both amplified the mismatch and created the truncation cliff in the first place.

The same gap at a safe length: a controlled comparison

The 16k run is not the only evidence on the sampler-trainer gap. The smaller 1.5B model was trained for 600 steps at a 2048-token generation limit twice under the same recipe, including the anchor replay on the slow dense circuit, once dense and once with the same 95 percent activation sparsity. The compressed run carried a gap near 14 nats, nearly five orders of magnitude above its dense control, for the entire run. It still matched dense on training score, 0.746 against 0.747 over the final 20 steps, and finished within 1.6 points on held-out accuracy, 0.661 against 0.677. The 16k run's gap was only about 25 percent higher and it collapsed. The gap is the enabling condition, not the trigger.

F · The sampler-trainer gap, three runs

Train-inference mismatch in nats per step, log scale.

1.5B dense, 2k limit 1.5B compressed, 2k limit 8B compressed, 16k limit
Sampler-trainer gap per step for three runs on a log scale Both compressed runs hold a flat gap, near 14.4 nats for the 1.5B model and near 17.9 for the 8B model, while the dense control sits near 0.0002 nats, nearly five orders of magnitude lower. 0.0001 0.001 0.01 0.1 1 10 100 nearly five orders of magnitude 8B mean 17.9 1.5B mean 14.4 dense mean 0.0002 0 150 300 450 600 step
Two flat high lines and one flat near-zero line. The gap that survives 600 steps and the gap that collapses differ by about 25 percent, 14.4 against 17.9 nats.

Reference divergence is what separates the two compressed runs, and it does so long before any score moves. Over the shared window of steps 1 to 189 it grows at 0.0147 nats per step at the 16k limit against 0.0016 at the 2048 limit, nine times faster at only a quarter more mismatch. The dense control accumulates 0.016 nats in 600 steps.

G · Divergence from the reference policy

Nats per step on a shared axis. The 16k line ends where the run was stopped.

1.5B dense, 2k limit 1.5B compressed, 2k limit 8B compressed, 16k limit
Divergence from the reference policy per step for three runs The 8B compressed run climbs to 2.79 nats by step 191. The 1.5B compressed run reaches 0.93 nats at step 600 on a flattening path. The dense control stays near zero, at 0.016 nats by step 600. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 2.79 at step 191 0.93 at step 600 0.016 at step 600 0 150 300 450 600 step
Same recipe, same kind of gap, very different damage rate. Divergence from the reference grows nine times faster at the long limit over the shared 189 steps.

What separates the length regimes is where each policy starts relative to its own limit. At 2048 the policy starts at about 38 percent of the limit with 11 to 12 percent of samples already truncated, and training shrinks generations away from the cap, so truncation decays to a stable 1 to 2 percent. At 15360 the policy starts at 5 percent of the limit with almost no truncation, then grows mean length 4.9 times into the cap. Truncation passes 1 percent at step 65 and peaks at 8.2 percent at step 159, about ten steps before the collapse begins. The cliff is a rising signal that arrives only once the policy has learned to write long. It cannot form when the tail is pressed against the limit from the first step and the pressure only falls.

H · Length against each run's own limit

Batch mean generation length as a share of the limit.

1.5B dense, 2k limit 1.5B compressed, 2k limit 8B compressed, 16k limit
Mean generation length as a share of each run's own limit The two short-limit runs start near 38 percent of their limit and settle near 25 to 29 percent. The 16k run starts at 5 percent, climbs to 26 percent by step 169, then crashes. 0.00 0.15 0.30 0.45 0 150 300 450 600 step
The short-limit runs shrink away from their cap. The 16k run grows into its cap, then crashes 65 percent in 22 steps.

I · Truncated fraction

Share of samples cut at the generation limit.

1.5B dense, 2k limit 1.5B compressed, 2k limit 8B compressed, 16k limit
Fraction of samples truncated at the generation limit for three runs The two short-limit runs start at 11 to 12 percent truncation and decay to 1 to 2 percent. The 16k run starts near zero, spikes to 8.2 percent at step 159, then reads exactly zero from step 176 onward. 0.00 0.05 0.10 0.15 0 150 300 450 600 step
Two lines start high and fall. One starts at zero, spikes late, and pins to exactly zero after the collapse.

J · Three runs at a glance

Run Generation limit Sampler-trainer gap Training score Held-out accuracy Outcome
1.5B dense, 2k limit 2048 tokens 0.0002 nats 0.747 at step 600 0.677 at step 600 Stable for 600 steps
1.5B compressed, 2k limit 2048 tokens 14.4 nats 0.746 at step 600 0.661 at step 600 Stable for 600 steps
8B compressed, 16k limit 15360 tokens 17.9 nats 0.872 peak, 0.453 at stop 0.802 at step 150 Collapsed near step 170
Gap values are run means. Training scores for the 600-step runs are means over the final 20 steps. The 16k run has a single held-out evaluation, at step 150.

One honest caveat. Model size, model family and generation limit all changed together between the small runs and the 16k run, so the collapse cannot be pinned on any one of them from this data alone. The 16k run's reference divergence also grew steadily from the opening steps, long before the truncation spike, so truncation was at most the trigger and not the sole cause. And the short-limit runs truncated 11 to 12 percent of samples early and still 1 to 2 percent at the end without collapsing, so the level of truncation by itself does not predict failure. The single-variable run that resolves this is the larger model at the 2048 limit, with the smaller model at the 16k limit as its mirror.

What this does not blame

The slow dense circuit. All nine replays used the correctly delayed batch with bounded retained state, and the decline began between replays rather than at one.

Model scale as a defence. When earlier variants of the compressed recipe failed on the smaller model they turned over near step 200, and this larger run turned over near step 180, so size did not buy stability and longer generations plausibly cost some. The comparison above shows the current recipe holding for 600 steps at the smaller scale.

What holds up

Under 95 percent activation sparsity the method still produced a 9-point held-out gain at step 150. Compression did not prevent learning.

The failure is an interaction between sequence length and the sampler-trainer gap, not evidence that the compressed path cannot train.

What a rerun changes

  • Token-level importance weighting, to close the ratio blindness. Earlier smaller-scale tests cut reference divergence by 40 to 78 percent.
  • A generation limit set with headroom above where length naturally settles, so the truncation cliff is never reached.
  • Frequent checkpoints, because on this run the best policy was never saved and is unrecoverable.