Compressing what crosses the stage boundary

Max-Collab · discussion #2 · 5 August 2026 · what a frozen 8B shows about activation compression, and the experiment plan.

This week we talked about the two-circuit network and its components, and mainly about the compression during rollout generation. We realised this part is important for two main things:

The two-circuit RL run already starts with a very high train-inference mismatch. The compression is applied only to the trainer, and the gap is way above the expected one.
Figuring out pipeline-parallel compression would also help the rollout generation in RL.

So the idea is to see how much we can compress, given a pretrained checkpoint, without overly distorting the outputs.

1 · What a frozen model shows

We took two untouched public 8B checkpoints, the post-trained release and its pretraining-only base, simulated eight pipeline stages inside a single model replica, and compressed the hidden state (4,096 values per token) at the seven stage boundaries. Two methods ran at matched payload: random masking, an unbiased random slice of the channels with the survivors rescaled (what our RL trainer applies today), and calibrated low-rank, a fixed per-boundary basis fitted on separate calibration text, sending only the top coefficients. Distortion is ΔNLL, the nats per token by which the model's own dense output becomes less likely under the compressed network (zero means unchanged). Everything is a teacher-forced probe on one held-out prompt with frozen weights, so it sizes the problem rather than benchmarks it. Full numbers: the quantitative report and the qualitative companion.

Figure 1. Left: distortion versus the fraction of each 4,096-value boundary vector sent. The calibrated basis improves steadily with rate, masking saturates until half is sent, and the base checkpoint compresses easier at every rate. Right: the calibrated basis on one boundary at a time (x axis: boundary after layer N) at 5%, post-trained model. The middle is most sensitive, and seven boundaries cost less than the sum of the parts.

Neither method is faithful at the RL training rate. Sending 5% costs about 17.4 nats per token under random masking and 11.7 under the calibrated basis, and both break in the recorded generations: masking emits character noise, low-rank drifts off the prompt and loops.
The calibrated basis wins at every rate, yet energy and precision are the wrong dials. At 5% the basis still holds roughly 99% of activation energy, because behavior lives in the low-energy tail behind a couple of giant directions, and full-precision coefficients moved distortion by under 0.03 nats, so a fixed byte budget should buy more directions, not more bits.
Calibration is a real lever on a pretrained model. The base checkpoint compresses much better (7.5 vs 11.7 nats at 5%), so post-training seems to have reshaped the activations, and broken outputs drift toward the calibration text's topics, a domain bias in the basis.

2 · Why this is the right lever for RL

An RL fine-tuning run of the same 8B on math problems, generating up to 16k tokens per response, improved for 150 steps and gained nine points of held-out accuracy, then gave it all back in about twenty, sparked by ordinary truncation feedback at the generation cap. The fuel is what matters here: rollouts were sampled by a dense engine while the trainer scored every token through compressed boundaries at the 5% rate. Same weights, two executions, about 18 nats per token apart, where a dense control puts the normal engine gap near 0.0002. The trainer never sees this gap, because its own consistency check compares the compressed view with itself and reads exactly zero. Longer sequences accumulate more of the per-token disagreement, so at the 16k budget the drift grew nine times faster than at 2k (a smaller model at short lengths carried the same mismatch for 600 steps unharmed).

Importance weights on each token cut the drift by 40 to 78 percent in earlier tests, but that only corrects the gap. Generating the rollouts through the same compressed forward pass removes it: the sampled and trained policies then match by construction at any length, and the capability this needs, faithful compressed generation, is the same one stage-split inference needs. The blocker is fidelity, which the plan below attacks.

3 · The experiment plan

Four questions about one object, the compressed boundary. The matrix is the first wave, more rows will follow, and every cell runs cheaply on a frozen model with simulated boundaries.

Experiment	What varies	What it decides
Q1 · How much can we compress before behavior moves?
Rate and length curves	2% to 50% sent, lengths to 16k, hundreds of prompts	the knee: the highest safe rate at each length
Q2 · Where can compression be applied?
Boundary sensitivity map	one boundary at a time, then pairs and counts	which depths tolerate it and how errors compound
Budget placement	allocation across depths, three or seven crossings	the best way to spend a fixed budget across depths
Q3 · Which method wins at matched bytes?
Trade precision for rank	8-bit and 4-bit coefficients with proportionally more directions	whether more directions at fewer bits wins
Outlier split	send the few giant channels exactly, compress the remainder	whether protecting outliers frees the budget
Temporal delta	encode the change from the previous token	the extra rate bought by redundancy across decode steps
Learned encoder	a small per-boundary autoencoder, model frozen	the gain over any linear basis
Q4 · What does calibration buy on a pretrained model?
Calibration design	corpus size, domain mix, mean removal, per-domain bases	held-out fidelity, domain-drift removal
Compression-aware adaptation	distill the dense model into its compressed self	behavior recovery and weight movement from base
Post-training compressibility	base versus post-trained checkpoints across model families	whether adaptation restores the lost structure

The bar every row is scored against

Free-running generation through compressed boundaries at 10% or less of the dense per-token payload in bytes, with task accuracy within about one point of dense on a real benchmark. Rows are screened with ΔNLL and generation fidelity first.

Provenance. Two untouched public Qwen3 8B checkpoints, seven simulated boundaries, matched payloads, measured values in Figure 1. Details: the quantitative report, recorded generations, and collapse analysis.