Back to the hub · system status · 2026-07-31

System status · 2026-07-31

The communication-efficient GRPO system

1 · The two-circuit system

The model is too large for one GPU, so it is split by pipeline parallelism into stages, and the stages live on GPUs connected over the ordinary internet. Everything that crosses a stage boundary during training, activations forward and gradients backward, is compressed to 5 percent of its dense size. Two circuits make that survivable: a fast circuit that runs GRPO end to end through the compressed pipeline on every optimizer step, and a slow anchor circuit that fires every 20 steps from deliberately stale weights and sends back one bit per parameter of direction. Over everything sits one constraint: post-training must not damage the base model.

Figure 1 · The two-circuit system

Left: the fast circuit, every optimizer step. Right: the anchor circuit, every 20 steps. Bottom: the firing schedule and the staleness contract.

FAST CIRCUIT compressed GRPO, every optimizer step prompts one batch sampler dense weights θt verifier r ∈ {0,1} group advantages  per prompt group training forward + backward through the 8-stage pipeline: stage 1 L1-4 stage 2 L5-8 stage 3 L9-12 stage 4 L13-16 stage 5 L17-19 stage 6 L20-22 stage 7 L23-25 stage 8 L26-28 every boundary, each direction: 77 of 1536 coordinates per token (5%) mask derived at both ends from PRF(seed, layer, step, ...): never transmitted, fresh each step GRPO objective clipped ratio + KL(π‖π_ref) G_comp compressed gradient sign-guided correction G = 0.25 G_comp + 0.75 |G_comp| ⊙ sign(M) optimizer step θt → θt+1 θ synced to the sampler after every step dense inside the fast loop: the rollout engine and evaluation. compressed: every gradient-bearing forward and backward above. ANCHOR CIRCUIT one dense device, fires every c = 20 steps stale checkpoint θ(t-20) * the newest weights the anchor may see are K = 20 steps old: the contract dense replay of matching rollouts rollouts generated by the weights it holds, fwd + bwd, no optimizer step → G_anc M: moving average of the stale-weight gradient M ← 0.25·M + 0.75·G_anc returns sign(M) only 1 bit per parameter, its entire output * a cheap weight forecast can track the newest weights and batch instead: appendix A θ checkpoints arrive 20 steps late rollout batches ride along: token ids + advantages sign(M): 1 bit per parameter, every 20 steps, held between fires The firing schedule and the staleness contract between fires the fast circuit runs alone, applying the sign from the last fire on every step 0 fire 20 fire 40 fire 60 t the checkpoint written at t = 20 is the one the fire at t = 40 uses (K = c = 20) compressed fast path dense anchor path sign-guided correction
One compression, one correction, one contract. The fast circuit exchanges 77 of 1536 activation coordinates per token at each of the seven boundaries, in both directions, and re-signs its gradient with the anchor's signal before the optimizer step. The anchor replays matching rollouts densely from its stale checkpoint and returns one bit per parameter. Nothing else crosses the network.

1.1 The fast circuit

Ordinary GRPO, end to end, every optimizer step. The sampler generates responses with full dense weights and a rule-based verifier scores them. The policy update then runs through the pipeline, eight stages and seven boundaries, with every boundary tensor compressed in both directions. All log-probability passes of an iteration (old log-probs, the policy update, the reference policy) run through the same mask, so the PPO ratio compares like with like. The rollout engine and evaluation are untouched, and after each step the updated weights are synced to the sampler.

1.2 The boundary compression: exact-k PRF masking

Each boundary keeps exactly 77 of the 1536 hidden coordinates per token, 5 percent, and scales the kept values by 20, so the compressed activation is an unbiased estimate of the dense one. The mask is derived independently at both ends of the link from a keyed pseudo-random function (PRF) of (seed, layer, step, sample, position, channel). Nothing about it is transmitted, the count never varies, and a fresh mask is drawn every step. The backward pass returns the gradient of the same 77 coordinates.

Figure 2 · The compression in one picture

keep exactly 77 of 1536 coordinates per token (5%), scale the kept values by 20 → unbiased step t kept, x20 step t+1 a different set next step stage k stage k+1 forward: the 77 kept values only backward: the gradient of the same 77 coordinates the mask itself never crosses the wire: both ends derive it from PRF(seed, layer, step, sample, position, channel)
A fresh random keep-set every step. The cartoon shows 1 of 16 cells kept. The real ratio is 77 of 1536, exactly, on every step and every boundary.

1.3 The anchor circuit

The anchor is one dense device off the critical path, and its job is to maintain M. Its inputs are stale by contract: shipping weights to a remote device takes real time, so the checkpoint it holds at step t was written at step t-20, never fresher. From those weights it replays rollouts that the same weights generated, with one uncompressed forward and backward and no optimizer step. The resulting clean gradient is folded into the moving average M, kept in full precision for every tensor in the model. The anchor's entire product is sign(M), one bit per parameter, refreshed every 20 steps and held in between.* The fast circuit merges that direction into every step's gradient before the optimizer consumes it:

G = 0.25 G_comp + 0.75 |G_comp| ⊙ sign(M)

The magnitudes stay the compressed gradient's own, which keeps the update correctly scaled for the optimizer. Three quarters of the signs come from the anchor's denoised direction.

The sign correction is what keeps the loop stable. Weakening the blend to half sign and half raw gradient, with everything else fixed, destabilizes training and costs 3.4 accuracy points at matched steps. Error the sign no longer removes reaches the sampler, degrades the data it generates, and returns as further error.

* A cheap weight forecast lets the anchor track the newest weights and replay the newest batch instead. Ablations preferred that configuration and the long runs in this report use it: appendix A.

2 · Results

Qwen2.5-Math-1.5B post-trained on MATH under the full system, always against a byte-matched dense control: an identical configuration with compression turned off, on the same hardware. Final checkpoints are evaluated through one harness, greedy decoding for the large sets and avg@8 at temperature 0.7 for the competition sets.

Training curves: dense against 95 percent compressed

A matched dense and compressed pair, every logged step, no smoothing.

dense PRF-compressed (95%)

Training reward

Mean verifiable score per step. The two runs rise together, about 0.35 to 0.74.

0 0.2 0.4 0.6 0.8 0 50 100 150 200 training step

Entropy (compression view)

The 20x rescale inflates the compressed reading to a flat band near 7.8. It is not the sampling entropy.

0 2 4 6 8 0 50 100 150 200 training step

Train-inference mismatch

Sampler-to-trainer gap in nats per token. Large from step 1 and flat. The dense pair sits at about 0.0002.

0 4 8 12 16 0 50 100 150 200 training step
Same learning, one honest difference. Reward rises together. The entropy readout and the train-inference gap look nothing alike between the two runs because the trainer reads them through the compression: section 3 explains why the large flat gap is the healthy signature rather than a warning.

Capability at step 600: dense against 95 percent compressed

Dense vs PRF-compressed accuracy per benchmark at step 600, in-domain and out-of-domain
Parity, in and out of domain. Every gap is within ±0.04 with mean |Δ| = 0.014. Compressed is higher on 6 of 10 benchmarks and dense on 4, with no systematic direction: run-to-run noise, not a compression penalty. Over the long run the two validation curves never separated by more than 0.016.

3 · The compression method and the train-inference mismatch

The sampler generates data with dense weights while the trainer computes its log-probabilities through the compression. These are two views of the same network, so the system trains under a permanent train-inference mismatch, and the mismatch is native rather than a defect: the compressed network is also the one generating its own training data. Sampling through the compression would remove it by construction, but a 95 percent masked view is not a usable generator, so the gap is handled on the training side.

The gap is large but essentially static. It is there at step 1, before any training, and what matters is its trend, not its level: over the 600-step run it drifted by less than a thousandth of a nat per step. The compression-view diagnostics that look alarming, the KL to the reference and the entropy readout, are inflated by the mask itself and do not measure damage. The same run that ends with a compression-view KL near 0.9 matches the dense control on every benchmark above, and its weights move by only 0.03 to 0.13 percent. Health is read from the trend of the gap and from dense-view evaluation.

Before the mask, the system compressed these boundaries the natural way, with a learnable low-rank projection. Each boundary carries an orthonormal basis Q of shape 1536 x 77, fitted to its own activations, and a token's activation x crosses the wire as 77 coefficients instead of 77 raw coordinates:

send y = xQ  (77 numbers),  reconstruct x̂ = yQᵀ,  backward through the same projector

The anchor owns the learning. During each dense replay it refreshes every basis with one step of power iteration on the activations it just saw, and broadcasts the result alongside sign(M):

Q ← orth( Xᵀ(XQ) )  once per fire, frozen in between

This buys more signal per byte than a random mask, and it fails at long horizons for one reason: whatever a frozen basis throws away is the same subspace on every step until the next refresh, so the compression error keeps pointing the same way. A persistent direction is exactly what a gradient method picks up and follows. The fresh random mask has no direction to find.

The collapse this produces is visible in the raw curves. Below are the two runs side by side at the same 95 percent budget: the learned-basis run loses the model at around step 160, while the mask run trains on (it is the long run of section 2, shown over its first 200 steps).

The learned basis collapses, the mask holds

Both runs at the same 95 percent compression budget, every logged step, no smoothing. The shaded band marks the collapse.

learned low-rank basis Q PRF mask (95%)

Training reward

The learned-basis run learns first, then loses answer quality in the collapse. The mask run keeps climbing.

collapse 0 0.2 0.4 0.6 0.8 0 50 100 150 200 training step

Entropy (compression view)

The learned basis cliffs from 5.5 to 0.03. The mask holds its flat band near 7.8.

collapse 0 2 4 6 8 0 50 100 150 200 training step

KL to the reference (compression view)

The learned basis accelerates to 9.8. The mask ends near 0.3.

collapse 0 2 4 6 8 10 0 50 100 150 200 training step

Train-inference mismatch

The learned basis turns a smaller initial gap into 30.9 nats and accelerating. The mask holds its band.

collapse 0 8 16 24 32 0 50 100 150 200 training step
Bias times coherence, not magnitude. The mask carries the larger mismatch from step 1 and holds it flat, and it stays healthy on every panel. The learned basis is better per step and worse per hour: its discarded subspace persists between refreshes, the error compounds, and reward, entropy, KL and the mismatch all break together at around step 160. Refreshing the basis more often only delays the failure, and it pays in anchor bandwidth, the one resource this deployment cannot spend freely.

We studied the learnable family extensively at this budget, faster basis refresh, low-rank-plus-residual hybrids, low-bit quantization, importance-sampling corrections, and the same picture appeared every time: a compression with a persistent direction eventually turns it into drift, and none beat the fresh mask on long-horizon stability. The mask has been the system default since 2026-07-29.

4 · Current status

The finding to carry away. A train-inference mismatch is unavoidable in this design, because compression is applied to the trainer and never to the sampler. What the mask changes is the character of that mismatch: the gap stays large but stationary, the model keeps learning under it, and capability lands at dense level. Collapse came only from compressions that gave the error a persistent direction. That makes the mismatch something to monitor and regulate rather than something to eliminate, and practical work proceeds on top of it.

Settled

The compression. Exact-k PRF masking is the boundary compression. Every challenger tested at the same budget lost on stability.

Long-horizon stability. 600 of 600 optimizer steps at 95 percent compression with no collapse and a steady optimizer.

Capability preservation. Parity with the dense control in and out of domain (the bar plot above).

The anchor. A sign-only return is enough, and weakening it destabilizes training.

The reading frame. Compression-view KL and entropy are not damage metrics. Health is read from the gap trend and dense-view evaluation.

Open

Variance, not bias. The mismatch cannot be removed at this budget, so the target is a lower-variance unbiased compression at the same wire cost.

A drift controller. A periodic compression-off probe of true drift is validated. The controller that acts on it has not yet run a full horizon.

Replication. Cross-seed replication of the long run is outstanding.

One recorded lead. Injecting one fully dense step every 50 failed its registered stability test, yet posted the best held-out accuracy we have seen (0.6914 at step 600 against 0.6774 dense). Single seed, flagged for replication, not adopted.

Where the system stands. A model split into eight stages can be post-trained with GRPO while every pipeline boundary carries 5 percent of its dense payload, plus one bit per parameter of anchor guidance every twenty steps. Over 600 steps this matches uncompressed training, in and out of domain. The compression question is settled. Keeping the mismatch stationary at lower cost is the open work.

Appendix A · Projecting the anchor's stale weights forward

The main text has the anchor replay rollouts that match the weights it holds. There is a stronger configuration. Replaying at weights that are K steps old reintroduces exactly the delay the schedule is meant to hide, and the anchor can compensate for it cheaply: keep the last two delayed checkpoints, c steps apart, and advance each parameter tensor linearly to the present,

θ̂(t) = θ(t-K) + (K/c) · (θ(t-K) - θ(t-K-c))

a two-checkpoint secant with no fitting and no extra traffic. The forecast weights then replay the newest batch. This works because of how RLVR moves weights: updates concentrate in a small, consistent part of the network[1] and steer along structured, spectrum-preserving directions[2], and we measure relative movement of only 0.03 to 0.13 percent over our runs, so the linear extrapolation stays accurate over a 20-step horizon. In ablations the secant ended highest of the three forecast variants we evaluated, with MATH test accuracy 0.671 against 0.667 for a progressive window and 0.663 for a four-checkpoint fit, and the long runs in this report use it.

The forecast in one picture

one weight coordinate (schematic) training step now (t) t-40 t-20 the true weight trajectory checkpoint θ(t-40) checkpoint θ(t-20) forecast θ̂(t) true θ(t) the anchor never sees anything newer than t-20. the secant through its two checkpoints lands close to the true weights, because the trajectory is nearly linear at this horizon

References

  1. Mukherjee et al., 2025. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. arXiv:2505.11711.
  2. Zhu et al., 2025. The Path Not Taken: RLVR Provably Learns Off the Principals. arXiv:2511.08567.

Communication-efficient RLVR · system status · 2026-07-31