Communication-efficient RLVR: the phase-one conclusion

Weekly update · week of 2026-07-20 · self-contained for other teams · base Qwen2.5-Math-1.5B on MATH · reinforcement learning from verifiable rewards with GRPO · follows the earlier two-circuit readouts.

learns to full strength matches uncompressed in and out of domain one open problem: a train and inference mismatch

Validated by Shamane

1. This week, in one line

The communication-efficient method learns the task to full strength, both in domain and out of domain, and matches an uncompressed run at the point where the base model saturates. Phase one also identified the one open problem: a train and inference mismatch that compression makes far larger than normal, and that shows up cleanly as a rising KL between the trained policy and the base policy. In the RLVR setting, on the datasets used here, we can manage the collapse this mismatch can cause. We have since conducted further analysis of this mismatch to guide future work, especially for realistic asynchronous RL settings.

2. The setup, for anyone new to this

To measure and understand every part of the system, our experimental setup is a plain verl GRPO run on math problems whose answers can be checked automatically, where the compression is applied only to the trainer while the vLLM inference engine stays normal. The goal was to train a stable two-circuit setup with otherwise standard settings.

The fast circuit. It compresses both the activations and the weight updates with a low-rank scheme, so far less data has to cross the network. This is where the communication savings come from.
The anchor circuit. On a fixed schedule it replays one dense, uncompressed update, and its own weight update is passed through a rank-one projection. That dense update pulls the fast circuit back toward where an ordinary run would be, correcting the drift that compression introduces.

The base model is Qwen2.5-Math-1.5B and the dataset is MATH, chosen so our numbers line up with the existing efficient-RLVR literature. Training uses about 7500 problems in batches of 512 prompts, and the runs in the results section go to 100 optimizer steps, which is just under seven passes over the training set and the point where the dense baseline saturates. We validate on MATH500, a held-out set of 500 problems, and we report per-benchmark accuracy across a wider suite at the end. Each prompt draws a group of 8 samples, with a prompt and response budget of 1024 and 3072 tokens.

The tuning knobs the two circuits carry were all settled in earlier weeks and are held fixed here. The activation compression rank is 77 and the anchor fires every 20 optimizer ticks with a matching freshness delay. This phase asked a simpler pair of questions. Does the whole system actually learn, and where are its limits?

The three update rules that matter

The whole method reduces to three small update rules. First, the fast circuit compresses each layer's activation matrix A through a shared low-rank basis Q. The anchor owns this basis, and over the run, on each anchor fire, Q is refreshed by one power-iteration step built from the activations the dense anchor replay just saw:

Q ← orth( A^T (A Q) ) The anchor accumulates the activation sketch during its dense replay and orthonormalizes it at the fire point, so Q tracks the directions the activations actually occupy, and the fast circuit reuses that shared basis between fires. The refresh rides on the anchor cadence rather than happening every step.

Second, every time the anchor fires, its raw dense gradient refreshes a running memory M of what the uncompressed gradient looks like:

M ← β · M + (1 − β) · G_dense, β = 0.25 We found this value of β to be the best through hyperparameter tuning. It weights the newest dense gradient at three quarters of the blend, so the memory follows the anchor closely while keeping some history.

Third, between anchor fires, every compressed gradient G is corrected by re-signing most of its magnitude with the signs stored in M:

G_corr = α · G + (1 − α) · |G| · sign(M), α = 0.25 The magnitudes come from the current compressed gradient and three quarters of the blend takes its signs from the dense memory. The value of α matters: raising it to 0.5, which trusts the compressed signs more, destabilized training in our sweeps.

The anchor weight projection

The anchor snapshots are stale by the anchor delay, so the update extrapolates linearly from the last snapshot to approximate the current fast-circuit weight, a two-checkpoint secant:

θ_proj = θ_t + s · (h / Δ) · (θ_t − θ_t−1)

We tested other fits, a four-checkpoint fit and a progressive schedule, and this linear secant was best, since compression adds a little randomness the higher-order fit chases and the linear one resists.

Figure 1. The two-circuit system, and the seam where the mismatch enters. The sampler runs the true, uncompressed weights and generates the answers. The trainer computes the GRPO gradient through the compressed forward pass. GRPO therefore improves one policy using data drawn from a slightly different one. The anchor periodically replays a dense update to pull the two back together, and its own weight update is additionally passed through a rank-one projection. In our early experiments the system could collapse when the anchor signal became too stale, and the projection exists to reduce exactly that staleness. The rest of this note is about the size of the gap between the two policies.

3. Results: compressed training learns as well as normal training

Every curve in this section is cut at 100 optimizer steps, the point where the dense baseline saturates, so the conditions are directly comparable and no late-run behavior is mixed in. The headline comparison is normal dense training with no compression against the full compressed system with the anchor and the weight projection. Previous no-projection ablations showed that the projection primarily improves stability, so the figures below focus on the projection-based system.

The cleanest view of the training signal is the per-step group score, the mean checked accuracy of the answers the sampler generates for each training batch. This is the critic score channel in the training logs, and every logged step is drawn below with no smoothing. Both learning conditions climb together from about 0.35 at step 1 to about 0.71 at step 100, and their curves are indistinguishable within batch noise. A third curve settles the anchor question. With the anchor circuit turned off, the same compressed trainer never learns and its score stays near 0.34 for the whole run. That point was settled in earlier weeks, so it appears here only as the reference line that shows the anchor is what makes compressed learning possible.

Figure 2. The training signal, exactly as logged. Group score per optimizer step, all 100 logged points per run with no smoothing. Normal dense training (green) ends at 0.709 and the anchor with projection (blue) at 0.715, indistinguishable within batch noise along the way. With the anchor off (red) the same compressed trainer stays near 0.34 and never learns.

Validation tells the same story. The figure below is the matched pair of runs, dense with no compression and the anchor with the weight projection, both validated every 25 steps from the same base checkpoint near 0.45. The two curves track each other the whole way. At step 25 they sit at 0.637 and 0.609, they meet near 0.65 by step 50, and by step 100 the anchor with projection reaches 0.679 against dense at 0.674, a hair above it. Both land more than twenty points above the base model, and the gap between them at any single point is within the sampling noise of a 500-problem measurement. This is the sense in which the projection variant is the stronger one to carry forward, even though the accuracy difference is small.

Figure 3. Dense and the anchor with projection land at the same ceiling. Validation accuracy on MATH500 for the matched pair, both validated every 25 steps and cut at step 100. The shared starting point is the base checkpoint near 0.45, measured once, since the starting model is identical across runs. Dense (green) ends at 0.674 and the anchor with projection (blue) at 0.679. Each validation is a 500-problem measurement, so single-point differences of one or two points are within noise.

Figure 4. Compressed matches or beats dense across the whole suite. Accuracy per benchmark, both measured at 100 steps. The compressed run (blue) is level with dense (green) on the held-out in-domain set and across the out-of-domain benchmarks, and is clearly ahead on the harder knowledge probe. These are saturation-point numbers with no late-run behavior mixed in. MATH500 is the true held-out set. The other math benchmarks are distribution shifts and the knowledge probe is non-math, grouped here for readability.

Reproducing the result on a second model and dataset: R1-Distill-1.5B on DeepScaleR

Everything above is one model on one dataset, Qwen2.5-Math-1.5B on MATH. To check the result is not specific to that pair, we ran the identical compressed setup on a second model and a second dataset: DeepSeek-R1-Distill-Qwen-1.5B on DeepScaleR. The method is unchanged, PowerSGD rank-77 activation compression with the rank-1 RELEX weight projection and the paired dense anchor, and it is compared against a dense control that differs only in having compression turned off. Both use the same GRPO recipe, a 512-prompt batch, eight rollouts per prompt, and a 4096-token response budget. We read the comparison at step 50, which is 25,600 training prompts, or 204,800 sampled rollouts at eight per prompt.

Figure 10. On the second pair, compressed training tracks dense step for step. Group score per optimizer step for the dense control (green) and the compressed run (blue) through step 75. The two curves rise together over the whole window, with the compressed run at or slightly above dense from the mid-forties onward. Same recipe on both, the only difference is whether the activations and the weight update run through the compression circuit.

Figure 11. At step 50 the compressed run matches or beats dense across the suite. Per-benchmark accuracy at step 50, greedy mean@1 for MATH500, Minerva, OlympiadBench and MMLU-STEM, and avg@8 at temperature 0.7 for AMC23 and the AIME sets. Compressed equals dense on the in-domain MATH500 (0.72) and on AMC23 (0.58), leads on MMLU-STEM (0.52 against 0.46) and AIME26, and sits within measurement noise everywhere else. The same in-domain and out-of-domain parity seen on the first pair holds on the second.

This is the phase-one result reproduced on a different base model and a different training set: at a matched compute point the compressed trainer learns as well as dense training, in and out of domain, on both pairs we have tried.

4. Further validation: measuring what the train and inference mismatch causes at this scale

4.1 A mismatch that is native to this setup

This system creates an obvious train and inference mismatch, the kind of gap the RL community works to reduce. It arises because the two sides of the loop run different computations. The sampler generates every answer with the true, uncompressed weights inside vLLM. The trainer then scores those same tokens through its compressed forward pass. The two sides therefore assign different probabilities to the same tokens from the first step onward, before any drift has had time to accumulate. Ordinary RLVR carries a small version of this gap, because the inference engine and the trainer kernels are never numerically identical. Compression multiplies it. The probability gap on sampled tokens is more than 300 times what a normal run shows, and the correlation between the sampler and the trainer log probabilities, near perfect in a dense run, sits at only about 0.15 under compression and falls toward zero at collapse. The full standalone analysis of this mismatch, including how it is measured, is in the earlier train and inference mismatch report.

Figure 5. Compression breaks the agreement between the two policies. Left, the correlation between the sampler and the trainer log-probabilities is near 1.0 for a dense run and only about 0.15 under compression, falling toward zero as the run collapses. Right, the probability gap at sampled tokens sits near 0.002 for dense and near 0.77 to 0.93 under compression, on a log scale, which is where the more than 300 times figure comes from. The gap is present from the very first step, before any drift accumulates.

The danger is that this can fail silently. At the point where the base model has already saturated the task, everything looks healthy, so nothing in the accuracy or the reward warns you that a problem is building. Here is why it can still break later. The low-rank compression applies a biased, coherent distortion, which nudges every update slightly in the same direction. The wedge between the trainer and the sampler accumulates step after step, and the effective policy drifts away from the base policy. To probe this we deliberately trained a few models for longer than the task needs. Some of them do collapse, but only well after the saturation point. In one longer run validation held at 0.671 at step 125 and then fell to 0.046 at step 150, and the training reward measured on the true-weight sampler dropped from about 0.68 to 0.17, so the policy really broke rather than only its compressed view. For the RLVR runs in this report, on these datasets, that risk is manageable, since the collapse sits well past the point where training is already done.

4.2 The KL readout: normal GRPO is flat, our setup is not

The cleanest running readout of the mismatch is the KL between the trained policy and the base policy, logged at every step. In ideal RLVR this should stay almost flat, because solving a task like this does not require moving the weights much, and normal GRPO training behaves exactly that way. The dense run's KL is 0.004 nats at step 50 and still only 0.009 at step 150. Our compressed system is a different picture. Its KL passes one nat around step 50, reaches about 6 nats by step 100 and keeps climbing. It is important to see that the size of this rise is also dataset and model dependent. On a second model family and dataset, the same climbing KL sits noticeably lower than it does here, but the point still holds, since it is far above the almost flat dense baseline. Both curves below are drawn from every logged step, the same points the training dashboard shows.

Two control runs sharpen the diagnosis. Normal RL does not apply dropout in the trainer, for exactly this mismatch reason, so we checked how the KL behaves when we train with attention dropout anyway. It stayed very low, near 0.11 nats and flat for the whole run. Dropout is zero-mean noise that is redrawn on every step, so it never builds a consistent wedge between the trainer and the sampler, and the perturbation averages out across the run. That observation motivated trying a PRF mask, a stochastic random-mask compression codec that is roughly similar to dropout in spirit though not exact, since it is a real compression method applied to the same tensors as the low-rank scheme. Its KL was also good, staying low and rising only slowly, and its end-task results held up well against the PowerSGD compression. The pair of controls points at one conclusion. Unbiased randomness that is redrawn each step is benign. A coherent low-rank bias that removes the same subspace on every step is what makes the KL run away.

Figure 6. Normal GRPO stays flat, the coherent codec climbs. Actor KL loss per optimizer step, every logged point, all runs cut at step 150, linear scale. Normal training (green) sits on the axis for the whole run, 0.009 nats at step 150, which is what ideal RLVR should look like since the weights barely need to move. The compressed system (blue) reaches 6.1 nats at step 100 and about 7.1 at step 150, with spikes to 8.6. The dropout control (grey) holds near 0.11 nats and the PRF random mask (purple) ends at 0.56, both hugging the flat line.

Figure 7. The same capability, from a coherent and an incoherent codec. Validation accuracy at each codec's best. PowerSGD (blue) reaches 0.671 at step 125, its peak just before it collapses in Figure 6. The stochastic PRF mask (purple) reaches 0.689 at step 150 and holds there. Both codecs reach the same end-task capability, but only the incoherent one keeps it.

4.3 Why asynchronous RL is where this bites hardest

Modern large-scale RL is moving toward asynchronous designs. The recent GLM-5 report and related work run generation and training on separate hardware, and the trainer consumes rollouts produced by a policy from a few updates back. This matters most in agentic RL, where a single rollout can be a long multi-turn interaction and the trainer cannot hold everything in lockstep, so it trains on whatever finished rollouts are available. In these systems the old log probabilities are usually not recomputed. The rollout engine, SGLang in GLM-5 and vLLM in our stack, records the token probabilities at generation time and the trainer reuses them directly as the behavior policy. Every update then passes through the standard clipped objective:

L(θ) = E[ min( r · Â, clip(r, 1 − ε, 1 + ε) · Â ) ], r = π_θ(token) / π_rollout(token) Â is the advantage. Clipping assumes the ratio r sits close to 1, so that only genuinely off-policy tokens are cut.

With the rollout engine providing the probabilities, any disagreement between the inference computation and the trainer computation lands directly in the ratio r. Our compression alone pushes r far from 1 on most tokens, before any staleness is even added. In the mismatch analysis, even a loose clip that lets the ratio drift by a factor of five already drops about 61 percent of the tokens. A standard clip is far tighter and would remove more than 65 percent, against well under one percent for a dense run. Losing most of the tokens starves every update. This is why the mismatch has to be answered before this technique meets a realistic asynchronous RL system.

Figure 8. Standard clipping would throw away most of the compressed update. Under a normal importance-ratio clip a dense run loses well under one percent of tokens, because the sampler and the trainer agree. Under compression the same clip would remove more than 65 percent of the tokens, which is why asynchronous RL is the setting where the mismatch bites hardest.

4.4 Why GRPO keeps learning through the mismatch, and how it can break later

The same clip from Section 4.3 explains both why the compressed system learns and where it eventually fails. GRPO maximizes a clipped surrogate with a small reference-KL penalty:

J(θ) = E[ min( r · Â, clip( r, 1−ε, 1+ε ) · Â ) ] − β · KL[ π_θ ‖ π_ref ] r = π_θ(token) / π_θ-old(token) is the importance ratio between the current policy and the policy a few gradient steps back. The highlighted clip caps how far one update may move any single token. Â is the group-normalized advantage, read from the true-weight vLLM rollouts.

In these runs the ratio r is taken between two trainer forward passes that share the same codec, so the large codec distortion is common-mode and cancels, r stays near one, and the clip almost never fires. The advantage Â still comes from the true-weight vLLM rollouts, so a real learning signal survives, and the ratio-one dense anchor supplies the direction it needs (Section 5). The failure comes later and from the same clip. The compression bias is coherent, so it accumulates instead of averaging out, the trainer policy drifts off the behavior policy, r is pulled away from one, and the clip fires on more and more tokens until the update starves. The two panels below track exactly this, near zero while the model learns and accelerating at the end, which is the early edge of the collapse in 4.1.

Figure 9. The clip barely fires while the model learns, then accelerates. Both panels are drawn from every logged step of the compressed run in Figure 6, cut at step 100. Left, the fraction of tokens hitting the clip boundary. Right, the within-update off-policy KL, the mean distance of the importance ratio from one. Both stay near zero while the reward climbs, then climb steeply over the final steps, which is the early signature of the accumulation failure rather than of healthy learning.

5. The phase-one verdict

What worked

The anchor is effective. With it, the compressed system learns to full strength and matches dense training in and out of domain. Without it, the same compressed trainer never learns at all.
Normal RLVR with reasoning does learn at this scale. The 1.5B math base model gains more than twenty points on MATH500 in under seven passes over the data, with an almost flat KL, which is the healthy reference every other run is judged against.
The weight projection can reduce staleness. Our early experiments showed that collapse can happen when the anchor signal becomes too stale, and the projection is a working lever against exactly that failure.

The open problem

The train and inference mismatch. It is native to the design, more than 300 times larger than in normal training, and it reads out as a KL that climbs while accuracy still looks fine. In this RLVR setting, on these datasets, the collapse it can cause sits past saturation and is manageable. In asynchronous RL it would gut the update through clipping, so it is not manageable there yet.

To take this technique beyond the setting shown here, and above all into asynchronous RL, we should treat the train and inference mismatch as the problem to solve head on rather than work around. The most immediate option is to apply the same compression inside vLLM, so the sampler and the trainer see the same computation and the gap closes by construction. The cost is that compressing the sampler could distort the sampling itself, and how much that matters is unknown. One further thought is that with an SSN model this would not be a problem at all, since the compressed computation would be the model itself on both sides of the loop. We leave that one open for discussion.