Weekly archive · Qwen Math on MATH
Weekly research report
This week I pushed the two-circuit method into much longer runs, the kind measured in hours rather than minutes, and the question stopped being whether it trains at all. It trains. The question became why it survives that long, and where the remaining headroom is. The long run held its full horizon while still learning, and at the end it lands on par with ordinary uncompressed RL, slightly ahead on several benchmarks.
This week every knob stayed exactly where it was. The only thing I changed was the compression method on the fast circuit: the learned PowerSGD subspace, or a random PRF mask. Both throw away about the same amount of information.
What decides a long run is whether they throw away the same thing twice.
Twenty steps on one subspace. The same remainder gets discarded every time, so the error stops being noise and becomes a direction the trainer can follow.
Same setup, anchor firing twice as often, run purely to test the story. If reuse is the problem, halving each piece of Q should buy time. It did, which is the cleanest confirmation we have.
Bigger error on any single step, but it points somewhere new each time. Random directions with a preserved average cancel rather than pile up, so the error stays noise and never competes with the reward.
If the drift builds gradually, why does the score fall off a cliff instead of sagging? Because GRPO is a loop, and the loop feeds the damage back into itself.
Compression sits inside the trainer's view of that loop, at the fourth box. Here is what a slowly growing bias there does to the rest of the circle:
|rt − 1| leaves the trust region, that token's surrogate is flat and it stops contributing gradient at all.Which is the real limitation of clipping. It is a guardrail on one step, not a controller across many. It bounds how far a single update can move and says nothing about where three hundred of them end up. That is the gap a compression bias walks straight through.
Only the combined standard clip fraction was logged in these runs. The loss also contains a separate dual-clip guard for extreme negative-advantage cases, but there is no W&B series for that branch here.
Five series, and it is worth being exact about what each one measures, because two of them are read through the codec rather than through the model. That distinction turns out to matter more than anything else in this report.
critic/score/mean: mean verifiable reward of sampled answers, not a learned critic value.actor/entropy: sharpness of the compressed trainer view, so not the entropy of the policy that actually samples. PRF and Q are not directly comparable here.actor/kl_loss: movement from the reference policy, also measured inside the trainer view. Dense RL raises this too, and it is the most misread number in the set.rollout_corr/kl: the train-inference gap, meaning the per-token disagreement in nats between the dense sampler and the compressed trainer. Trend matters more than level.actor/pg_clipfrac: share of tokens whose PPO surrogate is clipped. The run logger combines the ordinary upper and lower branches.The shaded region marks the failed Q run from the first clear score cliff through the entropy trough. Curves are reconstructed from W&B without smoothing.
The slower Q run learns first, then loses answer quality abruptly. The faster refresh and PRF runs hold.
The failed Q trainer view becomes extremely sharp. PRF stays diffuse because its mask changes the view.
Dense KL also rises. The failed Q run matters because its rise couples to the other failure signals.
PRF starts with a large but stationary wedge. The failed Q wedge accelerates with the score cliff.
Clipping pressure rises before the failed Q score cliff. The logger combines upper and lower clipping.
A spike is not enough by itself. The faster Q run also sees clipping pressure, but refresh breaks the persistent direction and the score survives.
All of that came from a bias with a direction. Here is where the direction comes from, and this is the place to say properly what the two compressors are.
à = AQQT. The columns of Q are the directions the activations actually used most. The anchor works them out by watching the activations during its dense pass and taking an orthonormal basis of what it saw. This is good compression, because it keeps the directions that carry the signal. The catch is that the fast circuit cannot recompute it, so Q does not change at all between anchor fires.(1536/77) · 1S ⊙ h, giving E[h̃] = h. Exact-k means that count never varies, which keeps the wire budget identical on every step. PRF means the subset comes from a keyed pseudo-random function of (seed, layer, step, sample, position, channel), so both ends generate the same mask and the mask never goes on the wire. Worse compression per step, and no memory at all.Now the mechanism. The backward pass returns through the same projector the forward pass went through, ∂L/∂A = (∂L/∂Ã) PQ with PQ = QQT. So the part that gets discarded is A(I − PQ), and while the anchor holds Q fixed that is the same subspace on every step of the interval. PRF drops a different random subset each step and preserves the mean, so there is no fixed direction to find.
Treat each step's codec error as a vector and add them up. The difference is a growth rate:
Q. Twenty steps of agreeing with itself.
Put a number on it. If the error holds one direction for L steps and then jumps, the accumulated drift over T steps grows like √(L·T). Fully held Q is the L = T corner and gives T. PRF is the L = 1 corner and gives √T. Everything the anchor cadence can do lives between those two.
That formula also says exactly what faster refresh buys, and it is less than it looks. Halving L divides the drift by √2, roughly 1.4x. It does not change the shape of the curve, it slides you a little way along it. The only cadence that reaches PRF's regime is L = 1, a dense pass on every single step, which is the one thing this whole method exists to avoid.
The runs agree:
| at step 200 | reference KL | train-inference gap | codec-view entropy | clipped tokens |
|---|---|---|---|---|
| Q, anchor every 20 | 9.8 | 30.9, accelerating | 0.03 | 13%, peak 17% |
| Q, anchor every 10 | 7.3 | 16.2, still climbing | 0.24 | 6%, peak 11% |
| Exact-k PRF | 0.7 | 14.6, flat | 7.84 | 1.6%, falling |
Halving the cadence cut the gap by about 1.9x, against the 1.4x the scaling alone predicts, so refresh is doing a little more than bookkeeping. It still lands in the same regime. The gap is climbing when the run ends, at 5.9 → 16.2 with no sign of settling. Entropy has still fallen from 5.5 to 0.24. Reference KL has still reached 7.3. And the improvement is paid for in anchor bandwidth, the one resource this deployment cannot spend freely.
PRF is not on that curve at all. Its gap starts high, 13.8 nats on step 1 before any training has happened, and then moves +0.8 across the whole 200 steps. The level is large and the slope is nearly zero, and the slope is the part that matters. Its clipped-token fraction actually falls, 2.5% down to 1.6%, which is the most useful line in the table. PRF is not spending its trust region fighting its own codec, so nearly all of it stays available for learning. That is the window the long run went on to use.
This is the correction I most want to land. Reference KL and the train-inference mismatch are different quantities, they do not move together, and neither one is a straight readout of how far the weights actually went.
Look at the table again. PRF ends at KL 0.7 with a 14.6 nat gap. Q at cadence 10 ends at KL 7.3 with a 16.2 nat gap. Ten times the KL for a comparable gap. The long-horizon runs sharpen it further. Over 600 steps the compressed run's reference KL rose from 0 to 0.899 while its gap moved only 13.88 to 14.66, and 87 percent of even that came from the sampler sharpening rather than the training view drifting.
The reason is that actor/kl_loss is measured through the codec. It is a reading taken with an instrument that is itself part of the system being measured. Refresh Q or redraw a mask and the number moves while the weights sit still. And the weights really are sitting still. In every regime here, cosine similarity to the base model stays near 1.0 and relative parameter change is 0.03 to 0.13 percent, which is under the 0.195 percent floor at which a bf16 weight can even record a change. Divide the codec-view KL by a codec-off measurement of the same quantity and the ratio grows from 17 to 54 over the run, so it is not a fixed scale factor either. Part of that number is the policy. Part of it is the lens.
The entropy column is the same fact from the other side. 7.84 next to dense RL's 0.11 looks alarming and is not, because it is the entropy of the codec view, not of the policy that writes the answers. Sampling stays dense, with full weights. What we have is a sharp sampler being trained through a deliberately blurred lens.
So the compressed trainer is not a slightly degraded copy of the dense one. It works on its own plane, where KL and entropy both read high by construction, and it stays there rather than drifting off it. PRF's entropy moves 7.80 to 7.84 across 200 steps, which is flat. The question worth asking is therefore not how to drag those numbers down to dense values. It is whether the policy learns on that plane.
It does. Training reward goes 0.34 to 0.75 and MATH validation 0.451 to 0.661, with the trainer view diffuse the entire time. Staying diffuse is not a cost we are tolerating either. It is the earlier property seen from the other side, because a view that never commits hard to one direction is a view whose errors never turn into an objective.
With one refinement, which is the most useful thing the week produced. The mismatch does not have to be small. Run 90 held roughly 14 nats of it for 13 hours and learned the whole way. It has to be stationary. A large settled gap is a working regime. A small growing one is a countdown. That moves the target from minimizing mismatch, which the compression ratio will never allow, to regulating its slope, which now looks reachable.
Weight movement is small in both Q runs, so magnitude does not separate them. What separates them is location: the failing run shifts its movement toward earlier blocks, the surviving run stays more centered, and PRF stays later-heavy. The difference is spatial direction, which is the same story the arrows tell, measured in a different place.
Everything so far was diagnosed on 200-step runs. The real question was whether the explanation holds at the horizon we actually care about, so the exact-k PRF run was taken to 600 steps, 12 hours 52 minutes on a single H200, against a protocol-identical dense control on the same hardware.
It completed the full horizon with no reward or entropy cliff, held its fixed wire budget on every step, and its gap stayed stationary throughout, exactly as the stationarity argument predicts. Validation improved early and stayed there.
Both runs. Final points read from the training logs.
Both runs. critic/score/mean per step, raw (no smoothing).
Both step-600 checkpoints on the same benchmark harness (greedy mean@1 for the large sets, avg@8 for the competition sets).
The week leaves us with a target that is stated the right way round. We are not trying to make the mismatch small, because at 95 percent compression it will not be small. We are trying to keep it stationary, and to be able to measure it honestly while a codec is in the way.
That is the program now open as issue #93. In short: a periodic codec-off measurement so we can read real drift rather than the lens, a controller that acts on that clean signal instead of on a fixed threshold, and a few codec variants that keep the no-persistent-direction property while carrying less per-step noise than a random mask does. Exact-k PRF stays the baseline everything has to beat. Cross-seed replication of this week's result is still outstanding.
Communication-efficient RLVR weekly report · real W&B trajectories · no smoothing
Verified by Shamane