Weekly archive · Qwen Math on MATH
Weekly research report
This week I finished the long-run check that had been overdue. The exact-k PRF fast circuit completed the full horizon without a collapse, kept its fixed activation budget, and held the MATH validation gain. The useful question now is why fresh random compression survives where a reused PowerSGD Q direction can fail.
Fresh PRF masks can carry the compressed fast circuit through a long RL run while learning, but a reused Q subspace can turn the same training loop into a directional failure. The evidence points to persistent direction, not compression magnitude alone.
The two Q runs use the same directional codec. Their main difference is how long the anchor state is allowed to remain stale. The PRF run redraws the compression direction instead.
The same preferred subspace remains active long enough for the mismatch to become a training direction.
This run still becomes rough, but the next anchor arrives before the transient locks into a collapse.
The instantaneous error is large, but its direction changes with each mask and does not become a second objective.
GRPO is a repeated feedback loop. The dense sampler generates answers, verifiable rewards rank them, and the trainer changes token probabilities so better answers become more likely. Those weights return to the sampler and the loop repeats. Compression sits inside the trainer view of that loop.
PPO clipping limits one update relative to the stored old policy. It does not limit the mismatch that can accumulate across many outer RL steps between the dense sampler and the compressed trainer.
Only the combined standard clip fraction was logged in these runs. The loss also contains a separate dual-clip guard for extreme negative-advantage cases, but there is no W&B series for that branch here.
The slow Q bias can therefore accumulate while the sampled reward still looks healthy. Once enough token ratios cross the clipping bounds together, more of the surrogate becomes flat. The useful update signal shrinks, the compressed actor becomes sharply concentrated, and the next dense rollout can become narrower. That weakens the within-group reward contrast that GRPO needs. The visible score drop is sudden even though the directional drift was building gradually.
critic/score/mean: mean verifiable reward of sampled answers, not a learned critic value.actor/entropy: sharpness of the compressed trainer view. PRF is not directly comparable with Q here.actor/kl_loss: movement from the reference policy inside the trainer view. Dense KL rises too.rollout_corr/kl: gap between dense sampling and compressed training. Trend matters more than level.actor/pg_clipfrac: share of tokens whose PPO surrogate is clipped. The run logger combines the ordinary upper and lower branches.The shaded region marks the failed Q run from the first clear score cliff through the entropy trough. Curves are reconstructed from W&B without smoothing.
The slower Q run learns first, then loses answer quality abruptly. The faster refresh and PRF runs hold.
The failed Q trainer view becomes extremely sharp. PRF stays diffuse because its mask changes the view.
Dense KL also rises. The failed Q run matters because its rise couples to the other failure signals.
PRF starts with a large but stationary wedge. The failed Q wedge accelerates with the score cliff.
Clipping pressure rises before the failed Q score cliff. The logger combines upper and lower clipping.
A spike is not enough by itself. The faster Q run also sees clipping pressure, but refresh breaks the persistent direction and the score survives.
PowerSGD does not make the whole model gradient low rank. It inserts a projector at each compressed pipeline boundary. While the basis is fixed, both the activation and the boundary gradient pass through the same preferred subspace.
What is directional: unless the activation already lies in the Q subspace, the same residual is repeatedly removed. Reusing Q makes this error coherent across RL updates.
What is mean-preserved: each boundary coordinate is preserved over fresh masks, so the first-order activation error has no persistent direction. The full model output and parameter gradient are not guaranteed to be unbiased because the network is nonlinear.
The dense sampler chooses tokens with full weights, but the trainer scores them through a compressed pipeline. With Q, the trainer repeatedly sees the policy through the same lens and can move the weights toward that lens. The sampling policy and training policy then separate in a directed way. With PRF, each trainer view is noisy but the noise distribution is stationary. The dense sampler can still sharpen while the codec-side view stays diffuse. This is why PRF entropy can remain high without meaning that the actual sampled policy failed to learn.
The weight evidence does not support calling RLVR movement simply messy, high rank, or sparse. The total movement is small in both Q runs, yet the failing run shifts its movement toward earlier blocks while the surviving run stays more centered and PRF remains later-heavy. The difference is spatial direction, not simply how much the weights moved.
The modified exact-k PRF run keeps a fixed sparse payload, redraws the mask each step, and uses constant inverted rescaling. It completed the full horizon without a reward or entropy cliff. Validation improved early and stayed inside the learned band.
The final checkpoint comes from the training log because W&B omitted the last row.
There is no single collapse band that works across codecs. Dense KL can rise and succeed. PRF can carry a large stationary sampler gap and succeed. The controller has to watch a multivariate health envelope and react to a coordinated turn.
Communication-efficient RLVR weekly report · real W&B trajectories · no smoothing