Two-circuit communication-efficient GRPO, Qwen2.5-1.5B, Big-Math
Each curve is a training run under a different communication-efficient configuration, against a dense (uncompressed) baseline. Metric: critic/score/mean, 9-step moving average, one seed.
“Stale signed EMA every step” is the standard stale-anchor signed-EMA gradient correction, applied on every optimizer step. “Every 20 steps” means that same correction is applied only when the anchor fires (once per 20 steps). The other axis is whether a weight projection is added on top (none, linear, or adaptive).
| Configuration | mean | peak | final | vs dense |
|---|---|---|---|---|
| Dense (no compression) | 0.504 | 0.606 | 0.580 | 0.000 |
| No projection, stale signed EMA every step | 0.445 | 0.568 | 0.440 | -0.059 |
| No projection, stale signed EMA every 20 steps (fires only) | 0.433 | 0.528 | 0.483 | -0.071 |
| Linear weight projection, stale signed EMA every 20 steps | 0.409 | 0.489 | 0.443 | -0.095 |
| Linear weight projection, stale signed EMA every step | 0.378 | 0.507 | 0.358 | -0.125 |
| Adaptive projection (method 1), stale signed EMA every 20 steps | 0.437 | 0.520 | 0.447 | -0.066 |
| Adaptive projection (method 1), stale signed EMA every step | 0.418 | 0.533 | 0.303 | -0.086 |
| Adaptive projection (method 2), stale signed EMA every step | 0.376 | 0.510 | 0.376 | -0.128 |
| Weight-trajectory analysis | GSM8K | Big-Math |
|---|---|---|
| Linearity (predictive linear-fit R²) | ≈ 0.54 | ≈ −1.0 (no linear structure) |
| Direction coherence (consecutive-update alignment) | ≈ 0.86 | ≈ 0.15 (near-orthogonal) |
On GSM8K the weights move coherently and roughly linearly, so an intervention could help there: applying the signed EMA only on fire steps, or adding a projection, gave a measurable lift. On Big-Math the updates are near-orthogonal with essentially no linear structure, so there is no consistent direction to project along or average, and no variant beats plain stale signed EMA every step. Big-Math simply moves more randomly.
Appendix
The same configurations on GSM8K, whose weight trajectory is coherent and roughly linear. Same metric, critic/score/mean, 9-step moving average, one seed. On GSM8K the pattern reverses from Big-Math: correcting every step with the stale reference collapses without projection and fades with it, while correcting only on fires, or adding a projection, tracks the dense ceiling.
| Configuration | mean | peak | final |
|---|---|---|---|
| Dense (no compression) | 0.799 | 0.906 | 0.863 |
| No projection, stale signed EMA every 20 steps (fires only) | 0.485 | 0.766 | 0.719 |
| No projection, stale signed EMA every step | 0.401 | 0.664 | 0.372 |
| Linear weight projection, stale signed EMA every 20 steps | 0.590 | 0.804 | 0.762 |
| Linear weight projection, stale signed EMA every step | 0.583 | 0.854 | 0.646 |
Even though the optimization path is a coherent near-straight line, a reference frozen twenty steps back points where the weights used to be, and because the correction is a systematic directional bias rather than random noise, re-applying that stale direction on every step steadily pushes training the wrong way and the bias compounds until it collapses; refreshing the reference or projecting it forward realigns it with where the weights are now, so the lift returns.
Values are the training reward (critic/score/mean), not held-out accuracy; single seed, 9-step moving average. Some Big-Math runs were stopped early once the ordering was clear. Coherence and linearity figures from the offline weight-trajectory ablation.