Two-circuit communication-efficient GRPO · Qwen2.5-1.5B · Big-Math · anchor latency 20/20

How much correction, and how fresh an anchor? A signed-EMA (α, β) sweep

Seven communication-efficient training runs that vary only two knobs of the gradient-correction merger, plus one dense (uncompressed) control as the ceiling. One seed, 75 steps, no validation — we read the training reward (critic/rewards/mean) and the gradient norm (actor/grad_norm, the stability signal). Data pulled from Weights & Biases.

Verdict. Every run that applies some correction from a fresh-enough anchor trains stably at this latency. The best communication-efficient setting is α=0.25, β=0.00 (freshest anchor memory) — highest final reward among the compressed runs and the most stable gradients — with the incumbent (β=0.50) essentially tied. Dense stays the ceiling (~0.05–0.10 reward above any compressed run and far steadier).

Did anything collapse? Only one: the pink curve, α=0.25 / β=0.90 (stalest anchor) — its reward falls away after step ~55 and its answers balloon in length. The “merger-off” floor does not collapse; it just plateaus low and noisy. So “nothing collapsed” is almost right — the exception is the stalest-anchor run.

The setup in plain terms

Training runs on two copies of the network at once. The fast circuit is the normal training pass, but the signal crossing the pipeline boundary is compressed (PowerSGD, ~5% of the bytes) — cheap to send, but the compression corrupts the sign of many small gradient coordinates. The anchor circuit is an occasional uncompressed pass run from a deliberately stale snapshot of the weights; it is slow but clean. A merger combines them right before each optimizer step:

G_corrected = α · G_fast + (1 − α) · |G_fast| · sign(M)

It keeps the size of each gradient coordinate from the fast (compressed) pass, but takes the sign from M, a running average of the clean anchor gradients. Magnitude from the cheap signal, direction from the trustworthy-but-stale one.

α — how much correction

1.0 = no correction (use the compressed gradient as-is). 0.0 = take the sign entirely from the anchor. 0.25 = mostly trust the clean sign, keep a little of the raw signal.

β — how fresh the anchor is

The decay of the running average M. 0.0 = freshest (M is just the newest anchor fire). 0.9 = stalest (a long, smooth, slow-moving average of old fires).

cadence / delay_K = 20 / 20

The anchor fires every 20 optimizer ticks (~7 times over the run) from weights that are ~10 steps old. This is the hard “longer coordination interval” regime where staleness bites.

The eight runs

Run (color = curve)	α	β	reward avg	peak	min	final (last-10)	grad avg	grad max	resp len
Dense — comm-eff OFFceiling reference	—	—	0.493	0.578	0.400	0.512	0.17	0.2	523
α=0.75, β=0.50light-touch correction	0.75	0.50	0.445	0.535	0.350	0.467	2.70	10.9	472
α=0.25, β=0.00 — freshest Mbest comm-eff	0.25	0.00	0.455	0.560	0.354	0.461	1.52	3.2	588
α=0.50, β=0.50α interior	0.50	0.50	0.458	0.565	0.341	0.458	2.78	16.2	557
α=0.25, β=0.50 — incumbentGSM8K-tuned default	0.25	0.50	0.448	0.571	0.344	0.451	1.52	3.3	556
α=1.00, β=0.50 — merger OFFno correction (floor)	1.00	0.50	0.414	0.493	0.352	0.434	4.16	18.9	489
α=0.00, β=0.50 — max correctionsign fully from anchor	0.00	0.50	0.446	0.564	0.353	0.414	1.91	4.0	524
α=0.25, β=0.90 — stalest Mcollapses late	0.25	0.90	0.435	0.533	0.316	0.371	2.33	8.4	805

Reward stats over steps 11–75 (after the shared warm-up, where the runs actually differ); final = mean of steps 66–75; gradient-norm over 11–75; response length = mean tokens over steps 66–75. Sorted by final reward. Higher reward is better; lower / less-spiky gradient norm is more stable.

Training reward over time

critic/rewards/mean — higher is better (5-step moving average)hover a legend entry to isolate a curve; dot = last step

The dense control (gray dashed) sits clearly on top and keeps climbing. The compressed runs bunch together in the middle — and because a single run's reward bounces by ~0.2 step-to-step, the ordering within that bunch is mostly noise. The one curve that clearly breaks away is the pink stalest-anchor run, which peels downward on the right.

Gradient norm — the real stability signal

actor/grad_norm — lower & steadier is more stable (log scale)steps 1–10 are the shared cold-start warm-up before the merger engages

Reward is noisy, so gradient norm is the cleaner way to see what is going well. All compressed runs share a big warm-up transient (steps 1–10, up near 200) while the anchor memory is still cold; the merger only engages once the anchor has fired (~step 10), after which the norms drop sharply. The dense run never has that transient — it is flat near 0.17 throughout, which is exactly why it is the ceiling.

What works, and why

Correction helps, up to a point. With no correction (α=1.0, merger off) the gradient is the noisiest of all — average norm 4.2, spiking near 19 — and reward is the lowest plateau. Adding correction (lowering α) both lifts reward and calms the gradient. But taking the sign entirely from the anchor (α=0.0) fades late: 100% reliance on a stale sign eventually drags. The sweet spot is a little correction, α≈0.25.
Anchor freshness is the difference between stable and collapsing. Holding α=0.25 and sweeping the anchor memory β: freshest (β=0.0) is best and most stable, the incumbent (β=0.5) is tied, and the stalest (β=0.9) collapses — final reward 0.371, its worst dip 0.316, and response length blows up to ~805 tokens (vs ~500 elsewhere). A correction sign averaged over many old fires points where the weights used to be; at 20/20 latency that error compounds until training runs the wrong way.
The two knobs interact. Correction is only safe when the sign it injects is fresh. Moderate correction + fresh anchor (α=0.25, β=0.0) is the stable, high-reward corner; heavy reliance on a stale anchor (β=0.9, or α=0.0) is where it breaks. Stability (gradient norm) tracks this directly: tightest for α=0.25 with fresh/medium β, worst for merger-off and the stalest run.
Dense is the ceiling and the calmest run. It costs the compressed method ~0.05–0.10 reward and ~10× the gradient stability — the price of communicating in ~5% of the bytes.

This lines up with the earlier anchor-delay finding on this hub: the same signed-EMA merger is near the dense ceiling at short latency (K=5) but has historically fallen below the no-merger floor at K=20. What this sweep adds: at 20/20 the merger still holds as long as the anchor memory stays fresh (β small); it is specifically the stale memory (β=0.9) that reproduces the old collapse.

Recommendation

Use α=0.25, β=0.00 (moderate correction, freshest anchor memory) as the Big-Math default at 20/20 latency; the incumbent β=0.50 is a safe equivalent. Avoid long anchor memory (β≥0.9) and avoid full sign replacement (α=0.0). Because the top compressed runs are within reward noise of each other, a natural next step is a small confirmation grid around this corner (α ∈ {0.25, 0.50} × β ∈ {0.0, 0.25}).

Values are training reward (critic/rewards/mean), not held-out accuracy; single seed. Reward curves are a 5-step moving average; gradient-norm is raw on a log axis. Data read from Weights & Biases (project verl_compression_research, group hyperparam_tuning_bigmath_signed_ema_c20d20); the last 1–2 steps of most runs were dropped by the async uploader and could not be backfilled (the training box was reclaimed after the sweep), so curves end at step 73–74. Substrate was identical across the seven compressed runs (anchor fired 7×, byte ratio 0.050, no cold-M fallbacks after warm-up); the dense control ran with the method fully off.