Communication-Efficient Pipeline-Parallel GRPO trained at 20/20 without collapse

LA-WARMUP-V2 · issue #59 · Qwen2.5-1.5B-Instruct + GSM8K · 1×H200 · vanilla GRPO (no-KL, no-entropy), flat LR 1e-6, 100 steps, val every 25 · locked substrate: PowerSGD rank-77, anchor cadence = delay_K = 20, signed_ema (α=0.25, β=0.50), resp=1024 · final deliverable, 2026-07-04

Contents: 1 Result · 2 What each run is · 3 Mechanism · 4 The four exercises E1–E4 · 5 E4 & softening · 6 Q-health · 7 Anchor-owned Q (C8a) · 8 Comm efficiency · 9 Findings · 10 Theory · 11 Implementation · 12 Caveats · 13 Links

1 · The result — validation accuracy per arm

Held-out GSM8K accuracy (val-core/openai/gsm8k/acc/mean@1, 1319 prompts) at the four validation checkpoints. DENSE is the dashed ceiling; the collapse arms (C7 on current code, A0 historical) both dive to 0.064. C2 holds a flat near-dense plateau; C1 overshoots at step 75 then softens; C8a rises steadily but pays a warmup tax.

Final ranking by val@100: DENSE 0.776 > C2 0.738 > C8a 0.686 > C1 0.582 > C7 0.390 (collapsed).

Results table

val@0 is identical across the config-only comm-eff arms (0.083); DENSE has no anchor so its val@0 is not run. Every number is drawn from the per-run train.log / wandb (see §13).

2 · What each run actually is

Shared substrate (identical across every arm): Qwen2.5-1.5B-Instruct + GSM8K, 1×H200, vanilla GRPO (no KL, no entropy bonus), constant LR 1e-6 (no decay), 100 training steps, val every 25 (the epoch-1→2 data boundary is at step 58 = 7473/128). The comm-eff arms add PowerSGD rank-77 activation compression, anchor cadence = delay_K = 20, and the signed_ema merger (α=0.25, β=0.50), resp=1024. The arms differ only in the four knobs below — everything else is byte-identical. ("step=" in the comm_eff logs is an optimizer tick = 2× the training step.)

The four knobs that vary: anchor.owns_q (who rebuilds the PowerSGD basis Q), anchor.lookahead_* (whether/when the projector engages), anchor.warmup_mode (no_correct = skip the anchor pass while M would be stale vs stale_correct = today's raw stale correction), and spectral.cadence (fold the held M every tick vs only on fresh-M fires).

3 · The mechanism — project → measure → correct → feedback

The fast net trains on PowerSGD-compressed (rank-77) boundary activations: its per-target gradient keeps the right magnitude but has corrupted signs on the coordinates the codec dropped. A clean "anchor" gradient could repair those signs — but in pipeline-parallel the anchor lags by K=20 ticks, where the stale weights have rotated ~orthogonal to the live ones (k-collapse, the mechanism that killed A0). The method makes the anchor gradient both fresh (via linear projection) and clean (measured at the projected point on the current batch), then folds only its signs into the trusted fast-net magnitude.

The 5 SOLID invariants (all verified live at 0 / dev=0 in the logs): no-peek staleness cross-rank determinism (dev=0) magnitude/sign separation cold-M guard anchor isolation

4 · The four exercises E1–E4

The redesign is defined by four decisions, each posed as an exercise and verified against a specific run.

5 · E4 head-to-head + softening decomposition

C1 folds the held M every tick (signs up to 19 ticks stale between fires); C2 folds it only on the fire tick when sign(M) is fresh. Side-by-side, the two share the same early trajectory but diverge in the epoch-2 window: C2 plateaus, C1 gives back its overshoot.

6 · Q-health — the cold-Q blowup and its cure (E1)

Two per-step signals from the CSVs. (a) reconstruction error: fast-owned Q (C1) drops from 0.975 to ~0.03 within ~7 steps and stays flat. The owns_q arms (C7, C8a) hold Q frozen at 0.975 through step 9, then snap to ~0.04 at the first fire (step 10) when the anchor rebuilds Q. (b) gradient norm (log scale): C1/C2 settle near ~1–2; C7 blows to the hundreds–thousands during the cold-Q window and later spikes into the ignition that precedes its collapse.

Panel (b) is drawn on a log axis so the fast-owned arms' calm ~1–2 range and C7's early hundreds/thousands and its late ignition spike are visible on one plot. C7's grad_norm hit 1201 at step 5 and spikes again near the collapse ignition; the ignition precedes the val@75 dive to 0.064.

7 · Can the anchor own Q if projection keeps it fresh? (C8a vs C7)

C7 and C8a start identically — both owns_q=true, both begin with a cold/stale Q rebuilt at stale weights. The single difference is projection: C7 never projects (raw stale θ + stale rollouts); C8a projects from step 30, so the anchor rebuilds Q from the projected θ̂ (≈θ_t, fresh) instead of θ[t−K]. The outcome is the whole story of k-collapse.

Collapse signature — the degenerate long-rambling mode (C7)

C7 doesn't just lose accuracy; it drifts into a degenerate policy. Its response length balloons from ~250 to ~1000 tokens and its entropy caves toward ~0.4 — the classic long-rambling collapse — while healthy arms hold resp_len ~200–250.

C7 response length climbs across steps ~55–75 to ~1000 tokens as the policy degenerates, then partially unwinds to ~730 during the erratic partial recovery; entropy over the same window falls to ~0.35–0.42. The healthy arms (C1 dashed shown) stay in a tight ~200–250 band.

8 · Communication efficiency — near-dense at 5% of the traffic

The training-path boundary traffic (activation + gradient across pipeline-stage boundaries) is bytes_ratio = 0.050 of the dense-equivalent — a ≈20× reduction. The whole point of the study is that this buys near-dense quality: C2's 0.738 sits within 0.038 of the dense 0.776.

Headline: ~20× less boundary communication for a final-accuracy gap of just 0.038 against the dense ceiling — the project's north star. The method pays an early-learning tax (val@25: dense 0.751 vs C2 0.510) that the correction largely closes by step 50 (C2 0.732 vs dense 0.773).

9 · Empirical findings (all 9, with evidence)

10 · Theoretical claims — why the mechanism holds

The 4-stage loop (§3) rests on four design choices, each with a stated reason:

• Linear projection. RLVR weights move ~linearly (R²≈0.9) and the linear fit holds ~600 steps ≫ K=20, so a two-snapshot extrapolation θ̂ recovers the live point accurately at the operating staleness. At h=g the projector reduces exactly to the frozen AsyncPP seed 2θ[τ0]−θ[τ1].
• Sign-only correction. The codec corrupts signs, not magnitudes; an approximate M must not be allowed to set the step size. So magnitude comes from the fast net (trustworthy) and only the sign comes from the anchor (the information the codec destroyed).
• Matched current-step rollouts. Because θ̂ ≈ θ_t, the anchor measures on the same data distribution the fast net is training on — a same-batch reference, not a stale one.
• Fast-owned Q. Q must span the LIVE activation subspace. An anchor-built Q at stale weights spans a rotated / wrong subspace at high K — the k-collapse. Fast Q is self-consistent at the live point, refreshed every step. (C8a shows projection can also keep an anchor-owned Q fresh, at a warmup cost.)

The 5 SOLID invariants, all verified live in the logs:

1. No-peek staleness — every anchor input (both snapshots, the residual) is ≥K-stale; nothing reads the live weights.
2. Cross-rank determinism — cross_rank_max_rel_dev = 0 for both Q and the learned residual (DP-identical across ranks).
3. Magnitude/sign separation — magnitude strictly from G_noisy, sign strictly from M.
4. Cold-M guard — ‖M‖ ≤ eps ⇒ gradient passes through unchanged (no correction from an unset M).
5. Anchor isolation — anchor takes no optimizer step, generates no rollouts, recomputes no rewards, and corrects no gradient outside the merger (all four counters = 0).

11 · Implementation

All changes are additive; defaults are byte-identical to upstream verl (with the method's knobs at their defaults, training is unchanged). Commit 7a9653a0 on vast-ai-workload.

• Two new knobs on comm_eff.anchor: warmup_mode ∈ {stale_correct (default), no_correct} and lookahead_min_snapshots ∈ {−1 (default), [2, n_points]}.
• 4 config-validation rules (e.g. no_correct is illegal with owns_q=true).
• New counter warmup_no_correct_skips; the ring gains min_points — a single readiness gate shared by the no_correct skip and the projected-vs-fallback branch.
• 66 unit tests pass (15 new + no regressions); the GPU micro-smoke gated every arm.

7 files touched:

12 · Honest caveats

• Early compression tax. At val@25 dense is 0.751 while C1/C2 are 0.477/0.510 — compressed grads plus the no-correct warmup learn slower early. Most of the gap closes by step 50, but it is real.
• C1's late softening. C1 gives back its overshoot (0.762@75 → 0.582@100). The dense decomposition attributes ~0.01 of this to generic RL overfitting and the remainder to the every-tick stale-sign correction — a real cost of the C1 configuration (not present in C2). A recommended follow-up is an LR-decay arm to confirm the tail can be recovered.
• C8a's cold-Q warmup tax. Anchor-owned Q is cold/stale over steps 1–29, so C8a (0.686) lands below fast-owned C2 (0.738). Anchor-owned + projection is viable but not the best default.
• The double-skip. Projection first fires at step 30, not step 20 — both fires at ticks 20 & 40 resolve to the same snapshot_tick=19 under delay_K=20 replay granularity, so the second is skipped. This is a replay-granularity artifact and is benign (warmup_no_correct_skips locks at 2).

13 · Runs & links

wandb project: verl_compression_research (entity shamanework-pl).

Final deliverable for issue #59 (LA-WARMUP-V2). Self-contained (inline CSS + inline SVG, no external assets); renders offline. All headline numbers from research/runs/LA-WARMUP-V2/FINDINGS.md; all full-resolution curves computed from the per-step CSVs in research/runs/LA-WARMUP-V2/data/.