← All runs · 90-prf-exactk-600 · mid-run diagnosis

#90 mid-run diagnosis: the KL climb is a codec-view artifact, the run is healthy, and the mismatch roadmap

RUN COMPLETE 600/600: NO COLLAPSE mid-run analysis 2026-07-24 no intervention: run untouched v2: fix decided + implemented (sections 8-9) v3: capability = dense, in-domain + OOD (section 10)

Companion artifacts (CSV pulls used for every figure): artifacts/90-prf-exactk-600-mismatch-diagnosis/

1 · TLDR: the four questions, answered

v2 addendum: the operator asked for a decided fix, not a ranking. Section 8 now contains the decision (P0-P3), backed by a kill table computed from this run's own logs (the IS/bypass/calibration family is measurably impossible at this compression: mean token-IS weight ~0.001) and a verified literature sweep (section 9). The fix package is implemented on exp/mismatch-control-kit.
  1. Is the current run OK? Yes. Let it finish untouched. Every dense-view health signal is green at step 599/600: val MATH acc went 0.451 (step 0, base) → 0.661 (150) → 0.663 (300) and is holding; train score 0.35 → 0.73; grad norm is 1.3-2.0 and falling; the kl_loss climb is linear and decelerating (slope 0.0020/step over steps 60-150, 0.0015/step over 150-330). The climbing actor/kl_loss and actor/entropy live in the codec view (both nets measured through the 95% mask); they are not the collapse signature. The plan gate "kl_loss well under 0.2" is formally breached (0.50 @ 599) but that number was calibrated on 100-step history; every mechanism-level precursor of a real break is absent (section 4 checklist). Projected codec-view kl_loss at step 600 if the current slope holds: about 0.9.
  2. Can the anchor do more, within the comm budget? Partly, but not by replaying harder: anchor replay measurably does not bend the KL curve (dKL over the 2 steps after each of the 16 fires: 0.0037 vs 0.0027 elsewhere, i.e. noise). The anchor bounds weight drift; the wedge is a sampler-vs-trainer view gap, not weight drift. The cheapest real lever is not on the slow→fast wire at all: the sampler's own token logprobs are already local to the fast circuit (zero bytes) and enable decoupled token-IS / rejection / calibration. The one genuinely new anchor lever this analysis proposes: the anchor is the natural calibrator; it already recomputes dense logprobs on replay batches, so shipping the fast circuit's codec-view logprobs to it (1 float/token, ~2.6 MB per replay) lets it fit a systematic-bias correction and broadcast back ~28 bytes. Sign-of-M remains available (193 MB vs 3.09 GB full M, 16x) but M is anchor-owned today and its absence on the fast side is not what causes the mismatch.
  3. bypass_model / inference-engine probs / systematic bias / better selection? bypass_mode=true today would not diverge, it would silently stop learning: the mean per-token log-ratio between codec-view trainer and dense sampler is -14.4 nats, so ratios are ~6e-7; positive-advantage tokens get e^-14-scaled gradients and negative-advantage tokens get exactly zero through the clip. The viable form is the decoupled mode already in the codebase: token-IS on PowerSGD (clvaf683) cut kl_loss 64% (9.84 → 3.54 @200), prevented entropy collapse (1.78 vs 0.027) and kept +2.6 val points at 200; its cost is an agreement filter (mean IS weight ~0.2). Treating the mismatch as a systematic bias is the right next idea and is measurable for a few logging lines (section 6.3): if a fitted global/per-layer calibration absorbs most of the 14.4 nats, calibrated correction becomes feasible; if dispersion stays huge, only filtering works. "Send the most important activations" is the coherence trap: every importance/structure-adding lever measured in #89 paid for it in KL (rms_match +38x, FRLR +5-16x); random selection is the feature. We are not using arXiv 2603.19470 (ALP); it is an off-policy-correction paper whose zero-mean-noise theory independently predicts exactly our PowerSGD-collapses / PRF-benign split (section 6.5).
  4. RL gradients are sparse-but-high, so is this theoretically fixable? True for the weight/update channel, and that channel is already solved (movement at collapse is 0.03-0.13% with cosine ~1.0; rank-1 RELEX + signed-EMA ride exactly this). The activation channel is different: hidden states are dense, and what breaks training is not lost magnitude but directional bias times coherence. Sparsity arguments do not transfer; unbiasedness arguments do. Fix = keep the random codec + police the policy accounting (IS/RS/calibration), not "send bigger coordinates".

2 · The run at a glance

itemvalue
Codecprf_mask, p=0.95, exact-k random 77/1536 dims kept per token (order-statistic on the PRF draw; top-k by magnitude is deliberately forbidden), constant rescale 1/(1-p)≈20x, pp_size=8 → 7 masked boundaries (layers 3,7,11,15,18,21,24), mask keyed on (seed, layer, step, sample_id, position_id, channel): bit-identical across the old-logprob / train / reference forwards of one step, fresh across steps
Viewsrollouts: vLLM, dense weights (uncompressed); trainer old/new/ref logprobs: codec view (mask_recompute=true, mask_reference=true)
GRPObatch 128 = ppo_mini 128 → exactly one on-policy optimizer tick per step; rollout n=8; 1024/2048 lengths; lr 1e-6; kl_loss_coef 0.001 (low_var_kl)
Two-circuitanchor paired dense replay cadence/delay 20/20 ticks (= every 20 steps here; 16 fires by step 337), rank-1 RELEX W2, signed-EMA M (beta_anc 0.25, alpha 0.25); anchor_owns_Q=false (PRF has no Q)
Correctionrollout correction strictly OFF (rollout_is=null, bypass_mode=false); vLLM logprobs logged for diagnostics only (calculate_log_probs=true)
StatusCOMPLETE: 600/600 steps in 12h52m (mean 89.4 s/it; mid-run check at step 337 saw 68.7 s/it, GPU 91% util, 127/144 GB). Box left running; no teardown authorized.
CheckpointsSAVE_FREQ=100 via launch-shell override (launcher default is -1) + comm_eff R2 sink with delete_local: six checkpoints (steps 100-600) uploaded to s3://shamane-pluralis/autonomous-harness-rlvr-compression/90-prf-exactk-600/prf-exactk/checkpoints/; sink drain n_uploaded=66 n_errors=0, independently verified by bucket listing (global_step_600 = 10 objects, 19.47 GB). Nothing irreplaceable remains on the box.

3 · Reading the four worrying panels correctly

The four panels that triggered this analysis measure three different policies. Keeping them separate is the whole diagnosis:

metriccomparesviewwhat it is doing
actor/kl_loss 0 → 0.50actor vs frozen referenceboth through the same per-step maskreal policy movement as seen through the codec, growing linearly, decelerating; the 0.001 coefficient makes its loss contribution ~5e-4
actor/entropy 7.808 → 7.834actor onlycodec viewstructural: the 20x constant rescale inflates codec-view entropy to ~7.82 (dense control 0.18, dropout 0.14, #89 frame). The +0.026 drift in 330 steps (+7.5e-5/step) is cosmetic. This is NOT the sampling entropy; the dense policy is sharpening (rollout_log_ppl 0.84 → 0.13)
actor/pg_clipfrac, pg_clipfrac_lower, ppo_kl ≡ 0actor-new vs actor-oldcodec vs codeczero by construction: one optimizer tick per step and a bit-identical mask across the old and train forwards force ratio ≡ 1. It is proof the within-step identity holds, and it also means PPO's ratio/clip machinery is inert: it cannot see, and therefore cannot correct, the sampler-trainer wedge
rollout_corr/kl 13.9 → 14.4dense vLLM sampler vs codec-view trainercross-viewthe wedge diagnostic, decomposed below: the growth is ~87% benign sharpening

3.1 The wedge is flat; the sharpening is what rises

rollout_corr/kl = training_log_ppl - rollout_log_ppl (per-token nats). The codec-view side (training_log_ppl, the wedge proper) has moved 14.46 → 14.39 (min, step 52) → 14.74: a ±0.1-nat band over 599 steps. The dense side (rollout_log_ppl) fell 0.84 → 0.09 because the policy is getting more confident on MATH: normal, wanted RL behavior. Of the +0.81-nat rise in the gap, +0.71 (87%) is the dense policy sharpening and only +0.10 (13%) is codec-side drift. Contrast PowerSGD (rose): its wedge itself explodes 4.4 → 29.5 nats while the policy sharpens into its own codec's preferences, and val crashes.

Decomposing rollout_corr/kl: wedge (flat) vs sharpening (falls) [nats/token] PowerSGD training_log_ppl (wedge grows, then explodes) run 90 training_log_ppl (codec wedge: 14.4-14.75 band) run 90 rollout_corr/kl run 90 rollout_log_ppl (dense policy sharpening) 0 5 10 11.93 15 20 25 30 0 100 200 300 400 500 600 global step uniform over 152k vocab = 11.93

Note the level: the codec-view trainer assigns on average e^-14.5 ≈ 5e-7 probability to the tokens the dense sampler actually picked, which is below uniform (uniform over the 152k vocab = 11.93 nats). At 95% masking the codec view is not "the policy plus noise"; it is a different, nearly-random policy whose gradient is nonetheless useful in expectation because the mask is fresh, zero-mean and incoherent across steps. That is the entire PRF-vs-PowerSGD story in one number.

4 · Verdict: healthy, with named tripwires

actor/kl_loss (codec-view reference KL, log scale) PowerSGD r77 (tb4m0k23, no IS) PowerSGD r77 + token-IS (clvaf683) run 90: PRF exact-k p=0.95 (woqs8zra) 0.001 0.01 0.1 1 10 0 100 200 300 400 500 600 global step dense control @40 (#89 frame) run-90 points below 0.001 clipped to axis floor
actor/entropy (codec view, log scale) PowerSGD r77 (collapses to 0.027) PowerSGD + token-IS (recovers) run 90: PRF exact-k (structural 7.8 band) 0.03 0.1 0.3 1 3 8 0 100 200 300 400 500 600 global step dense control @40
val-core MATH-lighteval acc/mean@1 (dense view: the metric that matters) PowerSGD (crashes at 200) PowerSGD + token-IS run 90 PRF exact-k (0.451 base, rising through 0.673@450) 0.45 0.5 0.55 0.6 0.65 0.7 0 100 200 300 400 500 600 global step run-90 step-0 value read from on-box train.log
critic/score/mean (train reward) PowerSGD PowerSGD + token-IS run 90 PRF exact-k 0.3 0.4 0.5 0.6 0.7 0 100 200 300 400 500 600 global step

4.1 Collapse-signature checklist (run 90 vs the PowerSGD break)

signalPowerSGD tb4m0k23 (breaks by 200)run 90 @ 599read
kl_loss shapeaccelerating: 0.11@25 → 1.15@50 → 6.12@100 → 9.84@200linear, decelerating: slope 0.0020 → 0.0015/stepno break
entropycliff DOWN 5.54 → 0.027 (deterministic)flat structural band 7.81 → 7.83, drifting UPno break
ppo_kl / clipfrac0.21 / 12.7% (within-step identity broken)exactly 0 / exactly 0identity intact
grad_normspikes to 29.9, settles 17.51.3-2.0, decliningno break
wedge (training_log_ppl)4.4 → 29.5 nats14.4 → 14.6 (flat)no break
train scorepeak 0.726@150 → 0.583@2000.35 → 0.73, plateau holdingno break
val MATH acc0.667@150 → 0.561@2000.451@0 → 0.661@150 → 0.663@300capability gained + held
Verdict for question 1: the run is doing what issue #90 was launched to prove: 95% activation compression training long without collapse. The kl_loss number in the plan gate (0.2) was set from 100-step history and should be re-judged at close-out on the signals above, primarily val at 450/600 and the absence of every break signature. Do not stop, do not modify.
Final outcome (run completed 2026-07-24, 600/600 steps, 12h52m at 89.4 s/it): every tripwire below was audited over steps 335-600 after completion and none fired: worst 30-step kl_loss slope 0.00245 (bar 0.004); entropy minimum 7.831, never falling (ended 7.846, drifting up); wedge maximum 14.75 (bar 15.0); ppo_kl exactly 0 on every step; val rising through 0.6733 at step 450 and 0.6613 at step 600 (from train.log; WandB drops the final step; the 1.2-point dip from the 450 peak is inside the ~2.1-point 1-sigma noise of a 499-problem single-sample eval, not a cliff). Final codec-view kl_loss 0.899 at step 599, within 1% of the linear projection made at step 337. Train score ended ~0.736. The 600-step no-collapse validation the issue was launched for is delivered; the only breached criterion is the mis-calibrated "kl_loss well under 0.2" plan text, re-scoped in this report.

4.2 Tripwires for the remaining steps (observe only)

If a tripwire fires, the action is still not intervention mid-run; it is labelling the close-out verdict and the follow-up arm correctly.

5 · Question 2: the anchor, the wire, and what actually fits through it

5.1 What the anchor can and cannot fix

Measured on this run: the 16 anchor replay fires (every 20 ticks) produce no detectable bend in the codec-view KL (mean dKL over the 2 steps after a fire 0.0037 vs 0.0027 elsewhere). This confirms the #88 exoneration from the other side: the anchor's job is bounding weight-space drift (which it does; at these horizons total weight movement is fractions of a percent), while the kl_loss climb is the policy adapting to the mask distribution and sharpening, seen through the codec. More replay, smarter replay timing, or a stronger M transaction cannot remove a wedge that lives in the forward-pass view, not in the weights. Within the fixed cadence/delay 20/20 contract, the anchor is already doing its part.

5.2 The communication budget, itemized (this config, 659k tokens/step)

channeldirectionpayloadbytesstatus
PP boundary activationsfast↔fast77/1536 bf16 values per token; indices derived from the shared PRF seed, never sent; backward is masked by the same chain rule101 MB per boundary per pass vs 2.02 GB dense (20.0x)the budget being validated
mask identityfast↔fast64-bit seed + step counter8 Bfree by design
rank-1 RELEX weight syncslow→fastu,v per weight matrix~3 MB bf16 per firein use
signed-EMA Manchor-owned, not shippedif ever shipped: sign(M) 1 bit/param + per-tensor scale193 MB vs 3.09 GB full bf16 (16x)available, but M absence on the fast side is NOT the mismatch cause
rollout batch for paired replayfast→slowtoken ids + rewards~2.6 MB per firein use
vLLM behavior logprobslocal to fast circuit1 float/token0 wire bytesalready logged; unused by the objective
codec-view logprobs to anchor (proposed, 6.3)fast→slow1 float/token on replay batches~2.6 MB per fireenables anchor-side calibration fit
calibration broadcast (proposed, 6.3)slow→fasttemperature + bias per boundary (14 floats), or rank-1 logit correction28 B, or ~300 KBthe "eliminate systematic bias" lever
The punchline for question 2: the cheapest strong lever costs zero wire bytes, because the quantity you need (the sampler's token logprobs) is produced by vLLM on the fast circuit and already travels with the batch. Everything in section 6 (IS, rejection, calibration measurement) is computational, not communicational. The user's instinct is right that signs are 16x cheaper than M; but no variant of shipping M addresses the wedge, so spend that budget only if a future design needs fast-side momentum for a different reason.

5.3 Why dense rollouts are the correct realistic design (not an artifact)

In the deployed two-circuit system the community GPUs do rollouts on full local replicas (a 1.5-2B policy fits on any of them; only weights need syncing, which rank-1 RELEX already carries), while training crosses the compressed pipeline. Sampling through the codec instead would make the mismatch zero by construction but would sample from the e^-14.5 near-random codec policy: garbage data, and our own #89 frame shows the codec view is not a usable generator. PP-decode across internet links is also latency-hopeless independent of compression. So the asymmetry this run trains under (dense sampler, compressed trainer) is the realistic one, the wedge is intrinsic to the design, and the correct place to handle it is the training-side objective. That is question 3.

6 · Question 3: inference-engine probs, systematic bias, and what to send

6.1 bypass_mode=true would be gradient starvation, not divergence

With bypass_mode=true, old_log_prob := vLLM logprobs (dense view) while new logprobs stay codec-view. The initial ratio is then r = exp(logp_codec - logp_dense) with mean log-ratio -14.4 nats → r ~ 6e-7 on typical tokens. Through the PPO-clip objective: for A>0 the unclipped branch is active and the gradient is scaled by r ≈ 0; for A<0 the min picks the constant clipped branch and the gradient is exactly zero. Net effect: learning silently stops, with the residual signal concentrated on the rare tokens where the codec view happens to agree with the sampler (the sampled-token chi-square estimate rollout_corr/chi2_token sits at -0.9 to -0.98 late in the run: essentially no such tokens are being sampled any more). The same holds for loss_type=reinforce with truncated IS weights (min(r, 2) ≈ 0). Do not use bypass_mode while the trainer forward is a 95% codec view. It is the right tool for the numeric dense-vs-dense mismatch it was built for (probs correlation ~0.97, not ~0).

6.2 The decoupled token-IS route is proven on our own data, and what it costs

clvaf683 ran the exact machinery this fork already has (rollout_is=token, threshold 2.0, decoupled mode) on top of the PowerSGD codec, whose wedge was 4-6 nats when correction started:

@200 stepsPowerSGD, no ISPowerSGD + token-IS
actor/kl_loss9.84 (accelerating)3.54 (-64%)
actor/entropy0.027 (collapsed)1.78 (recovering)
train score0.583 (crashed from 0.726)0.684 (held)
val MATH acc0.5610.587 (peaked 0.635@125)
mean IS weightn/a0.19-0.30 (an agreement filter: effective step size cut ~4-5x, grad_norm 20x lower)

Mechanism: at multi-nat wedges token-IS does not "correct" the distribution in the textbook sense (the ESS is tiny); it acts as a soft filter that routes gradient through the tokens where the two views agree, killing the self-reinforcing bias loop. That is why it stops PowerSGD's collapse. Under PRF's 14.4-nat wedge the filter would be much more aggressive; since run 90 is healthy without it, switching it on now would mostly slow learning. The right use is as insurance on longer horizons or coherent codecs, in the step-size-preserving form: IcePop band + batch normalization (rollout_is=token, rollout_is_threshold="0.5_2.0", rollout_is_batch_normalize=true), which zeroes disagreement tokens and renormalizes the surviving weights to mean 1 instead of shrinking the whole step. All of it is config-only and zero wire bytes.

6.3 Treating the mismatch as a systematic bias: the decisive cheap measurement

Decompose the per-token cross-view gap d_t = logp_dense(a_t) - logp_codec(a_t) = b + eps_t, where b is a systematic, slowly-varying calibration term (mean shift and/or temperature of the codec logits) and eps_t is zero-mean per-token noise. We know E[d] ≈ 14.4 nats; nobody has measured the dispersion of eps after fitting b. That single number decides the road:

Cost: ~6 lines in compute_offpolicy_metrics (log p10/p50/p90 quantiles and the post-fit residual std of d_t), zero GPU passes, zero wire. In the deployed two-circuit design the anchor is the natural place to fit b: it already recomputes dense logprobs on the replay batches; ship it the codec-view logprobs (2.6 MB per fire) and it returns the fit. The fit itself is closed-form (mean/temperature regression on logits it already has).

6.4 "Send the most important activation info" is the coherence trap

The intuition "we are losing info, send the important part" was tested to destruction in #89: every lever that added structure or importance to the selection paid for it in KL, because the failure mode is not lost magnitude, it is directional bias times cross-step coherence. rms_match rescale: entropy down but KL +38x. FRLR low-rank residual correction (rank 16/32/48 at the same 77-value budget): entropy down 2.4x but KL +5-16x, and slowing its Q refresh made KL sawtooth: the closer the codec tracks "important" directions, the more coherent its bias, the faster the reference-KL climbs. The two survivors of the whole search are exactly the two levers that preserve randomness: exact-k (fixes the keep count, keeps selection random; the codec this run validates) and antithetic draws (-7% KL, weak). Magnitude top-k is banned in the codec code on purpose. The realistic best-case for "smarter selection" at this budget is FRLR r48 with fast Q refresh, and it was goal-nearest but still KL-climbing when the #89 budget gate closed. Improving what we send is therefore a variance-reduction game with a measured ceiling, not the fix for the wedge.

6.5 arXiv 2603.19470 (ALP): are we already using it? No, and what to take from it

"Adaptive Layerwise Perturbation" (Amazon/UIUC, 2026) targets train-inference mismatch and staleness in LLM RL by adding learnable zero-mean Gaussian noise to every layer input during the update forward (sigma ~1e-4, learned; fresh per token per forward) and optimizing a single unified ratio pi_perturbed-train / pi_vLLM: their numerator is perturbed, their denominator is raw inference-engine logprobs. On Qwen2.5-Math-1.5B RLVR it beats GRPO/Bypass/MIS baselines by 1.5-4 points with visibly better stability.

7 · Question 4: "RL gradients are sparse but high, so this should be fixable"

Half-right, and the half matters:

8 · The fix, decided (v2 addendum, 2026-07-24)

Version 1 of this report ranked options. This addendum replaces the ranking with a decision, after (i) computing the feasibility numbers directly from the run's own logs, (ii) a verified literature sweep (section 9), and (iii) implementing the resulting kit on branch exp/mismatch-control-kit (local, CPU-tested).

8.1 What exactly is rising, and which piece fixes which curve

rising curvedecomposition (measured)the fix
actor/kl_loss 0 → 0.90true dense-view drift multiplied by a mask gain G ≈ 50-100x; grows whenever the policy learns, for any working codecP1 (measure + bound the true KL with a controller) makes it provably harmless; P2 (cut codec noise at the source) shrinks G itself, which is what visibly flattens the panel
rollout_corr/kl 13.9 → 14.6687% benign sharpening (rollout_log_ppl 0.84 → 0.09) + a flat wedge (training_log_ppl 14.4-14.6); the falling mean IS weight E[ρ] 0.05 → 0.001 is the same dynamicthe sharpening component should NOT be fixed (it is learning); the wedge component is what P2 attacks at the source

8.2 The kill table: corrections that are measurably impossible here

From the k3-kl estimator identity, E[ρ] = k3_kl - kl + 1 gives the exact untruncated mean token-IS weight between the codec-view trainer and the dense sampler, per step, from already-logged data:

quantityPowerSGD (where token-IS worked)run 90 PRF p=0.95verdict
mean token-IS weight E[ρ]0.196 median (logged rollout_is_mean)0.0007 median late-run (0.05 at step 1, falling)token-IS / IcePop / any ratio band: dead (≤0.2% of gradient signal survives)
dispersion of the log-gap dbimodal with a large agreement modetwo-moment lognormal fit: σ(d) ≈ 4.0 natsmean-calibration + IS: dead (post-calibration ESS ≈ e-σ² ≈ 1e-7)
bypass ratio πcodecdense~e-4..-6, marginal~e-14.4 ≈ 6e-7bypass_mode: dead (gradient starvation, 6.1)

Consequence: for the PRF codec at this compression, the entire policy-reaccounting family (bypass, TIS, MIS, IcePop, GSPO, calibrated IS) cannot fix the curves, because there is almost no per-token overlap left to reweight. The fix has to reduce the per-token view noise itself. (These tools remain valid and proven for the milder coherent-codec regime: clvaf683.)

8.3 The prescription

  1. P0 (this run): change nothing. Finish to 600; judge on the section-4 checklist; the curves as they stand are codec-view projections of healthy learning.
  2. P1 (control plane; permanent for all comm-eff runs): measure the true KL and bound it by construction.
    • Log-ratio distribution metrics every step (rollout_corr/logratio_p10|p50|p90|std, agree_frac_1nat): implemented on exp/mismatch-control-kit.
    • Dense-view probe every 25 steps: rerun the old-logprob + reference recomputes once with the codec disabled (no backward), logging probe/kl_dense, the gain probe/kl_gain = kl_loss/kl_dense, and the numeric vLLM-vs-trainer baseline. ~1-2% step-time overhead in the sim; in deployment this probe belongs to the anchor (it holds full weights and already receives the batch; 28 B comes back).
    • Adaptive kl_loss_coef: PI controller targeting probe/kl_dense ≤ 0.02 at horizon (Schulman-style adaptive beta, driven by the dense measurement, acting through the existing codec-view penalty). After this, real drift cannot run away no matter what the codec-view panel does; the codec-view panel is relabeled as G x dense-KL with G logged.
  3. P2 (fix at the source; the next codec experiment): replace 5%-random-keep with unbiased low-bit dense quantization: the sr_quant codec. All 1536 dims cross the boundary at 1-2 bits with blockwise scales (absmax per 32 dims) and PRF-keyed stochastic rounding, forward and backward. Zero-mean like PRF (the property that makes it collapse-safe), but bounded one-step noise instead of x20-or-x0 heavy tails: noise energy ~7x signal at 1-bit/1.87x bytes, ~0.9x at 2-bit/3.1x bytes, and ~5x at EXACT byte parity (2-bit on a random 493-dim subset), vs 19x heavy-tailed for the incumbent. Predicted: wedge 14.5 → low single digits, codec entropy 7.8 → ≤3, gain G collapses, and if the wedge lands under ~3 nats the IS toolbox re-activates as a safety net. Mandatory control arm: deterministic round-to-nearest at the same grid (NVFP4 found RN safer in their setting; our coherence mechanism predicts RN is the dangerous one here; measure it).

    Status: implemented, CPU-verified, and adversarially reviewed. Local branch exp/mismatch-control-kit (commits 001769fe metrics, d891b533 codec, 20f5b82b launcher plumbing, 9b4e87f9 review fix; ~1590 lines over 17 files; 136 tests pass including 38 new sr_quant tests covering unbiasedness, PRF determinism, cross-pass ratio identity, backward quantization, block scales, rn mode, and path confinement; a pre-GPU boot gate validates the knobs). A three-lens adversarial review panel (quantizer math, PRF/identity, config wiring) confirmed the codec clean; its one confirmed finding (the new quantile metric would crash on batches above torch.quantile's 2^24-element cap, latent for committed configs but zero-headroom for long-context arms) was reproduced on torch 2.12 and fixed in 9b4e87f9. Launch block for the arm:
    COMM_EFF_ENABLED=true COMM_EFF_COMPRESSION_TYPE=sr_quant \
    COMM_EFF_QUANT_BITS=1 COMM_EFF_QUANT_BLOCK_SIZE=32 COMM_EFF_QUANT_ROUNDING=sr \
    COMM_EFF_MASK_RECOMPUTE=true COMM_EFF_MASK_REFERENCE=true \
    COMM_EFF_MASK_PP_SIZE=8 COMM_EFF_ANCHOR_OWNS_Q=false \
    COMM_EFF_POWERSGD_FAST_Q_BOOTSTRAP=false \
    bash examples/grpo_trainer/run_qwen25_math_1p5b_rank1_relex_fsdp.sh
  4. P2b (exploratory, one arm): codec-view consistency (CVC). Add λ x CEcodec = -λ E[log πcodec(a)] on sampled tokens: this term literally IS the wedge, differentiable through the existing train forward, zero extra passes, zero wire. λ ≈ 0.003 with warmup, kill-guard on dense-side metrics (the codec view is below-uniform, so the early CE gradient points toward uniformizing it: watch rollout_ppl/val).
  5. P3 (insurance, coherent codecs only): token-IS as proven in clvaf683, unchanged.
  6. Standing no-gos (measured): bypass_mode under a codec view; token-IS/IcePop directly on PRF p=0.95; magnitude top-k selection; cross-step error feedback on activations (refuted conceptually and empirically, section 9); sampling through the codec; spending wire budget on shipping M.
In one sentence: stop trying to re-account a 14-nat per-token gap (measurably impossible), bound the true dense-view KL with a probe + controller (P1), and shrink the gap at its source by moving the byte budget from "few dims at full precision" to "all dims at low precision, unbiased" (P2): that is the fix.

9 · Literature check (verified sources)

9.1 Train-inference mismatch fixes in LLM RL: what survives at our gap

Every method below was validated at engine-level mismatch of 1e-4 to 1e-2 nats/token. Our regimes: (i) coherent codec 4-6 nats (PowerSGD), (ii) random codec ~14.4 nats bimodal (PRF p=0.95). The field's toolbox splits into three tiers; at regime (ii) only tier 3 survives.

tier / methodsourcemechanismat (i) 4-6 natsat (ii) 14 nats
1: token TISYao et al., OPT/NeurIPS 2025 (Flash-RL); adopted with C=5 in Qwen practices 2512.01374stop-grad coefficient min(πtrainsampler, C) on the surrogateworks as attenuation (proven here: clvaf683), ESS a few %weights ~3e-7: silently deletes the batch
1: CISPOMiniMax-M1, 2506.13585clip the IS weight, never zero token gradientssame attenuationsame collapse
1: CTPO2605.07331cumulative prefix ratios, √t-scaled log-space clipbest clipping schedule if token IS is keptprefix products of e-15 vanish
2: Seq-MIS / Geo-MaskLiu, Li et al., rl-collapse series (the verl rollout_corr machinery in this fork)reject sequences with out-of-band (geometric-mean) ratiosrho_geo = e-4..-6: rejects 100%rejects 100%
2: GSPOQwen, 2507.18071length-normalized sequence ratio; fixes zero-mean token noise (MoE flicker)dead: our bias is coherent, averaging preserves itdead
2: IcePopRing-1T, 2510.18855two-sided token masking of out-of-band ratios; failure declared at ~5% token-prob differenceour difference is ~100x their failure barkeeps only the ≤0.2% agreeing tokens
3: FP16 / fp32 headSea AI Lab, 2510.26788eliminate the numeric gap at the sourcethe school our fix belongs to: "align the views, do not reweight"
3: R3 routing replay2510.11370replay the inference engine's expert routing into the training forwardstrongest precedent: the answer to a structural forward perturbation is replaying the structure, not IS
3: QaRL2604.07853make the trainer's forward quantization-aware so it matches the quantized samplerour geometry inverted; validates the structural school under RL (+5.5 pts vs naive)
3: dropout practiceTRL #454/#2042; 2403.17031never let a stochastic perturbation into a ratio-bearing forward; nobody IS-corrects dropoutvalidates run 90's design: all three logprob passes share one mask, ratio stays 1
guardrail: adaptive KLZiegler 1909.08593 lineage; TorchRL AdaptiveKLControllerfeedback controller on the KL coefficient vs a targetcomplementary (P1), with the sweep's own caveat: the measured KL must come from an uncontaminated (dense) view or the controller chases codec noise
calibrate-then-IS(no source exists)debias the trainer view, then reweightconfirmed literature gap; our own two-moment fit (σ(d) ≈ 4 nats, ESS ~1e-7) says it would not work at (ii) anyway

Two further load-bearing notes from the sweep: Dropout-GRPO (2606.10184) formalizes the correct frame for our codec ("a masked network is a sampled policy, not a biased estimate of the dense one"), and Rethinking KL Regularization (2510.01555) shows the k3/low_var_kl loss estimator is itself a biased first-order approximation at large ratios: one more reason the codec-view kl_loss panel must be read through the P1 dense probe rather than at face value.

9.2 Activation compression for pipeline-parallel training

Verified anchors behind P2: SWARM (2301.11913) and Petals (2209.01188) ship 8-bit blockwise boundary activations across the internet losslessly (nothing published below 8 bits at the boundary; sr_quant probes that gap). ActNN (2104.14129) and GACT (2206.11357): unbiased stochastic ~2-bit activation quantization yields convergent, variance-characterized gradients (backward-storage setting). QSGD (1610.02132) supplies the unbiased quantizer and its variance bound, and its bucketing is why sr_quant uses blockwise scales; Gupta et al. 2015 (1502.02551) and Croci et al. 2022 are the canonical stochastic-rounding citations (E[q]=h, per-element variance ≤ Δ²/4). AQ-SGD (2206.01299) compresses activation deltas but requires per-sample recurrence across epochs, which fresh RL rollouts never provide (its DirectQ baseline: direct 2-3-bit boundary quantization degrades in SFT, the closest published control to our arm); Clapping (2509.19029) exists precisely because of that limitation. Rudakov et al. (2401.07788): boundary sparsification floor at ~10% TopK, and error feedback on activations does not help: the incumbent PRF 5% random-keep sits below that floor with the worst variance profile of the family. QuEST (2502.05003) trains transformers with 1-bit activations via Hadamard rotation (the extension knob if sr_quant wins); NVFP4 (2509.25149) is the counterpoint mandating the SR-vs-RN ablation. No published work combines low-bit boundary compression with RL, and none applies policy-ratio corrections to codec-induced mismatch: both points are unoccupied territory.

10 · Capability preserved: dense vs PRF-compressed (in-domain + OOD)

Headline: at step 600 the 95%-compressed PRF run matches the dense control on every benchmark — in-domain and out-of-domain — with all gaps within ±0.04 (mean |Δ| = 0.014) and no systematic direction. 95% activation compression reaches the same downstream capability as uncompressed training.

A byte-matched dense control (90-dense-600: identical config, only comm_eff.enabled=false) was trained 600 steps on MATH alongside the compressed run. Its in-domain val tracked the compressed run throughout (MATH-test acc @0/150/300/450/600 = 0.449 / 0.659 / 0.665 / 0.687 / 0.677, vs compressed 0.451 / 0.661 / 0.663 / 0.673 / 0.661). Both step-600 checkpoints were then pulled from R2 and evaluated through the same harness — greedy mean@1 for the large sets, avg@8 (temp 0.7) for the competition sets.

Dense vs PRF-compressed accuracy per benchmark at step 600, in-domain and out-of-domain
benchmarkdomaindensePRF (95%)Δ (PRF−dense)
MATH500in-domain0.6700.706+0.036
GSM8Kin-domain0.8130.831+0.018
MinervaOOD0.1620.184+0.022
OlympiadBenchOOD0.2400.235−0.005
MMLU-STEMOOD0.3840.393+0.009
AMC23OOD0.5470.541−0.006
AIME24OOD0.1790.163−0.017
AIME25OOD0.0500.058+0.008
AIME26OOD0.0920.104+0.013
HMMT25OOD0.0080.000−0.008

Compressed is higher on 6/10 benchmarks, dense on 4/10 — consistent with run-to-run noise, not a compression penalty. Checkpoints in R2 under .../90-prf-exactk-600/prf-exactk/ and .../90-dense-600/dense/; figure + numbers in artifacts/90-prf-exactk-600-mismatch-diagnosis/compare_dense_vs_prf_step600.{png,csv}. Dense run: WandB 90-dense-600/a134dxxx.

11 · Provenance