← All runs · 90-prf-exactk-600 · mid-run diagnosis
Companion artifacts (CSV pulls used for every figure): artifacts/90-prf-exactk-600-mismatch-diagnosis/
exp/mismatch-control-kit.actor/kl_loss and actor/entropy live in the codec view (both nets measured through the 95% mask); they are not the collapse signature. The plan gate "kl_loss well under 0.2" is formally breached (0.50 @ 599) but that number was calibrated on 100-step history; every mechanism-level precursor of a real break is absent (section 4 checklist). Projected codec-view kl_loss at step 600 if the current slope holds: about 0.9.bypass_mode=true today would not diverge, it would silently stop learning: the mean per-token log-ratio between codec-view trainer and dense sampler is -14.4 nats, so ratios are ~6e-7; positive-advantage tokens get e^-14-scaled gradients and negative-advantage tokens get exactly zero through the clip. The viable form is the decoupled mode already in the codebase: token-IS on PowerSGD (clvaf683) cut kl_loss 64% (9.84 → 3.54 @200), prevented entropy collapse (1.78 vs 0.027) and kept +2.6 val points at 200; its cost is an agreement filter (mean IS weight ~0.2). Treating the mismatch as a systematic bias is the right next idea and is measurable for a few logging lines (section 6.3): if a fitted global/per-layer calibration absorbs most of the 14.4 nats, calibrated correction becomes feasible; if dispersion stays huge, only filtering works. "Send the most important activations" is the coherence trap: every importance/structure-adding lever measured in #89 paid for it in KL (rms_match +38x, FRLR +5-16x); random selection is the feature. We are not using arXiv 2603.19470 (ALP); it is an off-policy-correction paper whose zero-mean-noise theory independently predicts exactly our PowerSGD-collapses / PRF-benign split (section 6.5).| item | value |
|---|---|
| Codec | prf_mask, p=0.95, exact-k random 77/1536 dims kept per token (order-statistic on the PRF draw; top-k by magnitude is deliberately forbidden), constant rescale 1/(1-p)≈20x, pp_size=8 → 7 masked boundaries (layers 3,7,11,15,18,21,24), mask keyed on (seed, layer, step, sample_id, position_id, channel): bit-identical across the old-logprob / train / reference forwards of one step, fresh across steps |
| Views | rollouts: vLLM, dense weights (uncompressed); trainer old/new/ref logprobs: codec view (mask_recompute=true, mask_reference=true) |
| GRPO | batch 128 = ppo_mini 128 → exactly one on-policy optimizer tick per step; rollout n=8; 1024/2048 lengths; lr 1e-6; kl_loss_coef 0.001 (low_var_kl) |
| Two-circuit | anchor paired dense replay cadence/delay 20/20 ticks (= every 20 steps here; 16 fires by step 337), rank-1 RELEX W2, signed-EMA M (beta_anc 0.25, alpha 0.25); anchor_owns_Q=false (PRF has no Q) |
| Correction | rollout correction strictly OFF (rollout_is=null, bypass_mode=false); vLLM logprobs logged for diagnostics only (calculate_log_probs=true) |
| Status | COMPLETE: 600/600 steps in 12h52m (mean 89.4 s/it; mid-run check at step 337 saw 68.7 s/it, GPU 91% util, 127/144 GB). Box left running; no teardown authorized. |
| Checkpoints | SAVE_FREQ=100 via launch-shell override (launcher default is -1) + comm_eff R2 sink with delete_local: six checkpoints (steps 100-600) uploaded to s3://shamane-pluralis/autonomous-harness-rlvr-compression/90-prf-exactk-600/prf-exactk/checkpoints/; sink drain n_uploaded=66 n_errors=0, independently verified by bucket listing (global_step_600 = 10 objects, 19.47 GB). Nothing irreplaceable remains on the box. |
The four panels that triggered this analysis measure three different policies. Keeping them separate is the whole diagnosis:
| metric | compares | view | what it is doing |
|---|---|---|---|
actor/kl_loss 0 → 0.50 | actor vs frozen reference | both through the same per-step mask | real policy movement as seen through the codec, growing linearly, decelerating; the 0.001 coefficient makes its loss contribution ~5e-4 |
actor/entropy 7.808 → 7.834 | actor only | codec view | structural: the 20x constant rescale inflates codec-view entropy to ~7.82 (dense control 0.18, dropout 0.14, #89 frame). The +0.026 drift in 330 steps (+7.5e-5/step) is cosmetic. This is NOT the sampling entropy; the dense policy is sharpening (rollout_log_ppl 0.84 → 0.13) |
actor/pg_clipfrac, pg_clipfrac_lower, ppo_kl ≡ 0 | actor-new vs actor-old | codec vs codec | zero by construction: one optimizer tick per step and a bit-identical mask across the old and train forwards force ratio ≡ 1. It is proof the within-step identity holds, and it also means PPO's ratio/clip machinery is inert: it cannot see, and therefore cannot correct, the sampler-trainer wedge |
rollout_corr/kl 13.9 → 14.4 | dense vLLM sampler vs codec-view trainer | cross-view | the wedge diagnostic, decomposed below: the growth is ~87% benign sharpening |
rollout_corr/kl = training_log_ppl - rollout_log_ppl (per-token nats). The codec-view side
(training_log_ppl, the wedge proper) has moved 14.46 → 14.39 (min, step 52) → 14.74: a ±0.1-nat band over 599 steps.
The dense side (rollout_log_ppl) fell 0.84 → 0.09 because the policy is getting more confident on MATH: normal, wanted RL behavior.
Of the +0.81-nat rise in the gap, +0.71 (87%) is the dense policy sharpening and only +0.10 (13%) is codec-side drift.
Contrast PowerSGD (rose): its wedge itself explodes 4.4 → 29.5 nats while the policy sharpens into its own codec's preferences, and val crashes.
Note the level: the codec-view trainer assigns on average e^-14.5 ≈ 5e-7 probability to the tokens the dense sampler actually picked, which is below uniform (uniform over the 152k vocab = 11.93 nats). At 95% masking the codec view is not "the policy plus noise"; it is a different, nearly-random policy whose gradient is nonetheless useful in expectation because the mask is fresh, zero-mean and incoherent across steps. That is the entire PRF-vs-PowerSGD story in one number.
| signal | PowerSGD tb4m0k23 (breaks by 200) | run 90 @ 599 | read |
|---|---|---|---|
| kl_loss shape | accelerating: 0.11@25 → 1.15@50 → 6.12@100 → 9.84@200 | linear, decelerating: slope 0.0020 → 0.0015/step | no break |
| entropy | cliff DOWN 5.54 → 0.027 (deterministic) | flat structural band 7.81 → 7.83, drifting UP | no break |
| ppo_kl / clipfrac | 0.21 / 12.7% (within-step identity broken) | exactly 0 / exactly 0 | identity intact |
| grad_norm | spikes to 29.9, settles 17.5 | 1.3-2.0, declining | no break |
| wedge (training_log_ppl) | 4.4 → 29.5 nats | 14.4 → 14.6 (flat) | no break |
| train score | peak 0.726@150 → 0.583@200 | 0.35 → 0.73, plateau holding | no break |
| val MATH acc | 0.667@150 → 0.561@200 | 0.451@0 → 0.661@150 → 0.663@300 | capability gained + held |
rollout_corr/training_log_ppl > 15.0 (wedge growth, currently 14.6);ppo_kl > 1e-2 sustained (within-step identity break);If a tripwire fires, the action is still not intervention mid-run; it is labelling the close-out verdict and the follow-up arm correctly.
Measured on this run: the 16 anchor replay fires (every 20 ticks) produce no detectable bend in the codec-view KL (mean dKL over the 2 steps after a fire 0.0037 vs 0.0027 elsewhere). This confirms the #88 exoneration from the other side: the anchor's job is bounding weight-space drift (which it does; at these horizons total weight movement is fractions of a percent), while the kl_loss climb is the policy adapting to the mask distribution and sharpening, seen through the codec. More replay, smarter replay timing, or a stronger M transaction cannot remove a wedge that lives in the forward-pass view, not in the weights. Within the fixed cadence/delay 20/20 contract, the anchor is already doing its part.
| channel | direction | payload | bytes | status |
|---|---|---|---|---|
| PP boundary activations | fast↔fast | 77/1536 bf16 values per token; indices derived from the shared PRF seed, never sent; backward is masked by the same chain rule | 101 MB per boundary per pass vs 2.02 GB dense (20.0x) | the budget being validated |
| mask identity | fast↔fast | 64-bit seed + step counter | 8 B | free by design |
| rank-1 RELEX weight sync | slow→fast | u,v per weight matrix | ~3 MB bf16 per fire | in use |
| signed-EMA M | anchor-owned, not shipped | if ever shipped: sign(M) 1 bit/param + per-tensor scale | 193 MB vs 3.09 GB full bf16 (16x) | available, but M absence on the fast side is NOT the mismatch cause |
| rollout batch for paired replay | fast→slow | token ids + rewards | ~2.6 MB per fire | in use |
| vLLM behavior logprobs | local to fast circuit | 1 float/token | 0 wire bytes | already logged; unused by the objective |
| codec-view logprobs to anchor (proposed, 6.3) | fast→slow | 1 float/token on replay batches | ~2.6 MB per fire | enables anchor-side calibration fit |
| calibration broadcast (proposed, 6.3) | slow→fast | temperature + bias per boundary (14 floats), or rank-1 logit correction | 28 B, or ~300 KB | the "eliminate systematic bias" lever |
In the deployed two-circuit system the community GPUs do rollouts on full local replicas (a 1.5-2B policy fits on any of them; only weights need syncing, which rank-1 RELEX already carries), while training crosses the compressed pipeline. Sampling through the codec instead would make the mismatch zero by construction but would sample from the e^-14.5 near-random codec policy: garbage data, and our own #89 frame shows the codec view is not a usable generator. PP-decode across internet links is also latency-hopeless independent of compression. So the asymmetry this run trains under (dense sampler, compressed trainer) is the realistic one, the wedge is intrinsic to the design, and the correct place to handle it is the training-side objective. That is question 3.
With bypass_mode=true, old_log_prob := vLLM logprobs (dense view) while new logprobs stay codec-view. The initial ratio is then r = exp(logp_codec - logp_dense) with mean log-ratio -14.4 nats → r ~ 6e-7 on typical tokens. Through the PPO-clip objective: for A>0 the unclipped branch is active and the gradient is scaled by r ≈ 0; for A<0 the min picks the constant clipped branch and the gradient is exactly zero. Net effect: learning silently stops, with the residual signal concentrated on the rare tokens where the codec view happens to agree with the sampler (the sampled-token chi-square estimate rollout_corr/chi2_token sits at -0.9 to -0.98 late in the run: essentially no such tokens are being sampled any more). The same holds for loss_type=reinforce with truncated IS weights (min(r, 2) ≈ 0). Do not use bypass_mode while the trainer forward is a 95% codec view. It is the right tool for the numeric dense-vs-dense mismatch it was built for (probs correlation ~0.97, not ~0).
clvaf683 ran the exact machinery this fork already has (rollout_is=token, threshold 2.0, decoupled mode) on top of the PowerSGD codec, whose wedge was 4-6 nats when correction started:
| @200 steps | PowerSGD, no IS | PowerSGD + token-IS |
|---|---|---|
| actor/kl_loss | 9.84 (accelerating) | 3.54 (-64%) |
| actor/entropy | 0.027 (collapsed) | 1.78 (recovering) |
| train score | 0.583 (crashed from 0.726) | 0.684 (held) |
| val MATH acc | 0.561 | 0.587 (peaked 0.635@125) |
| mean IS weight | n/a | 0.19-0.30 (an agreement filter: effective step size cut ~4-5x, grad_norm 20x lower) |
Mechanism: at multi-nat wedges token-IS does not "correct" the distribution in the textbook sense (the ESS is tiny); it acts as a soft filter that routes gradient through the tokens where the two views agree, killing the self-reinforcing bias loop. That is why it stops PowerSGD's collapse. Under PRF's 14.4-nat wedge the filter would be much more aggressive; since run 90 is healthy without it, switching it on now would mostly slow learning. The right use is as insurance on longer horizons or coherent codecs, in the step-size-preserving form: IcePop band + batch normalization (rollout_is=token, rollout_is_threshold="0.5_2.0", rollout_is_batch_normalize=true), which zeroes disagreement tokens and renormalizes the surviving weights to mean 1 instead of shrinking the whole step. All of it is config-only and zero wire bytes.
Decompose the per-token cross-view gap d_t = logp_dense(a_t) - logp_codec(a_t) = b + eps_t, where b is a systematic, slowly-varying calibration term (mean shift and/or temperature of the codec logits) and eps_t is zero-mean per-token noise. We know E[d] ≈ 14.4 nats; nobody has measured the dispersion of eps after fitting b. That single number decides the road:
Cost: ~6 lines in compute_offpolicy_metrics (log p10/p50/p90 quantiles and the post-fit residual std of d_t), zero GPU passes, zero wire. In the deployed two-circuit design the anchor is the natural place to fit b: it already recomputes dense logprobs on the replay batches; ship it the codec-view logprobs (2.6 MB per fire) and it returns the fit. The fit itself is closed-form (mean/temperature regression on logits it already has).
The intuition "we are losing info, send the important part" was tested to destruction in #89: every lever that added structure or importance to the selection paid for it in KL, because the failure mode is not lost magnitude, it is directional bias times cross-step coherence. rms_match rescale: entropy down but KL +38x. FRLR low-rank residual correction (rank 16/32/48 at the same 77-value budget): entropy down 2.4x but KL +5-16x, and slowing its Q refresh made KL sawtooth: the closer the codec tracks "important" directions, the more coherent its bias, the faster the reference-KL climbs. The two survivors of the whole search are exactly the two levers that preserve randomness: exact-k (fixes the keep count, keeps selection random; the codec this run validates) and antithetic draws (-7% KL, weak). Magnitude top-k is banned in the codec code on purpose. The realistic best-case for "smarter selection" at this budget is FRLR r48 with fast Q refresh, and it was goal-nearest but still KL-climbing when the #89 budget gate closed. Improving what we send is therefore a variance-reduction game with a measured ceiling, not the fix for the wedge.
"Adaptive Layerwise Perturbation" (Amazon/UIUC, 2026) targets train-inference mismatch and staleness in LLM RL by adding learnable zero-mean Gaussian noise to every layer input during the update forward (sigma ~1e-4, learned; fresh per token per forward) and optimizing a single unified ratio pi_perturbed-train / pi_vLLM: their numerator is perturbed, their denominator is raw inference-engine logprobs. On Qwen2.5-Math-1.5B RLVR it beats GRPO/Bypass/MIS baselines by 1.5-4 points with visibly better stability.
Half-right, and the half matters:
Version 1 of this report ranked options. This addendum replaces the ranking with a decision, after (i) computing the feasibility numbers directly from the run's own logs, (ii) a verified literature sweep (section 9), and (iii) implementing the resulting kit on branch exp/mismatch-control-kit (local, CPU-tested).
| rising curve | decomposition (measured) | the fix |
|---|---|---|
actor/kl_loss 0 → 0.90 | true dense-view drift multiplied by a mask gain G ≈ 50-100x; grows whenever the policy learns, for any working codec | P1 (measure + bound the true KL with a controller) makes it provably harmless; P2 (cut codec noise at the source) shrinks G itself, which is what visibly flattens the panel |
rollout_corr/kl 13.9 → 14.66 | 87% benign sharpening (rollout_log_ppl 0.84 → 0.09) + a flat wedge (training_log_ppl 14.4-14.6); the falling mean IS weight E[ρ] 0.05 → 0.001 is the same dynamic | the sharpening component should NOT be fixed (it is learning); the wedge component is what P2 attacks at the source |
From the k3-kl estimator identity, E[ρ] = k3_kl - kl + 1 gives the exact untruncated mean token-IS weight between the codec-view trainer and the dense sampler, per step, from already-logged data:
| quantity | PowerSGD (where token-IS worked) | run 90 PRF p=0.95 | verdict |
|---|---|---|---|
| mean token-IS weight E[ρ] | 0.196 median (logged rollout_is_mean) | 0.0007 median late-run (0.05 at step 1, falling) | token-IS / IcePop / any ratio band: dead (≤0.2% of gradient signal survives) |
| dispersion of the log-gap d | bimodal with a large agreement mode | two-moment lognormal fit: σ(d) ≈ 4.0 nats | mean-calibration + IS: dead (post-calibration ESS ≈ e-σ² ≈ 1e-7) |
| bypass ratio πcodec/πdense | ~e-4..-6, marginal | ~e-14.4 ≈ 6e-7 | bypass_mode: dead (gradient starvation, 6.1) |
Consequence: for the PRF codec at this compression, the entire policy-reaccounting family (bypass, TIS, MIS, IcePop, GSPO, calibrated IS) cannot fix the curves, because there is almost no per-token overlap left to reweight. The fix has to reduce the per-token view noise itself. (These tools remain valid and proven for the milder coherent-codec regime: clvaf683.)
rollout_corr/logratio_p10|p50|p90|std, agree_frac_1nat): implemented on exp/mismatch-control-kit.probe/kl_dense, the gain probe/kl_gain = kl_loss/kl_dense, and the numeric vLLM-vs-trainer baseline. ~1-2% step-time overhead in the sim; in deployment this probe belongs to the anchor (it holds full weights and already receives the batch; 28 B comes back).kl_loss_coef: PI controller targeting probe/kl_dense ≤ 0.02 at horizon (Schulman-style adaptive beta, driven by the dense measurement, acting through the existing codec-view penalty). After this, real drift cannot run away no matter what the codec-view panel does; the codec-view panel is relabeled as G x dense-KL with G logged.sr_quant codec. All 1536 dims cross the boundary at 1-2 bits with blockwise scales (absmax per 32 dims) and PRF-keyed stochastic rounding, forward and backward. Zero-mean like PRF (the property that makes it collapse-safe), but bounded one-step noise instead of x20-or-x0 heavy tails: noise energy ~7x signal at 1-bit/1.87x bytes, ~0.9x at 2-bit/3.1x bytes, and ~5x at EXACT byte parity (2-bit on a random 493-dim subset), vs 19x heavy-tailed for the incumbent. Predicted: wedge 14.5 → low single digits, codec entropy 7.8 → ≤3, gain G collapses, and if the wedge lands under ~3 nats the IS toolbox re-activates as a safety net. Mandatory control arm: deterministic round-to-nearest at the same grid (NVFP4 found RN safer in their setting; our coherence mechanism predicts RN is the dangerous one here; measure it).
exp/mismatch-control-kit (commits 001769fe metrics, d891b533 codec, 20f5b82b launcher plumbing, 9b4e87f9 review fix; ~1590 lines over 17 files; 136 tests pass including 38 new sr_quant tests covering unbiasedness, PRF determinism, cross-pass ratio identity, backward quantization, block scales, rn mode, and path confinement; a pre-GPU boot gate validates the knobs). A three-lens adversarial review panel (quantizer math, PRF/identity, config wiring) confirmed the codec clean; its one confirmed finding (the new quantile metric would crash on batches above torch.quantile's 2^24-element cap, latent for committed configs but zero-headroom for long-context arms) was reproduced on torch 2.12 and fixed in 9b4e87f9. Launch block for the arm:
COMM_EFF_ENABLED=true COMM_EFF_COMPRESSION_TYPE=sr_quant \
COMM_EFF_QUANT_BITS=1 COMM_EFF_QUANT_BLOCK_SIZE=32 COMM_EFF_QUANT_ROUNDING=sr \
COMM_EFF_MASK_RECOMPUTE=true COMM_EFF_MASK_REFERENCE=true \
COMM_EFF_MASK_PP_SIZE=8 COMM_EFF_ANCHOR_OWNS_Q=false \
COMM_EFF_POWERSGD_FAST_Q_BOOTSTRAP=false \
bash examples/grpo_trainer/run_qwen25_math_1p5b_rank1_relex_fsdp.shEvery method below was validated at engine-level mismatch of 1e-4 to 1e-2 nats/token. Our regimes: (i) coherent codec 4-6 nats (PowerSGD), (ii) random codec ~14.4 nats bimodal (PRF p=0.95). The field's toolbox splits into three tiers; at regime (ii) only tier 3 survives.
| tier / method | source | mechanism | at (i) 4-6 nats | at (ii) 14 nats |
|---|---|---|---|---|
| 1: token TIS | Yao et al., OPT/NeurIPS 2025 (Flash-RL); adopted with C=5 in Qwen practices 2512.01374 | stop-grad coefficient min(πtrain/πsampler, C) on the surrogate | works as attenuation (proven here: clvaf683), ESS a few % | weights ~3e-7: silently deletes the batch |
| 1: CISPO | MiniMax-M1, 2506.13585 | clip the IS weight, never zero token gradients | same attenuation | same collapse |
| 1: CTPO | 2605.07331 | cumulative prefix ratios, √t-scaled log-space clip | best clipping schedule if token IS is kept | prefix products of e-15 vanish |
| 2: Seq-MIS / Geo-Mask | Liu, Li et al., rl-collapse series (the verl rollout_corr machinery in this fork) | reject sequences with out-of-band (geometric-mean) ratios | rho_geo = e-4..-6: rejects 100% | rejects 100% |
| 2: GSPO | Qwen, 2507.18071 | length-normalized sequence ratio; fixes zero-mean token noise (MoE flicker) | dead: our bias is coherent, averaging preserves it | dead |
| 2: IcePop | Ring-1T, 2510.18855 | two-sided token masking of out-of-band ratios; failure declared at ~5% token-prob difference | our difference is ~100x their failure bar | keeps only the ≤0.2% agreeing tokens |
| 3: FP16 / fp32 head | Sea AI Lab, 2510.26788 | eliminate the numeric gap at the source | the school our fix belongs to: "align the views, do not reweight" | |
| 3: R3 routing replay | 2510.11370 | replay the inference engine's expert routing into the training forward | strongest precedent: the answer to a structural forward perturbation is replaying the structure, not IS | |
| 3: QaRL | 2604.07853 | make the trainer's forward quantization-aware so it matches the quantized sampler | our geometry inverted; validates the structural school under RL (+5.5 pts vs naive) | |
| 3: dropout practice | TRL #454/#2042; 2403.17031 | never let a stochastic perturbation into a ratio-bearing forward; nobody IS-corrects dropout | validates run 90's design: all three logprob passes share one mask, ratio stays 1 | |
| guardrail: adaptive KL | Ziegler 1909.08593 lineage; TorchRL AdaptiveKLController | feedback controller on the KL coefficient vs a target | complementary (P1), with the sweep's own caveat: the measured KL must come from an uncontaminated (dense) view or the controller chases codec noise | |
| calibrate-then-IS | (no source exists) | debias the trainer view, then reweight | confirmed literature gap; our own two-moment fit (σ(d) ≈ 4 nats, ESS ~1e-7) says it would not work at (ii) anyway | |
Two further load-bearing notes from the sweep: Dropout-GRPO (2606.10184) formalizes the correct frame for our codec ("a masked network is a sampled policy, not a biased estimate of the dense one"), and Rethinking KL Regularization (2510.01555) shows the k3/low_var_kl loss estimator is itself a biased first-order approximation at large ratios: one more reason the codec-view kl_loss panel must be read through the P1 dense probe rather than at face value.
Verified anchors behind P2: SWARM (2301.11913) and Petals (2209.01188) ship 8-bit blockwise boundary activations across the internet losslessly (nothing published below 8 bits at the boundary; sr_quant probes that gap). ActNN (2104.14129) and GACT (2206.11357): unbiased stochastic ~2-bit activation quantization yields convergent, variance-characterized gradients (backward-storage setting). QSGD (1610.02132) supplies the unbiased quantizer and its variance bound, and its bucketing is why sr_quant uses blockwise scales; Gupta et al. 2015 (1502.02551) and Croci et al. 2022 are the canonical stochastic-rounding citations (E[q]=h, per-element variance ≤ Δ²/4). AQ-SGD (2206.01299) compresses activation deltas but requires per-sample recurrence across epochs, which fresh RL rollouts never provide (its DirectQ baseline: direct 2-3-bit boundary quantization degrades in SFT, the closest published control to our arm); Clapping (2509.19029) exists precisely because of that limitation. Rudakov et al. (2401.07788): boundary sparsification floor at ~10% TopK, and error feedback on activations does not help: the incumbent PRF 5% random-keep sits below that floor with the worst variance profile of the family. QuEST (2502.05003) trains transformers with 1-bit activations via Hadamard rotation (the extension knob if sr_quant wins); NVFP4 (2509.25149) is the counterpoint mandating the SR-vs-RN ablation. No published work combines low-bit boundary compression with RL, and none applies policy-ratio corrections to codec-induced mismatch: both points are unoccupied territory.
A byte-matched dense control (90-dense-600: identical config, only comm_eff.enabled=false) was trained 600 steps on MATH alongside the compressed run. Its in-domain val tracked the compressed run throughout (MATH-test acc @0/150/300/450/600 = 0.449 / 0.659 / 0.665 / 0.687 / 0.677, vs compressed 0.451 / 0.661 / 0.663 / 0.673 / 0.661). Both step-600 checkpoints were then pulled from R2 and evaluated through the same harness — greedy mean@1 for the large sets, avg@8 (temp 0.7) for the competition sets.
| benchmark | domain | dense | PRF (95%) | Δ (PRF−dense) |
|---|---|---|---|---|
| MATH500 | in-domain | 0.670 | 0.706 | +0.036 |
| GSM8K | in-domain | 0.813 | 0.831 | +0.018 |
| Minerva | OOD | 0.162 | 0.184 | +0.022 |
| OlympiadBench | OOD | 0.240 | 0.235 | −0.005 |
| MMLU-STEM | OOD | 0.384 | 0.393 | +0.009 |
| AMC23 | OOD | 0.547 | 0.541 | −0.006 |
| AIME24 | OOD | 0.179 | 0.163 | −0.017 |
| AIME25 | OOD | 0.050 | 0.058 | +0.008 |
| AIME26 | OOD | 0.092 | 0.104 | +0.013 |
| HMMT25 | OOD | 0.008 | 0.000 | −0.008 |
Compressed is higher on 6/10 benchmarks, dense on 4/10 — consistent with run-to-run noise, not a compression penalty. Checkpoints in R2 under .../90-prf-exactk-600/prf-exactk/ and .../90-dense-600/dense/; figure + numbers in artifacts/90-prf-exactk-600-mismatch-diagnosis/compare_dense_vs_prf_step600.{png,csv}. Dense run: WandB 90-dense-600/a134dxxx.
shamanework-pl/90-prf-exactk-600/woqs8zra, full scan_history() pull 2026-07-24, 599 training rows; box checked read-only over SSH at step 337 (train.log tail, GPU/disk, tmux). No write of any kind was performed on the box or the run.quick-test/tb4m0k23 (PowerSGD r77 W2, 200 steps) and quick-test/clvaf683 (PowerSGD + decoupled token-IS 2.0, 200 steps), full history pulls; #89 reference frame from runs/89-prf-codec-autoresearch-benign-kl verdict (dense 0.181/0.0033, dropout 0.137/0.124 @40).origin/autonomous-harness-v1 (2ac5740d): verl/workers/comm_eff/activation_mask.py (PRF, exact-k order statistic, top-k forbidden), verl/trainer/ppo/rollout_corr_helper.py + ray_trainer.py (bypass vs decoupled modes, metric definitions), launcher chain run_prf_exactk_600.sh → run_qwen25_math_1p5b_rank1_relex_fsdp.sh → vast_comm_eff_engine_grpo.sh (rollout correction strictly off; calculate_log_probs=true).train.log line 1076; it never reached WandB (known step-0/final-step logging race).artifacts/90-prf-exactk-600-mismatch-diagnosis/; regenerate with the session scripts (pull_wandb.py + gen_report.py).exp/mismatch-control-kit (not pushed; the follow-up issues carry arms and gates).