Blockwise-freeze GRPO: train only the middle block (L11–15) for 75 steps on GSM8K + Big-Math

Close-out verdict (issue record)

VERDICT: PASS (gates green). At this short 75-step budget, block-freeze (train only L11-15) tracks full-parameter GRPO closely on both datasets; the small step-75 val gaps sit within run-to-run noise. This is a parity-at-low-budget result, not evidence that the block fails on the harder split.

C(block) = (S_frozen − S_base)/(S_dense − S_base) — 2-seed replication, train ONLY decoder block L11–15 (frozen) vs full-parameter (dense), vanilla GRPO comm-eff OFF, Qwen2.5-1.5B-Instruct:

Finding (corrected reading): at only 75 steps the frozen and dense runs are nearly indistinguishable on both datasets. Final-val gaps are small (GSM8K 0.011 to 0.036; Big-Math 0.026 to 0.034) and comparable to the run's own val noise: on Big-Math the frozen run actually equals or beats dense at step 50 (frozen 0.598 / 0.600 vs dense 0.578 / 0.580) and only trails by ~0.03 at step 75. Training-reward curves are near-parallel over all 75 steps (mean absolute gap ~0.04 on Big-Math, ~0.07 on GSM8K). The high GSM8K C and low Big-Math C are largely an artifact of the ratio: C's denominator on Big-Math (S_full - S_base ~ 0.07) is tiny, so a 0.02 val wobble swings C by ~0.3. So the earlier "clean negative on Big-Math" phrasing overstates a noise-level difference. Honest conclusion: with this short budget the middle-block-only freeze matches full GRPO on both splits; a longer run is needed to tell whether any real Big-Math gap emerges. Gates-green still makes this a PASS, but as parity-at-low-budget, not a dataset-dependent negative.

Gates (all green): 8/8 cells reached step 75; no NaN/non-finite grads; freeze-correctness verified (frozen cells ran TRAIN_LAYERS=11-15; ~15% params, structural optimizer-ckpt proof).

Caveat: GSM8K S_base=0.077 is a reward-format artifact (base model doesn't emit the #### answer format the GSM8K reward extracts; RL learns it by ~step 25), so C≈0.98 means the block recovers ~all of the (format-dominated) RL gain, not that block ≈ full capability. Big-Math base already emits \boxed (0.538), so its C reflects genuine reasoning gain.

Compute: 1×H200 team, spanned boxes 44365338→44376214 (reaper + disk-full recovery), both torn down. ~12 gpu-hr wall, ~$35 (approx; box2 dph unrecorded).

Code: PR shamanez/verl#24 (TRAIN_LAYERS freeze hook), merged to autonomous-harness-v1.

Report: /runs/64-middle-block-freeze-grpo.html

WandB: https://wandb.ai/shamanework-pl/64-middle-block-freeze-grpo

Analyst verdict (verdict.md)

Post-close correction: the archived text below reads Big-Math as a "clean negative (~half the gain)". The corrected reading in the Close-out verdict above supersedes it: at 75 steps the frozen and dense curves are near-identical on both datasets, the Big-Math step-75 gap (~0.03) is within val noise, and C is fragile there. The PASS stands; the interpretation is parity-at-low-budget, not a dataset-dependent negative.

Verdict — 64-middle-block-freeze-grpo (issue #64)

VERDICT: PASS

8-cell replication matrix {frozen (train ONLY decoder block L11-15), dense (full-param)}

x {gsm8k, bigmath} x {data.seed 42, 7}, comm-eff OFF on all, 75 steps, on branch

exp/64-dense-wandbfix (freeze-hook + #65 wandb final-step fix). Plus base-model val_only

on both datasets for S_base. Judged against run.json success_criteria + pass_rule

("PASS iff all gates green; a clean symmetric negative — C<0.80 with gates green — is PASS").

C(block) = (S_frozen - S_base) / (S_dense - S_base)

C values reproduce metrics/results_summary.json exactly (0.985 / 0.948 / 0.618 / 0.553).

S_base: gsm8k 0.0773313115996967 (results_summary.json; raw val_only in wandb 64-base-gsm8k-v2),

bigmath 0.538 (greppable at incoming.log step:0, and results_summary.json).

Seed spread:

• gsm8k: frozen 0.766490 vs 0.744503 = 0.0220 abs; C spread 0.985 vs 0.948 = 0.0366.
• bigmath: frozen 0.580 vs 0.580 = 0.000 abs (identical); C spread 0.618 vs 0.553 = 0.0650

(spread is driven entirely by the dense denominator: S_dense 0.606 s42 vs 0.614 s7, since frozen is identical).

Success criteria (gates)

• [x] All 8 cells reach step 75, step-75 val lands. 7/8 have a step:75 line in

metrics/*.valcore.txt; dense-bigmath-s7 step:75=0.614 is in metrics/incoming.log:6749

("Training Progress: 100%|...| 75/75", training/global_step:75). All 8 have a valid

step-75 val. Source: the 10 *.valcore.txt files + incoming.log:6749.

• [x] No NaN / non-finite grads. NaN/inf scan over incoming.log = 0 hits; every

actor/grad_norm finite and small (range ~0.064–0.166 across all logged steps; step-75

grad_norm 0.162 on dense-bigmath-s7). Gate green.

• [x] Freeze-correctness (frozen cells). launch_matrix.sh wires TRAIN_LAYERS=11-15 on

all 4 frozen cells (lines 91-99, 120-124); dense cells leave it unset and correctly emit

the "[TRAIN_LAYERS] unset/empty ... DENSE" guard (6 occurrences in incoming.log) — its

absence on frozen cells corroborates freeze active. Structural proof (per operator brief,

now off-box): optimizer ckpt = 234M / 15.2% params, 60 tensors. The freeze-ACTIVE

logger.info marker is a known un-captured launcher gap, not a failure.

• [x] C(block) per dataset per seed, 2-seed error bar. Computed above; PASS predicate

is dataset-split (see conclusion). Both splits are gates-green clean results.

• [x] S_dense = fresh dense-<dataset>-s<seed> step-75 val (measured natively, not the

old 0.7657 fixed ref). Old cross-checks match: freeze-block-l11-15-gsm8k step75=0.7627

(run.json cross_check 0.7627), freeze-block-l11-15-bigmath step75=0.568 (cross_check 0.568).

Metrics summary

• comm-eff OFF verified structurally: all runtime actor/comm_eff/* counters = 0.0

(mask_applications, anchor_backwards, spectral_corrections, powersgd_applications) at

step 75; config trace has actor_rollout_ref.actor.comm_eff.enabled=false.

• Vanilla GRPO confirmed (resolved override trace, incoming.log:6126, last-wins):

algorithm.adv_estimator=grpo, use_kl_in_reward=False, actor.use_kl_loss=False,

entropy_coeff=0 — no-KL / no-entropy, as required by the fixed control.

• gsm8k learns the reward format fast: frozen step-25 val already ~0.727–0.735 vs S_base 0.077.
• bigmath advantages non-degenerate at step 75 (critic/advantages spans ±2.47, reward mean 0.52).

Baseline comparison

Dense (full-param, comm-eff OFF) is the in-matrix apples-to-apples baseline (baseline_run:

"baseline"). Frozen-vs-dense at step 75: gsm8k frozen recovers 94.8–98.5% of the dense RL

gain over base; bigmath frozen recovers 55.3–61.8%. Old fixed-ref cross-checks

(S_full 0.7657 gsm8k; old frozen 0.7627 / 0.568) are consistent with the fresh natives.

Resolved-params excerpt / provenance

See resolved_params.txt + resolved_cmd.txt. RESOLVED_CONFIG_MISSING: the canonical

python3 -m verl.trainer.main_ppo set -x line was not synced locally (no train.log; box

44376214 torn down), so capture_resolved_config.py produced nothing — params were

reconstructed from the trainer's own Hydra override dump (incoming.log:6126) + launch_matrix.sh.

No plan-vs-ran divergence found: adv_estimator=grpo, comm_eff.enabled=false, TRAIN_LAYERS=11-15

(frozen), model Qwen2.5-1.5B-Instruct, 75 steps, seeds {42,7} — all match the run.json cells[] spec.

Notes

• RESOLVED_CONFIG_MISSING (see above) — provenance recovered from log dumps + launcher;

ground truth is consistent, but the set -x main_ppo trace itself is off-box.

• Benign, NOT failures (per operator brief, independently corroborated in-log):

1. All 8 cells' fail_*.flag (rc=1) are a wandb/DataLoader atexit teardown race AFTER

step 75 + val; every step-75 val is present, so the gate is judged on vals not flags.

(The flags were on the remote box; not present locally.)

2. First dense-bigmath-s7 attempt (PID 125495) crashed in _save_checkpoint with

"RuntimeError: basic_ios::clear: iostream error" + Ray "No space left on device"

(incoming.log:6126-6404) — a disk-full infra crash on checkpoint write, NOT numerical

divergence. The 8 cells filled the 200G disk; checkpoints were deleted and the affected

cells re-run on a clean disk. The dense-bigmath-s7 re-run (PID 159834) is clean at step 75.

3. base-gsm8k S_base was likewise re-measured post-crash (wandb 64-base-gsm8k-v2); clean.

• S_base format-artifact caveat: GSM8K S_base=0.077 is low because base Qwen2.5-1.5B-Instruct

does not emit the #### answer format the GSM8K reward extractor requires; RL learns the

format within ~25 steps (step-25 val ~0.73). So GSM8K C~0.98 does NOT mean "the block ~=

full model capability" — it means the block recovers ~all of the (largely format-driven) RL

gain. Big-Math base already emits \boxed (S_base=0.538), so its C reflects genuine reasoning

gain. C is internally consistent per dataset (identical eval config for base / frozen / dense).

Scientific conclusion

Training only the middle decoder block (L11-15) of Qwen2.5-1.5B-Instruct recovers essentially

all of the full-parameter GRPO gain on GSM8K (C = 0.985 / 0.948 at seeds 42 / 7; frozen seed

spread ~0.022 in raw val), but only about half of it on Big-Math (C = 0.618 / 0.553), a clean,

gates-green negative on the harder split. The result is therefore dataset-dependent: the middle

block is sufficient to carry the (format-dominated) RL improvement on GSM8K, but for genuine

multi-step math reasoning (Big-Math) roughly 40-45% of the dense gain requires parameters

outside L11-15. Both splits pass all gates (all cells reach step 75, no NaN/non-finite grads,

freeze-correctness holds), so per the plan's predicate — a clean symmetric negative with gates

green is a PASS — the experiment cleanly answered its question. VERDICT: PASS.

Provenance

# Resolved parameters — RECONSTRUCTED, not from the canonical `set -x` main_ppo trace.
# capture_resolved_config.py found no runs/64-middle-block-freeze-grpo/train.log and
# no 'python3 -m verl.trainer.main_ppo' set -x line in metrics/incoming.log (the box
# 44376214 was torn down; only the rolling tail was synced). => RESOLVED_CONFIG_MISSING.
#
# The values below are the GROUND-TRUTH resolved Hydra overrides printed by the trainer
# itself in metrics/incoming.log:6126 (cell 64-dense-bigmath-s7, the last cell), with
# last-write-wins applied (Hydra semantics). Per-cell env knobs come from launch_matrix.sh.
# Runtime comm_eff/* counters at step 75 are all 0.0 (mask/anchor/spectral) => comm-eff truly OFF.
#
# --- resolved Hydra overrides (from incoming.log:6126, last-wins) ---
algorithm.adv_estimator=grpo
algorithm.use_kl_in_reward=False
actor_rollout_ref.actor.use_kl_loss=False        # was =True earlier in the list; last-wins => False (no-KL)
actor_rollout_ref.actor.entropy_coeff=0          # no-entropy
actor_rollout_ref.actor.comm_eff.enabled=false   # comm-eff master switch OFF (runtime counters confirm 0.0)
actor_rollout_ref.model.path=Qwen/Qwen2.5-1.5B-Instruct
actor_rollout_ref.actor.optim.lr=1e-6
actor_rollout_ref.actor.ppo_mini_batch_size=64
actor_rollout_ref.rollout.n=8
data.train_batch_size=128
data.max_prompt_length=1024                       # bigmath cell (gsm8k accel cell differs; see launcher)
data.max_response_length=4096                     # bigmath cell (gsm8k accel resp=1024)
data.seed=7                                        # last cell; matrix uses {42,7}
trainer.total_training_steps=75
trainer.test_freq=25
trainer.save_freq=50
trainer.val_before_train=False
trainer.project_name=64-middle-block-freeze-grpo
trainer.experiment_name=64-dense-bigmath-s7        # LAST of 8 cells; see run.json cells[] for all
#
# --- per-cell env knobs (launch_matrix.sh) ---
# frozen cells: TRAIN_LAYERS=11-15   (dense cells: TRAIN_LAYERS unset => guard prints DENSE warning)
# gsm8k cells:  ACCEL_LAUNCHER vast_comm_eff_accel_base_qwen25_1p5b_grpo_gsm8k.sh
# bigmath cells: BASE_LAUNCHER vast_comm_eff_baseline_qwen25_1p5b_grpo_gsm8k.sh + BIG_KNOBS (resp=4096, dyn-bsz)
# all cells:    actor_rollout_ref.actor.comm_eff.enabled=false ; data.seed in {42,7}