The two-circuit method on harder math, and starting from a correct base model
Weekly update · week of 2026-07-06 · self-contained for other teams · follows last week’s weight-projection readout.
✓ Verified by Shamane
1. This week, in one line
The result that matters: even from the generic Qwen2.5-1.5B-Instruct base (not a math model), our two-circuit method stays stable on the harder, more random-looking Big-Math task, as long as we keep the staleness right.
Harder math is a zig-zag, random-looking regime. The two-circuit method rides it and stays on the dense line, and two knobs decide it: α (how strongly the clean anchor sign is applied) and β (how fresh the anchor memory M is). Only the stalest anchor collapses, and keeping β fresh with a moderate α is what prevents it. (see §3)
Weight projection is a separate, optional lever: it helped on the easy task and does nothing on the harder one (known since last week). Worth keeping for later: within a short window the weights still move in one rank-1 direction. (see §2)
On a correct base (a domain-adapted or distilled base model, here DeepSeek-R1-Distill-Qwen-1.5B with 16k reasoning) the two-circuit method holds the dense training curve, and adding weight projection tracks it just as closely (a hair ahead over the shared window). (see §4)
Training only one middle block reproduces almost the full-parameter training curve, even from a non-math base: a second way to cut pipeline-parallel traffic. (see §5)
This reshapes how we should evaluate RLVR (task-specific RL on a strong distilled base), which is the part I most want to discuss. (see §4)
2. Where we were, and what is new
The old world (known since last week). Weight projection, letting the stale side extrapolate its partner’s weights forward, wins on easy GSM8K and collapses on harder Big-Math. That part is settled.
The piece worth keeping (future work). Within a short window the weights still move in essentially one rank-1 direction on both tasks; only the step-to-step coherence that projection needs is lost on the harder task. So a window-local rank-1 view is a lever to revisit, not a dead end.
The new questions. Does the two-circuit method itself stay stable on the random-looking task and from a correct base, and what controls it?
Figure 1. Two timescales (schematic). Both tasks sit on the right: the accumulated weight change is one rank-1 direction on both (the part worth keeping). They split on the vertical axis, the alignment of consecutive updates: GSM8K 0.86 (a line can ride it) versus Big-Math 0.15 (near-orthogonal, nothing to project along). Rank-1 alone does not license projection.
3. The harder task is random, and the two-circuit method stays stable
Random-looking, still learning. On Big-Math the per-step updates point almost every-which-way (consecutive alignment 0.15), so the weight path zig-zags rather than marching. The reward still climbs; the zig-zag averages out around a small persistent drift by the end.
The two-circuit method needs no coherent direction. It keeps the magnitude of the cheap gradient and borrows only the sign from a fresh anchor, so it rides the persistent drift and ignores the zig-zag. That is exactly why it survives here, and why weight projection (which needs a coherent direction) does not add anything.
M freshness is the whole game. A stale anchor points where the weights used to be, and at longer cadence that error compounds until training runs the wrong way. Keep M fresh and the method holds; let it go stale and it collapses. That is the one setting that broke.
Figure 2. Big-Math training reward, generic base (real curves, cadence 20, single seed).α is how strongly the clean anchor sign is applied, β is how fresh the anchor memory M is. The two-circuit method with a fresh anchor (green) holds just under the dense ceiling; the same method with a stale anchor (pink, β=0.90) peels down after step 55. Weight projection (blue, refreshed periodically) does not beat the plain method. The knobs, not projection, are what control stability. (The projection curve is from the companion Big-Math projection sweep, matched in scale.)
For contrast: the easy, coherent task
On the easy task the trajectory is coherent, but staleness bites the same way. At cadence 20 the anchor reference is about 20 steps behind, and applying that stale sign on every step compounds a directional bias until it clearly collapses, while refreshing periodically (or projecting) holds. Same lesson as the hard task: keep the reference fresh.
Figure 3. The easy, coherent task (GSM8K), for contrast (real curves, single seed). At cadence 20 the anchor reference is about 20 steps stale. Applying that sign on every step (red) compounds a systematic directional bias and clearly collapses near step 65 before clawing back; refreshing only when the anchor fires (teal) or adding a projection (blue) holds on the dense line. The lever is the same as on the hard task: keep the reference fresh.
The surprise: our compression-and-efficiency work is more robust on the harder, random-looking regime than on the tidy easy one, because the load-bearing part never relied on a coherent path.
4. The big shift: correct base models change how we do RL research
This is the finding most likely to shape our RL work going forward, and especially how we evaluate it: the strong, efficient RLVR results in the literature depend on starting from a domain-adapted or distilled base model, so our efficiency claims have to be validated in that setting, not on a generic model. Worth a proper conversation.
We moved to the datasets and models the papers use, and looked into why. For hard math the RLVR papers do not start from a generic model; they start from a domain-adapted or distilled base already good at math (such as R1-Distill) and then run task-specific RL (as in DeepScaleR).
It is not cheating. It fits the picture that RL re-weights an existing skill rather than teaching a new one: the labs do task-specific RL and then merge, and before the RL there is almost always distillation (imitation learning). The datasets match: task-specific, verifiable-reward math sets sized to what the distilled base can already reach.
What we ran. The two-circuit method versus a dense control on DeepScaleR from DeepSeek-R1-Distill-Qwen-1.5B, with a 16k-token response budget.
Result: it holds the dense training curve. Both the two-circuit method and the two-circuit method plus weight projection track dense the whole way; over the shared window the projection variant is a hair ahead (within noise). On a hard task the held-out score (AIME) is noisy, so we read the training curve, where the signal is clean.
The same team also ran the projection idea on the newer Qwen3 models, and their released datasets and checkpoints look legitimate.
Figure 4. Correct base, hard task, long reasoning (run #63, real training reward). DeepSeek-R1-Distill-Qwen-1.5B, DeepScaleR, 16k tokens. The two-circuit method (green) and the two-circuit method plus weight projection (blue) both sit on the dense line (gray); over the shared window the projection variant is fractionally ahead (mean 0.495 versus 0.493, ahead on about half the steps, i.e. within noise). The projection arm was operator-cut at step 50. Held-out AIME is 0.21 for the compressed arm versus 0.25 for dense, inside the noise band of a 30-problem benchmark, at roughly 8 times the policy entropy of dense.
Why it works on a correct base (the part worth explaining)
A domain-adapted or distilled base already has the capability the RL is meant to elicit, so RL is re-weighting an existing skill, not installing a new one. Three consequences line up with what we see:
Small nudges. Each update is a small change to an already-good policy, so drift per step is small relative to competence and staleness at a given K bites less.
It rides the drift, not the noise. A fresh anchor averages away the near-orthogonal per-step noise and keeps the aggregate direction, so the compressed arm stays pointed the right way even when a single step looks random.
Capability survives a perturbed distribution. The compressed arms ran at roughly 8 times the policy entropy of dense yet held the reward line, because the anchor keeps the update direction honest and the skill the base already had is not lost.
5. A second efficiency axis: is one block enough?
A bigger lever than projection. Projection asks whether we can send less across the boundary. This asks whether we need to train the whole network at all. Recent work (Is One Layer Enough?, arXiv:2607.01232, on the same veRL and GRPO stack) shows RL gains concentrate in a few middle layers, and training that block alone can match full-parameter RL.
Why it matters. A blockwise scheme shrinks the gradient reduce and the optimizer state, and maps cleanly onto pipeline parallelism: train only one stage’s block and most stages exchange far less. We tried layer freezing back in the Mistral era and it worked; the reason is the one this project keeps hitting, RL does not move the weights very much. The interesting version is the non-extreme one, blockwise-independent training.
Result · block-only training tracks full-parameter training (issue #64, two-seed PASS)
Setup: freeze everything except the middle block (layers 11 to 15), train only that block with vanilla GRPO for 75 steps, comm-eff off, everything else identical to the full-parameter (dense) control. Base is the generic Qwen2.5-1.5B-Instruct, which is not a math model. Two seeds, both datasets. We read this as how closely the block-only curve tracks the full-parameter curve, not as a base-relative recovery ratio (a ratio distorts things when the base-to-full gap is small, or inflated by an answer-format quirk).
The curves land together. Block-only final validation sits right next to full-parameter on both datasets: about 0.76 versus 0.78 in-domain, and 0.58 versus 0.61 on the harder split, and it gets there on the same timescale.
Even from a non-math base. One middle block is enough to reproduce almost the entire full-parameter training curve, on a generic instruct model rather than a math-distilled one. That is the Is One Layer Enough? claim, holding on our stack; their released work checks out.
All gates green; vanilla GRPO, comm-eff off; a clean two-seed PASS. It green-lights blockwise-independent training as a real second comm-eff axis for pipeline parallelism.
Figure 5. Block-only versus full-parameter, final validation (run #64, seed-averaged over two seeds). Train only decoder layers 11 to 15 of Qwen2.5-1.5B-Instruct, comm-eff off, versus the full-parameter control. Block-only lands right next to full-parameter on both datasets (a gap of about 0.02 on the easier task, 0.03 on the harder one), from a generic non-math base.
6. Takeaways
The two-circuit method is the load-bearing part; projection is optional. On harder math the two-circuit method stays stable and projection adds nothing. What controls stability is anchor freshness: keep M fresh (fresh β, moderate α) and it holds, let it go stale and it collapses.
Robust where we expected fragile. The method holds up best on the harder, random-looking trajectories, because it never relied on a coherent path. Rank-1 within a window still holds on both tasks and is worth revisiting.
Correct base models are the right setting. On a domain-adapted or distilled 1.5B with 16k reasoning, the two-circuit method holds the dense training curve and weight projection matches it, keeping capability even at roughly 8 times the policy entropy.
Layer freezing is a promising second axis. Block-only training tracks the full-parameter curve closely on both datasets, even from a non-math base. RL barely moves the weights, so blockwise-independent training is a credible way to cut pipeline-parallel communication.
This reshapes how we define and evaluate RLVR. Task-specific RL from a strong distilled policy, not generic pre-training, is the frame that makes our efficiency results hold. This is the point I most want to talk through.
7. Next steps
Take the correct-base, hard-data runs to full length and across more model and dataset pairs, and read training curves plus a less noisy held-out metric.
Scope blockwise-independent training as a pipeline-parallel comm-eff axis, and check whether the harder-task gap closes from a correct base rather than the generic instruct model.
Revisit the window-local rank-1 direction as a lever (a recent, not global, line), since it survives on both tasks even where step-to-step projection does not.
Pin down the “why it works on a correct base” story with controlled runs on entropy, drift, and staleness against K.
Wang et al. (2026). Linear Dynamics in the RLVR Training of Large Language Models. arXiv:2601.04537.
Wei et al. (2026). You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (RELEX). arXiv:2605.21468.
Is One Layer Enough? Training a Single Transformer Layer Can Match Full-Parameter RL Training. arXiv:2607.01232.
Models: Qwen2.5-1.5B-Instruct (generic base), DeepSeek-R1-Distill-Qwen-1.5B (domain-adapted / distilled base). Datasets: GSM8K (easy), Big-Math filtered and DeepScaleR (hard). RL recipe: vanilla GRPO, no KL, no entropy.
Weekly update for the week of 2026-07-06. Curves are real training data (Big-Math and GSM8K reward from the offline sweeps and the live runs; run #63 from WandB; run #64 final validation, two seeds). The (α,β) sweep and run #63 are single-seed and, for #63, operator-truncated, so treat them as directional. Figure 1 is a schematic. Verified by Shamane.