Rank-1 Direction Weight Projection for the Delayed Anchor Circuit

01 Problem statement

The comm-efficient anchor circuit ships a full weight snapshot across the pipeline boundary only sparsely (every cadence steps) and forwards from a stale snapshot (delay_K steps old). The operating merger signed_ema reads only sign(M_anchor) per coordinate, where M_anchor is an EMA of gradients evaluated at the stale weight point θ[t−K]. As K grows, the gradient's sign pattern rotates away from the live weights and the correction degrades — the k-collapse, empirically appearing ~step 61 at cadence/delay_K = 20/20.

02 Current anchor-circuit behavior

Confirmed against the code (paths under /home/user/verl/). This answers the two questions directly: cadence = how often weights are exchanged (sparsity); delay_K = the lag. At fast tick 20 with delay_K=10, the anchor operates on the tick-10 snapshot.

03 Why the paper setting (RELEX) differs from ours

04 Proposed rank-1 projection method (growing-window)

Per target matrix, flatten to θ∈R^d. Store θ_base=θ_0 and consecutive-tick deltas d_k = θ_{t_k}−θ_{t_{k−1}}. Maintain the consecutive-delta Gram D[k,l]=⟨d_k,d_l⟩; prefix-sum → base-delta Gram A; re-base bilinearly. The Gram is additive over shards and group members (block/global pooling for free) and is d-independent in storage — the fork's offline TrajGram discipline ported online (rank1_traj.py:110-139).

# fire hook: fold in new anchor-visible snapshot θ_new @ tick t_new
d_new = θ_new − θ_prev                      # one fp32 diff (bf16-safe, no cancellation)
for k in retained: D[new,k]=D[k,new]=dot(d_k,d_new)
D[new,new]=dot(d_new,d_new); store d_new; θ_prev←θ_new; W←W+1
if W > W_cap: drop_oldest_and_rebase()       # fixed base θ0 + sliding recent fit window

# fit on demand (Gram only — no d-vector touched)
G  = symmetrize(B[window, window])           # W×W
λ,U = eigh(G)  (descending)
σ1 = sqrt(max(λ1,0));  c = σ1·u1             # coefficient trajectory c_i=⟨Δ_i,v1⟩
EVR1 = λ1/trace(G)                           # coherence signal #1

v1 = (1/σ1)·Σ_i u1[i]·Δ_i is materialized only to build θ̂ (one streamed O(d) pass), never for scoring.

4-checkpoint bootstrap

N0 = 4. A line needs ≥2 points; 4 gives a usable R² plus slack. Below fire 4 the projector is a strict no-op → warmup falls back to the raw stale anchor (unchanged from today). At cadence/delay_K=20/20, fires {20,40,60,80} record true-ticks {0,20,40,60} → first projection at step 80.

Does "keep growing ⇒ keeps improving" hold?

Only conditionally. Model c_i = a·t_i + b + ½q·(t_i−t̄)² + ε_i, Var(ε)=τ²:

05 Ordinary least-squares coefficient prediction

Closed form on real tick indices (handles sparse + variable staleness exactly; no uniform-spacing assumption):

t̄=mean(t_i); c̄=mean(c_i)
a = Σ(t_i−t̄)(c_i−c̄) / Σ(t_i−t̄)²   =  Cov(t,c)/Var(t)
b = c̄ − a·t̄
R² = 1 − SS_res/SS_tot

De-stale to "now", never lead:

h_eff = min(t_fast − t_W, h_safe)     # t_W = newest visible tick = t_fast − K
t_tgt = t_W + h_eff                    # ≤ t_fast  ⇒  ANCHOR NEVER LEADS
ĉ     = a·t_tgt + b

If h_safe < delay_K we deliberately land short of now (still lagging — conservative), never past it.

Weight reconstruction as an explicit linear combination of raw snapshots (the falsifiable "stays inside the snapshot span" contract, self-tested to 1e-9 / audited to 1e-6):

θ̂(t_tgt) = θ_base + ĉ·v1 = θ_base + Σ_i γ_i·Δ_i,   γ_i = (ĉ/σ1)·u1[i]

06 Alternative projection variants

Granularity — recommend per-tensor fit + block-pooled coherence gate

Ingredient comparison

07 Interaction with sign correction (additive)

Operating merger unchanged: G_corr = α·G_noisy + (1−α)·|G_noisy|·sign(M_anchor) (α=0.25, β_anc=0.50).

Two resolutions — ship both, distinct roles

• Resolution A · PRIMARY  (= user option #3 mechanism). Load θ̂[t] into the isolated anchor clone, run the one clean unmasked backward on the paired batch → M̂ = G_anchor(θ̂), feed RAW into the M_anchor EMA via the existing update_anchor. No currency mismatch — the gradient is genuinely evaluated at de-staled weights. A drop-in upgrade of the existing LookaheadProjector.project() seam.
• Resolution B · SECONDARY  (sign/gate only). M̂_dir = −a·v1 (weight velocity a·v1 ≈ −η·ḡ). Free (no extra backward), but no magnitude, no off-v1 structure. Use only to cross-check sign(M_anchor) and feed the coherence gate.

Additive decomposition (reduces exactly to signed_ema at full confidence)

w_i   = clip(|M_i| / (κ·quantile_p(|M|)), 0, 1)          # option #1: per-coord confidence
τ     = shrink_to_block(g(R², EVR1, fire_cos, sign_agree))  # option #3, ∈[0,1]
α_eff = α + (1−α)·(1−τ)
signed_i = w_i·|G_noisy_i|·sign(M_i) + (1−w_i)·G_noisy_i  # keep fast sign where M unsure
G_corr_i = α_eff·G_noisy_i + (1−α_eff)·signed_i

• DIRECTION → refreshes the sign input by moving the eval point (option #3). Applied to the anchor reference, never raw weights.
• COEFFICIENTS → set the de-stale distance + supply the coherence/trust τ.
• MAGNITUDE → per-coord confidence w_i (option #1) + per-tensor trust ‖M‖; output magnitude always stays |G_noisy|.

08 Cost analysis

Constants: 196 proj matrices (1.310 B params, 85%) + embed etc → 1.544 B total; bf16 snapshot = 3.09 GB; MLP (gate/up/down) = 88% of each layer.

Communication — per fire: weights up 3.09 GB + sign(M) down 0.19 GB ≈ 3.28 GB round-trip

Memory (Gram trick)

~10⁵× smaller. θ̂ is materialized on demand as a streamed O(d) linear combo of snapshots the anchor already retains (one transient 3 GB vector).

Compute per fire

Gram row update ~15 ms (HBM-resident) / eigh(T×T)×338 tensors < a few ms / OLS closed-form µs. Full-batch anchor backward ≈ ~5 s. Projection share < 1% at the full-batch op point. Negligible.

09 Failure modes

10 Implementation plan (no interface change, byte-identical OFF)

Extend the existing look-ahead seam (a look-ahead projector already exists but holds only 2–3 newest snapshots with fixed/learned linear extrapolation — the growing rank-1 SVD is additive to it).

11 Experiment plan (Big-Math)

Gate decision — commit GPU iff BOTH

• Big-Math (gate-correctness): ungated ratio ≥1.0 (projection harmful → gate necessary) AND gated ratio ≥0.999 (falls back to do-nothing, no worse). Fraction gated-ON ≈ 0.
• GSM8K (margin): ungated ratio ≤0.95 at h=K (EXP-47 got 0.940) AND ratio(W) slope ≤0 over W=4…W_max AND EVR1≥0.6, coef_r2≥0.7.
• Fail either → STOP / redesign.

Step 2 — online arms

Big-Math surface: gshasiri/Big-Math-RL-Verified-filtered, MAX_RESPONSE_LENGTH=4096, locked comm-eff accel base, cadence/delay_K=20/20, PowerSGD r=77, α=0.25 / β_anc=0.50, 200 steps (7 projected fires, W:4→10); GSM8K confirmation arm at resp 1024.

Metrics & thresholds

• val/score (primary); grad_norm stability (fail on NaN or >10× arm-A); comm bytes_ratio must equal arm A; coefficient-prediction error (retrospective, no peek); EVR/coherence; fraction gated-ON; off-path bitwise parity.
• Big-Math (safety, primary): arm C/D Δ ≥ −max(0.010, 2σ) (no regression); gated-ON ≈ 0.
• GSM8K (margin): arm C Δ ≥ max(0.010, 2σ); low coeff error; high EVR/coherence.
• Seeds: GSM8K 3, Big-Math 2 (safety-weighted); B/D on seed 0 as ablations.

12 Config toggles and safety guards (defaults OFF)

CommEffAnchorConfig (comm_eff.py:98-198):

CommEffSpectralConfig (comm_eff.py:201-435): signed_ema_conf_gate:bool=False, signed_ema_conf_floor:float=0.0 (option #1).

13 Recommended next steps

1. Run the GPU-free offline gate (§11 step 1) on EXP-43/57 (GSM8K) + EXP-58 (Big-Math). Reproduce EXP-60 degeneracy, measure ratio(W) monotonicity, and validate the gate collapses to do-nothing on Big-Math. No GPU until this clears both thresholds.
2. Implement the config surface (§12) + a byte-identical-OFF parity test (CPU parity + 1–2-step GPU weight-hash probe).
3. Implement GrowingWeightTrajRing + Rank1TrajProjector (Resolution A) porting rank1_traj math; wire the fire hook + update_anchor feed. Pin the eigenvector sign; compute-once-and-broadcast on the anchor node.
4. Implement the projected_anchor merger mode + option-#1 confidence gate additively over signed_ema (verify τ=1, w≡1 reduces byte-identical).
5. Run arm A vs C on Big-Math (safety, primary) at 20/20, 200 steps; require no-regression + gated-ON≈0 + comm parity.
6. Run the GSM8K confirmation arm for the coherent margin; add arms B/D ablations on seed 0.
7. Gate promotion on: Big-Math parity-or-better AND GSM8K Δ≥0.01 AND zero extra comm AND byte-identical OFF.