Two adaptive projectors, two different directions in weight space
Both cold-start at fixed_linear and learn a per-block correction from retrospective error — no peek at live weights. The real difference is what each one learns, and therefore which direction in weight space it can actually correct.
The frozen seed both adaptive modes start from — a plain linear extrapolation along the recent weight trajectory, with no feedback and nothing learned.
Projection
θ̂ = S₀ + (α·h/g) · (S₀ − S₁)
At the operating point (α = 1, h = g) this is the AsyncPP seed 2·S₀ − S₁ — coefficients (2, −1). A pure function of two stale snapshots: fully deterministic, no per-block state.
cost: 2 sources (S₀, S₁) · no θ̂ retained · 0 learned params — the cheapest mode
Its role
This is the cold start of both learned modes. β = 1 (step-scale) and r = 0 (offset) each reduce to exactly this line — so the adaptive modes only ever improve on fixed_linear, and their first fire is byte-identical to it.
Its limit
It is static: the same extrapolation for every block, forever. With no feedback it can't sense that a block over- or under-shot, so where the trajectory bends — k-collapse, Big-Math phase changes — it lands wrong and stays wrong. Closing that fixed gap is the whole job of the two adaptive modes below.
The anchor wants to sit at the live weights● θ_true, but only has stale snapshots on the trajectory S₁ → S₀. Offset can only shove the whole matrix along the all-ones diagonal — a direction the error barely has. Step-scale slides along the trajectory itself, landing on the closest point to θ_true. Drag β and watch it close the gap.
β1.35
dist step-scale25
dist offset / β=178
mode 2 · the offset residual
learned_linear_with_fixed_linear_cold_start
Learns
a per-block additive residual r (a DC offset), init 0.
Correction
θ̂ = fixed_linear + r (r broadcast over the whole matrix = r·𝟙)
Update rule
r += 0.1 · mean(θ_true − θ̂) · clip |r| ≤ 1e-3
The catch
mean(e) is the error's component along the all-ones direction. Weight-update matrices are ≈ zero-mean, so mean(e) ≈ 0 — the signal it trains on is nearly empty, and r stays tiny.
Geometry
shifts the matrix up/down by a constant — moves θ̂ only along the DC diagonal, not toward the trajectory.
Cost
3 sources (S₀,S₁,S₂) + θ̂ = 4 copies
mode 3 · the step scale NEW
learned_step_scale_with_fixed_linear_cold_start
Learns
a per-block multiplicative scale β on the extrapolation step, init 1.
⟨e, step⟩ projects the error onto the step actually taken — a genuine inner product, not a mean. Signed and ≈ nonzero: >0 = "under-shot, go further", <0 = "over-shot". It answers how far to project.
Geometry
slides θ̂ along the trajectory to the point closest to the live weights. β→0 degrades to raw-stale where the path isn't linear.
Cost
2 sources (S₀,S₁) + θ̂ = 3 copies (lighter)
Side by side
Same skeleton, one decisive swap: the basis direction the per-block scalar is learned along.
offset residual (mode 2)
step scale (mode 3)
Learned quantity
additive residual r
multiplicative scale β
Init (cold start)
r = 0
β = 1
Basis direction
𝟙 (all-ones / DC)
S₀ − S₁ (trajectory)
Training signal
mean(e) ≈ 0 → weak
⟨e, step⟩ ≠ 0 → informative
What it can fix
the mean level of a matrix
how far along the trajectory θ̂ sits
Bound
|r| ≤ 1e-3 (absolute)
β ∈ [0, 2] trust region
Source snapshots
3 (S₀,S₁,S₂)
2 (S₀,S₁)
Full-model copies
4
3
Order
second-order-ready
first-order
What they share
The step-scale mode is a drop-in sibling — it keeps every invariant the offset mode already guarantees.
≡Cold-start byte-identical to fixed_linear. r = 0 / β = 1 both reduce to the exact seed step, so the first fire matches mode 1.
=Cross-rank deterministic. every learned scalar is a reduction over DP-identical stale snapshots — no RNG, no device-local state.
⊘No-peek. trained only from a past prediction whose ground truth has aged into the stale ring, never the live weights.
▦Targets only 2D decoder matrices. norms / embeddings / lm_head take the raw θ[t−K] — the low-linearity exclusion.
Why the direction matters
The anchor's whole job is a clean reference gradient close to the fast network.
At staleness K the raw θ[t−K] has rotated nearly orthogonal to the live weights — the k-collapse — so a gradient computed there is a poor reference. Projection moves θ̂ back toward the live weights, so G_anchor lands near the fast net and the correction feedback becomes trustworthy.
The offset mode can only correct a DC level the error essentially doesn't have. The step-scale mode corrects the one thing that actually collapses: how far along the trajectory the anchor sits — learned directly from whether the last projection under- or over-shot.