Max-Collab · discussion #3 · 19 August 2026 · Sameera, AJ, Shamane, Max · PP boundary compression on the inference side as the direction.
We went further on the two-circuit method, and specifically on how far we can get by keeping inference exactly as it is while adding workarounds on the trainer. That framing exposed two angles for the next phase, and the meeting converged on one conclusion. The way to go is to understand the effect of compression at pipeline-parallel (PP) boundaries during inference, which type of compression works there, and how it scales. All four of us agreed.
1 · Where the trainer stands
The trainer-side story is now in good shape. The weight archaeology of the collapsed compressed-GRPO run traced the failure to the anchor correction being reused between refreshes and accumulating a coherent directional bias, and controlling the anchor signal by consuming it once per refresh removes the collapse with no optimizer surgery and no extra computation. What this establishes is that the trainer already tolerates large compression and train-inference mismatch once the correction signal is handled carefully. Inference is the opposite. Faithful compressed generation is still unsolved and an open problem, and discussion #2 showed that both of our methods break the model at the operating rate. Whatever we build on the inference side is therefore complementary to the trainer work, and we can actually apply it the moment it works.
2 · The two angles forward
Angle A. Keep inference as it is and harden the trainer. Keep the sampler dense and keep solving the problems we are facing in the two-circuit trainer as we scale, say from 8B to 32B. The open questions are whether the once-per-refresh anchor fix holds at larger scale and what new failure modes appear.
Angle B. Figure out how much pipeline-parallel inference can carry RL. We frame this as a PP-boundary communication-compression problem, asking what has to cross between stages during rollout generation and how small it can be made without distorting the rollouts. This is also the first step towards on-policy learning, both RL and on-policy distillation, running entirely pipeline-parallel.
Figure 1. The loop in one picture. Rollouts are generated through pipeline stages whose boundary activations are compressed (red) and feed the trainer (green). The green box is largely solved, and angle A scales it with inference kept dense. The red crossings are angle B and the open problem, and the gate is that what comes out must still be good rollouts.
3 · Outside signal from quantization-aware RL
We also discussed Moonshot AI's Kimi K3, which uses a native microscaling format combining MXFP4 weights with per-block scaling and MXFP8 activations, applied via quantization-aware training from the SFT stage onward. SGLang supports it day-0, computing per-token activation scales dynamically over native low-bit weights (backend details), with native execution on Blackwell and MI400-class tensor cores and inline dequantization on older parts. It has nothing to do with reusing the inference cost, but it does the important thing right. The compressed format is the native contract of the whole post-training loop, not a patch applied afterwards. There is already RL work in the same direction (MXFP8 / NVFP4 RL, and our own read of the K3 recipe is in the paper discussion). All of this points at the conclusion above. The open piece, and the one we can own, is compression at the PP boundary during inference.
4 · Initial research directions
exploratorynothing finalized This needs a bit of research first. The initial ideas were to check a few things.
4.1 · Candidate activation-compression methods at the PP boundary
A · Blockwise low-bit microscaling quantization (MXFP8 / NVFP4 style). Quantize each boundary vector with fine-grained per-block scales, optionally Hadamard-rotated to tame outliers, and dequantize on the receiving stage. This gives roughly 4 to 8x on the wire and is hardware-friendly (activation quant for TP inference, TAH-Quant at PP boundaries). The known risks are that outliers dominate the dynamic range and that decode is empirically less quantization-tolerant than prefill.
B · Structured sparsification beyond plain top-k. Transmit a structured column-subspace selection of the boundary activations. Our own PP-training work reports about 90% compression this way (beyond top-k). The unknown is that training averages per-step noise out over updates, while at inference each token's error feeds the next token's context with no averaging.
C · Phase-aware precision. Compress prefill-phase boundary traffic hard and keep decode conservative, with adaptive bit allocation per tile (Mix-Quant finds prefill quantization-redundant while full-pipeline quantization degrades). The catch is that rollout generation is decode-dominated, so the win is real but bounded, and adaptive schemes for decode are unproven.
A deeper idea, kept separate. A learned low-rank or autoencoder bottleneck at the cut point does not belong in the list above, since it changes the network rather than post-processing a frozen one, so we hold it as its own track. It is the retrofit version of our subspace networks, which constrain the layer projections to a slowly learned low-dimensional subspace so that the residual activation crossing each stage boundary reconstructs exactly on the next stage, up to 100x smaller (rank 40 out of hidden size 4096) at 2B to 8B scale. That result is about training. Whether the property carries to free-running rollout generation is precisely the open inference question here, and if it does, compressed PP inference becomes a property of the model instead of a patch on top of it.
4.2 · How these scale, and what they do to output quality
The gate on everything above
Whatever the method, it is only useful if the compressed sampler still produces good rollouts, meaning free-running generations whose task quality survives the boundary. Without good rollouts you cannot do RL or on-policy distillation at all. The rollouts are the fuel. The known failure mode sits exactly here, since boundary error accumulates token-to-token in autoregressive decoding and worsens with generation length. So rollout quality under compression, measured across scale (8B → 32B) and generation length, is the primary metric, not teacher-forced ΔNLL alone.
5 · Next steps
Trainer angle continues as-is. Carry the once-per-refresh anchor fix into the next scale step of the two-circuit runs, 8B first and then towards 32B, with inference kept dense.
Screen the boundary methods on frozen checkpoints. Matched byte budgets at the PP boundaries, scored on free-running rollout quality (task accuracy of the generations), not just teacher-forced ΔNLL.
Get the scaling curve. Repeat the screen across model sizes to see whether boundary compressibility improves or worsens with scale, and review at the next sync to pick the first PP-inference prototype.