Max-Collab
Weekly working notes from the collaboration with Max Ryabinin on extending PipelineRL. Each entry is a self-contained readout for team members who were not in the meeting.
2026-08-19Discussion #3Compressing pipeline-parallel inference for RL
The meeting converged on inference-side PP compression as the direction. Understanding what type of compression the stage boundaries tolerate during rollout generation, and how it scales, is the way to go, and all four of us agreed. The trainer side is already in good shape, since controlling the anchor signal removed the collapse and the trainer tolerates large compression and mismatch, so anything solved on inference is complementary and a first step towards fully-PP on-policy learning. Kimi K3's quantization-aware post-training points the same way. Candidate boundary-compression methods, a deeper subspace-networks track, the good-rollouts gate, and three next steps inside.
2026-08-05Discussion #2Compressing what crosses the stage boundary
Everything one pipeline stage tells the next is a single hidden-state vector per token. A frozen 8B shows how far today's compression is from faithful: a calibrated low-rank basis beats random masking at every rate, yet at the 5% operating point both change the model's behavior completely. Solving faithful compression for rollout generation would carry stage-split inference and training over ordinary internet links and remove the invisible sampler-trainer gap that collapsed the long-context run. The note ends with the first wave of experiments as a matrix.
2026-07-22Discussion #1Streaming PipelineRL: Max's layer-wise weight updates
PipelineRL already trains while it keeps generating. Max proposes streaming each new weight version out of the trainer piece by piece, at pipeline-stage boundaries as backprop finishes them, instead of shipping and loading the whole model at once. Off-policyness does not get worse, and the win grows with model size up to pipeline-parallel scale. The first step is to measure the stall.