Back to the hub · agentic training · 2026-08-05

Compressed reinforcement learning on an agentic task

Agentic training: where the train-inference gap stops being survivable

The same two-circuit method that trained for 600 steps without collapse on short single-turn generations was moved to an agentic task, where one trajectory is up to fifty separate interactions and each interaction is optimised as its own training row. The compressed run stopped completing any task by step six. The dense control on the identical task stayed flat. The train-inference gap that earlier reports flagged as the open risk measures 0.61 here against 0.006 dense, and it is already at full size before the first weight update.

Where this sits

This is a short extension of two earlier results, not a new method.

The system as it stands established that the two-circuit method works: 600 steps, no collapse, on a small model with short single-turn generations. Its recorded conclusion was that the circuit is sound and the train-inference mismatch is the outstanding problem.

Larger model, longer context then pushed generation length up. That run gained nine points of held-out accuracy and gave them all back in twenty steps, through a length feedback loop the compressed path could not see.

This run changes the task rather than the model size. It is the same conclusion arriving far sooner, and for a reason that is specific to agents.

What the run did

The outcome first, before any explanation of it. The compressed run was still improving at step three, then gave up everything it had gained and stopped moving at all.

A · critic/score/mean

Mean score per training step, both runs on one axis, zero marked.

compressed dense control
Mean task score per step, compressed against dense The dense control stays between plus 0.22 and plus 0.66 for every step it ran. The compressed run peaks near plus 0.9 at step three, crosses below zero at step six, and settles at exactly minus 0.1 from step seventeen onward. -0.2 0.25 0.5 0.75 1 0 penalty only, no task reward left to earn score turns negative dense run ends dense compressed 1 10 20 30 44 training step
The dense control never leaves positive territory over its twelve steps, ending at +0.59. The compressed run is still climbing at step three, peaking near +0.9, then crosses below zero at step six and lands on exactly −0.100 from step seventeen onward. That flat floor is the informative part. It is the score of a policy that earns no task reward at all and only collects the formatting penalty, it is identical for every trajectory in the batch, and being identical is what stops learning. The rest of this report is why it happened and how.

The gap, measured before any training happens

Generation runs dense on the serving copy. Training runs through the compressed path. So the same token has two probabilities: the one the sampler used when it chose it, and the one the trainer computes when it learns from it. The gap between them is the whole problem.

B · Train-inference probability gap

Mean absolute difference, per emitted token, between the sampler's probability and the trainer's.

compressed dense control
Train-inference probability gap per step, compressed against dense The compressed run starts at 0.61 and rises above 0.95. The dense control holds 0.006 for every step it ran. The compressed value falls near the end only because the policy has stopped producing usable tokens. 0 0.25 0.5 0.75 1 policy dead, gap no longer meaningful dense run ends compressed dense, 0.006 1 10 20 30 44 training step
At step one both runs sit at identical starting weights and no update has been applied, so the entire difference is the compressed forward pass itself: 0.61 against 0.006, a factor of a hundred. The dense control ran twelve steps, marked on the line, which brackets the collapse at step five. The compressed line falling after step 33 is not recovery, and the reason is worth stating because it is a real limitation of this particular measurement. See the note below.
Why this is not a small number. It is a probability difference, so it lives on a zero-to-one scale. A gap of 0.61 means the distribution the trainer is optimising is not a noisy copy of the policy that acted, it is a substantially different distribution. The reward, meanwhile, was earned by the policy that acted.
Why the gap appears to vanish once the policy is dead. The quantity is an absolute difference between two probabilities, so it can never exceed the larger of them. A collapsed policy spreads its mass thinly over 512 tokens of degenerate text, so the probability of each individual token is tiny on both paths, and the absolute difference between two tiny numbers is also tiny. The disagreement has not gone away. The maximum over tokens is still 1.000 on every step, including the steps where the mean reads 0.001, so individual tokens still disagree as completely as it is possible to disagree. What changed is that the mean is now dominated by the huge mass of near-zero probabilities. This floor is a property of measuring in probability space, and it is the clearest argument for preferring the log-ratio form, which has no such floor and would keep rising here.

This is not the rollout-correction KL. The single-turn math runs logged a log-ratio in nats, produced by the importance-sampling correction path. That path is not part of this build, so no such quantity was ever recorded on this task. The two are not interchangeable and neither can be derived from the other: an absolute probability gap says nothing about the ratio of the two probabilities, and the log-ratio is unbounded as either probability approaches zero. What is plotted here is the raw gap, which is the only form of the measurement this run produced.

How the collapse actually runs

It is not a blow-up or a loss spike. It is a length runaway that walks into a hard ceiling and takes the learning signal with it.

C · Tokens emitted per interaction

The mechanism panel. Length climbs into the per-interaction budget and stays there.

compressed dense control
Tokens emitted per interaction, compressed against dense The dense control stays between 90 and 97 tokens per interaction. The compressed run goes from 94 tokens to the 512-token ceiling by step five and stays there. 0 128 256 384 512 per-interaction budget, 512 tokens truncation begins no parseable actions left dense run ends compressed dense 1 10 20 30 44 training step
The dense control holds 90 to 97 tokens per interaction across all twelve steps it ran, ending where its marker sits. The compressed run goes from 94 tokens to the 512-token ceiling by step five and never comes back.

Putting chart A and chart C together, the chain is mechanical:

stepwhat happens
5Responses reach the 512-token budget and start being truncated.
6No episode in the batch completes its task, and the score crosses below zero.
17Not one emitted action is parseable by the environment. Truncated text is not a valid action.
17Every trajectory now scores exactly −0.100, so the group-relative advantage is constant and the policy-gradient term is zero. Learning has stopped on its own, which is why the score line is flat rather than still falling.

The run was carried to step 44 to check whether it would come back. It did not.

Why an agentic task should fail sooner, and does

A reasoning task and an agentic task put very different amounts of compressed forward pass between the reward and the gradient.

On a single-turn reasoning task, one training row is one whole answer. The compressed path is crossed once per answer, over one continuous span of tokens, and most of those tokens carry a loss term.

On an agentic task, one trajectory is up to fifty interactions, and each interaction is a separate training row with its own compressed forward pass and its own independent distortion. Two things follow. The prompt now dominates each row, because every interaction carries the whole accumulated history, so only a small minority of tokens in a row carry any loss at all while the distortion applies to all of them. And the interactions are not independent: what the policy writes at one turn becomes part of the prompt at the next. A distorted policy writes a worse history, and the next turn conditions on it.

That last part has no equivalent on a single-turn task. There the distortion is applied once and the episode ends. Here it is applied at every turn, inside a loop that feeds its own degraded output back in as input. So the same per-pass distortion that was survivable for 600 steps on short answers has fifty chances per trajectory to compound, and it compounds through the environment rather than only through the weights. Collapsing in single-digit steps is the expected outcome, not a surprising one.

The reading. The circuit is not broken and there is no implementation fault: the compression, the anchor and the correction all ran exactly as designed and were verified on every step. What the agentic surface changes is how much the train-inference gap gets to compound before the reward signal is consumed. On short single-turn generations that gap was a tolerable tax. Here it is the binding constraint.

What follows

Closing the gap is now ahead of anything else on the list. The direct move is to correct for it explicitly, so the trainer weights each token by the ratio between the sampler's probability and its own instead of assuming they agree. Reducing the amount of distortion per pass is the blunter alternative and costs communication. Either way the target is the 0.61 in chart B, not the length curve in chart C, which is only where the damage becomes visible.