Kimi K3 Post-Training Figures

Figure 1. Nine teachers from one checkpoint

Kimi K3, sections 4.1.1 and 4.1.2

π_SFT shared cold-start policy, quantisation-aware from here on

copied into three independent RL runs

General tasksreasoning, vision, search, knowledge work

General agentsassistant, deep research, writing

Coding agentsSWE, kernels, web development

anneal τ

maxlargest τ

T1π^{(gen, max)}

T2π^{(agt, max)}

T3π^{(cod, max)}

highτ lowered

T4π^{(gen, high)}

T5π^{(agt, high)}

T6π^{(cod, high)}

lowτ lowered again

T7π^{(gen, low)}

T8π^{(agt, low)}

T9π^{(cod, low)}

3 domains × 3 effort levels = 9 frozen teachers. Domains run in parallel. Effort levels are sequential: a trajectory scoring −1 once it exceeds τ · b₀(x) tokens, with τ annealed downward. None of the nine ships.

Figure 2. One student, nine frozen teachers, the student's own rollouts

Kimi K3, section 4.1.3; MOPD, section 3.1

π_SFT

assumed, per MOPD

trainable Student π_θ reads the effort level from context

Unified Kimi K33 domains, 3 effort levels, one model

1Draw prompt x with its labels: domain d, effort level e.

2Student samples the whole trajectory y from π_θ. Nothing is copied from a teacher.

3Router picks the one teacher matching (d, e).

▶trajectory y out

4Teacher prefills y in one forward pass, returns a log-probability per token.

◀one number per token back

5Update the student on every token, then resample from the updated student.

Nine frozen teachers

gen

agt

cod

max

T1

T2

T3

high

T4

T5

T6

low

T7

T8

T9

Forward pass only. Served outside the trainer like a reward model, so its cost hides behind student sampling.

On-policytraining states equal inference states, so no exposure bias

Densea signal at every token, not one scalar per trajectory

Parallelteachers never interact, a restart hits one lane only

Same origininitial per-token KL about 0.04, against about 0.19 for an outside teacher, whose run degraded

Figure 3. One scalar per token, and the loss it feeds

Kimi K3, equation 15; MOPD, equations 1 to 4

r_t = clip( sg[ log π_teacher^{(d, e)}(y_t | x, y_<t) π_θ(y_t | e, x, y_<t) ], −R_max, +R_max )

In words: at the token the student actually produced, take the teacher's log-probability of that token minus the student's own, freeze it into a plain number, and cap it. One scalar per token position: positive raises that token, negative lowers it, zero means the two already agree.

the loss that is actually optimised L(θ) = − E_{y ∼ π_θ} [ (1 / |y|) ∑_t r_t · log π_θ(y_t) ] Every token of the rollout contributes, divided by rollout length, averaged over rollouts drawn from the current student. r_t is a fixed weight, so the only quantity carrying a gradient is log π_θ(y_t).

teacherFrozen, forward pass only. Domain and effort are fixed in its weights, hence the superscripts.

±R_maxA hyperparameter, set by hand. A log-ratio has no lower bound, so it bounds one token's reward.

sgStop gradient. The bracket becomes a constant, so nothing propagates back through it.

log π_θThe only trainable term. Frozen: teachers, the sampled y, everything inside sg.

Runs on the on-policy RL machinery, but it is distillation, not reward maximisation

	On-policy RL with verifiable rewards	On-policy distillation (MOPD)
Who grades	an external verifier: tests pass, answer matches, rubric	a frozen teacher, scoring the token the student chose
Signal density	one scalar for the whole trajectory	one scalar at every token, so \|y\| of them
Rollouts per prompt	many, a group baseline is needed to assign credit	one is enough, MOPD trains with N = 1
Ceiling	can pass any existing model, no natural stopping point	bounded by the teacher, r_t vanishes at a match

Same update rule, different objective. Only the advantage computation changes, so it drops into a GRPO trainer, but the target is a teacher distribution rather than an outcome, and there is nothing here to reward hack.