Back to the hub · activation compression · 2026-08-05

Train-inference mismatch check

How much does activation compression change a frozen model?

Our training circuit keeps about 5% of hidden activation values at logical pipeline boundaries. During training, model weights can adapt to that bottleneck. We therefore tested the reverse condition: insert the same bottleneck into unchanged checkpoints at inference and observe the immediate change in behavior. This is a sensitivity test for train-inference mismatch, not a prediction of how a model trained end to end with compression will behave.

View the recorded outputs

The check

We tested unchanged Qwen3-8B and Qwen3-8B-Base checkpoints at seven boundaries after layers 4, 9, 14, 19, 23, 27, and 31.

Matched compression. PRF keeps exactly K features. Static SVD sends K BF16 coefficients in a prefit uncentered basis. K=205 out of 4,096 means 5.0049% of values are kept.

Where the hidden-state codec was inserted

The same seven intervention points were tested in separate PRF and static SVD arms.

Compression points along the 36-layer model path Two parallel 36-layer paths show the PRF arm and the static SVD arm. In each arm, a hidden-state codec is applied after layers 4, 9, 14, 19, 23, 27, and 31. Layers 32 through 35 then run without another compression point. Hidden-state codec after layer 4 9 14 19 23 27 31 PRF arm L0-4 L5-9 L10-14 L15-19 L20-23 L24-27 L28-31 L32-35 PRF after layer 4 PRF after layer 9 PRF after layer 14 PRF after layer 19 PRF after layer 23 PRF after layer 27 PRF after layer 31 Static SVD arm L0-4 L5-9 L10-14 L15-19 L20-23 L24-27 L28-31 L32-35 Static SVD after layer 4 Static SVD after layer 9 Static SVD after layer 14 Static SVD after layer 19 Static SVD after layer 23 Static SVD after layer 27 Static SVD after layer 31 36 layers total, indexed 0 to 35
The codec changes only the per-token hidden activation between layer groups, then reconstructs width 4,096 for the next group. Checkpoint weights remain unchanged, and KV-cache storage is not a codec target. This is a logical intervention map, not measured hardware or network transfer.

What happened. Keeping 5.0049% of activation values at all seven boundaries strongly changed next-token behavior for this prompt. Static SVD caused less distortion than PRF at every matched rate, although neither remained close to dense behavior.

Less activation kept, more policy distortion

Delta NLL measures how much less likely each checkpoint makes its own dense continuation after compression. Zero is dense parity. Lower is better.

PRF median across three step keys Static SVD, BF16

Qwen3-8B

Post-trained checkpoint

Qwen3-8B activation compression distortion sweep At 5.0049, 10.0098, 19.9951, 25, and 50 percent kept, PRF median delta NLL is 17.4352, 17.4803, 16.8355, 16.5094, and 11.6494. Static SVD delta NLL is 11.7077, 10.4617, 8.2171, 7.4892, and 4.1331. 20 15 10 5 0 5 10 20 25 50 Activation values kept (%) Delta NLL 5.0049% kept: Delta NLL 17.4352 10.0098% kept: Delta NLL 17.4803 19.9951% kept: Delta NLL 16.8355 25% kept: Delta NLL 16.5094 50% kept: Delta NLL 11.6494 5.0049% kept: Delta NLL 11.7077 10.0098% kept: Delta NLL 10.4617 19.9951% kept: Delta NLL 8.2171 25% kept: Delta NLL 7.4892 50% kept: Delta NLL 4.1331 PRF SVD
At the 5% operating point, Delta NLL is 17.44 for PRF and 11.71 for static SVD.

Qwen3-8B-Base

Pretraining-only checkpoint

Qwen3-8B-Base activation compression distortion sweep At 5.0049, 10.0098, 19.9951, 25, and 50 percent kept, PRF median delta NLL is 17.8954, 17.4909, 17.6302, 17.2645, and 10.5409. Static SVD delta NLL is 7.5329, 6.7826, 5.8444, 5.3235, and 2.8309. 20 15 10 5 0 5 10 20 25 50 Activation values kept (%) Delta NLL 5.0049% kept: Delta NLL 17.8954 10.0098% kept: Delta NLL 17.4909 19.9951% kept: Delta NLL 17.6302 25% kept: Delta NLL 17.2645 50% kept: Delta NLL 10.5409 5.0049% kept: Delta NLL 7.5329 10.0098% kept: Delta NLL 6.7826 19.9951% kept: Delta NLL 5.8444 25% kept: Delta NLL 5.3235 50% kept: Delta NLL 2.8309 PRF SVD
At the 5% operating point, Delta NLL is 17.90 for PRF and 7.53 for static SVD.

Rank and spectrum status by layer

The SVD basis was fitted separately at each boundary. Stable and effective rank values in the source run span 1.0 to 2.6, while the component counts and held-out coverage below vary sharply by boundary.

Components needed for calibration energy

The vertical scale is logarithmic. The dashed line marks 5% kept (K=205), the operating point used in the layer-wise test.

Components needed for 95% calibration energy Components needed for 99% calibration energy 5% kept, K=205

Qwen3-8B

Post-trained checkpoint

Qwen3-8B components needed for calibration energy by boundary After layers 4, 9, 14, 19, 23, 27, and 31, components needed for 95% calibration energy are 355, 1, 1, 1, 1, 1, and 2. Components needed for 99% are 1125, 1, 1, 1, 1, 44, and 520. Stable rank is 1.6, 1.0, 1.0, 1.0, 1.0, 1.0, and 1.1. Effective rank is 2.4, 1.0, 1.0, 1.0, 1.0, 1.1, and 1.2. 4096 1000 100 10 1 K=205 4 9 14 19 23 27 31 Boundary after layer Components, log scale After layer 4: K at 95% is 355 After layer 9: K at 95% is 1 After layer 14: K at 95% is 1 After layer 19: K at 95% is 1 After layer 23: K at 95% is 1 After layer 27: K at 95% is 1 After layer 31: K at 95% is 2 After layer 4: K at 99% is 1125 After layer 9: K at 99% is 1 After layer 14: K at 99% is 1 After layer 19: K at 99% is 1 After layer 23: K at 99% is 1 After layer 27: K at 99% is 44 After layer 31: K at 99% is 520
K=205 exceeds the 95% calibration threshold at every boundary except after layer 4.

Qwen3-8B-Base

Pretraining-only checkpoint

Qwen3-8B-Base components needed for calibration energy by boundary After layers 4, 9, 14, 19, 23, 27, and 31, components needed for 95% calibration energy are 415, 1, 1, 1, 2, 129, and 663. Components needed for 99% are 1239, 1, 2, 6, 347, 1331, and 2073. Stable rank is 1.6, 1.0, 1.0, 1.0, 1.1, 1.2, and 1.4. Effective rank is 2.6, 1.0, 1.0, 1.0, 1.1, 1.4, and 2.0. 4096 1000 100 10 1 K=205 4 9 14 19 23 27 31 Boundary after layer Components, log scale After layer 4: K at 95% is 415 After layer 9: K at 95% is 1 After layer 14: K at 95% is 1 After layer 19: K at 95% is 1 After layer 23: K at 95% is 2 After layer 27: K at 95% is 129 After layer 31: K at 95% is 663 After layer 4: K at 99% is 1239 After layer 9: K at 99% is 1 After layer 14: K at 99% is 2 After layer 19: K at 99% is 6 After layer 23: K at 99% is 347 After layer 27: K at 99% is 1331 After layer 31: K at 99% is 2073
The Base checkpoint needs more than K=205 for 95% calibration energy after layers 4 and 31.

K@95 and K@99 are spectrum thresholds, not evaluated codec settings. The Base layer 31 K@99 threshold is 2,073, beyond the stored 2,048-component basis.

Energy captured at 5% kept (K=205)

Calibration energy is measured on the states used to fit each basis. Dense held-out energy tests the same basis on separate dense states. The vertical axis begins at 65% so the layer differences remain visible.

Calibration energy Dense held-out energy

Qwen3-8B

Post-trained checkpoint

Qwen3-8B energy captured by K=205 at each boundary After layers 4, 9, 14, 19, 23, 27, and 31, calibration energy is 92.14, 99.99, 99.98, 99.95, 99.85, 99.44, and 98.30 percent. Dense held-out energy is 69.63, 99.92, 99.87, 99.73, 99.06, 96.53, and 91.66 percent. 100% 95% 85% 75% 65% 4 9 14 19 23 27 31 Boundary after layer Energy captured After layer 4: calibration energy 92.14% After layer 9: calibration energy 99.99% After layer 14: calibration energy 99.98% After layer 19: calibration energy 99.95% After layer 23: calibration energy 99.85% After layer 27: calibration energy 99.44% After layer 31: calibration energy 98.30% After layer 4: dense held-out energy 69.63% After layer 9: dense held-out energy 99.92% After layer 14: dense held-out energy 99.87% After layer 19: dense held-out energy 99.73% After layer 23: dense held-out energy 99.06% After layer 27: dense held-out energy 96.53% After layer 31: dense held-out energy 91.66%
Held-out coverage falls most sharply after layer 4, despite 92.14% calibration coverage at the same K.

Qwen3-8B-Base

Pretraining-only checkpoint

Qwen3-8B-Base energy captured by K=205 at each boundary After layers 4, 9, 14, 19, 23, 27, and 31, calibration energy is 90.98, 99.92, 99.83, 99.62, 98.68, 95.76, and 90.86 percent. Dense held-out energy is 69.52, 99.74, 99.55, 99.16, 97.23, 91.44, and 84.56 percent. 100% 95% 85% 75% 65% 4 9 14 19 23 27 31 Boundary after layer Energy captured After layer 4: calibration energy 90.98% After layer 9: calibration energy 99.92% After layer 14: calibration energy 99.83% After layer 19: calibration energy 99.62% After layer 23: calibration energy 98.68% After layer 27: calibration energy 95.76% After layer 31: calibration energy 90.86% After layer 4: dense held-out energy 69.52% After layer 9: dense held-out energy 99.74% After layer 14: dense held-out energy 99.55% After layer 19: dense held-out energy 99.16% After layer 23: dense held-out energy 97.23% After layer 27: dense held-out energy 91.44% After layer 31: dense held-out energy 84.56%
Held-out coverage is lower than calibration coverage at every boundary, with the largest gap after layer 4.

Layer-wise behavior at 5% kept (K=205)

Each connected point applies compression at one boundary only, ordered by layer. Dashed lines show the result when the same codec is applied at all seven boundaries. Delta NLL uses a fixed token history, and lower is better.

PRF median Static SVD All seven reference

Qwen3-8B

Post-trained checkpoint

Qwen3-8B layer-wise distortion at 5% kept After layers 4, 9, 14, 19, 23, 27, and 31, PRF median delta NLL is 8.5926, 9.6295, 12.7035, 12.8759, 11.5399, 10.6303, and 9.6159. Static SVD delta NLL is 2.8685, 2.6555, 3.8143, 5.9918, 7.6560, 7.0543, and 5.4305. Applying compression at all seven boundaries gives 17.4352 for PRF and 11.7077 for static SVD. 20 15 10 5 0 4 9 14 19 23 27 31 Boundary after layer Delta NLL all 7: 17.44 all 7: 11.71 After layer 4: Delta NLL 8.5926 After layer 9: Delta NLL 9.6295 After layer 14: Delta NLL 12.7035 After layer 19: Delta NLL 12.8759 After layer 23: Delta NLL 11.5399 After layer 27: Delta NLL 10.6303 After layer 31: Delta NLL 9.6159 After layer 4: Delta NLL 2.8685 After layer 9: Delta NLL 2.6555 After layer 14: Delta NLL 3.8143 After layer 19: Delta NLL 5.9918 After layer 23: Delta NLL 7.6560 After layer 27: Delta NLL 7.0543 After layer 31: Delta NLL 5.4305
Middle boundaries produced the largest one-boundary response, but the complete seven-boundary path shifted further for both codecs.

Qwen3-8B-Base

Pretraining-only checkpoint

Qwen3-8B-Base layer-wise distortion at 5% kept After layers 4, 9, 14, 19, 23, 27, and 31, PRF median delta NLL is 8.3916, 8.7693, 8.5936, 9.9200, 10.1801, 9.8775, and 9.3474. Static SVD delta NLL is 2.8977, 2.3226, 2.8945, 4.4791, 5.2255, 5.1092, and 4.6548. Applying compression at all seven boundaries gives 17.8954 for PRF and 7.5329 for static SVD. 20 15 10 5 0 4 9 14 19 23 27 31 Boundary after layer Delta NLL all 7: 17.90 all 7: 7.53 After layer 4: Delta NLL 8.3916 After layer 9: Delta NLL 8.7693 After layer 14: Delta NLL 8.5936 After layer 19: Delta NLL 9.9200 After layer 23: Delta NLL 10.1801 After layer 27: Delta NLL 9.8775 After layer 31: Delta NLL 9.3474 After layer 4: Delta NLL 2.8977 After layer 9: Delta NLL 2.3226 After layer 14: Delta NLL 2.8945 After layer 19: Delta NLL 4.4791 After layer 23: Delta NLL 5.2255 After layer 27: Delta NLL 5.1092 After layer 31: Delta NLL 4.6548
The one-boundary response is flatter in Base, while the complete seven-boundary path again moves further.

See the text itself. The metrics show the size of the shift. The companion report pairs dense references with recorded PRF and SVD generations at lighter and deeper compression.

View recorded outputs