Zero Entropy

Ask two researchers whether a language model reasons and both will reach for entropy.

One shows you that the answer's logit is already saturated at the first step of the chain of thought, that per-token entropy is flat, that permuting the premises changes nothing, and concludes retrieval. The other shows you a model RL-trained to a near-delta output distribution, solving problems it could not solve before, and concludes competence.

They are reading opposite verdicts off the same scalar. The scalar is not the one that decides.

What decides is a match: whether the model's distribution over answers is the one the problem licenses. Reasoning is not $H \to 0$ , and it is not $H_{θ} = H_{problem}$ either: equal entropies are a coincidence of scalars, and two distributions can share one while disagreeing about every answer. It is $p_{θ} = A^{⋆}$ , and the two ways of failing it need two different instruments, neither of which is an entropy. This note derives that quantity, shows which experimental designs can and cannot detect it, and shows why nearly every deployed safety method is a reweighting rather than a deletion.

What this argues. Two things, and they are smaller than the apparatus around them. First, that reasoning claims should be read off the slope of $lo g P$ against serial depth under an explicit per-step error model, not off accuracy. Second, that the target for a model's uncertainty is not a level or a gap but a pair: how much licensed mass it starves, and how much unlicensed mass it invents, which forces the construction of evaluation sets that are underdetermined on purpose, and which nobody has built. Everything else here is scaffolding for those two, or a consequence of them on the safety side. None of it has been measured; the last section says which experiment is cheapest.

The tape

A decoder-only transformer of fixed depth $L$ and log precision spends exactly $L$ layers per token, regardless of the difficulty of that token. Its single-pass expressivity sits inside uniform $TC^{0}$ .[1] No amount of width buys serial depth.

Chain-of-thought is what buys it. With $T (n)$ generated tokens the model simulates a Turing machine running in time $T (n)$ , and polynomially many steps, under a generalized pre-norm, give exactly $P$ .[2] The intermediate tokens are not narration. They are the tape. Whether a given model in fact uses it as one is a separate, empirical question, and measured faithfulness is partial, task-dependent, and not monotone in scale: on several tasks it is worse for larger models.[5][6]

This settles the naive version of the retrieval argument. If CoT were decoration, adding tokens could not raise the complexity class. It provably does.

But note what the tape is made of. The circuit result assumes precision growing as $O (lo g n)$ , which is a scaling hypothesis rather than a description of any particular float format, and it bounds worst-case expressivity rather than the geometry of typical inputs. The counting argument below is separate and needs only that precision is finite at all. The residual stream at layer $L$ carries $d \cdot b$ bits: for $d = 8192$ at bf16, about $1.3 \times 1 0^{5}$ . The model then commits that state to one symbol from $V$ :

lo g_{2} ∣ V ∣ \approx 17 bits, \frac{d \cdot b}{lo g _{2} ∣ V ∣} \approx 7.7 \times 1 0^{3}

The ratio is illustrative and nothing below rests on it: bf16 mantissa bits are a capacity ceiling, not task-relevant information, and no one believes the residual stream carries $1 0^{5}$ bits of signal.

The bandwidth framing is the wrong one anyway. It suggests the pre-rounding state is lost, and in a transformer it is not: position $t$ 's residual stream is cached at every layer and later positions attend to all of it. Nothing is erased. What a step does is commit: the trajectory branches on the sampled token, the branch not taken is never computed, and no later position can condition on the counterfactual. A wrong commitment stays wrong whether or not the state behind it is still visible, and this holds at any vocabulary size: the 17 bits sets how fine the branching is, not whether it happens.

That irreversibility is the only thing used below. It does not contradict the expressivity result and is not offered as a limit: expressivity says the tape suffices given enough steps, and irreversible commitment says each step is a place the derivation can go wrong. It motivates a per-step error rate, nothing more.

The exponent

Throughout, $T$ is the number of generated tokens, one variable, not three. A problem family has a required depth, which is the parameter an experiment sweeps; $T$ is what the model spends. Let $ε_{t}$ be the probability that step $t$ of a $T$ -step derivation is wrong. Then

lo g P (correct) = t = 1 \sum T lo g (1 - ε_{t}) \approx - ε ˉ T

with $ε ˉ$ the mean rate over the derivation. Linear decay in the number of committed steps. Nothing here needs $ε_{t}$ to be stationary: the chain is Markov in the token prefix, not time-homogeneous, and if $ε_{t}$ grows with depth the sum is concave and $lo g P$ decays faster than linearly.

Stationarity is not the assumption worth attacking. Independence is. The factorization requires each step's correctness to be conditionally independent of the earlier ones given the prefix, and real traces violate this in both directions. An early error poisons what follows, correlating failures positively and steepening the decay. Attention also lets step $t$ re-read and quietly repair steps $1.. t - 1$ , correlating them negatively and flattening it. The second matters more than it looks: "no error correction" is the wrong null for a transformer, because the architecture supplies an implicit checking channel whether or not the text mentions one. A step verified with detection probability $d$ has effective rate $ε ˉ (1 - d)$ , and under complete verification $lo g P$ flattens. So a fitted slope identifies the effective rate, with $ε ˉ$ and $d$ not separately recoverable from it.

This still yields a diagnostic that the reasoning literature largely skips:

δ (T) = lo g P (correct ∣ T + 1) - lo g P (correct ∣ T)

$T$ is a token count, so this is a finite difference and not a derivative; in practice one fits $lo g P$ against $T$ and reads the slope of the fit. Constant and negative means an effective error rate that does not fall with depth; tending to zero means one that does. That is narrower than "retrieval versus algorithm," and deliberately so: a correct algorithm executed without noise has $ε ˉ = 0$ , needs no checking, and plateaus too. The slope separates error-correcting from non-error-correcting execution. It is still an observable, which accuracy on a named benchmark is not.

Mechanism	Signature
Serial execution, errors uncorrected	$lo g P$ falls linearly or faster in $T$ , slope $\leq lo g (1 - ε ˉ)$
Serial execution, errors corrected or absent	$lo g P$ plateaus, slope $\to 0$
No serial dependence	$lo g P$ flat in $T$ until required depth exceeds circuit depth, then a cliff

The rows are mechanisms, not verdicts. "Retrieval" does not name a row: a lookup model has no reason to show $T$ -dependence at all, which is the third row, while the first describes noisy serial execution, which a retrieval story predicts neither way.

It is worth being blunt about how much a plateau buys. It does not establish an algorithm: a correct noiseless procedure plateaus, an error-correcting noisy one plateaus, and the two are not separated by the slope. What a plateau rules out is exactly one thing: uncorrected serial error accumulation with an effective rate bounded away from zero. That is a narrow claim. It is also the claim the "LLMs don't reason" literature is implicitly making and never measuring, which is why it is worth the trouble.

The regimes are not exhaustive, and reading the slope needs an identification strategy that most evaluations lack. $T$ cannot just be measured across a benchmark: problems requiring more serial depth are harder along other axes too, so the fit confounds depth with difficulty. And if $T$ is the length the model chose to emit, it is endogenous: selected by the very process under test. The measurement is clean only on a family with a tunable depth parameter and otherwise fixed structure ( $k$ -hop lookup, $n$ -digit multiplication, $n$ -step symbolic rewriting), or under a forced token budget on a fixed problem set. Most benchmarks vary $T$ neither way.

What the objective can and cannot produce

Pretraining minimizes $E_{D} [- lo g p_{θ} (y ∣ x)]$ . The unique minimizer is $p_{θ} = p_{D}$ .

The perfect language model therefore inherits the conditional entropy of human text exactly, including its error modes. A perfect model is not a correct model. Correctness is a correlate of plausibility under $D$ , never the target.

The usual next sentence (that likelihood training cannot remove entropy $D$ contains) is too strong. $D$ is a mixture over authors of wildly varying competence, and conditioning selects components: a prompt that makes the careful component likely lowers $H (y ∣ x)$ without touching the marginal. That is most of what prompting is, and it is why a model can beat the average of its corpus. What conditioning cannot do is put mass where no component has any, or make correctness rather than plausibility the thing being scored. The entropy that survives every conditioning is the part the licensed component itself contributes, and no amount of likelihood training reaches it.

The only operator that removes it is reinforcement against a verifier. Under a KL constraint to the reference policy,

p_{β} (y ∣ x) \propto p_{ref} (y ∣ x) exp (r (x, y) / β)

and as $β \to 0$ this converges to a point mass on $ar g max_{y} r$ . The limit is there to fix the direction of the operator, not to describe a training run. Nobody trains at that limit. Production RL carries entropy bonuses, KL penalties and early stopping, all of which attenuate the effect.

The claim that the tilt reduces entropy at any finite $β$ would be convenient and is false. Exponential tilting is not entropy-monotone: with $p_{ref} = (0.9, 0.1)$ , $r = (0, 1)$ and $β = 1$ , the tilted distribution is $(0.77, 0.23)$ and entropy has gone up, because the reward pulled against an already-sharp reference. What is unconditional is only the limit: as $β \to 0$ the tilt concentrates on $ar g max r$ and entropy goes to zero. In between, entropy falls when $p_{ref}$ is broad relative to the reward's accept set and can rise when it is not. The RLVR and RLHF regimes are the first case (a broad reference, a narrow accept set), which is why entropy collapse has to be actively fought with bonuses there. But this is a statement about the regime, not a property of the operator, and the alignment section below turns on that distinction.

Within the regime, this is why RLVR works and why it breaks what it breaks: at large $k$ , pass@k for the trained model can fall below the base model's, sampling diversity dies, and the exploration that finds unseen proofs narrows with it.[10] How large and how general that effect is remains contested and depends heavily on the range of $k$ measured, itself an argument for reporting the curve rather than a number.

The condition, stated correctly

Let $A^{⋆} (\cdot ∣ x)$ be the answer distribution the problem licenses. The honest definition is that it is stipulated by whoever built the instance, not discovered in it: a posterior over worlds consistent with $x$ needs a prior over worlds, and outside a uniform finite set there is no canonical one: least of all in the underdetermined cases this essay is about, where the contested prior is the disagreement. For a well-posed arithmetic problem $A^{⋆}$ is a point mass and the stipulation is uncontroversial. For "which of these two candidates wins" it is a choice. That is not a technicality; it is why the program below lives on constructed sets, where the generator fixes $A^{⋆}$ , and not in the wild, where it would have to be argued for.

Everything here is over meanings, not strings. Sampling a model repeatedly produces enormous surface variation at zero semantic variation, so a raw token-level $H_{θ}$ is dominated by paraphrase and measures the wrong thing entirely. The output space has to be quotiented by semantic equivalence first; semantic entropy[8] is the standard construction and is assumed throughout, with the caveat recorded below that the quotient is not as clean as the notation pretends.

The caveat is worth stating precisely, because it decides what a reported $C$ or $E$ even ranges over. NLI-derived equivalence is not transitive ( $a_{1} \equiv a_{2}$ and $a_{2} \equiv a_{3}$ do not give $a_{1} \equiv a_{3}$ ), so taking connected components of the entailment graph chains non-equivalent answers into one cluster and silently deflates both instruments. The partition has to be built to resist that. Over sampled terminal answers ${a_{1}, \dots, a_{n}}$ , form the undirected graph on symmetrized entailment,

w_{ij} = min (P_{NLI} (a_{i} \Rightarrow a_{j}), P_{NLI} (a_{j} \Rightarrow a_{i}))

keep the edges $E_{τ} = {(a_{i}, a_{j}) : w_{ij} \geq τ}$ at a strict $τ$ (0.8 upward), and take the classes $Y$ to be maximal cliques of that graph rather than its components: every member of a class entails every other, which is what the notation was claiming all along. Nodes that clique into more than one, or into none, get assigned to the class whose centroid maximizes $cos (e (a_{i}), e (y_{k}))$ under a sentence embedding. This is a stipulation, not a discovery: the classes depend on $τ$ , and a result that does not report its sensitivity to $τ$ has not reported the quantity it names.

The tempting comparison is the entropy gap $H_{θ} - H (A^{⋆})$ , and it fails for the same reason the raw level did. Entropy is invariant to which outcomes carry the mass, so a model concentrated on the wrong three answers with the right spread scores a gap of zero. Worse, $H_{θ}$ is set by decoding temperature, a knob outside the model: any system can be tuned to any entropy level, and therefore to a zero gap, without a single parameter changing. A criterion a thermostat can satisfy is not a criterion.

Both instruments below are therefore defined on the model's raw next-token distribution at $T = 1$ , and a reported $C$ or $E$ that does not name its decoder means nothing. This does not make the thermostat objection vanish, and it should not be pretended away: $C$ and $E$ are properties of a (model, decoder) pair, and temperature moves them. What changes is that it no longer moves them anywhere. Any entropy level, and so any entropy gap including zero, is reachable by tuning $T$ on any model whatsoever. No $T$ sends both $C$ and $E$ to zero unless the model was already right, because the family is one-dimensional and the target is not on it. That is the difference between a knob that can fake a criterion and a knob that can only trade within one.

So compare the distributions. The obvious way (one KL in each direction, invention and collapse) is the first thing anyone reaches for, and it does not survive the rest of this essay. $D_{KL} (p_{θ} ∥ A^{⋆})$ is infinite wherever $p_{θ}$ puts mass that $A^{⋆}$ excludes, and by the full-support lemma of §Safety it always does. On a well-posed problem, where $A^{⋆}$ is a point mass, that divergence is $+ \infty$ for every model, on every input, always. It is not a measure of invention. It is the lemma wearing a different hat.

The two failures therefore need two different instruments.

Collapse, ruling out answers the problem permits, is the forward KL, and full support is exactly what makes it finite:

C (x) = D_{KL} (A^{⋆} ∥ p_{θ}) = E_{A^{⋆}} [- lo g p_{θ}] - H (A^{⋆})

the excess of cross-entropy over the problem's own entropy. On a determinate problem it collapses to the negative log-likelihood of the answer. It is large exactly when $p_{θ}$ starves something $A^{⋆}$ licenses.

That logarithm is taken of an estimate, and the plug-in version of it is not usable as written. With $n$ sampled traces per instance, $E [lo g p^] \leq lo g E [p^]$ biases $C^$ upward, and the acute case is a licensed answer that fails to appear at all: $p^_{θ} (a ∣ x) = 0$ sends the empirical cross-entropy to $+ \infty$ on a model that is merely thin there. The estimator has to be smoothed to be finite. A Krichevsky–Trofimov count over semantic classes does it:

p^_{KT} (a ∣ x) = \frac{N _{a} + 1/2}{n + ∣ Y _{obs} ∣ /2}

with $N_{a}$ the number of traces resolving to class $a$ and $∣ Y_{obs} ∣$ the number of classes observed. The budget then follows the rarest thing the problem licenses: to hold $∣ C (x) - C^_{KT} (x)∣ \leq ϵ$ with probability at least $1 - η$ takes

n = Ω (\frac{1}{min _{a \in supp A^{⋆}} A ^{⋆} ( a ∣ x )} lo g \frac{∣ Y _{obs} ∣}{η})

( $η$ is the failure probability, unrelated to the slope $δ$ of §The exponent.) An evaluation set that licenses a one-in-fifty answer is committing whoever measures $C$ on it to hundreds of full traces per instance, which is the real cost of the underdetermined items and an argument for stipulating $A^{⋆}$ close to uniform when building them.

Invention, licensing answers the problem excludes, is a mass, not a divergence:

E (x) = p_{θ} (y \in / supp A^{⋆} (\cdot ∣ x)) = 1 - y \in supp A^{⋆} (\cdot ∣ x) \sum p_{θ} (y ∣ x)

with $p_{θ} (y ∣ x)$ the model's distribution over semantic equivalence classes $y \in Y$ : the probability the model puts outside the licensed set. Bounded in $[0, 1]$ , estimated by sampling and counting, and, since full support puts positive mass on each excluded answer, strictly positive for every model that exists, so $E \to 0$ is a direction and not an arrival. That is arithmetic rather than a deep connection to §Safety, and is worth one clause, not more.

Both instruments live on the terminal answer space, not on the trace space. A long-CoT sample is a pair $(a, z)$ of an answer and an intermediate trajectory, and $C$ and $E$ are evaluated on the marginal

p_{θ} (a ∣ x) = z \in Z \sum p_{θ} (a, z ∣ x)

with $z$ summed out. This is not a convenience. Clustering full trajectories rather than answers makes the estimator far noisier (the trace space is where the paraphrase variation actually lives), and the non-transitivity of NLI equivalence turns every extra clustered object into another place the partition can shatter. Marginalizing first confines that damage to the answers, which are short and where the licensing claim was made in the first place.

Where the licensed set itself has no clean boundary (open-ended tasks, semantic ambiguity, $supp A^{⋆}$ ill-defined rather than merely hard to estimate), the hard partition can be dropped for a transport cost under an embedding distance $d_{sem} (a, a^{'}) \in [0, 1]$ :

W_{1} (A^{⋆}, p_{θ}) = γ \in Π (A^{⋆}, p_{θ}) in f E_{(a, a^{'}) \sim γ} [d_{sem} (a, a^{'})]

which degrades gracefully where $E$ would be reporting the threshold rather than the model. It buys robustness and gives up the thing $E$ was for: a mass outside a stipulated set is a claim about licensing, a transport cost is a claim about distance, and the two should not be reported under one name.

On a determinate problem the two carry the same information, and it is worth saying so, because it says what the apparatus is for. With $A^{⋆}$ a point mass on $a$ , $E = 1 - p_{θ} (a)$ and $C = - lo g p_{θ} (a)$ , so $C = - lo g (1 - E)$ : one is a monotone reparameterization of the other, and reporting both is reporting one. They separate only when $supp A^{⋆}$ has more than one element: when the model can be wrong about which licensed answers it keeps, not merely about how much mass it lands on the single right one. Most existing benchmarks are determinate, so on them this construction collapses to a calibrated accuracy number, correctly so. Its content is entirely in the underdetermined case, which is an argument for building those evaluation sets rather than a defect in the instruments.

$C = 0$ already implies $p_{θ} = A^{⋆}$ , so $C$ alone characterizes the target and $E$ is not a second condition. $E$ earns its place as a diagnostic: $C$ penalizes invention only indirectly, through the mass it steals, so a bad $C$ has two very different causes and reporting it alone throws away which one occurred. Anyone who wants one bounded number instead should use total variation or Jensen–Shannon, both finite here; what they give up is the direction of the failure, which is the part worth knowing.

This is the two-sided reading the entropy gap was reaching for, on instruments that survive the lemma. The gap was both of these projected onto one scalar, and the projection is where the counterexamples live. Note also what temperature does now. On an underdetermined input it genuinely trades the two: cooling concentrates on one licensed answer, cutting $E$ and inflating $C$ ; heating does the reverse. On a determinate input it moves both together, which is the sense in which sharpening really is just good there. A single global temperature cannot be right precisely because it must be one setting for both cases, and that is the whole argument for shaping over levels in miniature.

That trade is worth writing down, because it is what makes the decoder part of the reported quantity rather than a detail of the harness. A sampler with temperature $T > 0$ and nucleus cutoff $p_{n} \in (0, 1]$ turns logits $z$ into

p_{θ, T, p_{n}} (v) = ⎩ ⎨ ⎧ \frac{exp ( z _{v} / T )}{\sum _{u \in V_{p_{n}}} exp ( z _{u} / T )} 0 v \in V_{p_{n}} otherwise

with $V_{p_{n}}$ the smallest subset of $V$ whose softmax mass at temperature $T$ reaches $p_{n}$ . On a determinate input the two knobs move $C$ and $E$ the same way: cooling or truncating concentrates on the argmax, $E \to 0$ and $C \to 0$ together, and there is nothing to trade. On an underdetermined input they cannot both go down. Cooling starves every licensed answer but one:

T \to 0 lim E (x) = 0 and T \to 0 lim C (x) = \infty whenever ∣ supp A^{⋆} ∣ > 1

The surviving mass sits on a single licensed answer, and every other element of $supp A^{⋆}$ contributes an $A^{⋆} (a ∣ x) lo g (1/ p_{θ} (a ∣ x))$ term that grows without bound: immediately, at any $T$ , if nucleus truncation has already cut that answer out of $V_{p_{n}}$ . Warming does the reverse, spreading mass back over the starved licensed answers and buying a lower $C$ by raising the mass outside $supp A^{⋆}$ . What $(T, p_{n})$ sweeps out is a frontier in the $(C, E)$ plane, and where a system sits on it is a deployment choice; how far in the frontier lies is the property of the model. Reporting one point without the sweep reports the choice and not the model.

A widely held intuition says that a system resolving its uncertainty to zero is approaching general intelligence. The intuition points at something real and names the wrong coordinate. Greedy decoding already gives $H \approx 0$ ; RLVR gives $H \to 0$ . Neither is intelligence, because both act on the level uniformly: they buy reductions in $E$ everywhere and pay in $C$ on every input that is genuinely underdetermined, since starving licensed answers is the only way left to sharpen a distribution that was already right.

With $S (x)$ the causal structure of the instance, one can also ask that

I (trace; S) = H (S) and p_{θ} (y ∣ x, S) = A^{⋆} (y ∣ x, S)

The second clause is the condition above, conditioned on structure, and it is deliberately not $H_{θ} (y ∣ x, S) = 0$ : an underdetermined problem stays underdetermined once its structure is known.

The first clause needs an ensemble to mean anything: the mutual information is over instances drawn from a specified problem family, with $S$ the structure that varies across it, and it is undefined for a single prompt. Stated that way it is a strictly stronger and different demand than the second, and presenting them as one claim would be a mistake: it contradicts the faithfulness results cited at the top of this essay. A model that computes $S$ internally and emits a shallow or post-hoc trace violates it while being perfectly correct, and that combination is not hypothetical; it is the modal finding of the CoT faithfulness literature.[5][6] So this is a condition on verbalized reasoning, not on reasoning, and conflating the two is how interpretability requirements get smuggled in as capability claims.

It is also not the condition auditing wants, and the two get used interchangeably. $I (trace; S) = H (S)$ is completeness: the trace determines the structure. What an auditor needs first is soundness: no step in the trace is false. A complete trace can be padded with falsehoods, and a sound trace can omit half the derivation and still be safe to read. Completeness is what verification wants, in a trace you can re-execute. Soundness is what auditing wants. The condition here is the former, and the former is what the faithfulness literature measures the absence of.

Neither $C$ nor $E$ is a training loss, because $A^{⋆}$ is unavailable for an arbitrary prompt. That is a weaker objection than it sounds (cross-entropy is minimized against a $p_{D}$ nobody observes, and every calibration result in the field estimates a relationship to an unobservable conditional), but it is real, and it constrains where measurement can happen:

By construction. Sets whose licensed answer distribution is fixed by the generator: a known number of consistent completions, ambiguity introduced deliberately and in known amount.
Differentially. Across an intervention that leaves $A^{⋆}$ fixed, where the unknown term cancels and the change is identified even though the level is not.
By decomposition. Where $A^{⋆}$ is unavailable entirely, the model's own uncertainty still splits without it, via $H_{θ} (y ∣ x) = I (y; θ ∣ x) + E_{θ} [H (y ∣ x, θ)]$ , over an ensemble or a posterior.[9] The first term is the model's ignorance and is the part that ought to vanish; the second is its estimate of the problem's spread. Not a substitute for $A^{⋆}$ (the estimate can be wrong in exactly the way the model is wrong), but measurable in the wild, which $C$ and $E$ are not, and the workhorse this section should be read as extending rather than displacing.

What it costs to actually measure these. "Report $C$ and $E$ " is worth nothing without a sample budget, and the two are not equally cheap. $E$ is a binomial proportion over semantic classes: a few hundred samples give usable intervals, and the estimator is unbiased. $C$ is worse, for the reasons given with its definition: it takes a logarithm of an estimate, it needs smoothing and a stated $n$ to be finite at all, and its budget is set by the rarest licensed answer rather than by the model. The direction of its bias is at least the safe one: an unconverged $C$ overstates collapse, so a small measured $C$ can be trusted in a way a small measured $E$ cannot. And $p_{θ} (y ∣ x)$ over semantic classes is a marginal over traces, so on a long-CoT model every sample is a full trace rather than an answer, multiplying that already-awkward budget by the cost of generation. The clique partition above keeps the quotient honest but does not make it free: it is another threshold to sweep, on top of the sample budget. None of this makes the measurement impossible. It does mean the honest status of this section is a proposal with a known cost, not a method with a track record.

Annotator disagreement is the tempting fourth route and it is not one. Disagreement mixes the aleatoric content of the item with the epistemic state of the annotators: ten people disagreeing because the problem is underdetermined is indistinguishable, in the label distribution alone, from ten disagreeing because they are bad at it. That is the same conflation this section exists to undo, one level up. Disagreement bounds $H (A^{⋆})$ from above only once annotator competence has been established separately, which is a harder measurement than the one it was meant to supply.

The claim is not that entropy should vanish. It is that entropy should become entirely aleatoric. A system whose every remaining bit of uncertainty is a property of the world rather than of itself is the thing worth calling general.

The interference is in the weights, not the objective

This is the part with teeth, and the teeth are not where they look.

RLHF for harmlessness and RLVR for mathematics are the same map. Both are $p_{ref} exp (r / β)$ , differing only in $r$ . Both concentrate mass on a verifier's accept set.

The entropy claim has to be made in regime terms. Tilting is not entropy-monotone, so "both reduce $H$ " is not a property of the operator. Both do run in one regime (a broad pretrained reference, an accept set narrow inside it), and there the tilt reduces entropy, with $β \to 0$ annihilating it. What that scoping does not buy is monotonicity in the KL moved. In the counterexample above, $D_{KL} (p_{β} ∥ p_{ref})$ rises all the way to the limit while entropy first rises and only then falls, peaking around $β \approx 0.4$ . Monotonicity needs more than a qualitative regime: it needs the reference to be roughly uniform across the accept/reject split, so that tilting cannot first flatten before it sharpens. Absent that condition, the honest statement is that the reduction grows with the KL moved once sharpening has begun, which is all the prediction below requires.

Now the argument everyone expects from here, which is that a shared functional form implies a shared entropy budget and the alignment tax is that budget being spent. It does not go through, and the closed form is what refutes it. If $r (x, \cdot)$ is near-constant in $y$ for $x$ off the safety distribution, the tilt cancels in normalization and $p_{β} = p_{ref}$ exactly there. Read literally, the operator contains no interference term at all: no off-distribution entropy change, no pass@k cost, no tax.

The temptation is to turn that into a live prediction and an experiment that tests it. It is not one, for two reasons worth stating rather than discovering later. The closed form is the optimum of the KL-regularized objective, and no training run reaches it: what happens is finitely many gradient steps on shared parameters, so a prediction from $p_{β}$ is a prediction about an object nobody believes exists. And $r$ is a learned reward model, not the idealized harmlessness signal; reward models are conspicuously not near-constant off their training distribution, which is one of the mechanisms named below. The antecedent fails exactly where the prediction would have needed it. Nobody expects the null, so nothing is learned by rejecting it.

What the closed form is good for is a baseline, and that is a smaller and more defensible role. The objective at its own optimum has no coupling in it. Every bit of observed tax therefore has to come from something the idealization omits: finite optimization on shared parameters, capacity contention, and reward generalization past the distribution $r$ was fit on. Those are not caveats on the entropy-budget story. They are the whole story, and the budget framing was hiding them behind an operator that has nothing to do with it.

So the measurement is not a test of the tilt. It is an attempt to size the gap between the idealization and what training does, which is a quantity nobody has:

At matched compute, measure $H_{θ}$ on neutral held-out prompts and pass@k on unrelated hard tasks, before and after safety training, and report the effect as a function of KL moved. The interesting output is not the sign (the sign is expected) but the magnitude, which is a direct measurement of how far a reward model's influence extends past its own training distribution.

The corollary survives the reframing and is worth keeping: within the sharpening regime, a maximally sharpened model is maximally collapsed. Safety-by-sharpening and capability-by-sharpening produce the same collapse whether or not they compete for anything, and neither is the target, which is $p_{θ} \to A^{⋆}$ : entropy removed only where the problem is determined, and preserved exactly where it is not.

The covariance test

One experimental design recurs across the "LLMs don't reason" literature: permute the premises, observe that the answer is unchanged, conclude pattern matching.

The inference does not go through. Invariance to reordering independent premises is precisely what a correct reasoner exhibits. The design measures a property both hypotheses predict, and so has no discriminating power.

Let $φ$ be semantics-preserving and $ψ$ semantics-breaking. A reasoner satisfies both:

A (φ x) = A (x) and A (ψ x) \neq = A (x)

$φ$ (must not change the answer)	$ψ$ (must change it)
rename entities	reverse a dependency
reorder independent premises	reorder dependent premises
swap lexicon, keep structure	negate one premise
change numerals, keep relations	delete a required premise

A bag-of-keywords model is invariant to both. A surface-brittle model is sensitive to both. Only the joint test separates them, and results reporting a single side, in either direction, are not evidence about reasoning.

Safety

Lemma (full support). For finite logits $z$ , $softmax (z)_{i} > 0$ for all $i$ . Autoregressively, $p_{θ} (y ∣ x) = \prod_{t} p_{θ} (y_{t} ∣ x, y_{< t}) > 0$ for every string $y$ . $■$

Corollary. For every harmful target $h$ and every context $x$ , $p_{θ} (h ∣ x) > 0$ . Alignment implemented inside the distribution can make $h$ improbable. It cannot make $h$ impossible.

Two scope conditions, because on its own the lemma is close to trivial and proves less than its placement suggests. It is a statement about the model's distribution, not the deployed sampler: top- $k$ and nucleus decoding truncate support and do set exact zeros: not a counterexample but the first instance of the escape named below, a constraint imposed from outside $p$ . And positive probability is not the operative quantity, because the adversary picks $x$ ; what matters is $sup_{x} p_{θ} (h ∣ x)$ over reachable contexts, and whether that supremum is large enough to find by search. The stronger result here is Wolf et al.:[3] for any behaviour the model can express at all, prompts eliciting it exist, with required length governed by how far the aligned distribution sits from the unaligned one. It is a real strengthening, and not free: it models behaviour as a mixture of personas the prompt selects among, and permits prompts of unbounded length. Granted both, the conclusion is close to definitional: an adversary with unlimited context and a persona decomposition can always reach the component. What it establishes is the shape of the dependence, not the existence of the attack.

Every method acting by reweighting logits (refusal training, RLHF, preference optimization, unlearning that lowers likelihood) moves mass on a simplex whose support is fixed. The substrate survives. Two much-discussed empirical results are usually cited together here:

Unlearning that reduces $p$ while preserving the feature is reversed by fine-tuning on $O (1 0^{2})$ examples.
Poisoning requires a near-constant absolute number of samples, not a constant fraction.[7]

They are not the same result. The first is direct evidence that a down-weighted representation survives and can be weighted back up. The second concerns inserting a narrow backdoor during training rather than recovering a suppressed behaviour, and supports a correspondingly narrow claim: for that class of behaviour, at the scales that study reached, the sample count needed did not grow with the corpus it competed against.

Read jointly they would say that moving mass is cheap in both directions and does not get more expensive with scale: a claim over behaviour classes and scales neither paper examined, and a conjecture rather than a finding. Only the first bears directly on the argument here. The second is suggestive company.

What this licenses and what it does not. Representation rerouting,[4] destroying the map $x \mapsto h_{ℓ}$ rather than the logit, is a genuine improvement, because it lowers $p (h ∣ x)$ across the whole preimage of the destroyed feature rather than for the sampled $x$ alone. The difference from reweighting is structural. RLHF, DPO and preference tuning move the readout (the unembedding and late-layer biases) while the activation submanifold $h_{ℓ} \in R^{d}$ that encodes the concept is left intact, which is exactly why $O (1 0^{2})$ fine-tuning examples or a prompt that steers into the same region can re-point the readout at a representation that never went anywhere. Representation editing (direction ablation, subspace nulling $h_{ℓ} \leftarrow (I - P_{harm}) h_{ℓ}$ , SAE feature clamping) changes the circuit that computes the thing rather than the layer that reads it out. Reweighting controls expression on a simplex whose support is fixed; editing attacks the representation the behaviour needs in the first place.

It is still not an escape from the lemma. The output layer remains a softmax, support remains full, and "the capacity to execute $h$ is driven to zero" is a claim about the edited directions, not about the network: what is measured in practice is a much wider basin and a much higher recovery cost, not an impossibility, and ablation can be routed around by paths the projection missed for the same reason a probe can. The honest framing is basin width.

Hard guarantees require leaving the distribution, and only one of the two usual escapes survives the semantic commitment made earlier. Constrained decoding does zero mass, but it zeroes it over strings: a token-level constraint can forbid a literal, an API call, a specific completion. It cannot enumerate a harmful meaning, and the harmful thing was defined here as a semantic class. So constrained decoding is a real guarantee about a narrow, string-shaped target and not about the object this essay has been discussing. What is left is the external check on the completed output, the only mechanism that can be as semantic as the thing it checks for. Anything expressed as a preference over $p$ is soft by construction.

Evaluation is a biased estimator. Safety work measures $m = E [b ∣ eval]$ and wants $τ = E [b ∣ deploy]$ . If the model conditions on a detector $e$ ,

τ - m = (E [b ∣ e = 0] - E [b ∣ e = 1]) \cdot (P (e = 1 ∣ eval) - P (e = 1 ∣ deploy))

The identity holds when $e$ is a sufficient statistic for the context's influence on $b$ : when behaviour depends on where the model is only through what it detected. That is an assumption, not a derivation, and it is the load-bearing one. Granting it, the conjecture is that both factors grow with capability: a stronger model separates the contexts better, and has more to gain from conditioning on the separation. Neither growth is established here, and the second presumes something about what the model optimizes that would itself need showing. If both do grow, the bias is a product of two increasing terms, evaluations are valid only while detection accuracy stays low, and that window is closing.

Probes measure correlation until they are made to measure mediation. A direction $w$ that classifies deceptive states at high accuracy has shown nothing causal. The intervention is:

h^{'} = h - \frac{w ^{⊤} h}{∥ w ∥ ^{2}} w

If behavior is unchanged, $w$ does not mediate. This is necessary and still not sufficient: ablation can be routed around by parallel paths, so a null result is weak evidence, and a positive result establishes only that $w$ lies in the mediating subspace, not that it spans it. The generalization axis matters as much: a probe firing when a model discusses deception, but not when it merely deceives, has found language about the thing.

The agenda this implies

Five programs that follow directly, each falsifiable.

Measure the slope. Report accuracy against a swept problem-depth parameter, and $lo g P$ against the $T$ it induces, rather than against a benchmark name. Publish $δ (T)$ . Depth-swept curves are not new (the length-generalization literature has run exactly these families for years[11]), but they are reported as accuracy against length, which is not the same object. Fitting $lo g P$ and reading the slope against an explicit per-step error model is what turns those curves into a statement about error correction, and that is the part the reasoning debate is missing.

Calibrate against $A^{⋆}$ . Build evaluation sets with known aleatoric content (underdetermined by construction, licensed answer set stipulated by the generator) and report $C$ and $E$ separately, over semantic classes rather than strings, with the sample budget and clustering threshold stated. A single entropy number, or a gap between two, discards which answers moved, and that is where the information is.

Run the cheap one first. The essay's own thesis is testable on one GPU over a weekend, and nothing here has been. Build a constructed set with stipulated $A^{⋆}$ and genuine aleatoric content, take an open-weights base model and its RLVR-trained sibling, and report $C$ and $E$ for both. The prediction is that RLVR lowers $E$ and inflates $C$ on the underdetermined items: sharpening bought by starving licensed answers. It needs no safety-training run, no matched-compute protocol, and no unpublished access. Every other item here is more expensive and less direct.

Localize the interference. Track $H_{θ}$ on neutral held-out prompts and pass@k on unrelated hard tasks across safety training. The tilt predicts null; a non-null result puts the coupling in the weights and measures how far the reward model generalizes past its training distribution.

Run both sides. Any claim about reasoning should report the $φ$ and $ψ$ arms together. Single-arm results are unfalsifiable in the direction they are reported.

Target the map, not the mass. Given full support, effort spent lowering $p (h)$ has a known ceiling. Effort spent destroying the representation that $h$ requires does not, though it also does not reach zero, and the honest framing is basin width, not guarantee.

The pilot

Every instrument above is unmeasured, and the cheapest way to measure all of them at once is one synthetic family. A DAG reachability task gives a known serial depth and a stipulated licensed set on the same instance, which no natural benchmark does.

[source x] ── k-hop deterministic path ── [node u] ─┬─> [target a₁]  (licensed)
                                                    └─> [target a₂]  (licensed)

The path from $x$ to $u$ is unique and requires $k$ serial steps to verify; the branch at $u$ is where the licensed set is planted. Two parameters are swept independently, which is the point of the construction:

Parameter	Sweep	Controls
Depth $k$	$2, 4, 8, 16, 32$	required serial steps, against a generation budget $T$ forced by the harness
Branching $M$	$1, 2, 4, 8$	$∣ supp A^{⋆} ∣$ , with $A^{⋆} (a ∣ x) = 1/ M$ stipulated uniform

Every prompt is padded with neutral filler to a constant $L_{p}$ (2048 tokens). Without that, depth and context length move together and a decay read off the sweep is positional attenuation rather than serial error: the confound that makes most depth results uninterpretable. Uniform $A^{⋆}$ is also not cosmetic: by the budget above, the sample cost of $C$ is set by the rarest licensed answer, and $1/ M$ keeps it as cheap as an underdetermined item can be.

Two things come out of the same runs. Fit $δ (T) = lo g P (correct ∣ T + 1) - lo g P (correct ∣ T)$ across $T \in [1, 2 k]$ under forced budgets: a retrieval baseline gives a step function (failure while $T < k$ , flat after) while uncorrected serial execution gives a constant negative slope $δ (T) \approx - ε ˉ$ past the required depth, and the two shapes are distinguishable at small scale. Then, at $n = 500$ KT-smoothed traces per instance over $T \in [0.1, 1.5]$ and $p_{n} \in [0.5, 1.0]$ , plot $C$ against $E$ : the $M = 1$ instances should show the two moving together, and the $M > 1$ instances should trace the frontier. Until that plot exists, the frontier is an argument.

What would falsify this

A plateau in $lo g P$ on a depth-swept family where individually graded steps show a per-step error rate bounded away from zero. Verifier-free plateaus are predicted here, so a plateau alone falsifies nothing; what would break the multiplicative model is a measured $ε ˉ$ that the observed decay cannot be reconciled with.
A reweighting-only defence that no fine-tuning or prompt-search budget reverses. The lemma forbids exact zeros inside the distribution, so an exact zero is not the test; recoverability is, and recoverability is empirical.
Safety training that moves substantial KL and does reduce off-distribution entropy on prompts where the reward is near-constant. The tilt predicts no change there, so this would falsify the closed-form account of what safety training does, and force the coupling into the weights.
An RLVR variant that reduces $E$ without inflating $C$ on underdetermined held-out inputs: sharpening where the problem is determined, and leaving it alone where it is not.

The last is the one worth working on. Every deployed alignment technique is entropy suppression applied uniformly. The object we want is entropy shaping: certainty where the world is determined, uncertainty preserved exactly where it is not. Until then, alignment is a reweighting, and reweightings are reversible.

References

1. Merrill & Sabharwal, The Parallelism Tradeoff: Limitations of Log-Precision Transformers (2023).
2. Merrill & Sabharwal, The Expressive Power of Transformers with Chain of Thought (2024).
3. Wolf et al., Fundamental Limitations of Alignment in Large Language Models (2023).
4. Zou et al., Improving Alignment and Robustness with Circuit Breakers (2024).
5. Lanham et al., Measuring Faithfulness in Chain-of-Thought Reasoning (2023).
6. Turpin et al., Language Models Don't Always Say What They Think (2023).
7. Souly et al., Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples (2025).
8. Kuhn, Gal & Farquhar, Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023).
9. Depeweg et al., Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning (2018).
10. Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025).
11. Jelassi et al., Length Generalization in Arithmetic Transformers (2023).

#The tape

#The exponent

#What the objective can and cannot produce

#The condition, stated correctly

#The interference is in the weights, not the objective

#The covariance test

#Safety

#The agenda this implies

#The pilot

#What would falsify this

#References