Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Temperature scaling is the only linear scaling method that never changes a model's hard predictions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:05 UTC pith:RXLRANBJ

load-bearing objection Theorem 5.1 is a solid, checkable result, but Corollary 1 disappears for Dirichlet calibration — a concrete counterexample kills the advertised uniqueness claim. the 3 major comments →

arxiv 2602.14862 v2 pith:RXLRANBJ submitted 2026-02-16 stat.ML cs.AIcs.ITcs.LGmath.ITstat.ME

The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling

Pierre-Alexandre Mattei , Bruno Loureiro This is my paper
classification stat.ML cs.AIcs.ITcs.LGmath.ITstat.ME MSC 62B1060E15
keywords temperature scalingclassifier calibrationaccuracy-preserving scalersmajorisationentropy monotonicityinformation projectionautoregressive language modelslogit linear scalers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper gives a systematic theoretical account of temperature scaling, the one-parameter method that multiplies logits by a scalar before the softmax. It proves that for classifiers, raising temperature increases uncertainty in a very strong sense, so entropy and related dispersion measures move monotonically. Its main structural result is a uniqueness theorem: among all linear re-calibrations, temperature scaling is the only one that leaves the argmax, and therefore accuracy, untouched; any more flexible accuracy-preserving scaler must be nonlinear. The paper also gives a geometric interpretation, showing the tempered model is the relative-entropy projection of the original onto the set of distributions with a target entropy. A toy autoregressive counterexample shows these guarantees do not automatically transfer to language models when temperature is applied per decoding step.

Core claim

For classification, the central claim is a characterisation theorem. If a linear map z ↦ Wz+b preserves the set of maximal coordinates for every vector z, then W must be a positive multiple of the identity plus a rank-one matrix with constant columns, and b must be a constant vector. Adding the same constant to all coordinates leaves the softmax unchanged, so the only such map is multiplication by a single positive temperature. It follows that the logit-linear scaler family and the log-probability-linear scaler family have exactly one common submodel that never changes hard predictions: temperature scaling. The paper also proves that the entropy of the tempered distribution has derivative −β

What carries the argument

The proof rests on three elementary tools. The first is the log-partition function of the exponential-family form of the tempered softmax: its first derivative is the expected logit and its second derivative is the variance of the logit under the tempered model, so entropy and likelihood derivatives collapse to variance computations. The second is a linear-algebra characterisation of argmax-invariant matrices: evaluating the invariance condition on the test vectors e_i, e_i+e_j, e_i+e_j+e_k, and 1 forces every column of W to be constant off the diagonal and forces the diagonal entries to differ from those constants by a common positive amount, leaving W=βI+1α^T. The third is majorisation, a

Load-bearing premise

The practical conclusion that temperature scaling is the only accuracy-preserving linear scaler rests on the assumption that accuracy preservation must hold for every possible logit vector in R^K, not just for the logits a real classifier produces on a finite calibration set.

What would settle it

Take a trained classifier, record its logits on a calibration set, and solve for a linear map Wz+b that keeps the argmax unchanged on those empirical logits while also changing the probabilities in a way temperature scaling cannot; existence of such a map would falsify the practical reading of the uniqueness theorem. Alternatively, on a real autoregressive model, compute the entropy of the per-token-tempered distribution over a fixed corpus and check whether any pair of temperatures has entropy increasing with temperature.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any linear scaler that preserves accuracy must be temperature scaling; more flexible accuracy-preserving calibrators have to abandon linearity.
  • For a fixed classifier input, temperature is a guaranteed uncertainty dial: entropy and every dispersion measure that respects majorisation move monotonically as the inverse temperature changes.
  • Raising temperature spreads the marginal class proportions, which can shift probability mass toward minority classes in imbalanced problems.
  • Autoregressive language models using per-token temperature do not inherit the monotonicity guarantee, and sequence-level entropy can rise when temperature rises.
  • Temperature scaling solves a constrained projection problem: it is the closest distribution to the original among all distributions with a desired entropy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The uniqueness theorem is proved for all logit vectors in R^K, but in practice a scaler is only required to preserve predictions on the finite, low-dimensional set of logits a given classifier emits; whether other linear maps satisfy that weaker property is not addressed by the paper.
  • A natural empirical test would fit arbitrary linear maps to a calibration set with a hard constraint that no prediction changes; finding feasible maps outside the characterised family would show that data-restricted accuracy preservation is strictly weaker than temperature scaling.
  • The language-model counterexample suggests a broader heuristic: if early-token uncertainty strongly determines later-token entropy, temperature can become a nonmonotonic diversity control, so its effect should be measured on the target corpus rather than assumed.
  • The projection characterisation could be adapted to other closeness measures or other constraints, such as matching a target confidence quantile, yielding new calibration methods that retain temperature scaling's geometric meaning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies elementary properties of temperature scaling for classifiers and language models. Its contributions are: (i) monotonicity of the entropy and of majorisation in the inverse temperature for a fixed input (Proposition 2 and Theorem 3.1); (ii) a toy autoregressive example where the so-called myopic temperature scaling has non-monotone entropy; (iii) an information-projection interpretation of temperature scaling (Theorem 4.1); and (iv) a characterization of argmax-preserving affine maps, W = βI + 1α^T, b = γ1 (Theorem 5.1), with the advertised conclusion that temperature scaling is the only accuracy-preserving linear scaler among matrix scaling and Dirichlet calibration (Corollary 1). The first three items are largely sound, but the Corollary 1 claim for Dirichlet calibration is false, and the marginal class-proportion majorisation claim in Section 3.3 is also false.

Significance. If restricted to its valid parts, the paper contains useful and checkable results: Theorem 5.1 has a clean, elementary proof and is a genuine characterization of full-domain argmax-preserving affine maps; Proposition 2 gives a short statistical-physics proof of entropy monotonicity; and the LLM myopic-temperature counterexample is explicit and arithmetically sound. The information-projection theorem is correctly attributed to prior work. However, the paper's central advertised claim—that temperature scaling is the only accuracy-preserving linear scaler, including for Dirichlet calibration—is disproved by a simple counterexample. A second claimed result, about class proportions becoming more spread out at lower temperature, is also false. These are load-bearing errors, not presentation issues.

major comments (3)
  1. [Section 5, Corollary 1 and Appendix G] Corollary 1 is false for Dirichlet calibration. The map z ↦ W log Softmax(z) + b is not affine in z, so Theorem 5.1 cannot be applied; Appendix G's 'essentially the same' is not a proof. Counterexample: K=2, W=[[2,-1],[0,0]], b=(0,-ln2). For π=(p,1-p), the scaled logits are (2 ln p - ln(1-p), -ln2); the first exceeds the second iff 2p^2 > 1-p iff p>1/2, so argmax is preserved for every p. Yet the logit difference ln2+2ln p-ln(1-p) is not β(ln p-ln(1-p)) for any β, so the map is not temperature scaling. This disproves the 'respectively' claim and the abstract's blanket statement.
  2. [Section 3.3, around Eq. (20)] The claim that β1<β2 implies p_{β2}(y) majorises p_{β1}(y) is false. Pointwise majorisation does not pass through mixtures when the component means change. Example: P(X=x1)=P(X=x2)=1/2, logits z1=1 and z2=-2. At β1=0.5, p_{β1}(y=1) = (σ(0.5)+σ(-1))/2 = 0.4457; at β2=3, p_{β2}(y=1) = (σ(3)+σ(-6))/2 = 0.4775. Thus H(p_{β1}) ≈ 0.687 < H(p_{β2}) ≈ 0.692, and p_{β2} is majorised by p_{β1}, the opposite of the paper's claim. Jensen's inequality points in the wrong direction for the paper's conclusion here.
  3. [Theorem 5.1 / Corollary 1, domain of applicability] The uniqueness conclusion for matrix scaling requires argmax preservation for every z∈R^K. In practice a scaler is checked only on the logits emitted by a given classifier or on a finite calibration set, which lie on a low-dimensional subset of R^K. The proof uses test vectors such as e_i, e_i+e_j, 1, and 0 that a real classifier may never produce, so a linear map could preserve hard predictions on the restricted set without being of the form βI+1α^T. The theorem is a correct full-domain statement, but the practical reading 'temperature scaling is the only accuracy-preserving linear scaler' is an overclaim unless the full-domain assumption is made explicit and defended.
minor comments (5)
  1. [Section 3.2, after Definition 1] The sentence 'π≺ρ implies H(π)≤H(ρ)' has the wrong sign: with φ(t)=t log t, Σφ(π)=-H(π), so π≺ρ implies H(π)≥H(ρ). The same sign error affects the following sentence about Rényi entropy. The later conclusion that entropy decreases with β is nevertheless correct.
  2. [Section 3.4, Appendix E, Figure 3] The value of π is inconsistent: the main text and Figure 3 say π=0.51, while Appendix E says π=0.55. Please state the exact parameter values used to generate the figure.
  3. [Section 3.3] The sentence 'increasing the temperature will make the class proportions more spread out' is opposite to the paper's own conclusion that H(p_β(y)) decreases with β. Clarify whether 'temperature' means β or 1/β.
  4. [Section 4, Theorem 4.1] The theorem should state the allowable range of h* (for example h*∈[0, log K]) and explicitly allow the boundary case β*=0 when h* is the uniform entropy.
  5. [Discussion] Typo: 'counterintuive' should be 'counterintuitive'.

Circularity Check

0 steps flagged

No significant circularity: central results are derived from first principles or properly attributed external prior work; self-citations are contextual only.

full rationale

The paper's main derivation chain is self-contained. Proposition 2 follows from Lemma 1, proven in Appendix B by direct differentiation of log Z; Theorem 5.1 is proven in Appendix F using explicit test vectors and does not presuppose its conclusion; Theorem 3.1 is attributed to Marshall and Olkin and Dabah and Tirer; Theorem 4.1 is attributed to Sgarro and to Girardin and Regnault. The only self-citations (Clarté et al., with co-author Loureiro) are background context or an alternative calibration criterion, and they are not load-bearing for any main theorem. No parameter is fitted to data and then renamed as a prediction, and no result reduces by construction to its own input. One non-circular correctness concern: Appendix G says the Dirichlet-calibration case of Corollary 1 is 'essentially the same' as matrix scaling, but the Dirichlet map z -> W log Softmax(z) + b is not affine in z, so Theorem 5.1 does not directly apply. This is a proof/extension gap, not a circularity, and therefore does not affect the circularity score.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The paper's central claims rest on standard convex-analysis/majorisation facts, two externally attributed theorems (Marshall–Olkin majorisation of tempered vectors; Sgarro/Girardin–Regnault projection characterization), a false implicit assumption that pointwise majorisation survives mixing (Section 3.3), and the standard autoregressive factorization for LLMs. No invented entities; the only hand-chosen numbers are the toy-model Bernoulli parameters.

free parameters (1)
  • Toy-model Bernoulli parameters π, ρ0, ρ1 = π=0.51 (§3.4) or 0.55 (Appendix E); ρ0=0.01; ρ1=0.51
    Chosen by hand to exhibit non-monotonic sequence entropy in the myopic temperature scaling counter-example; not fitted to data, but the inconsistency between the two stated values of π is a reproducibility defect.
axioms (5)
  • standard math Convex-function characterization of majorisation: π≺ρ iff Σφ(π_k) ≤ Σφ(ρ_k) for all convex φ (Definition 1)
    Hardy–Littlewood–Pólya/Karamata equivalence; the paper's entropy implication drawn from it has the wrong sign (Section 3.2).
  • standard math Marshall–Olkin majorisation: for 0<β1<β2, π^{β1} ≺ π^{β2} (Theorem 3.1)
    Cited from Marshall & Olkin (1965) / Marshall et al. (2011, p. 189), not proven in the paper; load-bearing for §3.2–3.3.
  • standard math Information-projection solution: minimizer of KL(q||p) at fixed H(q)=h⋆ is a tempered p_{β⋆} (Theorem 4.1)
    Attributed to Sgarro (1978) and Girardin & Regnault (2016, Thm 1); assumes p unimodal with full support.
  • ad hoc to paper Pointwise majorisation is closed under marginalisation over x (used in §3.3)
    False. Majorisation is not inherited by mixtures: p_β(y)=∫p_β(y|x)p(x)dx can become more uniform as β increases, e.g. logits (ln9,0) and (0,ln4) with equal weights give marginal (0.55,0.45) at β=1 and (0.523,0.477) at β=2. The paper's 'direct consequence of Jensen and Theorem 3.1' is an erroneous load-bearing step for the class-proportion claim.
  • domain assumption Autoregressive factorization p(x1,…,xT)=p(x1)∏p(xt+1|xt) (Eq. 21) as the LLM model
    Standard next-token LLM model; load-bearing for the §3.4 counter-example and the definition of myopic temperature scaling.

pith-pipeline@v1.3.0-alltime-deepseek · 26937 in / 43661 out tokens · 357289 ms · 2026-08-02T23:05:10.182172+00:00 · methodology

0 comments
read the original abstract

Temperature scaling is a simple method that allows to control the uncertainty of probabilistic models. It is mostly used in two contexts: improving the calibration of classifiers and tuning the stochasticity of large language models (LLMs). In both cases, temperature scaling is the most popular method for the job. Despite its popularity, a rigorous theoretical analysis of the properties of temperature scaling has remained elusive. We investigate here some of these properties. For classification, we show that increasing the temperature increases the uncertainty in the model in a very general sense (and in particular increases its entropy). However, for LLMs, we challenge the common claim that increasing temperature increases diversity. Furthermore, we introduce two new characterisations of temperature scaling. The first one is geometric: the tempered model is shown to be the information projection of the original model onto the set of models with a given entropy. The second characterisation clarifies the role of temperature scaling as a submodel of more general linear scalers such as matrix scaling and Dirichlet calibration: we show that temperature scaling is the only linear scaler that does not change the hard predictions of the model.

Figures

Figures reproduced from arXiv: 2602.14862 by Bruno Loureiro, Pierre-Alexandre Mattei.

Figure 1
Figure 1. Figure 1: Entropies of two tempered versions of a toy language model. Standard temperature scaling has the [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Temperature scaling as an information projection. In both cases, the initial model [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Terms involved in the computation of the entropy of the myopically scaled model. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 9 linked inside Pith

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Gemini Team

    G. Gemini Team. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Pith/arXiv arXiv

  3. [8]

    A. S. Mozafari, H. S. Gomes, W. Leão, S. Janny, and C. Gagné. Attended temperature scaling: a practical approach for calibrating deep neural networks.arXiv preprint arXiv:1810.11586,

    Pith/arXiv arXiv

  4. [12]

    doi: 10.1109/ICRA40945.2020.9197266. C. Tomani, D. Cremers, and F. Buettner. Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. InEuropean conference on computer vision, pages 555–569. Springer,

    arXiv 2020

  5. [13]

    doi: 10.1038/s41592-019-0686-2. J. Xie, A. Chen, Y. Lee, E. Mitchell, and C. Finn. Calibrating language models with adaptive temperature scaling. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18128–18138,

    doi:10.1038/s41592-019-0686-2 2024

  6. [1952]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network.NIPS 2014 Deep Learning Workshop, arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv 2014

  7. [1978]

    M. Shen, S. Das, K. Greenewald, P. Sattigeri, G. Wornell, and S. Ghosh. Thermometer: Towards universal calibration for large language models.arXiv preprint arXiv:2403.08819,

    Pith/arXiv arXiv

  8. [2007]

    Karpathy

    A. Karpathy. The unreasonable effectiveness of recurrent neural networks. Blog post,https://karpathy. github.io/2015/05/21/rnn-effectiveness/,

    2015

  9. [2018]

    Nakkiran, A

    P. Nakkiran, A. Bradley, A. Goliński, E. Ndiaye, M. Kirchhof, and S. Williamson. Trained on tokens, calibrated on concepts: The emergence of semantic calibration in LLMs.arXiv preprint arXiv:2511.04869,

    arXiv

  10. [2020]

    Rastogi, A

    A. Rastogi, A. Q. Jiang, A. Lo, G. Berrada, G. Lample, J. Rute, J. Barmentlo, K. Yadav, K. Khandelwal, K. R. Chandu, et al. Magistral.arXiv preprint arXiv:2506.10910,

    Pith/arXiv arXiv

  11. [2021]

    Berta, D

    E. Berta, D. Holzmüller, M. I. Jordan, and F. Bach. Rethinking early stopping: Refine, then calibrate.arXiv preprint arXiv:2501.19195, 2025a. E. Berta, D. Holzmüller, M. I. Jordan, and F. Bach. Structured matrix scaling for multi-class calibration. arXiv preprint arXiv:2511.03685, 2025b. J. Blasiok, P. Gopalan, L. Hu, and P. Nakkiran. When does optimizing...

    Pith/arXiv arXiv

  12. [2022]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. DeepSeek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  13. [2025]

    J. Domke. Moment-matching conditions for exponential families with conditioning or hidden data.arXiv preprint arXiv:2001.09771,

    Pith/arXiv arXiv 2001