REVIEW 3 major objections 3 minor 45 references
Dividing the likelihood-ratio statistic by a goodness-of-fit score stops confidence intervals from shrinking to zero when the model is wrong.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 09:34 UTC pith:LU6ZG7WG
load-bearing objection A clean, useful construction with a real caveat: self-limitation only kicks in if the GOF test can actually see the misspecification. the 3 major comments →
The Well-Tempered Likelihood: Honest Confidence Intervals for Misspecified Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that the statistic T(θ) = Λ(θ)/γ̂, where Λ is the usual likelihood-ratio test statistic and γ̂ is a goodness-of-fit statistic per degree of freedom evaluated at the best fit, gives confidence intervals that are honest under misspecification. When the model is correct, γ̂ tends to one and T recovers standard Wilks-based inference. When the model is wrong, both Λ and γ̂ grow linearly with the number of data points, so T remains O(1) and the resulting interval width plateaus at a constant floor rather than collapsing at the 1/√N rate. The floor corresponds to an effective sample size N_eff = N/γ̂ that saturates at a value set by the severity of the misspecification and the reso
What carries the argument
The central object is the ratio T(θ) = Λ(θ)/γ̂, formed by dividing the standard likelihood-ratio test statistic by a goodness-of-fit statistic evaluated at the best-fit point. The scaling property — γ̂ is O(1) under correct specification and O(N) under misspecification, growing in lockstep with Λ — is what converts an ever-shrinking interval into a self-limiting one. For the binned case the GOF is Pearson's chi-squared per degree of freedom, giving T that follows an F-distribution when the model is correct. For the unbinned case the GOF is a classifier two-sample test statistic converted to a shifted z-score, with per-N null parameters calibrated by permutation. The numerator and denominator
Load-bearing premise
The self-limiting property requires that the goodness-of-fit statistic actually grows at the same O(N) rate as the likelihood under the specific model defect present; if the GOF test is too weak to detect that defect, the tempering is inert and the interval keeps shrinking.
What would settle it
Generate data from a model with a known, small misspecification (e.g., a Gaussian with variance 1.3 fit to unit variance), bin at a very coarse level so the Pearson test has low power, and check whether the well-tempered interval still self-limits or instead keeps shrinking like 1/√N. A second test: train a deliberately low-capacity classifier (e.g., a shallow decision tree) on a high-dimensional misspecified dataset and compare the resulting N_eff plateau to that obtained with a high-capacity classifier; if the plateau disappears or shifts substantially, the safety property is training-depend
If this is right
- Under correct specification the well-tempered interval is essentially identical to the standard likelihood-ratio interval, so adopting it costs no nominal statistical power when the model is adequate.
- Under misspecification the interval width saturates at a constant floor proportional to √(D_χ²/(B−2)), so the method refuses to report precision beyond what the goodness-of-fit test can resolve.
- The effective sample size N_eff = N/γ̂ gives a data-driven measure of how much information the model actually supports, which could inform analysis design and systematic-error reporting.
- The method is agnostic to the source of misspecification — a classifier two-sample test detects any shape discrepancy between data and model in high-dimensional feature spaces.
- The framework extends beyond particle physics to any likelihood-based inference in cosmology, gravitational waves, or climate modeling where model error is a concern.
Where Pith is reading between the lines
- The self-limiting guarantee is only as strong as the GOF test's power: if the chosen test is blind to a particular defect, γ̂ stays O(1) and the interval collapses at the standard rate while wearing the appearance of protection. The paper acknowledges this in Sec. V, saying a weak test is 'more permissive.'
- The floor on interval width is set by the resolution of the GOF test, meaning that a more powerful classifier yields a wider, more conservative interval. This makes the reported uncertainty depend on the analyst's choice of test architecture, which is a modeling decision rather than an objective property of the data.
- One could test the method's behavior against a truly adversarial misspecification, such as a defect that a low-capacity classifier misses entirely, to see whether the self-limiting floor is robust or merely an artifact of the test's sensitivity.
- The duster form could be adapted for hypothesis testing, not just parameter estimation, to prevent overconfident exclusions of physically motivated models based on misspecified backgrounds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a modification of likelihood-based inference, the 'well-tempered likelihood', defined by dividing the standard likelihood-ratio statistic Λ(θ) by a goodness-of-fit statistic γ̂ evaluated at the best-fit point: T(θ)=Λ(θ)/γ̂. The authors argue that under correct specification γ̂→1 and standard Wilks inference is recovered, while under misspecification both Λ and γ̂ grow as O(N), so T remains O(1) and confidence intervals self-limit at a floor set by the model's inadequacy. Two concrete GOF choices are developed: a binned Pearson chi-squared statistic and an unbinned classifier two-sample statistic. The method is illustrated on a Gaussian location problem with an analytic floor formula and on a particle-physics-like measurement of α_s using a DCTR surrogate likelihood with a gradient-boosted decision tree GOF. Code and data are provided.
Significance. If the main scaling claim were unconditional, the well-tempered likelihood would be a valuable, simple safeguard against overconfident intervals under misspecification, with clear relevance to high-energy physics and beyond. The Gaussian proof of concept is explicit and internally consistent: the floor w_WT≈0.084 for σ0=1.3, B=50 follows from Dχ2(1.3)≈0.34 and N_eff≈140, and the analytical formulas are checkable. The public code and reproduction data are a definite strength. However, the central guarantee is conditional on the GOF test actually detecting the misspecification. The manuscript acknowledges this only in Sec. V, while the abstract and Eq. (10) present the O(N) scaling as unconditional. As a result, the advertised 'self-limiting guarantee' is overstated relative to what is proved.
major comments (3)
- [Abstract; Sec. II.B, Eq. (10)] Eq. (10) and the abstract state that under misspecification γ̂=O(N) unconditionally. This is false for a fixed GOF. For example, with fixed B bins, any misspecification that leaves all bin probabilities at the pseudo-true parameter unchanged (e.g., a within-bin rearrangement of the conditional density) gives Gbin(θ̂MLE)/(B−k−1)→1 under the model, so γ̂=O(1) and T(θ)=Λ(θ)/γ̂ inherits the standard O(N) growth and the interval collapses as 1/√N. The property in Eq. (10) is a property of the pair (misspecification, GOF), not of misspecification alone. The remark in Sec. V that 'a weak or underpowered test is more permissive' does not rescue the abstract's unconditional claim. Because self-limitation is the main advertised result, this conditionality must be stated as a formal hypothesis in Eq. (10), the abstract, and the summary of guarantees.
- [Sec. II.C, Eq. (13)] The statement 'T(θ0)/k follows an F_{k,B−k−1} distribution' is not correct as written, because γ̂ is clipped at 1 in Eq. (12). Unconditionally, T/k equals Λ/k on the event Gbin/(B−k−1)<1, and equals (Λ/k)/(Gbin/(B−k−1)) on the complement. The law is therefore a mixture, not a single F distribution. A conditional statement on Gbin/(B−k−1)>1 would be correct, but that is not what is written. This matters for finite-sample calibration, and it is inconsistent with Eq. (21), which uses the χ² quantile rather than the F quantile that the text advertises as a finite-sample correction.
- [Sec. II.D, Eq. (16)] The unbinned construction rests on the claim that the null standard deviation σ(N)=std[G_cls]_0 is approximately constant in N. This is load-bearing: G_cls is a sum over 2N event-level terms, so for a raw sum the null spread would grow as √N. A constant σ requires a nontrivial cancellation, e.g., classifier estimation error of order N^{−1/2}; the text only says 'Empirically, ν(N) is negative and approximately constant in N, and σ(N) is positive ... and likewise approximately constant.' If instead σ(N)∼√N (or any growing rate), then under misspecification z=(G_cls−ν)/σ=O(√N), so γ̂=O(√N) and the interval width scales as N^{−1/4}, not the advertised constant floor. The unbinned self-limiting claim therefore needs either a proof of the variance cancellation or a careful empirical scaling study of σ(N) over a wide range of N.
minor comments (3)
- [Sec. II.C, Eq. (12) vs. Sec. II.D, Eq. (16)] The two γ̂ constructions are structurally different: the binned version uses a scaled Pearson statistic directly, while the unbinned version uses a shifted z-score. This makes the analogy less direct than the text suggests. Consider unifying the notation or explicitly stating that the binned case is a special case of a more general 'GOF per degree of freedom'.
- [Sec. IV.D, Fig. 2c] The misspecified N_eff curve rises from ≈180 at N=500 to ≈480 at N=25000; this is far from the asymptotic plateau of the Gaussian example. The label 'self-limits' is an asymptotic claim; the finite-sample behavior shown is consistent with a slow approach to a floor, but the figure does not yet demonstrate saturation. The text should distinguish between 'approaching a floor' and 'at the floor'.
- [Sec. III.C, Eq. (21)] For moderate B, the F distribution has heavier tails than χ²_k/k, so using χ² quantiles in Eq. (21) is a further approximation that is not quantified. If the F-statistic claim is retained as a finite-sample correction, the numerical coverage studies should show the effect of clipping and the choice of quantile.
Circularity Check
No significant circularity: the O(N) scaling and self-limiting floor derive from separate asymptotic results; the underpowered-GOF caveat is an acknowledged scope condition, not a fit or self-definition.
full rationale
The derivation chain is self-contained. The well-tempered statistic is defined in Eq. (1) and concretized in Eqs. (12) and (16); it is not fitted. The O(N) growth of Lambda under misspecification follows from standard misspecified-MLE theory (White, Ref. [8]), and the O(N) growth of the binned Pearson GOF follows from the noncentrality calculation in Eq. (19), gamma_hat -> 1 + N D_chi2/(B-2). The claimed floor in Eq. (22), the Neff saturation in Eq. (23), and the Gaussian plateau w_WT ~ 0.084 are derived analytically and then checked against Monte Carlo, not obtained by fitting a parameter to the simulated plateau. In the unbinned case, nu(N) and sigma(N) are empirical null-calibration quantities, not fits to the misspecification signal; gamma_hat remains a test statistic. The paper explicitly flags the conditionality of the O(N) guarantee in Sec. V: 'a weak or underpowered test is more permissive, allowing narrower intervals.' That is a scope condition and correctness risk, not a circular step. The author-overlapping citations (DCTR, Ref. [26]; quark/gluon systematics, Ref. [25]) are tool and observable references used in the physics application; they are not used to prove the central scaling, to import a uniqueness theorem, or to smuggle in an ansatz. The construction is an engineered ratio with explicitly stated asymptotic ingredients, and no step reduces by definition to its own inputs.
Axiom & Free-Parameter Ledger
free parameters (6)
- B (number of bins) =
50 (Gaussian example)
- bin span =
[μ̂−5, μ̂+5]
- z-score shift form of γ̂ =
γ̂ = max(1, 1 + z)
- K (cross-validation folds) =
5
- M (permutations for null calibration) =
20
- GOF classifier / DCTR architecture choices =
GBDT for GOF; NN (256,256,256,128), GELU, dropout 0.05, LR 3e-4, AdamW for DCTR
axioms (7)
- standard math Wilks' theorem: under correct specification Λ(θ₀) → χ²_k
- domain assumption Under misspecification, θ̂_MLE → θ* = argmin D_KL(P‖f_θ) and Λ(θ*) = O(N)
- domain assumption Existence of GOF statistics with γ̂ = O(1) under correct specification and O(N) under misspecification (Eq. 10)
- domain assumption Independence of Λ(θ₀) and G_bin(θ̂_MLE) for the F_{k,B−k−1} null
- domain assumption Classifier convergence: ĉ/(1−ĉ) → p/f under universal approximation and sufficient capacity (Eq. 14)
- domain assumption DCTR surrogate ℓ(θ)−ℓ(θ₀) equals the true log-likelihood ratio (Eq. 25)
- domain assumption Pythia8.317/Monash faithfully forward-models e+e− → Z → hadrons for the study
invented entities (1)
-
N_eff (effective sample size)
no independent evidence
read the original abstract
Likelihood-based inference in particle physics, and in the physical sciences more broadly, relies on the assumption that the model accurately describes the data. When the model is misspecified, though, standard confidence intervals shrink to zero width with increasing data, producing overconfident and potentially misleading constraints. We propose the well-tempered likelihood, which divides the likelihood-ratio test statistic by a goodness-of-fit (GOF) statistic evaluated at the best-fit point. Under correct specification, the GOF is $\mathcal{O}(1)$ and standard inference is recovered. Under misspecification, both the likelihood and the GOF scale as $\mathcal{O}(N)$ for $N$ data points, so their ratio remains $\mathcal{O}(1)$ and the resulting well-tempered confidence interval self-limits at a floor determined by the model's inadequacy, essentially reducing the effective sample size. In other words, \textit{all models are correct, as long as your dataset is small enough}. We present binned and unbinned formulations of the well-tempered likelihood---the latter based on a classifier two-sample test---and demonstrate our method on a Gaussian example and on a measurement of the strong coupling constant using synthetic electron-positron collisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Note that under variance misspecification, the mean is still unbiased:θ ∗ = θ0 since ¯X→µ 0 regardless ofσ 0
Ifσ 0 >1 the standard interval undercovers, and ifσ 0 <1 it overcovers. Note that under variance misspecification, the mean is still unbiased:θ ∗ = θ0 since ¯X→µ 0 regardless ofσ 0. This Gaussian proof of concept focuses on coverage/width distortion, while Sec. IV below illustrates both bias and coverage issues. A. Binned W ell-T empered Statistic To demo...
-
[2]
the model cannot distinguishα s values differing by less than δ
usingB= 50 bins. Shaded bands show 16th–84th percentile ranges over 2000 pseudoexperiments.(a)68% CL interval half-width onµversusN. The gray dashed line shows the standard likelihood-ratio interval (∝1/ √ N); solid lines show the well-tempered intervals. Under correct specification (σ 0 = 1, blue) both agree. Under misspecification (σ 0 = 1.3, red), the ...
arXiv 2000
-
[3]
K. Cranmer, J. Brehmer, and G. Louppe, The frontier of simulation-based inference, Proc. Nat. Acad. Sci.117, 30055 (2020), arXiv:1911.01429 [stat.ML]
Pith/arXiv arXiv 2020
-
[4]
R. Barlow, Systematic errors: Facts and fictions, inCon- ference on Advanced Statistical Techniques in Particle Physics(2002) pp. 134–144, arXiv:hep-ex/0207026
Pith/arXiv arXiv 2002
-
[5]
G. Aadet al.(ATLAS), Jet energy scale and resolution measured in proton–proton collisions at √s= 13 TeV with the ATLAS detector, Eur. Phys. J. C81, 689 (2021), arXiv:2007.02645 [hep-ex]
Pith/arXiv arXiv 2021
-
[6]
V. Khachatryanet al.(CMS), Jet energy scale and reso- lution in the CMS experiment in pp collisions at 8 TeV, JINST12(02), P02014, arXiv:1607.03663 [hep-ex]
-
[7]
Szewc, Defining a Minimum Resolution for Unbinned Analyses, (2026), arXiv:2606.28498 [hep-ph]
M. Szewc, Defining a Minimum Resolution for Unbinned Analyses, (2026), arXiv:2606.28498 [hep-ph]
Pith/arXiv arXiv 2026
-
[8]
Navaset al.(Particle Data Group), Review of particle 11 physics, Phys
S. Navaset al.(Particle Data Group), Review of particle 11 physics, Phys. Rev. D110, 030001 (2024)
2024
-
[9]
P. J. Huber, The Behavior of Maximum Likelihood Esti- mates under Nonstandard Conditions, Proc. Fifth Berke- ley Symp. Math. Stat. Prob.1, 221 (1967)
1967
-
[10]
White, Maximum Likelihood Estimation of Misspeci- fied Models, Econometrica50, 1 (1982)
H. White, Maximum Likelihood Estimation of Misspeci- fied Models, Econometrica50, 1 (1982)
1982
-
[11]
Gr¨ unwald and T
P. Gr¨ unwald and T. van Ommen, Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It, Bayesian Analysis12, 1069 (2017)
2017
-
[12]
J. W. Miller and D. B. Dunson, Robust Bayesian In- ference via Coarsening, J. Am. Stat. Assoc.114, 1113 (2019)
2019
-
[13]
S. S. Wilks, The Large-Sample Distribution of the Like- lihood Ratio for Testing Composite Hypotheses, Annals Math. Statist.9, 60 (1938)
1938
-
[14]
W. G. Cochran, The distribution of quadratic forms in a normal system, with applications to the analysis of covariance, Math. Proc. Cambridge Phil. Soc.30, 178 (1934)
1934
-
[15]
T. S. Ferguson,A Course in Large Sample Theory(Chap- man & Hall, 1996)
1996
-
[16]
M. Williams, How good are your fits? Unbinned mul- tivariate goodness-of-fit tests in high energy physics, JINST5, P09004, arXiv:1006.3019 [hep-ex]
-
[17]
J. H. Friedman, On multivariate goodness of fit and two sample testing, eConfC030908, THPD002 (2003)
2003
-
[18]
D. Lopez-Paz and M. Oquab, Revisiting Classifier Two- Sample Tests, inInternational Conference on Learning Representations(2017) arXiv:1610.06545 [stat.ML]
Pith/arXiv arXiv 2017
-
[19]
Neyman and E
J. Neyman and E. S. Pearson, On the Problem of the Most Efficient Tests of Statistical Hypotheses, Phil. Trans. Roy. Soc. Lond. A231, 289 (1933)
1933
-
[20]
J. L. W. V. Jensen, Sur les fonctions convexes et les in´ egalit´ es entre les valeurs moyennes, Acta Mathematica 30, 175 (1906)
1906
-
[21]
Bierlichet al., A comprehensive guide to the physics and usage of PYTHIA 8.3, SciPost Phys
C. Bierlichet al., A comprehensive guide to the physics and usage of PYTHIA 8.3, SciPost Phys. Codeb.2022, 8 (2022), arXiv:2203.11601 [hep-ph]
Pith/arXiv arXiv 2022
-
[22]
P. Skands, S. Carrazza, and J. Rojo, Tuning PYTHIA 8.1: the Monash 2013 Tune, Eur. Phys. J. C74, 3024 (2014), arXiv:1404.5630 [hep-ph]
Pith/arXiv arXiv 2013
-
[23]
M. Cacciari, G. P. Salam, and G. Soyez, The anti-k t jet clustering algorithm, JHEP04, 063, arXiv:0802.1189 [hep-ph]
-
[24]
A. Roy, J. Pivarski, and C. W. Freer, An array-oriented Python interface for FastJet, J. Phys. Conf. Ser.2438, 012011 (2023), arXiv:2202.03911 [hep-ex]
Pith/arXiv arXiv 2023
-
[25]
M. Cacciari, G. P. Salam, and G. Soyez, FastJet User Manual, Eur. Phys. J. C72, 1896 (2012), arXiv:1111.6097 [hep-ph]
Pith/arXiv arXiv 2012
-
[26]
CMS Collaboration, Performance of quark/gluon dis- crimination in 8 TeV pp data, CMS-PAS-JME-13-002 (2013)
2013
-
[27]
P. Gras, S. H¨ oche, D. Kar, A. Larkoski, L. L¨ onnblad, S. Pl¨ atzer, A. Si´ odmok, P. Skands, G. Soyez, and J. Thaler, Systematics of quark/gluon tagging, JHEP07, 091, arXiv:1704.03878 [hep-ph]
-
[28]
A. Andreassen and B. Nachman, Neural Networks for Full Phase-space Reweighting and Parameter Tuning, Phys. Rev. D101, 091901 (2020), arXiv:1907.08209 [hep- ph]
Pith/arXiv arXiv 2020
-
[29]
K. Cranmer, J. Pavez, and G. Louppe, Approximating Likelihood Ratios with Calibrated Discriminative Classi- fiers, (2015), arXiv:1506.02169 [stat.AP]
Pith/arXiv arXiv 2015
-
[30]
P. Baldi, K. Cranmer, T. Faucett, P. Sadowski, and D. Whiteson, Parameterized neural networks for high- energy physics, Eur. Phys. J. C76, 235 (2016), arXiv:1601.07913 [hep-ex]
Pith/arXiv arXiv 2016
-
[31]
K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition 10.1109/CVPR.2016.90 (2015), arXiv:1512.03385 [cs.CV]
Pith/arXiv arXiv 2016
-
[32]
J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer Normal- ization, (2016), arXiv:1607.06450 [stat.ML]
Pith/arXiv arXiv 2016
-
[33]
D. Hendrycks and K. Gimpel, Gaussian Error Linear Units (GELUs), (2016), arXiv:1606.08415 [cs.LG]
Pith/arXiv arXiv 2016
-
[34]
Srivastava, G
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, J. Machine Learning Res.15, 1929 (2014)
1929
-
[35]
D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, inInternational Conference on Learning Representations(2014) arXiv:1412.6980 [cs.LG]
Pith/arXiv arXiv 2014
-
[36]
I. Loshchilov and F. Hutter, Decoupled Weight Decay Regularization (2017) arXiv:1711.05101 [cs.LG]
Pith/arXiv arXiv 2017
-
[37]
R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, A Limited Memory Algorithm for Bound Constrained Optimiza- tion, SIAM J. Sci. Comput.16, 1190 (1995)
1995
-
[38]
C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization, ACM Trans. Math. Softw.23, 550 (1997)
1997
-
[39]
P. Virtanenet al., SciPy 1.0–Fundamental Algorithms for Scientific Computing in Python, Nature Meth.17, 261 (2020), arXiv:1907.10121 [cs.MS]
Pith/arXiv arXiv 2020
-
[40]
A. Paszkeet al., PyTorch: An Imperative Style, High-Performance Deep Learning Library, (2019), arXiv:1912.01703 [cs.LG]
Pith/arXiv arXiv 2019
-
[41]
J. H. Friedman, Greedy function approximation: A gra- dient boosting machine., Annals Statist.29, 1189 (2001)
2001
-
[42]
T. Chen and C. Guestrin, XGBoost: A Scalable Tree Boosting System 10.1145/2939672.2939785 (2016), arXiv:1603.02754 [cs.LG]
arXiv 2016
-
[43]
Hastie, R
T. Hastie, R. Tibshirani, and J. Friedman,The Elements of Statistical Learning, 2nd ed., Springer Series in Statis- tics (Springer, 2009)
2009
-
[44]
Gretton, K
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨ olkopf, and A. Smola, A Kernel Two-Sample Test, J. Machine Learning Res.13, 723 (2012)
2012
-
[45]
G. J. Sz´ ekely and M. L. Rizzo, Energy statistics: A class of statistics based on distances, J. Statist. Plann. Infer- ence143, 1249 (2013)
2013
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.