{"id":"3c1fc9a6-aebc-47b8-ae14-2afc2fc264bd","arxiv_id":"2607.20718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Dividing the likelihood-ratio statistic by a goodness-of-fit statistic stops confidence intervals from collapsing under model misspecification, making them self-limit at a floor set by the model's inadequacy.","lead":"When a physics model fits the data imperfectly, standard error bars shrink to nothing as data grows, giving false confidence. This paper divides the usual test statistic by a goodness-of-fit score, so error bars stop at an honest floor set by how badly the model fails — and return to normal when the model is adequate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-limiting guarantee depends on GOF power: Eq. (10) asserts O(N) growth of γ̂ for any misspecification, but a defect invisible to the chosen GOF leaves γ̂=O(1) and the interval collapses at 1/√N.","rationale":"The reader's weakest-assumption identification is the same as mine: Eq. (10) is the load-bearing premise. If the GOF statistic lacks power against the actual defect, γ̂=O(1), the tempering is inert, and the interval collapses at the standard rate while appearing protected. This is not a manufactured concern; the paper's own Sec. V admits the dependence on GOF power, but the abstract and the scaling argument in Sec. II.B present it as a general result. The proposed within-bin Gaussian test directly probes the condition in a case where the binned Pearson statistic is blind by construction. The paper's independent support—the exact Gaussian floor calculation and the open code—does not resolve this concern; those checks use misspecifications that the GOF was designed to detect. The conditional nature of the guarantee is enough to keep the reader's CONDITIONAL verdict: the construction is promising and the proof-of-concept is internally consistent, but the abstract should state the dependence on GOF power, and the unbinned scaling for classifier-based GOF deserves an explicit proof or a comprehensive power study. No change to the verdict is needed because the reader already flagged this exact condition as the weakest assumption.","tokens_in":15731,"tokens_out":13638,"duration_ms":114740,"concrete_test":"In the Sec. III Gaussian setup (model N(μ,1), B=50 bins over [μ̂−5,μ̂+5]), generate data from a distribution that has N(0,1) bin probabilities exactly but a different within-bin conditional density—e.g., two narrow Gaussians at the edges of each bin with equal mass, symmetric about 0. Fit N(μ,1), compute γ̂=max(1,Gbin/(B−2)) and the 68% CL well-tempered half-width wWT at N=10^3, 10^4, 10^5 for 1000 pseudoexperiments. If γ̂ remains O(1) and wWT continues to shrink as 1/√N instead of plateauing, Eq. (10)/(12) fails for a misspecification the binned GOF cannot resolve, and the abstract's unconditional self-limiting claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety property is the O(N) scaling of the GOF statistic under misspecification (Eq. (10)). This is not a property of misspecification alone; it is a property of the pair (misspecification, GOF test). For the binned Pearson statistic with fixed B bins, any model defect that preserves all bin probabilities at the pseudo-true parameter gives Gbin(θ̂MLE)/(B−k−1)→1, so γ̂=O(1) and T(θ)=Λ(θ)/γ̂ inherits the standard 1/√N collapse. The same failure occurs for a classifier GOF when the classifier family cannot represent the density-ratio difference: Gcls can remain O(1) even though p≠f(θ*). Section II.D invokes 'universal approximation and sufficient capacity' and Section V concedes that 'a weak or underpowered test is more permissive, allowing narrower intervals', but the abstract and Eq. (10) state the scaling as unconditional. Because the self-limiting floor and Neff saturation are the main advertised guarantees, this conditionality is load-bearing. A simple demonstration is a Gaussian misspecification localized inside bins: replace the conditional density in each of the B bins by a narrow two-point distribution with the same bin probabilities as N(0,1). The Pearson GOF at θ̂MLE≈0 is blind (γ̂→1), while the true sampling variance of the mean is inflated, so the standard interval undercovers and the well-tempered interval would not self-limit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a modification of likelihood-based inference, the 'well-tempered likelihood', defined by dividing the standard likelihood-ratio statistic Λ(θ) by a goodness-of-fit statistic γ̂ evaluated at the best-fit point: T(θ)=Λ(θ)/γ̂. The authors argue that under correct specification γ̂→1 and standard Wilks inference is recovered, while under misspecification both Λ and γ̂ grow as O(N), so T remains O(1) and confidence intervals self-limit at a floor set by the model's inadequacy. Two concrete GOF choices are developed: a binned Pearson chi-squared statistic and an unbinned classifier two-sample statistic. The method is illustrated on a Gaussian location problem with an analytic floor formula and on a particle-physics-like measurement of α_s using a DCTR surrogate likelihood with a gradient-boosted decision tree GOF. Code and data are provided.","tokens_in":16080,"tokens_out":9544,"duration_ms":83700,"significance":"If the main scaling claim were unconditional, the well-tempered likelihood would be a valuable, simple safeguard against overconfident intervals under misspecification, with clear relevance to high-energy physics and beyond. The Gaussian proof of concept is explicit and internally consistent: the floor w_WT≈0.084 for σ0=1.3, B=50 follows from Dχ2(1.3)≈0.34 and N_eff≈140, and the analytical formulas are checkable. The public code and reproduction data are a definite strength. However, the central guarantee is conditional on the GOF test actually detecting the misspecification. The manuscript acknowledges this only in Sec. V, while the abstract and Eq. (10) present the O(N) scaling as unconditional. As a result, the advertised 'self-limiting guarantee' is overstated relative to what is proved.","major_comments":[{"comment":"Eq. (10) and the abstract state that under misspecification γ̂=O(N) unconditionally. This is false for a fixed GOF. For example, with fixed B bins, any misspecification that leaves all bin probabilities at the pseudo-true parameter unchanged (e.g., a within-bin rearrangement of the conditional density) gives Gbin(θ̂MLE)/(B−k−1)→1 under the model, so γ̂=O(1) and T(θ)=Λ(θ)/γ̂ inherits the standard O(N) growth and the interval collapses as 1/√N. The property in Eq. (10) is a property of the pair (misspecification, GOF), not of misspecification alone. The remark in Sec. V that 'a weak or underpowered test is more permissive' does not rescue the abstract's unconditional claim. Because self-limitation is the main advertised result, this conditionality must be stated as a formal hypothesis in Eq. (10), the abstract, and the summary of guarantees.","section":"Abstract; Sec. II.B, Eq. (10)"},{"comment":"The statement 'T(θ0)/k follows an F_{k,B−k−1} distribution' is not correct as written, because γ̂ is clipped at 1 in Eq. (12). Unconditionally, T/k equals Λ/k on the event Gbin/(B−k−1)<1, and equals (Λ/k)/(Gbin/(B−k−1)) on the complement. The law is therefore a mixture, not a single F distribution. A conditional statement on Gbin/(B−k−1)>1 would be correct, but that is not what is written. This matters for finite-sample calibration, and it is inconsistent with Eq. (21), which uses the χ² quantile rather than the F quantile that the text advertises as a finite-sample correction.","section":"Sec. II.C, Eq. (13)"},{"comment":"The unbinned construction rests on the claim that the null standard deviation σ(N)=std[G_cls]_0 is approximately constant in N. This is load-bearing: G_cls is a sum over 2N event-level terms, so for a raw sum the null spread would grow as √N. A constant σ requires a nontrivial cancellation, e.g., classifier estimation error of order N^{−1/2}; the text only says 'Empirically, ν(N) is negative and approximately constant in N, and σ(N) is positive ... and likewise approximately constant.' If instead σ(N)∼√N (or any growing rate), then under misspecification z=(G_cls−ν)/σ=O(√N), so γ̂=O(√N) and the interval width scales as N^{−1/4}, not the advertised constant floor. The unbinned self-limiting claim therefore needs either a proof of the variance cancellation or a careful empirical scaling study of σ(N) over a wide range of N.","section":"Sec. II.D, Eq. (16)"}],"minor_comments":[{"comment":"The two γ̂ constructions are structurally different: the binned version uses a scaled Pearson statistic directly, while the unbinned version uses a shifted z-score. This makes the analogy less direct than the text suggests. Consider unifying the notation or explicitly stating that the binned case is a special case of a more general 'GOF per degree of freedom'.","section":"Sec. II.C, Eq. (12) vs. Sec. II.D, Eq. (16)"},{"comment":"The misspecified N_eff curve rises from ≈180 at N=500 to ≈480 at N=25000; this is far from the asymptotic plateau of the Gaussian example. The label 'self-limits' is an asymptotic claim; the finite-sample behavior shown is consistent with a slow approach to a floor, but the figure does not yet demonstrate saturation. The text should distinguish between 'approaching a floor' and 'at the floor'.","section":"Sec. IV.D, Fig. 2c"},{"comment":"For moderate B, the F distribution has heavier tails than χ²_k/k, so using χ² quantiles in Eq. (21) is a further approximation that is not quantified. If the F-statistic claim is retained as a finite-sample correction, the numerical coverage studies should show the effect of clipping and the choice of quantile.","section":"Sec. III.C, Eq. (21)"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid: the headline guarantee is conditional on GOF power, and the current abstract and Eq. (10) overstate it. This is fixable by reframing the contribution and adding formal conditions. The σ(N)=O(1) issue for the unbinned GOF is more than a presentation issue: without a proof or scaling study, the unbinned self-limiting claim is not established. The paper's analytic Gaussian example and reproducibility are strong, and I would be willing to reconsider after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth reading. The core construction is simple and genuinely new as far as I know: divide the standard likelihood-ratio statistic Λ(θ) by a goodness-of-fit statistic γ̂ evaluated at the best-fit point. Under correct specification γ̂→1 and you recover Wilks; under misspecification, if the GOF has power, γ̂ grows with N at the same rate as Λ, so T = Λ/γ̂ stays O(1) and the confidence interval self-limits instead of collapsing. The binned Pearson case is worked out cleanly, and they note it reproduces the PDG S-factor for weighted averages. The unbinned classifier version is a natural extension.\n\nWhat they do well: the Gaussian proof of concept is internally consistent—I checked the announced floor w_WT≈0.084 against D_χ²(1.3)≈0.34 at B=50, and it holds; the MC matches the analytic curve. They ship code and data, and they are transparent about AI use and the concurrent work by Szewc. That's real credit.\n\nThe soft spot is precisely what the stress-test note flags. Eq. (10) states γ̂ = O(N) under misspecification, but scaling is a joint property of the misspecification and the GOF test. If the misspecification hides inside bins—say, a two-point distribution in each bin with bin probabilities equal to the model's—the Pearson statistic is blind, γ̂ stays O(1), and the well-tempered interval inherits the standard 1/√N collapse. Same story for a classifier with insufficient capacity. The authors acknowledge this in Sec. V as a modeling choice, but the abstract and conclusions present the guarantee as unconditional. That's an honest mismatch, fixable by stating the condition and either proving the scaling for a useful test class or giving the failure mode.\n\nTwo smaller gaps: the unbinned G_cls scaling is asserted under 'universal approximation,' not proven, and the physics demo leans on a stack of approximations—DCTR surrogate, GBDT GOF, M=20 permutations, 25 pseudoexperiments per point. It's illustrative, not precision validation.\n\nWho benefits: HEP folks and anyone doing simulation-based inference, plus statisticians interested in misspecification. I'd send this to review; a competent referee can push on the conditional guarantee. I'd also bring it to reading group. Cite it if you work on likelihood robustness.","headline":"A clean, useful construction with a real caveat: self-limitation only kicks in if the GOF test can actually see the misspecification.","tokens_in":16661,"tokens_out":4206,"would_cite":true,"duration_ms":34640,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F10","62F25","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Dividing the likelihood-ratio statistic by a goodness-of-fit score stops confidence intervals from shrinking to zero when the model is wrong.","keywords":["misspecified models","confidence intervals","likelihood ratio","goodness-of-fit","self-limiting intervals","effective sample size","classifier two-sample test","Wilks theorem"],"falsifier":"Generate data from a model with a known, small misspecification (e.g., a Gaussian with variance 1.3 fit to unit variance), bin at a very coarse level so the Pearson test has low power, and check whether the well-tempered interval still self-limits or instead keeps shrinking like 1/√N. A second test: train a deliberately low-capacity classifier (e.g., a shallow decision tree) on a high-dimensional misspecified dataset and compare the resulting N_eff plateau to that obtained with a high-capacity classifier; if the plateau disappears or shifts substantially, the safety property is training-depend","tokens_in":15505,"feed_emoji":"⚖️","tokens_out":1432,"duration_ms":13238,"temperature":0.7,"pith_summary":"This paper proposes a simple fix for a deep problem in likelihood-based inference: when the model does not perfectly describe the data, standard confidence intervals keep shrinking as more data arrive, eventually becoming misleadingly precise and centered on a biased value. The fix, dubbed the well-tempered likelihood, is to divide the usual likelihood-ratio test statistic by a goodness-of-fit statistic evaluated at the best-fit point. Under a correct model the divisor is about one and standard inference is unchanged; under a misspecified model both numerator and denominator grow with the dataset size, so their ratio stays bounded and the interval self-limits at a floor set by the model's inadequacy. The paper demonstrates the behavior in a Gaussian example where the floor can be computed analytically and in a particle-physics measurement of the strong coupling constant using classifier-based goodness-of-fit. The key claim is that this construction honestly reports 'the model cannot distinguish parameter values differing by less than this' rather than a spuriously precise number.","feed_headline":"Divide by goodness-of-fit to keep wrong-model intervals honest","feed_subtitle":"New method stops confidence intervals from shrinking to zero when the model is misspecified, recovering standard inference when it is correc","key_machinery":"The central object is the ratio T(θ) = Λ(θ)/γ̂, formed by dividing the standard likelihood-ratio test statistic by a goodness-of-fit statistic evaluated at the best-fit point. The scaling property — γ̂ is O(1) under correct specification and O(N) under misspecification, growing in lockstep with Λ — is what converts an ever-shrinking interval into a self-limiting one. For the binned case the GOF is Pearson's chi-squared per degree of freedom, giving T that follows an F-distribution when the model is correct. For the unbinned case the GOF is a classifier two-sample test statistic converted to a shifted z-score, with per-N null parameters calibrated by permutation. The numerator and denominator","core_discovery":"The paper claims that the statistic T(θ) = Λ(θ)/γ̂, where Λ is the usual likelihood-ratio test statistic and γ̂ is a goodness-of-fit statistic per degree of freedom evaluated at the best fit, gives confidence intervals that are honest under misspecification. When the model is correct, γ̂ tends to one and T recovers standard Wilks-based inference. When the model is wrong, both Λ and γ̂ grow linearly with the number of data points, so T remains O(1) and the resulting interval width plateaus at a constant floor rather than collapsing at the 1/√N rate. The floor corresponds to an effective sample size N_eff = N/γ̂ that saturates at a value set by the severity of the misspecification and the reso","pith_inferences":["The self-limiting guarantee is only as strong as the GOF test's power: if the chosen test is blind to a particular defect, γ̂ stays O(1) and the interval collapses at the standard rate while wearing the appearance of protection. The paper acknowledges this in Sec. V, saying a weak test is 'more permissive.'","The floor on interval width is set by the resolution of the GOF test, meaning that a more powerful classifier yields a wider, more conservative interval. This makes the reported uncertainty depend on the analyst's choice of test architecture, which is a modeling decision rather than an objective property of the data.","One could test the method's behavior against a truly adversarial misspecification, such as a defect that a low-capacity classifier misses entirely, to see whether the self-limiting floor is robust or merely an artifact of the test's sensitivity.","The duster form could be adapted for hypothesis testing, not just parameter estimation, to prevent overconfident exclusions of physically motivated models based on misspecified backgrounds."],"forward_implications":["Under correct specification the well-tempered interval is essentially identical to the standard likelihood-ratio interval, so adopting it costs no nominal statistical power when the model is adequate.","Under misspecification the interval width saturates at a constant floor proportional to √(D_χ²/(B−2)), so the method refuses to report precision beyond what the goodness-of-fit test can resolve.","The effective sample size N_eff = N/γ̂ gives a data-driven measure of how much information the model actually supports, which could inform analysis design and systematic-error reporting.","The method is agnostic to the source of misspecification — a classifier two-sample test detects any shape discrepancy between data and model in high-dimensional feature spaces.","The framework extends beyond particle physics to any likelihood-based inference in cosmology, gravitational waves, or climate modeling where model error is a concern."],"fun_headline_variants":["GOF-ratio likelihood: intervals that admit model misspecification","When model fits poorly, interval width stops shrinking","Self-rescaling likelihood tames overconfident intervals","Divide and conquer: GOF ratio prevents interval collapse","Honest confidence intervals from wrong models? Try GOF ratio"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The self-limiting property requires that the goodness-of-fit statistic actually grows at the same O(N) rate as the likelihood under the specific model defect present; if the GOF test is too weak to detect that defect, the tempering is inert and the interval keeps shrinking.","fun_headline_variants_meta":{"raw":{"variants":["GOF-ratio likelihood: intervals that admit model misspecification","When model fits poorly, interval width stops shrinking","Self-rescaling likelihood tames overconfident intervals","Divide and conquer: GOF ratio prevents interval collapse","Honest confidence intervals from wrong models? Try GOF ratio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001187,"raw_usage":{"total_tokens":4756,"prompt_tokens":779,"completion_tokens":3977,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":3896}},"tokens_in":523,"tokens_out":3977,"duration_ms":23041,"temperature":1.0,"reasoning_tokens":3896,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:34:27.961847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate data from a model with a known, small misspecification (e.g., a Gaussian with variance 1.3 fit to unit variance), bin at a very coarse level so the Pearson test has low power, and check whether the well-tempered interval still self-limits or instead keeps shrinking like 1/√N. A second test: train a deliberately low-capacity classifier (e.g., a shallow decision tree) on a high-dimensional misspecified dataset and compare the resulting N_eff plateau to that obtained with a high-capacity classifier; if the plateau disappears or shifts substantially, the safety property is training-depend","supporting_citations":[],"review_version":1}