Machine Learning · arXiv:2610.21567 v1 · submitted 15 Oct 2026 · 64 references
Sharp and Stable Label-Free Calibration Certificates from Logical Coherence
The paper says
“Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing.”
more of the abstract
For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A rigorous label-free calibration lower-bound framework with honest limitations; the math is solid and the paper delivers what it promises. This line is wrong
3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 64 references say.
3 minor objections
The paper says, at Section 4 (Corollary 11)
Pith says: minor objection 1 of 3
In Corollary 11 the ratio L_H/L_marg is displayed as min{1,Hε}->0 as ε->0, but the ratio is undefined at ε=0 since L_marg(P_F^0)=0; the statement should be quantified for ε>0 with the limit taken as ε↓0.
The paper says, at Section 5 (after Proposition 15)
Pith says: minor objection 2 of 3
The paragraph discussing a step η=√(G/T_max) fixed for a budget says the regret bound is √(GT) only at T=T_max, whereas Proposition 15's displayed bound uses η=√(G/T); please mark explicitly that the fixed-step statement is a separate observation and not part of the proposition.
show 1 more
The paper says, at Section 6.2 / Abstract
Pith says: minor objection 3 of 3
The phrase 'the eighth pair did not pass the pilot' in the abstract can be misread as a failed certificate; Section 6.2 and Table 16 make clear that Gate B step (4) decided not to collect confirmation data for Jev on DBpedia-14. Consider using the latter wording in the abstract as well.
What would settle it
As a further robustness check, one could take a real-data stream with known labels, compute the certificates under the assumed logical structure, and then systematically introduce errors into C (e.g., drop one label, add a spurious relation) to verify that the certificates behave as the theory predicts: positive when the structure is wrong and Θ_w=0, and non-increasing when C is weakened. This would empirically validate the scope sensitivity admitted in §8.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper gives a clean minimax characterization of the sharp label-free lower bound on calibration residuals, for finitely supported laws of the answers. It unifies several earlier ideas (Dutch-book incoherence, feasible posteriors, mutual calibration) under one roof and adds the stability analysis and the Lipschitz-stake certificate that is continuous in the data. The four-point example separating mean, marginal, and joint stages is instructive. The preregistered empirical study, while small (7 confirmatory certificates on 2 datasets), is honestly reported and shows the method works on real predictors.
The soft spots are the ones the authors themselves flag. The approach depends critically on the logical structure entered into the coherent polytope—enter a wrong relation and the certificate refutes the wrong thing. They state this clearly, but it means the method is only as good as the semantics you encode. Second, the bounds are lower bounds only: a certificate of zero means “not refuted,” not “calibrated.” The online learning guarantee (Theorem 16) is worst-case and can be vacuous (they show this in Figure 3). The empirical comparisons between stake-learning methods are secondary and partly post hoc. None of these is a fatal flaw; they are honest scope conditions.
Who should read this? Anyone working on label-free evaluation of probabilistic predictors, especially for LLMs or typed decision models. The paper deserves a serious peer review because the mathematics is careful, the claims are properly qualified, and the empirical protocol is preregistered. It does not oversell.
Recommendation: send to review. The paper is complete, the arguments are self-contained, and the limitations are discussed rather than hidden. A good referee will push on the sensitivity to the logical structure and on the real-world cost of specifying it, but the core contribution stands.
The strongest objection
The paper's central claim—that label-free calibration certificates can be derived from the law of the answers and the logical structure—rests on the assumption that the logical structure encoded in the coherent polytope C is correct. The authors openly acknowledge this limitation (§8) and illustrate its consequences with a synthetic example (Table 8). The mathematical development (Theorem 4, Theorem 12, etc.) is rigorous, with clear statements of when finite support is required and when it is extended via Lipschitz stakes. The empirical study is preregistered and honestly reported, with all confirmatory certificates positive. The admitted assumptions (exhaustive label set, correct logical relations) are genuine scope conditions, but they are neither hidden nor internally inconsistent with the arguments. No technical flaw or unaddressed weakness has been found that would undermine the main results within their stated scope. The paper is complete as a theoretical framework with a practical implementation; its limitations are properly scoped and do not invalidate the central claim.
The reader correctly identifies the correctness of the logical structure as the weakest assumption. I agree that this is the most delicate point in the chain. However, the paper explicitly acknowledges this limitation and does not claim that the certificates are valid when the structure is wrong. The theory holds conditional on that assumption, and the empirical study operates under it. No internal inconsistency or hidden flaw was detected in the mathematics or the experimental design. Therefore, the reader's ACCEPT verdict with high confidence is appropriate, and no adjustment is needed.
The referee's report
If accepted, the paper provides a unified minimax treatment of label-free calibration refutation, connects it to existing work on incoherence, feasible joint posteriors, and mutual calibration, and quantifies the price of stability in closed form. The proofs are complete and the standard tools are applied correctly: finite LP duality in Theorem 4, a correct Sion's theorem argument over a compact set of compatible laws in Theorem 12, Lipschitz extension in Lemma 20, and valid martingale/concentration arguments in Theorem 16 and Proposition 17. The online construction is not circular: stakes are predictable and the certificates control Type I error through the laws of the answers alone. The preregistration, hash-pinned reproducibility, and honest reporting of the failed pilot gate and post hoc analyses are exemplary. The main scope condition—that the coherent polytope C and the exhaustive-label semantics correctly encode the intended logical relations—is explicitly acknowledged in Section 8 and is a delimitation of the empirical interpretation rather than a flaw in the theoretical core.
Pith's reading of the paper (a machine's summary, not the authors')
The paper establishes that the logical structure of the questions a probabilistic predictor answers, together with the law of its outputs, can refute calibration without any observed labels. A minimax theorem identifies the sharp label-free lower bound on any information-restricted calibration residual as the largest expected profit a bettor can make by settling every input in its worst world. For marginal calibration this sharp bound is unstable — it jumps under tiny perturbations of the answers — but restricting the stakes to Lipschitz functions yields a stable bound that is sharp for a smooth calibration residual and positive under exactly the same conditions as the marginal bound. The stable bound is computable from i.i.d. data via online gradient ascent, and a preregistered experiment on four predictors and two data sets found all confirmatory certificates positive. The result is a unifying refutation side of calibration hierarchies, bringing together incoherence measures, feasible joint posteriors, and mutual calibration.
What carries the argument. The coherent polytope C, the set of probability vectors consistent with the logical structure of the events. The Dutch book without outcomes dwu(F) = min_ℓ Σ_i w_i u_i(F_i)(F_i - z^ℓ_i), which measures the profit a bettor can guarantee by using stakes u that depend only on permitted information. For finitely supported laws the sharp bound is the value of a finite linear program; for Lipschitz-constrained stakes the bound becomes (1+H)-Lipschitz in the Wasserstein distance. Projected online gradient ascent on a finite set of interpolation nodes makes the bound computable.
Core claim. The paper proves that for finitely supported laws of the answers, the sharp label-free lower bound on an information-restricted calibration residual equals the supremum, over stakes using only that information, of the expected profit from a Dutch book settled without seeing outcomes. This minimax theorem (Theorem 4) subsumes known results on incoherence, feasible joint posteriors, and mutual calibration. On bundles of logically related events the mean, marginal, and joint bounds separate strictly (Theorem 7). The sharp marginal bound is discontinuous in the Wasserstein distance (Proposition 9) and no continuous label-free lower bound stays within a constant factor of it (Corollary 11). Replacing unrestricted stakes with H-Lipschitz functions produces a stable bound L_H that is (1+H)-Lipschitz, sharp for a smooth residual Γ_H, and positive for exactly the same population laws as the marginal bound (Theorem 12). For i.i.d. streams, projected online gradient ascent on a grid of nodes yields fixed-horizon certificates whose shortfall is bounded by grid, learning, and statistical terms (Theorem 16), and predictable betting gives anytime-valid certificates (Proposition 17).
The premise it rests on. The coherent polytope C correctly encodes the true logical relationships among the queried events, because a relation entered wrongly makes a statistically valid certificate refute the wrong calibration notion.
If the paper is right (Pith's inferences, not the authors')
Label-free audits can provide guaranteed lower bounds on calibration error without ever seeing labels, identifying predictors that are provably miscalibrated.
The strict separation of mean, marginal, and joint bounds means an audit must specify which calibration notion it refutes; reporting the average distance to the coherent set overstates what can be proved about marginal calibration.
Stabilising the bound with Lipschitz stakes sacrifices a uniform constant factor but preserves the ability to detect miscalibration in the population, and the factor is known in closed form on simple examples.
The fixed-horizon and anytime-valid certificates make the approach practical: given a stream of bundles, one can continuously monitor whether a predictor remains below a chosen threshold of miscalibration.
The minimax framework could be extended to conditional events or ordinal predictions by enriching the coherent polytope, though the number of worlds may grow combinatorially.
The choice of Lipschitz constant H controls the trade-off between tightness and stability; the real-data results suggest that modest H (1 or 2) and a coarse grid (G=6) already give positive certificates, implying the price of stability is often small in practice.
The paper’s decomposition of the shortfall into grid, learning, and statistical terms guides practitioners in allocating budget: the worst-case bound can be vacuous, but the empirical performance is much better, hinting that a refined analysis might yield tighter guarantees.
The approach complements labelled audits: a positive certificate shows where labels are needed, while a zero certificate is inconclusive and does not imply calibration.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.67
“Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.93 · human participants 0.10 · controlled comparison 0.08 · randomized assignment 0.03 · states quantitative result 0.66 · compares with prior results 0.23 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. | background 0.87 | 0.07 | 0.03 | asserted 0.94 |
| For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. | result 0.49 | 0.78 | 0.21 | asserted 1.00 |
| On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. | result 0.90 | 0.73 | 0.08 | asserted 0.99 |
| The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. | result 0.95 | 0.79 | 0.13 | asserted 1.00 |
| Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. | result 0.77 | 0.80 | 0.26 | asserted 1.00 |
| On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. | method 0.72 | 0.70 | 0.14 | asserted 1.00 |
| In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). | result 0.96 | 0.94 | 0.13 | asserted 1.00 |
| A certificate shows only that calibration is at least this bad; zero means "not refuted". | interpretation 0.81 | 0.29 | 0.14 | asserted 1.00 |
What kind of paper the classifier reports this is: theoretical result 0.81 · evidence from proof or derivation 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,320 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
References (64)
- [1] Donald W. K. Andrews and Xiaoxia Shi. Inference based on conditional moment inequalities. Econometrica, 81 0 (2): 0 609--666, 2013. doi:10.3982/ECTA9370 on Pithdoi:10.3982/ecta9370
- [2] Isaiah Andrews. Revealed rationality: Label-free evaluation and regularization from representation theorems. arXiv preprint arXiv:2608.05015, 2026 on PithPith's reviewarXiv
- [3] Isaiah Andrews and Suproteem Sarkar. Dutch books for language models. arXiv preprint arXiv:2609.02797, 2026 on PithPith's reviewarXiv
- [4] Ivane Antonov, Sohom Mukherjee, Richard Pibernik, and Yo Joong Choe. Bet on features: Anytime-valid and feature-aware auditing of conditional quantile forecasters. arXiv preprint arXiv:2607.11653, 2026 on PithPith's reviewarXiv
- [5] Itai Arieli, Yakov Babichenko, Fedor Sandomirskiy, and Omer Tamuz. Feasible joint posterior beliefs. Journal of Political Economy, 129 0 (9): 0 2546--2594, 2021. doi:10.1086/714993 on Pithdoi:10.1086/714993
- [6] Sebastian Arnold, Alexander Henzi, and Johanna F. Ziegel. Sequentially valid tests for forecast calibration. The Annals of Applied Statistics, 17 0 (3): 0 1909--1935, 2023. doi:10.1214/22-AOAS1697 on Pithdoi:10.1214/22-aoas1697
- [7] Jaros aw B asiok and Preetum Nakkiran. Smooth ECE : Principled reliability diagrams via kernel smoothing. In The Twelfth International Conference on Learning Representations ( ICLR ) , 2024 on Pith
- [8] Jaros aw B asiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing (STOC), pages 1727--1740, 2023 on Pith
- [9] Jochen Br \"o cker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135 0 (643): 0 1512--1519, 2009. doi:10.1002/qj.456 on Pithdoi:10.1002/qj.456
- [10] Martin Br \"u ckerhoff and Nicolas Juillet. Instability of martingale optimal transport in dimension d 2 . Electronic Communications in Probability, 27: 0 1--10, 2022. doi:10.1214/22-ECP463 on Pithdoi:10.1214/22-ecp463
- [11] Philippe Casgrain, Martin Larsson, and Johanna Ziegel. Sequential testing for elicitable functionals via supermartingales. Bernoulli, 30 0 (2): 0 1347--1374, 2024. doi:10.3150/23-BEJ1634 on Pithdoi:10.3150/23-bej1634
- [12] Xiaohong Chen, Ashesh Rambachan, and Elie Tamer. Partial identification from LLM prompts. arXiv preprint arXiv:2606.15031, 2026 on PithPith's reviewarXiv
- [13] Yo Joong Choe and Aaditya Ramdas. Comparing sequential forecasters. Operations Research, 72 0 (4): 0 1368--1387, 2024. doi:10.1287/opre.2021.0792 on Pithdoi:10.1287/opre.2021.0792
- [14] Ben Chugg, Santiago Cortes-Gomez, Bryan Wilder, and Aaditya Ramdas. Auditing fairness by betting. In Advances in Neural Information Processing Systems 36 ( NeurIPS 2023) , pages 6070--6091, 2023. doi:10.52202/075280-0266 on Pithdoi:10.52202/075280-0266
- [15] A. Philip Dawid. The well-calibrated B ayesian. Journal of the American Statistical Association, 77 0 (379): 0 605--610, 1982. doi:10.1080/01621459.1982.10477856 on Pithdoi:10.1080/01621459.1982.10477856
- [16] A. Philip Dawid, Morris H. DeGroot, and Julia Mortera. Coherent combination of experts' opinions. Test, 4 0 (2): 0 263--313, 1995 on Pith
- [17] Glauber De Bona and Marcelo Finger. Measuring inconsistency in probabilistic logic: Rationality postulates and D utch book interpretation. Artificial Intelligence, 227, 2015 on Pith
- [18] Daniel Dragonevskiy. Understanding as no-arbitrage: Bounded D utch books as a definition and training objective for language models. arXiv preprint arXiv:2609.39341, 2026 on PithPith's reviewarXiv
- [19] Cynthia Dwork, Michael P. Kim, Omer Reingold, Guy N. Rothblum, and Gal Yona. Outcome indistinguishability. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 1095--1108, 2021. doi:10.1145/3406325.3451064 on Pithdoi:10.1145/3406325.3451064
- [20] Lukas Fluri, Daniel Paleka, and Florian Tram \`e r. Evaluating superhuman models with consistency checks. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning ( SaTML ) , pages 194--232, 2024. doi:10.1109/SaTML59370.2024.00017 on Pithdoi:10.1109/satml59370.2024.00017
show 44 more
- [21] Dean P. Foster and Sergiu Hart. Smooth calibration, leaky forecasts, finite recall, and N ash dynamics. Games and Economic Behavior, 109: 0 271--293, 2018. doi:10.1016/j.geb.2017.12.022 on Pithdoi:10.1016/j.geb.2017.12.022
- [22] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69 0 (2): 0 243--268, 2007 on Pith
- [23] Parikshit Gopalan and Lunjia Hu. Calibration through the lens of indistinguishability. ACM SIGecom Exchanges , 23 0 (1): 0 51--79, 2025. doi:10.1145/3821553.3821560 on Pithdoi:10.1145/3821553.3821560
- [24] Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar, and Pranay Tankala. The importance of being smoothly calibrated. arXiv preprint arXiv:2603.16015, 2026 on PitharXiv
- [25] Jason Hartline, Lunjia Hu, and Yifan Wu. A perfectly truthful calibration measure. arXiv preprint arXiv:2508.13100, 2025 a on PithPith's reviewarXiv
- [26] Jason Hartline, Yifan Wu, and Yunran Yang. Smooth calibration and decision making. In Symposium on Foundations of Responsible Computing ( FORC 2025) , volume 329 of Leibniz International Proceedings in Informatics ( LIPIcs ) , pages 16:1--16:26, 2025 b . doi:10.4230/LIPIcs.FOR... on Pithdoi:10.4230/lipics.forc.2025.16
- [27] Alexander Henzi and Johanna F. Ziegel. Valid sequential inference on probability forecast performance. Biometrika, 109 0 (3): 0 647--663, 2022. doi:10.1093/biomet/asab047 on Pithdoi:10.1093/biomet/asab047
- [28] Alexander Henzi, Marius Puke, Timo Dimitriadis, and Johanna Ziegel. A safe H osmer-- L emeshow test. The New England Journal of Statistics in Data Science, pages 175--189, 2024. doi:10.51387/23-NEJSDS56 on Pithdoi:10.51387/23-nejsds56
- [29] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58 0 (301): 0 13--30, 1963 on Pith
- [30] Lunjia Hu, Arun Jambulapati, Kevin Tian, and Chutong Yang. Testing calibration in nearly-linear time. In Advances in Neural Information Processing Systems 37 ( NeurIPS 2024) , pages 116022--116059, 2024. doi:10.52202/079017-3683 on Pithdoi:10.52202/079017-3683
- [31] Hazem Ibrahim and Yasir Zaki. Evaluating decision models for text annotation in computational social science. arXiv preprint arXiv:2609.24574, 2026 on PithPith's reviewarXiv
- [32] Saman Sarker Joy. Do system one decisions add up? A study of probabilistic coherence. arXiv preprint arXiv:2609.33971, 2026 on PithPith's reviewarXiv
- [33] Sham M. Kakade and Dean P. Foster. Deterministic calibration and N ash equilibrium. Journal of Computer and System Sciences, 74 0 (1): 0 115--130, 2008. doi:10.1016/j.jcss.2007.04.017 on Pithdoi:10.1016/j.jcss.2007.04.017
- [34] Wooyoung Kim, Koohyun Kwon, Soonwoo Kwon, and Sokbae Lee. The identification power of smoothness assumptions in models with counterfactual outcomes. Quantitative Economics, 9 0 (2): 0 617--642, 2018. doi:10.3982/QE545 on Pithdoi:10.3982/qe545
- [35] Yuqing Kong, Mingyu Song, Yizhou Wang, and Yifan Wu. Calibration without ground truth. arXiv preprint arXiv:2601.19862, 2026 on PitharXiv
- [36] Anany Kotawala. Locally coherent, globally incoherent: Bounding compositional incoherence in multi-component LLM agents. arXiv preprint arXiv:2605.30335, 2026 on PithPith's reviewarXiv
- [37] Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems (NeurIPS), 2019 on Pith
- [38] Keyi Li, Yihao He, and Quanyi Li. Beyond calibration: Do a typed-decision model's probabilities obey the probability axioms? arXiv preprint arXiv:2609.33209, 2026 a on PithPith's reviewarXiv
- [39] Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. JEV -as-a-judge: Accept when confident, escalate when unsure. arXiv preprint arXiv:2609.26550, 2026 b on PithPith's reviewarXiv
- [40] Charles F. Manski. Partial Identification of Probability Distributions. Springer Series in Statistics. Springer, New York, 2003. doi:10.1007/b97478 on Pithdoi:10.1007/b97478
- [41] Charles Marx, Sofian Zalouk, and Stefano Ermon. Calibration by distribution matching: Trainable kernel calibration metrics. In Advances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
- [42] Krish Matta, Atharv Naphade, and Andy Zou. Rethinking uncertainty evaluation in large language models. arXiv preprint arXiv:2607.19367, 2026 on PithPith's reviewarXiv
- [43] Mehryar Mohri, Jon Schneider, and Yifan Wu. Coherence mechanisms for provable self-improvement. arXiv preprint arXiv:2511.08440, 2025 on PitharXiv
- [44] Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tram \`e r. Consistency checks for language model forecasters. In The Thirteenth International Conference on Learning Representations ( ICLR ) , 2025 on Pith
- [45] Anastasios Panagiotelis, George Athanasopoulos, Puwasala Gamakumara, and Rob J. Hyndman. Forecast reconciliation: A geometric view with new insights on bias correction. International Journal of Forecasting, 37 0 (1): 0 343--359, 2021. doi:10.1016/j.ijforecast.2020.06.004 on Pithdoi:10.1016/j.ijforecast.2020.06.004
- [46] Teodora Popordanoska, Gorjan Radevski, Tinne Tuytelaars, and Matthew B. Blaschko. Estimating calibration error under label shift without labels. arXiv preprint arXiv:2312.08586, 2023 on PithPith's reviewarXiv
- [47] Nico Potyka. Linear programs for measuring inconsistency in probabilistic logics. In Principles of Knowledge Representation and Reasoning ( KR ) , pages 568--577, 2014 on Pith
- [48] Joel B. Predd, Robert Seiringer, Elliott H. Lieb, Daniel N. Osherson, H. Vincent Poor, and Sanjeev R. Kulkarni. Probabilistic coherence and proper scoring rules. IEEE Transactions on Information Theory, 55 0 (10): 0 4786--4792, 2009. doi:10.1109/TIT.2009.2027573 on Pithdoi:10.1109/tit.2009.2027573
- [49] Amir Rafe and Subasish Das. Calibrated decisions at scale: Converting police crash narratives into probabilistic crash variables with a S ystem O ne model ( J ev). arXiv preprint arXiv:2609.24052, 2026 on PithPith's reviewarXiv
- [50] Johannes Resin, Lu Yang, and Tilmann Gneiting. Hierarchies of calibration: Classification meets regression. arXiv preprint arXiv:2606.03245, 2026 on PithPith's reviewarXiv
- [51] Leo Richter, Xuanli He, Pasquale Minervini, and Matt J. Kusner. An auditing test to detect behavioral shift in language models. In The Thirteenth International Conference on Learning Representations ( ICLR ) , 2025 on Pith
- [52] Mark J. Schervish, Teddy Seidenfeld, and Joseph B. Kadane. Measures of incoherence: How not to gamble if you must. In Bayesian Statistics 7, pages 385--401. Oxford University Press, 2003. doi:10.1093/oso/9780198526155.003.0021 on Pithdoi:10.1093/oso/9780198526155.003.0021
- [53] Teddy Seidenfeld. Calibration, coherence, and scoring rules. Philosophy of Science, 52 0 (2): 0 274--294, 1985. doi:10.1086/289244 on Pithdoi:10.1086/289244
- [54] Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70 0 (2): 0 1178--1203, 2024. doi:10.1109/TIT.2023.3305867 on Pithdoi:10.1109/tit.2023.3305867
- [55] Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8 0 (1): 0 171--176, 1958 on Pith
- [56] Julia Staffel. Measuring the overall incoherence of credence functions. Synthese, 192 0 (5): 0 1467--1493, 2015. doi:10.1007/s11229-014-0640-x on Pithdoi:10.1007/s11229-014-0640-x
- [57] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type-safe is not error-free: Typed decision models follow the option name, not the definition bound to it. arXiv preprint arXiv:2609.26758, 2026 on PithPith's reviewarXiv
- [58] TypeSafe AI . Jev 1.13 jaggedness. https://web.archive.org/web/20260929101607/https://docs.typesafe.ai/model-jaggedness/jev-1.13.md, 2026. Version last reviewed 2026-09-17, archived 2026-09-29. The examples cited are absent from the version last reviewed 2026-10-02 on Pith
- [59] Tim van Erven and Jairo Cugliari. Game-theoretically optimal reconciliation of contemporaneous hierarchical time series forecasts. In Modeling and Stochastic Learning for Forecasting in High Dimensions, Lecture Notes in Statistics, pages 297--317. Springer, 2015. doi:10.1007/9... on Pithdoi:10.1007/978-3-319-18732-7_15
- [60] Jean Ville. \'E tude Critique de la Notion de Collectif . PhD thesis, Paris, 1939 on Pith
- [61] Jason Z. Wang. The verification tax: Fundamental limits of AI auditing in the rare-error regime. arXiv preprint arXiv:2604.12951, 2026 on PithPith's reviewarXiv
- [62] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86 0 (1): 0 1--27, 2024. doi:10.1093/jrsssb/qkad009 on Pithdoi:10.1093/jrsssb/qkad009
- [63] Jian-Qiao Zhu and Thomas L. Griffiths. Incoherent probability judgments in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society ( CogSci ) , 2024 on Pith
- [64] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning (ICML), 2003 on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment