Pith.

Machine Learning · arXiv:2610.21567 v1 · submitted 15 Oct 2026 · 64 references

Sharp and Stable Label-Free Calibration Certificates from Logical Coherence

Takahiro Kato (NP-hard, Inc.)

The paper says

“Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing.”

more of the abstract

For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A rigorous label-free calibration lower-bound framework with honest limitations; the math is solid and the paper delivers what it promises. This line is wrong

3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 64 references say.

3 minor objections

The paper says, at Section 4 (Corollary 11)

Pith says: minor objection 1 of 3

In Corollary 11 the ratio L_H/L_marg is displayed as min{1,Hε}->0 as ε->0, but the ratio is undefined at ε=0 since L_marg(P_F^0)=0; the statement should be quantified for ε>0 with the limit taken as ε↓0.

The paper says, at Section 5 (after Proposition 15)

Pith says: minor objection 2 of 3

The paragraph discussing a step η=√(G/T_max) fixed for a budget says the regret bound is √(GT) only at T=T_max, whereas Proposition 15's displayed bound uses η=√(G/T); please mark explicitly that the fixed-step statement is a separate observation and not part of the proposition.

show 1 more

The paper says, at Section 6.2 / Abstract

Pith says: minor objection 3 of 3

The phrase 'the eighth pair did not pass the pilot' in the abstract can be misread as a failed certificate; Section 6.2 and Table 16 make clear that Gate B step (4) decided not to collect confirmation data for Jev on DBpedia-14. Consider using the latter wording in the abstract as well.

What would settle it

As a further robustness check, one could take a real-data stream with known labels, compute the certificates under the assumed logical structure, and then systematically introduce errors into C (e.g., drop one label, add a spurious relation) to verify that the certificates behave as the theory predicts: positive when the structure is wrong and Θ_w=0, and non-increasing when C is weakened. This would empirically validate the scope sensitivity admitted in §8.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 32 s · 59,847 tokens read, 2,866 written
  3. Editor wrote the verdict above 14 s · 60,596 tokens read, 1,002 written
  4. Referee minor revision: 3 minor 2 min 59 s · 60,221 tokens read, 14,469 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper gives a clean minimax characterization of the sharp label-free lower bound on calibration residuals, for finitely supported laws of the answers. It unifies several earlier ideas (Dutch-book incoherence, feasible posteriors, mutual calibration) under one roof and adds the stability analysis and the Lipschitz-stake certificate that is continuous in the data. The four-point example separating mean, marginal, and joint stages is instructive. The preregistered empirical study, while small (7 confirmatory certificates on 2 datasets), is honestly reported and shows the method works on real predictors.

The soft spots are the ones the authors themselves flag. The approach depends critically on the logical structure entered into the coherent polytope—enter a wrong relation and the certificate refutes the wrong thing. They state this clearly, but it means the method is only as good as the semantics you encode. Second, the bounds are lower bounds only: a certificate of zero means “not refuted,” not “calibrated.” The online learning guarantee (Theorem 16) is worst-case and can be vacuous (they show this in Figure 3). The empirical comparisons between stake-learning methods are secondary and partly post hoc. None of these is a fatal flaw; they are honest scope conditions.

Who should read this? Anyone working on label-free evaluation of probabilistic predictors, especially for LLMs or typed decision models. The paper deserves a serious peer review because the mathematics is careful, the claims are properly qualified, and the empirical protocol is preregistered. It does not oversell.

Recommendation: send to review. The paper is complete, the arguments are self-contained, and the limitations are discussed rather than hidden. A good referee will push on the sensitivity to the logical structure and on the real-world cost of specifying it, but the core contribution stands.

The strongest objection

The paper's central claim—that label-free calibration certificates can be derived from the law of the answers and the logical structure—rests on the assumption that the logical structure encoded in the coherent polytope C is correct. The authors openly acknowledge this limitation (§8) and illustrate its consequences with a synthetic example (Table 8). The mathematical development (Theorem 4, Theorem 12, etc.) is rigorous, with clear statements of when finite support is required and when it is extended via Lipschitz stakes. The empirical study is preregistered and honestly reported, with all confirmatory certificates positive. The admitted assumptions (exhaustive label set, correct logical relations) are genuine scope conditions, but they are neither hidden nor internally inconsistent with the arguments. No technical flaw or unaddressed weakness has been found that would undermine the main results within their stated scope. The paper is complete as a theoretical framework with a practical implementation; its limitations are properly scoped and do not invalidate the central claim.

The reader correctly identifies the correctness of the logical structure as the weakest assumption. I agree that this is the most delicate point in the chain. However, the paper explicitly acknowledges this limitation and does not claim that the certificates are valid when the structure is wrong. The theory holds conditional on that assumption, and the empirical study operates under it. No internal inconsistency or hidden flaw was detected in the mathematics or the experimental design. Therefore, the reader's ACCEPT verdict with high confidence is appropriate, and no adjustment is needed.

The referee's report

If accepted, the paper provides a unified minimax treatment of label-free calibration refutation, connects it to existing work on incoherence, feasible joint posteriors, and mutual calibration, and quantifies the price of stability in closed form. The proofs are complete and the standard tools are applied correctly: finite LP duality in Theorem 4, a correct Sion's theorem argument over a compact set of compatible laws in Theorem 12, Lipschitz extension in Lemma 20, and valid martingale/concentration arguments in Theorem 16 and Proposition 17. The online construction is not circular: stakes are predictable and the certificates control Type I error through the laws of the answers alone. The preregistration, hash-pinned reproducibility, and honest reporting of the failed pilot gate and post hoc analyses are exemplary. The main scope condition—that the coherent polytope C and the exhaustive-label semantics correctly encode the intended logical relations—is explicitly acknowledged in Section 8 and is a delimitation of the empirical interpretation rather than a flaw in the theoretical core.

Pith's reading of the paper (a machine's summary, not the authors')

The paper establishes that the logical structure of the questions a probabilistic predictor answers, together with the law of its outputs, can refute calibration without any observed labels. A minimax theorem identifies the sharp label-free lower bound on any information-restricted calibration residual as the largest expected profit a bettor can make by settling every input in its worst world. For marginal calibration this sharp bound is unstable — it jumps under tiny perturbations of the answers — but restricting the stakes to Lipschitz functions yields a stable bound that is sharp for a smooth calibration residual and positive under exactly the same conditions as the marginal bound. The stable bound is computable from i.i.d. data via online gradient ascent, and a preregistered experiment on four predictors and two data sets found all confirmatory certificates positive. The result is a unifying refutation side of calibration hierarchies, bringing together incoherence measures, feasible joint posteriors, and mutual calibration.

What carries the argument. The coherent polytope C, the set of probability vectors consistent with the logical structure of the events. The Dutch book without outcomes dwu(F) = min_ℓ Σ_i w_i u_i(F_i)(F_i - z^ℓ_i), which measures the profit a bettor can guarantee by using stakes u that depend only on permitted information. For finitely supported laws the sharp bound is the value of a finite linear program; for Lipschitz-constrained stakes the bound becomes (1+H)-Lipschitz in the Wasserstein distance. Projected online gradient ascent on a finite set of interpolation nodes makes the bound computable.

Core claim. The paper proves that for finitely supported laws of the answers, the sharp label-free lower bound on an information-restricted calibration residual equals the supremum, over stakes using only that information, of the expected profit from a Dutch book settled without seeing outcomes. This minimax theorem (Theorem 4) subsumes known results on incoherence, feasible joint posteriors, and mutual calibration. On bundles of logically related events the mean, marginal, and joint bounds separate strictly (Theorem 7). The sharp marginal bound is discontinuous in the Wasserstein distance (Proposition 9) and no continuous label-free lower bound stays within a constant factor of it (Corollary 11). Replacing unrestricted stakes with H-Lipschitz functions produces a stable bound L_H that is (1+H)-Lipschitz, sharp for a smooth residual Γ_H, and positive for exactly the same population laws as the marginal bound (Theorem 12). For i.i.d. streams, projected online gradient ascent on a grid of nodes yields fixed-horizon certificates whose shortfall is bounded by grid, learning, and statistical terms (Theorem 16), and predictable betting gives anytime-valid certificates (Proposition 17).

The premise it rests on. The coherent polytope C correctly encodes the true logical relationships among the queried events, because a relation entered wrongly makes a statistically valid certificate refute the wrong calibration notion.

If the paper is right (Pith's inferences, not the authors')

Label-free audits can provide guaranteed lower bounds on calibration error without ever seeing labels, identifying predictors that are provably miscalibrated.

The strict separation of mean, marginal, and joint bounds means an audit must specify which calibration notion it refutes; reporting the average distance to the coherent set overstates what can be proved about marginal calibration.

Stabilising the bound with Lipschitz stakes sacrifices a uniform constant factor but preserves the ability to detect miscalibration in the population, and the factor is known in closed form on simple examples.

The fixed-horizon and anytime-valid certificates make the approach practical: given a stream of bundles, one can continuously monitor whether a predictor remains below a chosen threshold of miscalibration.

The minimax framework could be extended to conditional events or ordinal predictions by enriching the coherent polytope, though the number of worlds may grow combinatorially.

The choice of Lipschitz constant H controls the trade-off between tightness and stability; the real-data results suggest that modest H (1 or 2) and a coarse grid (G=6) already give positive certificates, implying the price of stability is often small in practice.

The paper’s decomposition of the shortfall into grid, learning, and statistical terms guides practitioners in allocating budget: the worst-case bound can be vacuous, but the empirical performance is much better, hinting that a refined analysis might yield tighter guarantees.

The approach complements labelled audits: a positive certificate shows where labels are needed, while a zero certificate is inconclusive and does not imply calibration.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.67

“Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.93 · human participants 0.10 · controlled comparison 0.08 · randomized assignment 0.03 · states quantitative result 0.66 · compares with prior results 0.23 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. background 0.87 0.07 0.03 asserted 0.94
For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. result 0.49 0.78 0.21 asserted 1.00
On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. result 0.90 0.73 0.08 asserted 0.99
The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. result 0.95 0.79 0.13 asserted 1.00
Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. result 0.77 0.80 0.26 asserted 1.00
On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. method 0.72 0.70 0.14 asserted 1.00
In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). result 0.96 0.94 0.13 asserted 1.00
A certificate shows only that calibration is at least this bad; zero means "not refuted". interpretation 0.81 0.29 0.14 asserted 1.00

What kind of paper the classifier reports this is: theoretical result 0.81 · evidence from proof or derivation 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,320 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Four-point separation (Theorem 7). (a) The bundle
Figure 2 of the paper
Figure 2. Sharpness versus stability on the two-point example
Figure 3 of the paper
Figure 3. Where the fixed-T certificate loses ground (Lemma 14, Proposition 15, Theo￾rem 16). (a) A predictor answers “A?” and “not A?” with the same prob￾ability x ∈ {0.05, . . . , 0.95}, uniformly; its true label law is the one attain￾ing LH. With H = 6 and a uniform grid of G = 6 nodes, LH − bT splits into the grid gap LH − LH,G, the learning gap LH,G − ρT and the statistical gap ρT − bT , where ρT is the average population value of the stakes used and bT = T −1 P t≤T dt − aT ; curves are medians over 10 seeds with 10–90% bands, and the reported bound is max{0, bT }. The worst-case…
Figure 4 of the paper
Figure 4. Anytime-valid lower bounds (Proposition 17) on the preregistered confirmatory
Figure 5 of the paper
Figure 5. Power design on a synthetic bundle shaped like the main one (
Figure 6 of the paper
Figure 6. A blind spot of every label-free audit. One stream of bundles from the predictor of

Figures are the authors', reproduced from the paper.

References (64)
  1. [1] Donald W. K. Andrews and Xiaoxia Shi. Inference based on conditional moment inequalities. Econometrica, 81 0 (2): 0 609--666, 2013. doi:10.3982/ECTA9370 on Pithdoi:10.3982/ecta9370
  2. [2] Isaiah Andrews. Revealed rationality: Label-free evaluation and regularization from representation theorems. arXiv preprint arXiv:2608.05015, 2026 on PithPith's reviewarXiv
  3. [3] Isaiah Andrews and Suproteem Sarkar. Dutch books for language models. arXiv preprint arXiv:2609.02797, 2026 on PithPith's reviewarXiv
  4. [4] Ivane Antonov, Sohom Mukherjee, Richard Pibernik, and Yo Joong Choe. Bet on features: Anytime-valid and feature-aware auditing of conditional quantile forecasters. arXiv preprint arXiv:2607.11653, 2026 on PithPith's reviewarXiv
  5. [5] Itai Arieli, Yakov Babichenko, Fedor Sandomirskiy, and Omer Tamuz. Feasible joint posterior beliefs. Journal of Political Economy, 129 0 (9): 0 2546--2594, 2021. doi:10.1086/714993 on Pithdoi:10.1086/714993
  6. [6] Sebastian Arnold, Alexander Henzi, and Johanna F. Ziegel. Sequentially valid tests for forecast calibration. The Annals of Applied Statistics, 17 0 (3): 0 1909--1935, 2023. doi:10.1214/22-AOAS1697 on Pithdoi:10.1214/22-aoas1697
  7. [7] Jaros aw B asiok and Preetum Nakkiran. Smooth ECE : Principled reliability diagrams via kernel smoothing. In The Twelfth International Conference on Learning Representations ( ICLR ) , 2024 on Pith
  8. [8] Jaros aw B asiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing (STOC), pages 1727--1740, 2023 on Pith
  9. [9] Jochen Br \"o cker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135 0 (643): 0 1512--1519, 2009. doi:10.1002/qj.456 on Pithdoi:10.1002/qj.456
  10. [10] Martin Br \"u ckerhoff and Nicolas Juillet. Instability of martingale optimal transport in dimension d 2 . Electronic Communications in Probability, 27: 0 1--10, 2022. doi:10.1214/22-ECP463 on Pithdoi:10.1214/22-ecp463
  11. [11] Philippe Casgrain, Martin Larsson, and Johanna Ziegel. Sequential testing for elicitable functionals via supermartingales. Bernoulli, 30 0 (2): 0 1347--1374, 2024. doi:10.3150/23-BEJ1634 on Pithdoi:10.3150/23-bej1634
  12. [12] Xiaohong Chen, Ashesh Rambachan, and Elie Tamer. Partial identification from LLM prompts. arXiv preprint arXiv:2606.15031, 2026 on PithPith's reviewarXiv
  13. [13] Yo Joong Choe and Aaditya Ramdas. Comparing sequential forecasters. Operations Research, 72 0 (4): 0 1368--1387, 2024. doi:10.1287/opre.2021.0792 on Pithdoi:10.1287/opre.2021.0792
  14. [14] Ben Chugg, Santiago Cortes-Gomez, Bryan Wilder, and Aaditya Ramdas. Auditing fairness by betting. In Advances in Neural Information Processing Systems 36 ( NeurIPS 2023) , pages 6070--6091, 2023. doi:10.52202/075280-0266 on Pithdoi:10.52202/075280-0266
  15. [15] A. Philip Dawid. The well-calibrated B ayesian. Journal of the American Statistical Association, 77 0 (379): 0 605--610, 1982. doi:10.1080/01621459.1982.10477856 on Pithdoi:10.1080/01621459.1982.10477856
  16. [16] A. Philip Dawid, Morris H. DeGroot, and Julia Mortera. Coherent combination of experts' opinions. Test, 4 0 (2): 0 263--313, 1995 on Pith
  17. [17] Glauber De Bona and Marcelo Finger. Measuring inconsistency in probabilistic logic: Rationality postulates and D utch book interpretation. Artificial Intelligence, 227, 2015 on Pith
  18. [18] Daniel Dragonevskiy. Understanding as no-arbitrage: Bounded D utch books as a definition and training objective for language models. arXiv preprint arXiv:2609.39341, 2026 on PithPith's reviewarXiv
  19. [19] Cynthia Dwork, Michael P. Kim, Omer Reingold, Guy N. Rothblum, and Gal Yona. Outcome indistinguishability. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 1095--1108, 2021. doi:10.1145/3406325.3451064 on Pithdoi:10.1145/3406325.3451064
  20. [20] Lukas Fluri, Daniel Paleka, and Florian Tram \`e r. Evaluating superhuman models with consistency checks. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning ( SaTML ) , pages 194--232, 2024. doi:10.1109/SaTML59370.2024.00017 on Pithdoi:10.1109/satml59370.2024.00017
show 44 more
  1. [21] Dean P. Foster and Sergiu Hart. Smooth calibration, leaky forecasts, finite recall, and N ash dynamics. Games and Economic Behavior, 109: 0 271--293, 2018. doi:10.1016/j.geb.2017.12.022 on Pithdoi:10.1016/j.geb.2017.12.022
  2. [22] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69 0 (2): 0 243--268, 2007 on Pith
  3. [23] Parikshit Gopalan and Lunjia Hu. Calibration through the lens of indistinguishability. ACM SIGecom Exchanges , 23 0 (1): 0 51--79, 2025. doi:10.1145/3821553.3821560 on Pithdoi:10.1145/3821553.3821560
  4. [24] Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar, and Pranay Tankala. The importance of being smoothly calibrated. arXiv preprint arXiv:2603.16015, 2026 on PitharXiv
  5. [25] Jason Hartline, Lunjia Hu, and Yifan Wu. A perfectly truthful calibration measure. arXiv preprint arXiv:2508.13100, 2025 a on PithPith's reviewarXiv
  6. [26] Jason Hartline, Yifan Wu, and Yunran Yang. Smooth calibration and decision making. In Symposium on Foundations of Responsible Computing ( FORC 2025) , volume 329 of Leibniz International Proceedings in Informatics ( LIPIcs ) , pages 16:1--16:26, 2025 b . doi:10.4230/LIPIcs.FOR... on Pithdoi:10.4230/lipics.forc.2025.16
  7. [27] Alexander Henzi and Johanna F. Ziegel. Valid sequential inference on probability forecast performance. Biometrika, 109 0 (3): 0 647--663, 2022. doi:10.1093/biomet/asab047 on Pithdoi:10.1093/biomet/asab047
  8. [28] Alexander Henzi, Marius Puke, Timo Dimitriadis, and Johanna Ziegel. A safe H osmer-- L emeshow test. The New England Journal of Statistics in Data Science, pages 175--189, 2024. doi:10.51387/23-NEJSDS56 on Pithdoi:10.51387/23-nejsds56
  9. [29] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58 0 (301): 0 13--30, 1963 on Pith
  10. [30] Lunjia Hu, Arun Jambulapati, Kevin Tian, and Chutong Yang. Testing calibration in nearly-linear time. In Advances in Neural Information Processing Systems 37 ( NeurIPS 2024) , pages 116022--116059, 2024. doi:10.52202/079017-3683 on Pithdoi:10.52202/079017-3683
  11. [31] Hazem Ibrahim and Yasir Zaki. Evaluating decision models for text annotation in computational social science. arXiv preprint arXiv:2609.24574, 2026 on PithPith's reviewarXiv
  12. [32] Saman Sarker Joy. Do system one decisions add up? A study of probabilistic coherence. arXiv preprint arXiv:2609.33971, 2026 on PithPith's reviewarXiv
  13. [33] Sham M. Kakade and Dean P. Foster. Deterministic calibration and N ash equilibrium. Journal of Computer and System Sciences, 74 0 (1): 0 115--130, 2008. doi:10.1016/j.jcss.2007.04.017 on Pithdoi:10.1016/j.jcss.2007.04.017
  14. [34] Wooyoung Kim, Koohyun Kwon, Soonwoo Kwon, and Sokbae Lee. The identification power of smoothness assumptions in models with counterfactual outcomes. Quantitative Economics, 9 0 (2): 0 617--642, 2018. doi:10.3982/QE545 on Pithdoi:10.3982/qe545
  15. [35] Yuqing Kong, Mingyu Song, Yizhou Wang, and Yifan Wu. Calibration without ground truth. arXiv preprint arXiv:2601.19862, 2026 on PitharXiv
  16. [36] Anany Kotawala. Locally coherent, globally incoherent: Bounding compositional incoherence in multi-component LLM agents. arXiv preprint arXiv:2605.30335, 2026 on PithPith's reviewarXiv
  17. [37] Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems (NeurIPS), 2019 on Pith
  18. [38] Keyi Li, Yihao He, and Quanyi Li. Beyond calibration: Do a typed-decision model's probabilities obey the probability axioms? arXiv preprint arXiv:2609.33209, 2026 a on PithPith's reviewarXiv
  19. [39] Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. JEV -as-a-judge: Accept when confident, escalate when unsure. arXiv preprint arXiv:2609.26550, 2026 b on PithPith's reviewarXiv
  20. [40] Charles F. Manski. Partial Identification of Probability Distributions. Springer Series in Statistics. Springer, New York, 2003. doi:10.1007/b97478 on Pithdoi:10.1007/b97478
  21. [41] Charles Marx, Sofian Zalouk, and Stefano Ermon. Calibration by distribution matching: Trainable kernel calibration metrics. In Advances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
  22. [42] Krish Matta, Atharv Naphade, and Andy Zou. Rethinking uncertainty evaluation in large language models. arXiv preprint arXiv:2607.19367, 2026 on PithPith's reviewarXiv
  23. [43] Mehryar Mohri, Jon Schneider, and Yifan Wu. Coherence mechanisms for provable self-improvement. arXiv preprint arXiv:2511.08440, 2025 on PitharXiv
  24. [44] Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tram \`e r. Consistency checks for language model forecasters. In The Thirteenth International Conference on Learning Representations ( ICLR ) , 2025 on Pith
  25. [45] Anastasios Panagiotelis, George Athanasopoulos, Puwasala Gamakumara, and Rob J. Hyndman. Forecast reconciliation: A geometric view with new insights on bias correction. International Journal of Forecasting, 37 0 (1): 0 343--359, 2021. doi:10.1016/j.ijforecast.2020.06.004 on Pithdoi:10.1016/j.ijforecast.2020.06.004
  26. [46] Teodora Popordanoska, Gorjan Radevski, Tinne Tuytelaars, and Matthew B. Blaschko. Estimating calibration error under label shift without labels. arXiv preprint arXiv:2312.08586, 2023 on PithPith's reviewarXiv
  27. [47] Nico Potyka. Linear programs for measuring inconsistency in probabilistic logics. In Principles of Knowledge Representation and Reasoning ( KR ) , pages 568--577, 2014 on Pith
  28. [48] Joel B. Predd, Robert Seiringer, Elliott H. Lieb, Daniel N. Osherson, H. Vincent Poor, and Sanjeev R. Kulkarni. Probabilistic coherence and proper scoring rules. IEEE Transactions on Information Theory, 55 0 (10): 0 4786--4792, 2009. doi:10.1109/TIT.2009.2027573 on Pithdoi:10.1109/tit.2009.2027573
  29. [49] Amir Rafe and Subasish Das. Calibrated decisions at scale: Converting police crash narratives into probabilistic crash variables with a S ystem O ne model ( J ev). arXiv preprint arXiv:2609.24052, 2026 on PithPith's reviewarXiv
  30. [50] Johannes Resin, Lu Yang, and Tilmann Gneiting. Hierarchies of calibration: Classification meets regression. arXiv preprint arXiv:2606.03245, 2026 on PithPith's reviewarXiv
  31. [51] Leo Richter, Xuanli He, Pasquale Minervini, and Matt J. Kusner. An auditing test to detect behavioral shift in language models. In The Thirteenth International Conference on Learning Representations ( ICLR ) , 2025 on Pith
  32. [52] Mark J. Schervish, Teddy Seidenfeld, and Joseph B. Kadane. Measures of incoherence: How not to gamble if you must. In Bayesian Statistics 7, pages 385--401. Oxford University Press, 2003. doi:10.1093/oso/9780198526155.003.0021 on Pithdoi:10.1093/oso/9780198526155.003.0021
  33. [53] Teddy Seidenfeld. Calibration, coherence, and scoring rules. Philosophy of Science, 52 0 (2): 0 274--294, 1985. doi:10.1086/289244 on Pithdoi:10.1086/289244
  34. [54] Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70 0 (2): 0 1178--1203, 2024. doi:10.1109/TIT.2023.3305867 on Pithdoi:10.1109/tit.2023.3305867
  35. [55] Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8 0 (1): 0 171--176, 1958 on Pith
  36. [56] Julia Staffel. Measuring the overall incoherence of credence functions. Synthese, 192 0 (5): 0 1467--1493, 2015. doi:10.1007/s11229-014-0640-x on Pithdoi:10.1007/s11229-014-0640-x
  37. [57] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type-safe is not error-free: Typed decision models follow the option name, not the definition bound to it. arXiv preprint arXiv:2609.26758, 2026 on PithPith's reviewarXiv
  38. [58] TypeSafe AI . Jev 1.13 jaggedness. https://web.archive.org/web/20260929101607/https://docs.typesafe.ai/model-jaggedness/jev-1.13.md, 2026. Version last reviewed 2026-09-17, archived 2026-09-29. The examples cited are absent from the version last reviewed 2026-10-02 on Pith
  39. [59] Tim van Erven and Jairo Cugliari. Game-theoretically optimal reconciliation of contemporaneous hierarchical time series forecasts. In Modeling and Stochastic Learning for Forecasting in High Dimensions, Lecture Notes in Statistics, pages 297--317. Springer, 2015. doi:10.1007/9... on Pithdoi:10.1007/978-3-319-18732-7_15
  40. [60] Jean Ville. \'E tude Critique de la Notion de Collectif . PhD thesis, Paris, 1939 on Pith
  41. [61] Jason Z. Wang. The verification tax: Fundamental limits of AI auditing in the rare-error regime. arXiv preprint arXiv:2604.12951, 2026 on PithPith's reviewarXiv
  42. [62] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86 0 (1): 0 1--27, 2024. doi:10.1093/jrsssb/qkad009 on Pithdoi:10.1093/jrsssb/qkad009
  43. [63] Jian-Qiao Zhu and Thomas L. Griffiths. Incoherent probability judgments in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society ( CogSci ) , 2024 on Pith
  44. [64] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning (ICML), 2003 on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment