Proper Calibeating

Dean P. Foster; Sergiu Hart

REVIEW 2 major objections 2 minor 1 cited by

Proper calibration of forecasts with respect to all proper scoring rules is equivalent to universal no-regret when best replying to them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 14:58 UTC pith:FSSZHGDE

load-bearing objection The paper cleanly extends calibration to all proper scoring rules via uniform convergence and shows this yields equivalence to universal no-regret, but the uniform step is the part that needs checking. the 2 major comments →

arxiv 2605.26703 v2 pith:FSSZHGDE submitted 2026-05-26 econ.TH cs.GTcs.LGstat.ML

Proper Calibeating

Dean P. Foster , Sergiu Hart This is my paper

classification econ.TH cs.GTcs.LGstat.ML

keywords proper calibrationcalibeatingproper scoring rulesno regretforecastingdecision making under uncertaintyuniversal calibration

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper extends calibrated forecasts and calibeating from the quadratic scoring rule to the full class of proper scoring rules. It defines proper-calibration and proper-calibeating via uniform convergence of errors to zero across all bounded proper scoring rules. Standard calibration always implies proper-calibration, while ordinary calibeating does not imply proper-calibeating. The authors supply procedures that guarantee proper-calibeating and proper-multicalibeating, and they prove that proper-calibration is equivalent to achieving universal no-regret when a decision maker best-responds to the forecasts.

Core claim

Proper-calibration requires that forecast errors converge uniformly to zero over the entire class of bounded proper scoring rules. Calibration implies proper-calibration, but calibeating need not imply proper-calibeating. The central result is that proper-calibration is equivalent to universal no-regret when a decision maker best-replies to the forecasts under uncertainty.

What carries the argument

Uniform convergence of errors to zero over the class of all bounded proper scoring rules.

Load-bearing premise

The definitions require uniform convergence of errors to zero over the entire class of all bounded proper scoring rules.

What would settle it

A sequence of forecasts that satisfies standard calibration (or calibeating) yet produces non-vanishing errors for some bounded proper scoring rule other than the quadratic one, or a decision maker who achieves no regret without satisfying proper-calibration.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Procedures exist that guarantee proper-calibeating.
Procedures exist that guarantee proper-multicalibeating.
Proper-calibration ensures no regret for any best reply to the forecast.
The equivalence links forecast quality directly to regret-free decision making under uncertainty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The uniform-over-all-proper-scoring-rules requirement may be relaxed to finite subclasses while preserving the no-regret equivalence in restricted decision settings.
The same uniform convergence idea could be applied to other forecast properties such as sharpness or resolution.
Online algorithms that achieve proper-calibeating could be used as black-box predictors inside larger mechanism-design problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The paper cleanly extends calibration to all proper scoring rules via uniform convergence and shows this yields equivalence to universal no-regret, but the uniform step is the part that needs checking.

read the letter

The paper takes calibration and calibeating, originally tied to the quadratic score, and generalizes them by requiring forecast errors to converge uniformly to zero over the full class of bounded proper scoring rules. This produces the notions of proper-calibration and proper-calibeating. They prove that ordinary calibration always implies proper-calibration, while calibeating does not always imply proper-calibeating. They also give explicit constructions for achieving the proper versions, including a multicalibeating variant.

The equivalence result is the clearest new piece: proper-calibration lines up exactly with universal no-regret for a decision maker who best-replies to the forecast. The uniform quantification over every bounded proper score is what makes the link work, and the paper flags that the extension is non-trivial precisely because the original calibeating does not carry over.

The uniform requirement is strong by design. It may exceed what a given decision problem needs if the decision maker's utilities only induce a subclass of scoring rules. The stress-test concern about whether uniform convergence precisely matches the no-regret condition is worth watching in the proofs; if the equivalence holds only because of the broad class, some applications could use a weaker version.

Readers already familiar with calibration and no-regret learning in economics or machine learning will get the most from the separation of the two notions and the decision-theoretic link. The work is grounded in the existing literature and the authors have a record of careful results in this area. It deserves a serious referee because the equivalence, if the details check out, connects forecasting to robust decision making in a direct way.

Recommendation: send it to peer review.

Referee Report

2 major / 2 minor

Summary. The paper extends calibration and calibeating from the quadratic scoring rule to the class of all bounded proper scoring rules, defining proper-calibration and proper-calibeating via uniform convergence of errors to zero over this class. It proves that (standard) calibration always implies proper-calibration while calibeating need not imply proper-calibeating, constructs procedures that guarantee proper-calibeating and proper-multicalibeating, and establishes an equivalence between proper-calibration and universal no-regret when a decision maker best-replies to the forecasts.

Significance. If the equivalence holds, the work supplies a decision-theoretic characterization of a strengthened calibration notion and clarifies the relationship between forecast evaluation and regret in online decision problems. The explicit separation between calibration and proper-calibration, together with the constructions for achieving the stronger property, are useful technical contributions.

major comments (2)

[equivalence result (final section)] The equivalence between proper-calibration and universal no-regret (final result) is established using uniform convergence over the entire class of bounded proper scoring rules. Because any fixed decision maker's utility induces only a subclass of scoring rules, it is not immediate that the uniform quantification is equivalent to (rather than strictly stronger than) the no-regret condition; a direct argument showing necessity or a counter-example separating the two notions is needed to confirm the claimed equivalence.
[Section establishing that calibeating need not imply proper-calibeating] The statement that calibeating need not imply proper-calibeating is used to motivate the uniform extension. The concrete counter-example or construction demonstrating this separation should be checked against the later no-regret equivalence to ensure internal consistency of the quantifiers.

minor comments (2)

[Definitions section] Notation for the class of bounded proper scoring rules and the uniform norm should be introduced once and used consistently.
[Introduction] The abstract and introduction both state the three main results; a short roadmap paragraph at the end of the introduction would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the relationships between the notions in the paper. We address each major comment below and plan revisions to strengthen the manuscript.

read point-by-point responses

Referee: [equivalence result (final section)] The equivalence between proper-calibration and universal no-regret (final result) is established using uniform convergence over the entire class of bounded proper scoring rules. Because any fixed decision maker's utility induces only a subclass of scoring rules, it is not immediate that the uniform quantification is equivalent to (rather than strictly stronger than) the no-regret condition; a direct argument showing necessity or a counter-example separating the two notions is needed to confirm the claimed equivalence.

Authors: We agree that the distinction between uniform convergence over the full class and the per-utility no-regret condition merits explicit verification. The manuscript's equivalence claim relies on the fact that every bounded utility induces a proper scoring rule (via the standard construction), but the uniform quantifier is used to obtain a single statement that works for all decision makers simultaneously. In the revision we will insert a short direct argument (likely as a new lemma) establishing necessity: if a forecaster fails proper-calibration, then there exists some utility for which the best-replying decision maker incurs linear regret. This will confirm that the uniform condition is equivalent rather than strictly stronger. revision: yes
Referee: [Section establishing that calibeating need not imply proper-calibeating] The statement that calibeating need not imply proper-calibeating is used to motivate the uniform extension. The concrete counter-example or construction demonstrating this separation should be checked against the later no-regret equivalence to ensure internal consistency of the quantifiers.

Authors: We will re-examine the counter-example (currently in the section following the definition of proper-calibeating) to verify that the quantifiers align with those appearing in the no-regret equivalence. In particular, we will confirm that the construction uses a fixed utility (hence a single scoring rule) while the equivalence result quantifies over all utilities; a brief remark will be added to the revised text to make this distinction explicit and to note that the separation does not affect the later equivalence. revision: partial

Circularity Check

0 steps flagged

No significant circularity; definitions and equivalence are independently derived.

full rationale

The paper introduces proper-calibration and proper-calibeating as new extensions requiring uniform convergence of errors to zero over the class of all bounded proper scoring rules. It proves that standard calibration implies proper-calibration (but not conversely for calibeating), provides guarantees for the new notions, and demonstrates an equivalence to universal no-regret under best-reply decision making. These steps are presented as mathematical results rather than reductions to fitted parameters, self-definitions, or load-bearing self-citations. No quoted equations or steps in the provided text show any claim reducing by construction to its own inputs; the uniform quantification is an explicit definitional choice whose consequences (including the equivalence) are then established separately. The derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on standard domain assumptions from scoring-rule theory with no free parameters or invented entities.

axioms (2)

domain assumption Proper scoring rules are those for which the best forecast is the true distribution.
This is the defining property used to extend calibration and calibeating.
domain assumption All proper scoring rules under consideration are bounded.
Boundedness is invoked to make uniform convergence well-defined over the whole class.

pith-pipeline@v0.9.1-grok · 5657 in / 1157 out tokens · 59268 ms · 2026-06-29T14:58:05.035379+00:00 · methodology

0 comments

read the original abstract

The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of $\textit{proper}$ scoring rules (for which the best forecast is the true distribution) and define $\textit{proper-calibration}$ and $\textit{proper-calibeating}$ by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Optimal Recalibration of an Online Predictor
stat.ML 2026-07 accept novelty 8.0

(ε, ε²)-recalibration is achievable in Θ(ε⁻³) rounds and this is optimal; the same rate gives simultaneous calibration and calibeating.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

When Does Optimizing a Proper Loss Yield Calibration?

Blasiok, J., P. Gopalan, L. Hu, and P. Nakkiran (2023), “When Does Optimizing a Proper Loss Yield Calibration?” inAdvances in Neural Information Processing Systems36, 42386–42413. Brier, G. W. (1950), “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review78, 1–3. Chen, Y., Z. Huang, M. I. Jordan, and H. Luo (2026), “Calibeat...

work page arXiv 2023
[2]

Stable Reliability Diagrams for Probabilistic Classifiers,

50When thecanddbinnings coincide, (U2) is precisely (ii). 41 Dimitriadis, T., T. Gneiting, and A. I. Jordan (2021), “Stable Reliability Diagrams for Probabilistic Classifiers,”Proceedings of the National Academy of Sciences118, e2016191118. Foster, D. P. and S. Hart (2018), “Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamics,”Games and ...

work page arXiv 2021
[3]

Asymptotic Calibration,

Foster, D. P. and R. V. Vohra (1998), “Asymptotic Calibration,”Biometrika85, 379–390. Gneiting, T. and A. E. Raftery (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,”Journal of the American Statistical Association102, 359–378. Gopalan, P., A. T. Kalai, O. Reingold, V. Sharan, and U. Wieder (2022), “Omnipredic- tors,” inInnovations in Th...

work page arXiv 1998

[1] [1]

When Does Optimizing a Proper Loss Yield Calibration?

Blasiok, J., P. Gopalan, L. Hu, and P. Nakkiran (2023), “When Does Optimizing a Proper Loss Yield Calibration?” inAdvances in Neural Information Processing Systems36, 42386–42413. Brier, G. W. (1950), “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review78, 1–3. Chen, Y., Z. Huang, M. I. Jordan, and H. Luo (2026), “Calibeat...

work page arXiv 2023

[2] [2]

Stable Reliability Diagrams for Probabilistic Classifiers,

50When thecanddbinnings coincide, (U2) is precisely (ii). 41 Dimitriadis, T., T. Gneiting, and A. I. Jordan (2021), “Stable Reliability Diagrams for Probabilistic Classifiers,”Proceedings of the National Academy of Sciences118, e2016191118. Foster, D. P. and S. Hart (2018), “Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamics,”Games and ...

work page arXiv 2021

[3] [3]

Asymptotic Calibration,

Foster, D. P. and R. V. Vohra (1998), “Asymptotic Calibration,”Biometrika85, 379–390. Gneiting, T. and A. E. Raftery (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,”Journal of the American Statistical Association102, 359–378. Gopalan, P., A. T. Kalai, O. Reingold, V. Sharan, and U. Wieder (2022), “Omnipredic- tors,” inInnovations in Th...

work page arXiv 1998