Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Adaptive Conformal Inference through the Lens of Blackwell Approachability

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Pith's one-line read Adaptive conformal inference is recast as a repeated two-player vector-valued game, and a calibration-based opportunistic approachability algorithm, BOACI, is shown to achieve almost surely asymptotic coverage of at least 1−α together with

desk verdict Fresh game-theoretic reformulation and a new impossibility result, but the headline 'best of many worlds' guarantee is conditional on an unimplemented exact calibration oracle. read the letter →

arxiv 2510.15824 v2 pith:SNUK5TXJ submitted 2025-10-17 stat.ML cs.LG

classification stat.MLcs.LG MSC 91A2062G15
keywords AdaptiveconformalinferenceBlackwellapproachabilityopportunisticefficientpredictionsetsonlinecoverageguaranteeregimeswitchingcalibratedforecaster
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to give adaptive conformal inference (ACI) a unified theoretical foundation by viewing it as a repeated two-player vector-valued game. Modeling the learner's choice of discrete miscoverage level and the opponent's choice of the smallest threshold that excludes the realized outcome, the paper analyzes which pairs of (miscoverage rate, average interval length) are attainable, using Blackwell's approachability theorem. It then constructs BOACI, an algorithm that combines a calibrated forecaster with an opportunistic approachability response, and proves that BOACI achieves, almost surely, whenever the opponent is statistically Q-restricted (a very mild restriction on adversarial play), a validity guarantee (limsup miscoverage ≤ α) and an efficiency guarantee that is an oracle bound depending only on Q. Because the adversarial and exchangeable settings are particular cases of statistical restrictions, the same algorithm recovers classical conformal efficiency on exchangeable data, maintains validity under fully adversarial data, and gives meaningful efficiency bounds for intermediate regimes such as drift or regime switching. This 'best of many worlds' behavior matters because existing ACI methods typically offer validity for arbitrary sequences but have no simultaneous efficiency guarantees.

What carries the argument

The central object is a finite two-player vector-valued game: the learner chooses a discrete miscoverage level a ∈ A (equivalently, a conformal interval), the opponent chooses b ∈ B, the smallest discretized threshold below which the realized outcome would be covered, and the payoff is m(a,b) = (1_{b≤a}, L(C_a)) ∈ [0,1]×[0,2L], with L(C_a) the interval length. Two mechanisms do the work: (i) Blackwell's approachability theorem, whose dual condition characterizes exactly which convex sets of average payoffs are attainable, and (ii) an opportunistic variant based on a calibrated forecaster that outputs at each round a forecast z_t ∈ Δ(B) of the opponent's next action; the learner responds with

What would settle it

Run BOACI on exchangeable, i.i.d. test scores using a finite-grid ε-calibrated forecaster (for example, histogram forecasts over a coarse grid of Δ(B)) for a long horizon: if empirical coverage is maintained at 1−α but the average interval length converges to a value strictly above the classical conformal length L(C_α)—rather than to L(C_α) up to grid rounding—then the practical version does not inherit the claimed guarantee; a success, conversely, would be the first evidence that ε-calibration suffices.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central result (Theorem 5) is: if the opponent's play is statistically Q-restricted—that is, the empirical distribution of the opponent's mixed actions converges in average distance to a convex set Q of distributions over B—then the BOACI strategy satisfies lim_{T→∞} d(m̄_T, S_Q) = 0 almost surely, where m̄_T is the average of the payoff vectors (indicator of miscoverage, interval length) and S_Q is the set of pairs with miscoverage ≤ α and length ≤ max_{q∈Q} φ^+_q(α). Two consequences follow immediately: the limsup of average miscoverage is at most α (coverage is at least 1−α), and the limsup of average interval length is at most the oracle bound max_{q∈Q} φ^+_

Load-bearing premise

The guarantee rests on the existence of a forecaster that is calibrated over every Borel subset of the simplex of opponent actions; the paper provides no implementable instance of such a perfect calibration oracle, and the computationally feasible ε-calibrated version is only suggested (in Remark 5 and the conclusion), not proven to preserve the theorem's conclusion.

Editorial extensions

If this is right

  • Coverage at level 1−α is guaranteed almost surely for any data sequence, without any stochastic assumption on the outcomes; efficiency is then added wherever the data exhibit enough regularity.
  • In exchangeable settings, BOACI beats standard ACI on efficiency: it converges to the classical conformal interval length L(C_α), eliminating the positive step-size penalty that standard ACI pays.
  • For regime-switching or slowly drifting score distributions, the average interval length converges to a bound that is strictly better than the trivial 2L when the regimes are informative, and the guarantee degrades gracefully to the adversarial vacuous bound as the regime set expands.
  • The same geometry yields impossibility results: in fully adversarial settings no strategy can simultaneously keep validity and beat the trivial length bound, and two natural regret-based efficiency objectives (pinball-loss regret and best-valid-constant regret) are shown to be unattainable once the finite grid of conformal scores is taken into account.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ε-calibrated forecaster variant can be proven to satisfy a finite version of the approachability guarantee, BOACI becomes immediately deployable; the natural next test is to run it on real time-series benchmarks and compare its average interval length against ACI and its variants under drift.
  • Because the formulation is built on an abstract (A, B, m) game, the same opportunistic-approachability scheme may transfer to other sequential set-prediction problems—quantile tracking, classification with abstention, or distribution prediction—where a binary validity event and a scalar cost compete.
  • The statistical Q-restriction is an unusually weak assumption (only average convergence of the opponent's empirical play to Q), so the theorem likely holds for a broad class of nonstationary processes; an explicit rate of convergence of d(m̄_T, S_Q) in terms of the calibration convergence would sharpen this and is not provided.
  • A practical open question suggested by the paper: how to choose Q adaptively—the guarantee's sharpness is bounded by the tightest Q one can posit, and the convex-hull examples show that learning Q online would interpolate smoothly between the exchangeable and adversarial extremes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies adaptive conformal inference (ACI) through Blackwell approachability. It first reformulates the sequential conformal prediction problem as a repeated finite two-player game in which the learner chooses a discretized miscoverage level a_t and the opponent chooses the threshold b_t, with vector payoff encoding miscoverage and interval length. It then introduces the notion of a statistically Q-restricted opponent (Definition 1), defines the efficiency benchmark max_{q∈Q} φ⁺_q(α) and the target set S_Q (Eq. 13), and proposes BOACI (Algorithm 1), a calibration-based algorithm using the response function p*(q)=δ_{level(α,q)} (Eq. 15). The main theoretical result (Theorem 5) states that if the opponent is Q-restricted, BOACI's average payoff converges to S_Q, yielding the validity bound (19) and efficiency bound (20); Corollary 2 recovers the exchangeable guarantee. The proof checks that p* is piecewise continuous (Lemma 5) and that the closed convex image of Q under the goal function is contained in S_Q (Lemma 6), then applies the opportunistic approachability theorem of Bernstein et al. (2014).

Significance. If the main theorem is established for a practically available subroutine, the paper would provide a genuinely useful 'best of many worlds' result: adversarial validity plus efficiency that adapts to the degree of stochasticity of the data sequence. The game-theoretic reformulation is elegant and connects ACI to a mature body of approachability theory. The authors are careful to check the two hypotheses needed to instantiate Bernstein et al.'s general theorem, and the paper ships a self-contained proof sketch in Appendix C. However, the central guarantee is conditional on the existence of a calibrated forecaster satisfying the very strong Definition 3 over the entire simplex Δ(B). The paper neither constructs such a forecaster nor analyzes the ε-calibrated version it recommends in practice, and it explicitly defers that analysis to future work. This is a substantial gap for a paper whose abstract promises a strategy 'that ensures validity while adapting the efficiency' in real settings. There is also an unexamined timing/information issue: in the conformal game the opponent observes a_t before choosing b_t, and a_t is a deterministic function of the forecast z_t, while the standard

major comments (3)
  1. [§4.2, Algorithm 1; Remark 5; Conclusion] Theorem 5 and Corollary 2 are proven only under Definition 3: an exact calibrated forecaster over Δ(B) that satisfies the calibration condition for every Borel set G. The paper gives no implementation of such a forecaster and does not verify that a computable ε-calibrated forecaster inherits the guarantees. Remark 5 and the Conclusion explicitly defer the ε-calibrated analysis to future work. Therefore the stated guarantees (19)–(20) are not established for any implementable instance of BOACI; the theorem is a conditional existence result. This is load-bearing because the abstract and contributions present BOACI as an algorithm that 'ensures validity while adapting the efficiency' in practical scenarios. Please either provide an ε-calibrated version with a formal guarantee, or clearly scope the claims and state the result as oracle-based.
  2. [Definition 3; Algorithm 1; Appendix C, first inequality] There is a protocol mismatch. In the conformal game the opponent observes the learner's action a_t before choosing b_t, and in Algorithm 1 a_t is the deterministic function level(α,z_t) of the forecast z_t. The calibration condition in Definition 3 is required to hold for every strategy of the opponent, but the standard calibration theorems (Foster; Bernstein et al.) assume that the outcome b_t is independent of the realized forecast z_t given the forecaster's mixed distribution. Here the opponent can condition b_t on a_t, i.e. on a function of z_t, so it is not established that a calibrated forecaster satisfying Definition 3 exists in this game. The proof of Theorem 5 uses calibration to replace b_t by z_t in the average payoff; that step is unjustified without a formal protocol and an existence proof under this feedback structure. Please specify the timing precisely and either prove th
  3. [Definition 1; §2.2; §C.2] Statistical Q-restriction is defined for a sequence (q_t) of mixed actions 'output by the opponent,' but the game in §2.2 is defined with pure actions b_t ∈ B. It is never stated what q_t is (conditional distribution of b_t given history? Some latent randomization?) or whether it is observed. This matters because Lemmas 9–11 and the proof of Theorem 5 use the joint distribution of (q_t,z_t) and the identity E[q|z]=z for calibrated forecasts. Without a clear definition of q_t, it is impossible to verify whether the definitions are consistent with the actual implementation, where only b_t is observed. Please clarify what q_t is and how it relates to the pure actions and to the calibrated forecasts.
minor comments (5)
  1. [§4.2, after Theorem 5] The text says 'these results hold pathwise, i.e., for any possible sequence of the opponent's actions.' The theorem is stated as an almost-sure convergence with respect to the forecaster's randomization; this is not a deterministic pathwise statement. Please rephrase.
  2. [Algorithm 1, step 2] The line 'Play a_t ∼ p⋆(z_t)' is misleading because p⋆(z_t) is a point mass at level(α,z_t). Write 'Play a_t = level(α,z_t)' or state that p⋆ is deterministic.
  3. [Lemma 5] Typo: 'response-fonction' should be 'response function.' Also, the partition R_j uses strict/non-strict inequalities; consider stating the boundary convention explicitly.
  4. [§3.1, Theorem 2] Theorem 2 is an informal restatement of Zaffran et al. but is labeled as a theorem with an unspecified constant λ and 'strong assumptions.' It would be clearer to call it an informal reformulation, as done in the text before it.
  5. [§2.1, paragraph after Fact 1] The notation L(C_a) is used both for the length of the conformal set and for the upper bound L on residuals; this is a minor notational overloading that could confuse readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BOACI's guarantee is a conditional theorem built on externally published approachability/calibration results; the only self-citation is background, and the efficiency benchmark is an oracle quantity not fitted to the algorithm.

full rationale

The derivation chain is self-contained in the crucial sense that all load-bearing ingredients are external, published results (Blackwell 1956; Bernstein et al. 2014; Foster 1999) or are proven in the paper from those ingredients. The central Theorem 5 applies Bernstein et al.'s opportunistic approachability theorem to the finite ACI game. The paper checks the theorem's hypotheses: p* is piecewise continuous (Lemma 5); the closed convex image of Q under the goal function is contained in S_Q (Lemma 6); calibration transfers Q-restrictedness from opponent mixed actions to forecasts (Lemma 11). No equation defining S_Q or the efficiency objective is fitted to the algorithm's output; max_{q in Q} phi+_q(alpha) is an oracle benchmark computed from Q, which is unknown to BOACI. The only self-citation, Mannor & Stoltz (2010), appears in related work as background on calibration and is not used in the proof of Theorem 5. The manuscript explicitly flags that exact calibration over Delta(B) may be computationally heavy and defers epsilon-calibrated analysis to future work (Remark 5; Conclusion). That is an implementation/conditionality gap, not circularity: the theorem is conditional on a calibrated forecaster satisfying Definition 3, and no implementable instance is claimed. Thus no step reduces by construction to its inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

Central claims rest on standard external results plus domain assumptions about bounded scores, an unknown restriction set Q, and the availability of an exact calibrated forecaster. No fitted free parameters are introduced beyond the user-chosen target level alpha and the assumed bound L.

free parameters (1)
  • L (upper bound on residual scores) = assumed known; practical heuristic L = 2 max_{t in D_calib} s_t
    The finite-game payoffs, target sets, and impossibility results all scale with L; the paper assumes a known constant L in Section 2.1 rather than fitting it.
assumptions (5)
  • domain assumption Calibration scores are almost surely distinct and bounded by a known constant L, and conformal sets are based on absolute residuals.
    Section 2.1; the finite game and payoffs depend on this.
  • domain assumption The opponent's play is statistically Q-restricted for some unknown convex Q: (1/T) sum d(q_t, Q) -> 0 a.s.
    Definition 1; the efficiency guarantee (20) holds only under this.
  • domain assumption There exists a calibrated forecaster over Delta(B) satisfying Definition 3 for every Borel set G.
    Algorithm 1 and Theorem 5 rely on it; the paper defers epsilon-calibration to future work.
  • standard math Blackwell's approachability theorem and the opportunistic approachability theorem of Bernstein et al. (2014) are valid.
    Theorems 1 and 4 are used as black boxes.
  • domain assumption Under exchangeable scores, b_t is uniform on B (Lemma 2).
    Used for Corollaries 1-2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Conformal Inference through the Lens of Blackwell Approachability." pith.science (2026). https://pith.science/paper/SNUK5TXJ

@misc{pith2026251015824,
  author       = {Pith},
  title        = {Pith review of: Adaptive Conformal Inference through the Lens of Blackwell Approachability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNUK5TXJ}},
  note         = {Machine review of arXiv:2510.15824}
}
read the original abstract

This article considers an online version of conformal inference, called adaptive conformal inference [ACI] and introduced by Gibbs and Cand\`es (2021): prediction sets are issued sequentially, after observing features and before the outcomes are revealed. These sets are evaluated both in terms of validity (the fraction of rounds where the outcome was lying in the prediction set) and efficiency (the average lengths of the prediction sets). The two criteria point to different directions (validity favors larger sets). We also target a wide range of scenarios, with exchangeable data and arbitrary data (lack of any stochastic guarantees) as two extremes. A series of existing strategies for ACI typically guarantee that empirical coverage converges to the desired level for arbitrary sequences, but they generally lack simultaneous efficiency guarantees. To provide a unified study, we first formulate ACI as a repeated two-player game with finite action sets and vector-valued payoffs encoding validity and efficiency. Building on this reformulation, we introduce a strategy based on Blackwell approachability and on its opportunistic extension by Bernstein et al. (2014) that ensures validity while adapting the efficiency of the prediction intervals to the underlying degree of stochasticity of the opponent player. The resulting guarantee is "best of many worlds": it recovers the relevant efficiency guarantees in exchangeable and adversarial settings, and provides guarantees in intermediate settings that arise in typical applications such as the forecasting of time series.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 3 linked inside Pith

  1. [1]

    Conformal pid control for time series prediction

    Anastasios Angelopoulos, Emmanuel Candes, and Ryan J Tibshirani. Conformal pid control for time series prediction. Advances in neural information processing systems, 36: 0 23047--23074, 2023

  2. [2]

    Gradient equilibrium in online learning: Theory and applications, 2025

    Anastasios N Angelopoulos, Michael I Jordan, and Ryan J Tibshirani. Gradient equilibrium in online learning: Theory and applications, 2025. Preprint, arXiv:2501.08330

  3. [3]

    Online conformal prediction with decaying step sizes

    Anastasios Nikolas Angelopoulos, Rina Barber, and Stephen Bates. Online conformal prediction with decaying step sizes. In International Conference on Machine Learning, pages 1616--1630. PMLR, 2024

  4. [4]

    Conformal prediction beyond exchangeability

    Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 0 (2): 0 816--845, 2023

  5. [5]

    Response-based approachability and its application to generalized no-regret algorithms, 2013

    Andrey Bernstein and Nahum Shimkin. Response-based approachability and its application to generalized no-regret algorithms, 2013

  6. [6]

    Opportunistic approachability and generalized no-regret problems

    Andrey Bernstein, Shie Mannor, and Nahum Shimkin. Opportunistic approachability and generalized no-regret problems. Mathematics of Operations Research, 39 0 (4): 0 1057--1083, 2014

  7. [7]

    Improved online conformal prediction via strongly adaptive online learning

    Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning, pages 2337--2363. PMLR, 2023

  8. [8]

    An analog of the minimax theorem for vector payoffs

    David Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Mathematics, 6 0 (1): 0 1--8, 1956

Show all 23 references
  1. [9]

    A conformal martingales approach for recurrent concept drift

    Charalambos Eliades and Harris Papadopoulos. A conformal martingales approach for recurrent concept drift. In Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2025), pages 706--724. PMLR, 2025

  2. [10]

    A proof of calibration via blackwell's approachability theorem

    Dean P Foster. A proof of calibration via blackwell's approachability theorem. Games and Economic Behavior, 29 0 (1-2): 0 73--78, 1999

  3. [11]

    Adaptive conformal inference under distribution shift

    Isaac Gibbs and Emmanuel J Cand \`e s. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 0 1660--1672, 2021

  4. [12]

    Conformal inference for online prediction with arbitrary distribution shifts

    Isaac Gibbs and Emmanuel J Cand \`e s. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25 0 (162): 0 1--36, 2024

  5. [13]

    A general class of adaptive strategies

    Sergiu Hart and Andreu Mas-Colell. A general class of adaptive strategies. Journal of Economic Theory, 98 0 (1): 0 26--54, 2001

  6. [14]

    Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman

    J. Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018

  7. [15]

    A geometric proof of calibration

    Shie Mannor and Gilles Stoltz. A geometric proof of calibration. Mathematics of Operations Research, 35 0 (4): 0 721--727, 2010

  8. [16]

    Approachability in unknown games: Online learning meets multi-objective optimization

    Shie Mannor, Vianney Perchet, and Gilles Stoltz. Approachability in unknown games: Online learning meets multi-objective optimization. In Conference on Learning Theory, pages 339--355. PMLR, 2014

  9. [17]

    Calibration and internal no-regret with partial monitoring, 2010

    Vianney Perchet. Calibration and internal no-regret with partial monitoring, 2010. Preprint arXiv:1006.1746

  10. [18]

    The relationship between no-regret learning and online conformal prediction, 2025

    Ramya Ramalingam, Shayan Kiyani, and Aaron Roth. The relationship between no-regret learning and online conformal prediction, 2025. Preprint arXiv:2502.10947

  11. [19]

    Online conformal prediction with efficiency guarantees, 2025

    Vaidehi Srinivas. Online conformal prediction with efficiency guarantees, 2025. Preprint arXiv:2507.02496

  12. [20]

    o fstr \

    Johan Hallberg Szabadv \'a ry and Tuwe L \"o fstr \"o m. Beyond conformal predictors: Adaptive conformal inference with confidence predictors. Pattern Recognition, page 111999, 2025

  13. [21]

    Algorithmic Learning in a Random World

    Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005

  14. [22]

    Uniform conditional variability ordering of probability distributions

    Ward Whitt. Uniform conditional variability ordering of probability distributions. Journal of Applied Probability, 22 0 (3): 0 619--633, 1985

  15. [23]

    Adaptive conformal predictions for time series

    Margaux Zaffran, Olivier F \'e ron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pages 25834--25866. PMLR, 2022

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.