REVIEW 3 major objections 5 minor 23 references
Adaptive Conformal Inference through the Lens of Blackwell Approachability
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Adaptive conformal inference is recast as a repeated two-player vector-valued game, and a calibration-based opportunistic approachability algorithm, BOACI, is shown to achieve almost surely asymptotic coverage of at least 1−α together with
desk verdict Fresh game-theoretic reformulation and a new impossibility result, but the headline 'best of many worlds' guarantee is conditional on an unimplemented exact calibration oracle. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a finite two-player vector-valued game: the learner chooses a discrete miscoverage level a ∈ A (equivalently, a conformal interval), the opponent chooses b ∈ B, the smallest discretized threshold below which the realized outcome would be covered, and the payoff is m(a,b) = (1_{b≤a}, L(C_a)) ∈ [0,1]×[0,2L], with L(C_a) the interval length. Two mechanisms do the work: (i) Blackwell's approachability theorem, whose dual condition characterizes exactly which convex sets of average payoffs are attainable, and (ii) an opportunistic variant based on a calibrated forecaster that outputs at each round a forecast z_t ∈ Δ(B) of the opponent's next action; the learner responds with
What would settle it
Run BOACI on exchangeable, i.i.d. test scores using a finite-grid ε-calibrated forecaster (for example, histogram forecasts over a coarse grid of Δ(B)) for a long horizon: if empirical coverage is maintained at 1−α but the average interval length converges to a value strictly above the classical conformal length L(C_α)—rather than to L(C_α) up to grid rounding—then the practical version does not inherit the claimed guarantee; a success, conversely, would be the first evidence that ε-calibration suffices.
Extended reading notes
Core claim
On the paper's own terms, the central result (Theorem 5) is: if the opponent's play is statistically Q-restricted—that is, the empirical distribution of the opponent's mixed actions converges in average distance to a convex set Q of distributions over B—then the BOACI strategy satisfies lim_{T→∞} d(m̄_T, S_Q) = 0 almost surely, where m̄_T is the average of the payoff vectors (indicator of miscoverage, interval length) and S_Q is the set of pairs with miscoverage ≤ α and length ≤ max_{q∈Q} φ^+_q(α). Two consequences follow immediately: the limsup of average miscoverage is at most α (coverage is at least 1−α), and the limsup of average interval length is at most the oracle bound max_{q∈Q} φ^+_
Load-bearing premise
The guarantee rests on the existence of a forecaster that is calibrated over every Borel subset of the simplex of opponent actions; the paper provides no implementable instance of such a perfect calibration oracle, and the computationally feasible ε-calibrated version is only suggested (in Remark 5 and the conclusion), not proven to preserve the theorem's conclusion.
Editorial extensions
If this is right
- Coverage at level 1−α is guaranteed almost surely for any data sequence, without any stochastic assumption on the outcomes; efficiency is then added wherever the data exhibit enough regularity.
- In exchangeable settings, BOACI beats standard ACI on efficiency: it converges to the classical conformal interval length L(C_α), eliminating the positive step-size penalty that standard ACI pays.
- For regime-switching or slowly drifting score distributions, the average interval length converges to a bound that is strictly better than the trivial 2L when the regimes are informative, and the guarantee degrades gracefully to the adversarial vacuous bound as the regime set expands.
- The same geometry yields impossibility results: in fully adversarial settings no strategy can simultaneously keep validity and beat the trivial length bound, and two natural regret-based efficiency objectives (pinball-loss regret and best-valid-constant regret) are shown to be unattainable once the finite grid of conformal scores is taken into account.
Reading between the lines
- If the ε-calibrated forecaster variant can be proven to satisfy a finite version of the approachability guarantee, BOACI becomes immediately deployable; the natural next test is to run it on real time-series benchmarks and compare its average interval length against ACI and its variants under drift.
- Because the formulation is built on an abstract (A, B, m) game, the same opportunistic-approachability scheme may transfer to other sequential set-prediction problems—quantile tracking, classification with abstention, or distribution prediction—where a binary validity event and a scalar cost compete.
- The statistical Q-restriction is an unusually weak assumption (only average convergence of the opponent's empirical play to Q), so the theorem likely holds for a broad class of nonstationary processes; an explicit rate of convergence of d(m̄_T, S_Q) in terms of the calibration convergence would sharpen this and is not provided.
- A practical open question suggested by the paper: how to choose Q adaptively—the guarantee's sharpness is bounded by the tightest Q one can posit, and the convex-hull examples show that learning Q online would interpolate smoothly between the exchangeable and adversarial extremes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies adaptive conformal inference (ACI) through Blackwell approachability. It first reformulates the sequential conformal prediction problem as a repeated finite two-player game in which the learner chooses a discretized miscoverage level a_t and the opponent chooses the threshold b_t, with vector payoff encoding miscoverage and interval length. It then introduces the notion of a statistically Q-restricted opponent (Definition 1), defines the efficiency benchmark max_{q∈Q} φ⁺_q(α) and the target set S_Q (Eq. 13), and proposes BOACI (Algorithm 1), a calibration-based algorithm using the response function p*(q)=δ_{level(α,q)} (Eq. 15). The main theoretical result (Theorem 5) states that if the opponent is Q-restricted, BOACI's average payoff converges to S_Q, yielding the validity bound (19) and efficiency bound (20); Corollary 2 recovers the exchangeable guarantee. The proof checks that p* is piecewise continuous (Lemma 5) and that the closed convex image of Q under the goal function is contained in S_Q (Lemma 6), then applies the opportunistic approachability theorem of Bernstein et al. (2014).
Significance. If the main theorem is established for a practically available subroutine, the paper would provide a genuinely useful 'best of many worlds' result: adversarial validity plus efficiency that adapts to the degree of stochasticity of the data sequence. The game-theoretic reformulation is elegant and connects ACI to a mature body of approachability theory. The authors are careful to check the two hypotheses needed to instantiate Bernstein et al.'s general theorem, and the paper ships a self-contained proof sketch in Appendix C. However, the central guarantee is conditional on the existence of a calibrated forecaster satisfying the very strong Definition 3 over the entire simplex Δ(B). The paper neither constructs such a forecaster nor analyzes the ε-calibrated version it recommends in practice, and it explicitly defers that analysis to future work. This is a substantial gap for a paper whose abstract promises a strategy 'that ensures validity while adapting the efficiency' in real settings. There is also an unexamined timing/information issue: in the conformal game the opponent observes a_t before choosing b_t, and a_t is a deterministic function of the forecast z_t, while the standard
major comments (3)
- [§4.2, Algorithm 1; Remark 5; Conclusion] Theorem 5 and Corollary 2 are proven only under Definition 3: an exact calibrated forecaster over Δ(B) that satisfies the calibration condition for every Borel set G. The paper gives no implementation of such a forecaster and does not verify that a computable ε-calibrated forecaster inherits the guarantees. Remark 5 and the Conclusion explicitly defer the ε-calibrated analysis to future work. Therefore the stated guarantees (19)–(20) are not established for any implementable instance of BOACI; the theorem is a conditional existence result. This is load-bearing because the abstract and contributions present BOACI as an algorithm that 'ensures validity while adapting the efficiency' in practical scenarios. Please either provide an ε-calibrated version with a formal guarantee, or clearly scope the claims and state the result as oracle-based.
- [Definition 3; Algorithm 1; Appendix C, first inequality] There is a protocol mismatch. In the conformal game the opponent observes the learner's action a_t before choosing b_t, and in Algorithm 1 a_t is the deterministic function level(α,z_t) of the forecast z_t. The calibration condition in Definition 3 is required to hold for every strategy of the opponent, but the standard calibration theorems (Foster; Bernstein et al.) assume that the outcome b_t is independent of the realized forecast z_t given the forecaster's mixed distribution. Here the opponent can condition b_t on a_t, i.e. on a function of z_t, so it is not established that a calibrated forecaster satisfying Definition 3 exists in this game. The proof of Theorem 5 uses calibration to replace b_t by z_t in the average payoff; that step is unjustified without a formal protocol and an existence proof under this feedback structure. Please specify the timing precisely and either prove th
- [Definition 1; §2.2; §C.2] Statistical Q-restriction is defined for a sequence (q_t) of mixed actions 'output by the opponent,' but the game in §2.2 is defined with pure actions b_t ∈ B. It is never stated what q_t is (conditional distribution of b_t given history? Some latent randomization?) or whether it is observed. This matters because Lemmas 9–11 and the proof of Theorem 5 use the joint distribution of (q_t,z_t) and the identity E[q|z]=z for calibrated forecasts. Without a clear definition of q_t, it is impossible to verify whether the definitions are consistent with the actual implementation, where only b_t is observed. Please clarify what q_t is and how it relates to the pure actions and to the calibrated forecasts.
minor comments (5)
- [§4.2, after Theorem 5] The text says 'these results hold pathwise, i.e., for any possible sequence of the opponent's actions.' The theorem is stated as an almost-sure convergence with respect to the forecaster's randomization; this is not a deterministic pathwise statement. Please rephrase.
- [Algorithm 1, step 2] The line 'Play a_t ∼ p⋆(z_t)' is misleading because p⋆(z_t) is a point mass at level(α,z_t). Write 'Play a_t = level(α,z_t)' or state that p⋆ is deterministic.
- [Lemma 5] Typo: 'response-fonction' should be 'response function.' Also, the partition R_j uses strict/non-strict inequalities; consider stating the boundary convention explicitly.
- [§3.1, Theorem 2] Theorem 2 is an informal restatement of Zaffran et al. but is labeled as a theorem with an unspecified constant λ and 'strong assumptions.' It would be clearer to call it an informal reformulation, as done in the text before it.
- [§2.1, paragraph after Fact 1] The notation L(C_a) is used both for the length of the conformal set and for the upper bound L on residuals; this is a minor notational overloading that could confuse readers.
Circularity Check
No significant circularity: BOACI's guarantee is a conditional theorem built on externally published approachability/calibration results; the only self-citation is background, and the efficiency benchmark is an oracle quantity not fitted to the algorithm.
full rationale
The derivation chain is self-contained in the crucial sense that all load-bearing ingredients are external, published results (Blackwell 1956; Bernstein et al. 2014; Foster 1999) or are proven in the paper from those ingredients. The central Theorem 5 applies Bernstein et al.'s opportunistic approachability theorem to the finite ACI game. The paper checks the theorem's hypotheses: p* is piecewise continuous (Lemma 5); the closed convex image of Q under the goal function is contained in S_Q (Lemma 6); calibration transfers Q-restrictedness from opponent mixed actions to forecasts (Lemma 11). No equation defining S_Q or the efficiency objective is fitted to the algorithm's output; max_{q in Q} phi+_q(alpha) is an oracle benchmark computed from Q, which is unknown to BOACI. The only self-citation, Mannor & Stoltz (2010), appears in related work as background on calibration and is not used in the proof of Theorem 5. The manuscript explicitly flags that exact calibration over Delta(B) may be computationally heavy and defers epsilon-calibrated analysis to future work (Remark 5; Conclusion). That is an implementation/conditionality gap, not circularity: the theorem is conditional on a calibrated forecaster satisfying Definition 3, and no implementable instance is claimed. Thus no step reduces by construction to its inputs.
Assumptions & free parameters
free parameters (1)
- L (upper bound on residual scores) =
assumed known; practical heuristic L = 2 max_{t in D_calib} s_t
assumptions (5)
- domain assumption Calibration scores are almost surely distinct and bounded by a known constant L, and conformal sets are based on absolute residuals.
- domain assumption The opponent's play is statistically Q-restricted for some unknown convex Q: (1/T) sum d(q_t, Q) -> 0 a.s.
- domain assumption There exists a calibrated forecaster over Delta(B) satisfying Definition 3 for every Borel set G.
- standard math Blackwell's approachability theorem and the opportunistic approachability theorem of Bernstein et al. (2014) are valid.
- domain assumption Under exchangeable scores, b_t is uniform on B (Lemma 2).
Cite this review
Pith. "Pith review of Adaptive Conformal Inference through the Lens of Blackwell Approachability." pith.science (2026). https://pith.science/paper/SNUK5TXJ
@misc{pith2026251015824,
author = {Pith},
title = {Pith review of: Adaptive Conformal Inference through the Lens of Blackwell Approachability},
year = {2026},
howpublished = {\url{https://pith.science/paper/SNUK5TXJ}},
note = {Machine review of arXiv:2510.15824}
}
read the original abstract
This article considers an online version of conformal inference, called adaptive conformal inference [ACI] and introduced by Gibbs and Cand\`es (2021): prediction sets are issued sequentially, after observing features and before the outcomes are revealed. These sets are evaluated both in terms of validity (the fraction of rounds where the outcome was lying in the prediction set) and efficiency (the average lengths of the prediction sets). The two criteria point to different directions (validity favors larger sets). We also target a wide range of scenarios, with exchangeable data and arbitrary data (lack of any stochastic guarantees) as two extremes. A series of existing strategies for ACI typically guarantee that empirical coverage converges to the desired level for arbitrary sequences, but they generally lack simultaneous efficiency guarantees. To provide a unified study, we first formulate ACI as a repeated two-player game with finite action sets and vector-valued payoffs encoding validity and efficiency. Building on this reformulation, we introduce a strategy based on Blackwell approachability and on its opportunistic extension by Bernstein et al. (2014) that ensures validity while adapting the efficiency of the prediction intervals to the underlying degree of stochasticity of the opponent player. The resulting guarantee is "best of many worlds": it recovers the relevant efficiency guarantees in exchangeable and adversarial settings, and provides guarantees in intermediate settings that arise in typical applications such as the forecasting of time series.
Reference graph
Works this paper leans on
-
[1]
Conformal pid control for time series prediction
Anastasios Angelopoulos, Emmanuel Candes, and Ryan J Tibshirani. Conformal pid control for time series prediction. Advances in neural information processing systems, 36: 0 23047--23074, 2023
2023
-
[2]
Gradient equilibrium in online learning: Theory and applications, 2025
Anastasios N Angelopoulos, Michael I Jordan, and Ryan J Tibshirani. Gradient equilibrium in online learning: Theory and applications, 2025. Preprint, arXiv:2501.08330
arXiv 2025
-
[3]
Online conformal prediction with decaying step sizes
Anastasios Nikolas Angelopoulos, Rina Barber, and Stephen Bates. Online conformal prediction with decaying step sizes. In International Conference on Machine Learning, pages 1616--1630. PMLR, 2024
2024
-
[4]
Conformal prediction beyond exchangeability
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 0 (2): 0 816--845, 2023
2023
-
[5]
Response-based approachability and its application to generalized no-regret algorithms, 2013
Andrey Bernstein and Nahum Shimkin. Response-based approachability and its application to generalized no-regret algorithms, 2013
2013
-
[6]
Opportunistic approachability and generalized no-regret problems
Andrey Bernstein, Shie Mannor, and Nahum Shimkin. Opportunistic approachability and generalized no-regret problems. Mathematics of Operations Research, 39 0 (4): 0 1057--1083, 2014
2014
-
[7]
Improved online conformal prediction via strongly adaptive online learning
Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning, pages 2337--2363. PMLR, 2023
2023
-
[8]
An analog of the minimax theorem for vector payoffs
David Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Mathematics, 6 0 (1): 0 1--8, 1956
1956
Show all 23 references
-
[9]
A conformal martingales approach for recurrent concept drift
Charalambos Eliades and Harris Papadopoulos. A conformal martingales approach for recurrent concept drift. In Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2025), pages 706--724. PMLR, 2025
2025
-
[10]
A proof of calibration via blackwell's approachability theorem
Dean P Foster. A proof of calibration via blackwell's approachability theorem. Games and Economic Behavior, 29 0 (1-2): 0 73--78, 1999
1999
-
[11]
Adaptive conformal inference under distribution shift
Isaac Gibbs and Emmanuel J Cand \`e s. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 0 1660--1672, 2021
2021
-
[12]
Conformal inference for online prediction with arbitrary distribution shifts
Isaac Gibbs and Emmanuel J Cand \`e s. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25 0 (162): 0 1--36, 2024
2024
-
[13]
A general class of adaptive strategies
Sergiu Hart and Andreu Mas-Colell. A general class of adaptive strategies. Journal of Economic Theory, 98 0 (1): 0 26--54, 2001
2001
-
[14]
Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman
J. Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018
2018
-
[15]
A geometric proof of calibration
Shie Mannor and Gilles Stoltz. A geometric proof of calibration. Mathematics of Operations Research, 35 0 (4): 0 721--727, 2010
2010
-
[16]
Approachability in unknown games: Online learning meets multi-objective optimization
Shie Mannor, Vianney Perchet, and Gilles Stoltz. Approachability in unknown games: Online learning meets multi-objective optimization. In Conference on Learning Theory, pages 339--355. PMLR, 2014
2014
-
[17]
Calibration and internal no-regret with partial monitoring, 2010
Vianney Perchet. Calibration and internal no-regret with partial monitoring, 2010. Preprint arXiv:1006.1746
2010 arXiv
-
[18]
The relationship between no-regret learning and online conformal prediction, 2025
Ramya Ramalingam, Shayan Kiyani, and Aaron Roth. The relationship between no-regret learning and online conformal prediction, 2025. Preprint arXiv:2502.10947
2025 arXiv
-
[19]
Online conformal prediction with efficiency guarantees, 2025
Vaidehi Srinivas. Online conformal prediction with efficiency guarantees, 2025. Preprint arXiv:2507.02496
2025
-
[20]
o fstr \
Johan Hallberg Szabadv \'a ry and Tuwe L \"o fstr \"o m. Beyond conformal predictors: Adaptive conformal inference with confidence predictors. Pattern Recognition, page 111999, 2025
2025
-
[21]
Algorithmic Learning in a Random World
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005
2005
-
[22]
Uniform conditional variability ordering of probability distributions
Ward Whitt. Uniform conditional variability ordering of probability distributions. Journal of Applied Probability, 22 0 (3): 0 619--633, 1985
1985
-
[23]
Adaptive conformal predictions for time series
Margaux Zaffran, Olivier F \'e ron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pages 25834--25866. PMLR, 2022
2022
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.