Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Improving Coverage in Combined Prediction Sets with Weighted p-values

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Weighted p-value aggregation interpolates combined prediction-set coverage between the classical 1−2α worst case and the single-model 1−α guarantee, and a learned scaling factor preserves finite-sample validity when the weights depend on…

desk verdict Useful extension of weighted p-value aggregation to data-dependent weights, but the main finite-sample proof has a gap that needs fixing before the claims as written are taken at face value. read the letter →

arxiv 2505.11785 v3 pith:BLPHAY4J submitted 2025-05-17 cs.LG cs.AIstat.ML

Gina Wong , Drew Prinster , Suchi Saria , Rama Chellappa , Anqi Liu This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords conformalpredictionweightedp-valuessetaggregationdata-dependentweightsmixture-of-expertsfinite-samplecoverageguaranteeworst-slicelocalvalidity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to combine several conformal prediction sets, each valid at level $1-\alpha$, into one aggregate set without losing the whole guarantee. The standard symmetric answer costs a factor of two, giving $1-2\alpha$ worst-case coverage. The paper shows that weighting the underlying p-values rather than the sets interpolates the guarantee between $1-2\alpha$ and $1-\alpha$, depending on how concentrated the weights are. For the harder case where the weights themselves are learned from data, as in mixture-of-experts routing, it introduces a scaling correction computed from an auxiliary merging set that restores a finite-sample coverage guarantee. Experiments on real regression and classification data show that the resulting sets keep coverage on hard data slices and demographic subgroups where plain split conformal undercovers.

What carries the argument

The load-bearing object is the weighted-average p-value function $\bar p(x,y)=\sum_k w_k \hat p_k(x,y)$ built from per-model conformal p-values, together with the p-value averaging theorem the framework is built on: for fixed weights, scaling the weighted average by $A_v=\min(1/v,2)$ turns it into a valid p-variable. For data-dependent weights this scaling no longer applies, and the argument is carried by the correction factor $m^*$, the smallest scalar making $m^* P_{\mathrm{all}}$ satisfy the p-variable inequality, computed as the supremum ratio of the distribution function of $P_{\mathrm{all}}$ to its argument. Its empirical version $\hat m^*$ is the maximum such ratio over an auxiliary merging set, and the Dvoretzky-Kiefer-Wolfowitz inequality converts the estimation error into the finite-sample slack $\epsilon+\delta$ in Proposition 5.2. The mechanism matters because a linear scaling preserves the relative proportions of the learned weights, which a rank-based CDF transformation would destroy, while still repairing the validity condition that the weighted average alone violates.

What would settle it

Run the scaled weighted-aggregation procedure at $\alpha=0.1$ with an exchangeable merging set of 40 points and $\delta=0.1$ over many trials; Proposition 5.2 caps expected miscoverage at about 0.39, so observing miscoverage consistently above that level on any dataset would falsify the guarantee's assumptions, whereas much lower miscoverage would show the bound is pessimistic. Separately, the practical adaptivity claim is falsifiable by measuring worst-slice coverage on a large held-out test set: if the weighted-aggregation set undercovers the worst slab at least as much as split conformal on any of the reported datasets, the central empirical claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a prediction set formed by thresholding the weighted average $\bar p(x,y)=\sum_k v_k \hat p_k(x,y)$ of per-model conformal p-values at level $\alpha$ has coverage at least $1-\min(1/v,2)\alpha$, where $v$ is the largest weight. When one set dominates, with $v>1/2$, this beats the classical $1-2\alpha$ aggregation bound, and at $v=1$ it recovers the individual set's $1-\alpha$ guarantee. For data-dependent weights $W(x)$, the same weighted average is no longer a valid p-variable, but multiplying it by the smallest scalar $m^* = \sup_\delta F_{P_{\mathrm{all}}}(\delta)/\delta$ that forces $P\{m^* P_{\mathrm{all}} \le \alpha\}\le \alpha$ for all $\alpha$ restores validity, giving a $1-\alpha$ prediction set in the infinite-sample limit. Estimating $m^*$ from the empirical CDF of $P_{\mathrm{all}}$ on a merging set yields the finite-sample coverage guarantee $1-(\alpha+\epsilon+\delta)$ with worst-case statistical tolerance $\epsilon=\sqrt{\log(2/\delta)/(2|S_{\mathrm{merge}}|)}$. Applied to mixture-of-experts, where router weights depend on the input, the method delivers prediction sets whose coverage follows the dominant expert locally, with improved worst-slice and per-group coverage relative to split conformal and CQR baselines.

Load-bearing premise

The formal guarantee for data-dependent weights is only $1-(\alpha+\epsilon+\delta)$ where $\epsilon$ is a worst-case statistical error that shrinks as the merging set grows, and with the merging sets used in the main experiments (about 40 points) the proved bound is roughly $1-(\alpha+0.29)$, so the paper's adaptive-coverage results depend on the unstated assumption that the real distribution of the aggregated p-values is better behaved than that worst-case bound.

Editorial extensions

If this is right

  • With data-independent weights, any weighted aggregation of conformal sets has coverage at least $1-\min(1/v,2)\alpha$, so a single dominant set with $v\ge 1/2$ already improves on the classical $1-2\alpha$ guarantee.
  • Because the aggregation operates on p-values after calibration, it can be layered on top of locally adaptive score functions such as CQR, so the two sources of adaptivity compose rather than compete.
  • In the mixture-of-experts setting, the finite-sample guarantee makes learned router weights a safe input to set aggregation, yielding sets whose coverage tracks the expert the router most trusts at each input.
  • The merging-set size controls a concrete trade-off: larger $|S_{\mathrm{merge}}|$ shrinks the worst-case tolerance $\epsilon$, and the paper's ablations show empirical coverage approaching the nominal level, with roughly 160 points sufficing for about 3% overcoverage.
  • Across the paper's datasets, weighted aggregation narrows the gap between marginal and worst-slice coverage compared with split conformal, and in demographic subgroup analyses it meets coverage for groups that split conformal undercovers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same linear-repair idea would extend to data-dependent weights in other aggregation rules, such as majority vote or rank-based averaging, though those would sacrifice the weight-proportion semantics the authors rely on.
  • Because the guarantee marginalizes over the merging set, the adaptive-coverage behavior is an empirical property rather than a distribution-free one; a natural stress test is measuring worst-slice coverage as $|S_{\mathrm{merge}}|$ shrinks well below the 40-point range used in the main experiments, where the proved bound is weak ($\epsilon\approx 0.19$ at $\delta=0.1$).
  • Since the linear scaling preserves the router's weight proportions, the aggregate prediction set can be read as expressing the router's own judgment about which expert is responsible for a test point, which suggests using the correction factor itself as a diagnostic for router confidence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for aggregating K conformal prediction sets by taking a weighted average of the associated p-value functions. For data-independent weights, Proposition 4.1 applies a result of Vovk and Wang to show that the aggregated set has coverage at least 1 - min(1/v, 2)α, where v is the largest weight. For data-dependent weights, the paper defines a correction factor m* as the minimal scaling that makes the weighted average a p-variable, and proposes to estimate m* from a merging set using an empirical CDF (Proposition 5.2), yielding a finite-sample coverage guarantee of 1 - (α + ε + δ). The method is applied to mixture-of-experts models, where routing weights are used to aggregate expert p-value functions, and experiments on synthetic and UCI data report improved worst-slice coverage relative to split conformal and adaptive baselines. The central theoretical contribution is the finite-sample guarantee for data-dependent weights; the paper also provides detailed proofs in Appendices A.1-A.5 and extensive ablations in Appendices B-E.

Significance. If the finite-sample guarantee is established, the paper makes a useful contribution: it extends asymmetric prediction-set aggregation from fixed weights to data-dependent weights, connects the Vovk-Wang averaging result to conformal prediction, and demonstrates a practically relevant application to mixture-of-experts with improved local validity. The treatment of the data-independent case is clean and the experimental study is thorough, including comparisons to conservative CQR, localized conformal prediction, and demographic subgroup analyses. The paper also gives detailed derivations of the equivalence in (7) and of the empirical correction factor, which are helpful. However, the central finite-sample result (Proposition 5.2) currently has a genuine proof gap in the application of the DKW inequality, and the assumptions connecting the marginal quantity m* to the empirical estimator bm* are not stated precisely. The gap is repairable, but it affects the paper's main theoretical claim and therefore requires a major revision.

major comments (4)
  1. [Appendix A.3, Eqs. (14)-(16)] The DKW step in the proof of Proposition 5.2 is not justified as written. Conditional on the calibration set S_cal, the merging samples P_i^all are i.i.d. from the conditional law F_{S_cal} of P_all for an exchangeable point; unconditionally they are dependent, and the marginal CDF F_Pall is a mixture over S_cal. DKW controls sup_x |\hat F(x) - F_{S_cal}(x)|, not sup_x |\hat F(x) - F_Pall(x)|, and these two quantities can differ by an amount that does not vanish with |S_merge| (for example, K=1 with |S_cal|=1). Equation (14) therefore cannot use F_Pall for a probability that is conditional on S_merge, and the key inequality (15) is unsupported. The theorem is repairable by conditioning on S_cal throughout and defining the target CDF as the conditional law, but the published proof of the central finite-sample claim has a real gap.
  2. [Section 5, Propositions 5.1 and 5.2] The object m* in Proposition 5.1 is defined through the marginal CDF F_Pall, whereas the empirical bm* in Proposition 5.2 estimates the scaling appropriate to the conditional law of P_all given S_cal. These are different quantities in general, so the two propositions are not connected as stated. A repaired statement should either define m* conditionally on S_cal and then marginalize over S_cal, or prove that bm* estimates the marginal m*; the first route is the natural one and preserves the coverage claim.
  3. [Section 5, Proposition 5.2 assumptions] The DKW argument also requires the weights W^(i) used on the merging set to be independent across i and independent of the test point conditional on S_cal, for example when they are computed from a training split disjoint from S_merge. As stated, W may depend on {(X_i,Y_i)}_{i in [n]} union {X_{n+1}}, which includes S_merge, and then the merging samples are not i.i.d. This assumption needs to be made explicit; the MoE application satisfies it only because the routing network is trained on S_train, which should be stated in the theorem or in Section 6.
  4. [Section 6 and Appendix E.1] The main UCI experiments set |S_merge|=40, so with delta=0.1 the DKW tolerance is epsilon approx 0.19 and the guarantee proved in Proposition 5.2 is only about 1-(alpha+0.29). The paper's practical conclusions about adaptive coverage at this merging size therefore rest on empirical behavior that is not captured by the finite-sample theorem. I recommend either using a larger merging set in the headline experiments, reporting the bound alongside the empirical coverage, or explicitly qualifying the practical claims as heuristic.
minor comments (5)
  1. [Section 5, first paragraph] The text says P1,...,PK are in U, 'the set of all uniformly distributed random variables', which conflicts with the definition of a p-variable in (1) and with the fact that split conformal p-values are generally conservative rather than exactly uniform; please clarify that U denotes p-variables or state the precise-p-variable assumption.
  2. [Algorithm 1] Algorithm 1 presents the coverage guarantee for the m* correction, but the experiments use the m† and m‡ corrections from equations (10) and (11); the algorithm should specify which correction factor is being computed.
  3. [Equation (7) and Appendix A.4] The equality m* = sup_{delta>0} F_Pall(delta)/delta requires that F_Pall has no mass at 0; this holds for conformal p-values, but the condition should be stated before the equivalence is used.
  4. [Figure 3 legend] The legend contains an incomplete label 'WA targeted (0, ]'; the intended significance-level range should be written out.
  5. [Appendix A.3] The Dvoretzky-Kiefer-Wolfowitz inequality is used without a citation; please add the standard reference for the DKW bound in the proof of Proposition 5.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the finite-sample weighted p-value guarantee rests on external Vovk–Wang theory and a DKW bound, not on self-citation or a fitted input renamed as a prediction.

full rationale

I walked the derivation chain in Sections 4, 5, and Appendix A. Proposition 4.1 is a direct application of Vovk and Wang's Proposition 9, an external theorem, and it is not supported by any self-citation. Proposition 5.1 is definitional in the sense that m* is defined as the minimal scaling that makes m*Pall a p-variable, so the stated 1-alpha coverage follows immediately from the definition of a p-variable; however, the paper presents this as a construction rather than as an empirical prediction, and the load-bearing finite-sample claim is Proposition 5.2. In Proposition 5.2, bm* is estimated from a separate merging set, but the coverage guarantee is not forced by that fit alone: the proof additionally applies the Dvoretzky–Kiefer–Wolfowitz inequality to control the empirical CDF and then marginalizes over the merging set, giving the 1-(alpha+epsilon+delta) bound. No fitted parameter is renamed as a prediction, and no load-bearing claim reduces to a self-citation. The paper's references [17, 18, 19] are external works by Gasparin, Ramdas, Vovk, and Wang, not the present authors. The reader's note about a possible DKW conditioning gap in the proof of Proposition 5.2 concerns whether the DKW event is stated relative to the correct distribution; that is a proof-correctness concern, not circularity. Therefore the derivation is self-contained with respect to the circularity patterns considered here.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No ad hoc free parameters are fitted to make the derivation work; the correction factor m* is the minimal scaling that makes the weighted average a p-variable, estimated from a separate merging set with a DKW-controlled error. The method relies on standard conformal exchangeability and external results (Vovk-Wang, DKW). The merging set and correction factor are not new entities, only data splits and scalars.

assumptions (5)
  • domain assumption Training, calibration, merging, and test points are jointly exchangeable.
    Used throughout Sections 4-5 and Appendix A to guarantee that each p-value function bp_k is a p-variable and that the empirical CDF on S_merge estimates the test distribution.
  • standard math Vovk and Wang (2020), Proposition 9: for weights v with maximum v, A_v = min(1/v, 2) makes A_v Σ v_k P_k a valid p-variable for arbitrary dependent p-variables.
    Invoked in Appendix A.1 to prove Prop 4.1.
  • standard math Dvoretzky-Kiefer-Wolfowitz inequality bounds the sup distance between empirical and true CDF.
    Used in Appendix A.3 to prove the finite-sample guarantee in Prop 5.2.
  • domain assumption The learned weights W are functions of the input (and possibly the training data) but not of the unobserved test label Y_{n+1}.
    Required so that the prediction set does not use label information; stated in Section 5 and used in the MoE application where W(x) comes from the routing network.
  • standard math Each conformal p-value function bp_k is super-uniform, i.e., a p-variable, under exchangeability.
    Standard split conformal result, stated in Section 2 and used to define the p-value functions in (2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Coverage in Combined Prediction Sets with Weighted p-values." pith.science (2026). https://pith.science/paper/BLPHAY4J

@misc{pith2026250511785,
  author       = {Pith},
  title        = {Pith review of: Improving Coverage in Combined Prediction Sets with Weighted p-values},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLPHAY4J}},
  note         = {Machine review of arXiv:2505.11785}
}
abstract

Conformal prediction quantifies the uncertainty of machine learning models by augmenting point predictions with valid prediction sets. For complex scenarios involving multiple trials, models, or data sources, conformal prediction sets can be aggregated to create a prediction set that captures the overall uncertainty, often improving precision. However, aggregating multiple prediction sets with individual $1-\alpha$ coverage inevitably weakens the overall guarantee, typically resulting in $1-2\alpha$ worst-case coverage. In this work, we propose a framework for the weighted aggregation of prediction sets, where weights are assigned to each prediction set based on their contribution. Our framework offers flexible control over how the sets are aggregated, achieving tighter coverage bounds that interpolate between the $1-2\alpha$ guarantee of the combined models and the $1-\alpha$ guarantee of an individual model depending on the distribution of weights. Importantly, our framework generalizes to data-dependent weights, as we derive a procedure for weighted aggregation that maintains finite-sample validity even when the weights depend on the data. This extension makes our framework broadly applicable to settings where weights are learned, such as mixture-of-experts (MoE), and we demonstrate through experiments in the MoE setting that our methods achieve adaptive coverage.

Figures

Figures reproduced from arXiv: 2505.11785 by the authors.

Figure 1
Figure 1. Left: Storm forecasting example with different models tracking humidity, sea temperature, rainfall, and wind shear over time. Below, an abstract representation of how models vary in predictive strength across the input space (colored regions). At the given test point (black), the red and green models dominate, so their prediction sets matter most. Right: Model prediction sets are combined with weighted aggregation. … view at source ↗
Figure 2
Figure 2. Left: Network diagram for MoE. For traditional MoE, the aggregation module takes a weighted sum of the outputs from each expert. To learn weight-dependent prediction sets, we instead propose to combine the prediction sets of each expert by weighted p-value. Right, top row: Comparison of split conformal prediction sets with those learned from weighted aggregation. Weighted aggregation allows overall coverage to follo… view at source ↗
Figure 3
Figure 3. Local validity experiments comparing split conformal to weighted aggregation using absolute residual [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Mean coverage compared to the size of the merging set [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Coverage (left) and prediction set size (right) for different feature assignment methods and weighted [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Mean coverage across significance levels [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Local validity experiments comparing split conformal (green) to weighted aggregation (orange, purple, [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Coverage (top) and prediction set size (bottom) for split conformal (green) and weighted aggregation [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Coverage (top) and prediction set size (bottom) for split conformal and weighted aggregation variants [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Comparison between WA targeted (0, 0.1] and CQR at matched empirical marginal coverage. Each row corresponds to a dataset, with plots for marginal coverage, WS coverage, ∆ coverage, and prediction set size from left to right. After matching marginal coverage, WA targe…
Figure 11
Figure 11. Figure 11: Comparison between weighted aggregation and adaptive baselines. Each row corresponds to a dataset, [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Regression experiments of our linear transformation method with ECDF and ECDF-DKW. Each row [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Coverage (top) and prediction set size (bottom) for subgroups with top 50th percentile racial [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Finite-sample correction ϵ used in ECDF-DKW as a function of the merging set size |Smerge| and user-specified significance level δ. This correction is the offset required to ensure the finite sample guarantee of (1 − α)(1 − δ) in Stutz et al. [33]. Dashed lines mark α…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 31 canonical work pages

  1. [1]

    Transformers in vision: A survey.ACM computing surveys (CSUR), 54 (10s):1–41, 2022

    Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey.ACM computing surveys (CSUR), 54 (10s):1–41, 2022

    work page 2022

  2. [2]

    Recent advances in natural language processing via large pre-trained language models: A survey.ACM Computing Surveys, 56(2): 1–40, 2023

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey.ACM Computing Surveys, 56(2): 1–40, 2023

    work page 2023

  3. [3]

    Foundations & trends in multimodal machine learning: Principles, challenges, and open questions.ACM Computing Surveys, 56 (10):1–42, 2024

    Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions.ACM Computing Surveys, 56 (10):1–42, 2024

    work page 2024

  4. [4]

    Improving the accuracy of medical diagnosis with causal machine learning.Nature communications, 11(1):3923, 2020

    Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. Improving the accuracy of medical diagnosis with causal machine learning.Nature communications, 11(1):3923, 2020

    2020

  5. [5]

    The clinician and dataset shift in artificial intelligence.New England Journal of Medicine, 385(3):283–286, 2021

    Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane, and Suchi Saria. The clinician and dataset shift in artificial intelligence.New England Journal of Medicine, 385(3):283–286, 2021

    2021

  6. [6]

    Tackling prediction uncertainty in machine learning for healthcare.Nature Biomedical Engineering, 7(6):711–718, 2023

    Michelle Chua, Doyun Kim, Jongmun Choi, Nahyoung G Lee, Vikram Deshpande, Joseph Schwab, Michael H Lev, Ramon G Gonzalez, Michael S Gee, and Synho Do. Tackling prediction uncertainty in machine learning for healthcare.Nature Biomedical Engineering, 7(6):711–718, 2023

    work page 2023

  7. [7]

    Thomas Grote and Philipp Berens. Uncertainty, evidence, and the integration of machine learning into medical practice.The Journal of Medicine and Philosophy: A Forum for Bioethics and Philosophy of Medicine, 48(1):84–97, 2023

    work page 2023

  8. [8]

    Machine learning techniques in additive manufacturing: a state of the art review on design, processes and production control.Journal of Intelligent Manufacturing, 34(1):21–55, 2023

    Sachin Kumar, T Gopi, N Harikeerthana, Munish Kumar Gupta, Vidit Gaur, Grzegorz M Krolczyk, and ChuanSong Wu. Machine learning techniques in additive manufacturing: a state of the art review on design, processes and production control.Journal of Intelligent Manufacturing, 34(1):21–55, 2023

    work page 2023

Show all 53 references
  1. [9]

    Machine learning for industrial sensing and control: A survey and practical perspective.Control Engineering Practice, 145:105841, 2024

    Nathan P Lawrence, Seshu Kumar Damarla, Jong Woo Kim, Aditya Tulsyan, Faraz Amjad, Kai Wang, Benoit Chachuat, Jong Min Lee, Biao Huang, and R Bhushan Gopaluni. Machine learning for industrial sensing and control: A survey and practical perspective.Control Engineering Practice,...

    2024

  2. [10]

    Physics-informed machine learning: case studies for weather and climate modelling.Philosophical Transactions of the Royal Society A, 379(2194):20200093, 2021

    Karthik Kashinath, M Mustafa, Adrian Albert, JL Wu, C Jiang, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, R Wang, Ashesh Chattopadhyay, A Singh, et al. Physics-informed machine learning: case studies for weather and climate modelling.Philosophical Transactions of the Royal Soc...

    2021

  3. [11]

    Pushing the frontiers in climate modelling and analysis with machine learning

    Veronika Eyring, William D Collins, Pierre Gentine, Elizabeth A Barnes, Marcelo Barreiro, Tom Beucler, Marc Bocquet, Christopher S Bretherton, Hannah M Christensen, Katherine Dagon, et al. Pushing the frontiers in climate modelling and analysis with machine learning. Nature Cl...

    2024

  4. [12]

    Machine learning for climate physics and simulations.arXiv preprint arXiv:2404.13227, 2024

    Ching-Yao Lai, Pedram Hassanzadeh, Aditi Sheshadri, Maike Sonnewald, Raffaele Ferrari, and Venkatramani Balaji. Machine learning for climate physics and simulations.arXiv preprint arXiv:2404.13227, 2024. 11

    2024 arXiv

  5. [13]

    Springer, 2005

    Vladimir V ovk, Alexander Gammerman, and Glenn Shafer.Algorithmic learning in a random world, volume 29. Springer, 2005

    2005

  6. [14]

    Inductive confidence machines for regression

    Harris Papadopoulos, Kostas Proedrou, V olodya V ovk, and Alex Gammerman. Inductive confidence machines for regression. InMachine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19–23, 2002 proceedings 13, pages 345–356. Springer, 2002

    2002

  7. [15]

    Cross-conformal predictors.Annals of Mathematics and Artificial Intelligence, 74:9–28, 2015

    Vladimir V ovk. Cross-conformal predictors.Annals of Mathematics and Artificial Intelligence, 74:9–28, 2015

    2015

  8. [16]

    Predictive inference with the jackknife+.Annals of Statistics, 2021

    Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+.Annals of Statistics, 2021

    2021

  9. [17]

    Merging uncertainty sets via majority vote.arXiv preprint arXiv:2401.09379, 2024

    Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote.arXiv preprint arXiv:2401.09379, 2024

    2024 arXiv

  10. [18]

    Conformal online model aggregation.arXiv preprint arXiv:2403.15527, 2024

    Matteo Gasparin and Aaditya Ramdas. Conformal online model aggregation.arXiv preprint arXiv:2403.15527, 2024

    2024

  11. [19]

    Combining p-values via averaging.Biometrika, 107(4): 791–808, 2020

    Vladimir V ovk and Ruodu Wang. Combining p-values via averaging.Biometrika, 107(4): 791–808, 2020

    2020

  12. [20]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  13. [21]

    Distribution-free predictive inference for regression.Journal of the American Statistical Associ- ation, 113(523):1094–1111, 2018

    Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression.Journal of the American Statistical Associ- ation, 113(523):1094–1111, 2018

    2018

  14. [22]

    Selection and aggregation of conformal prediction sets.Journal of the American Statistical Association, pages 1–13, 2021

    Yachong Yang and Arun Kumar Kuchibhotla. Selection and aggregation of conformal prediction sets.Journal of the American Statistical Association, pages 1–13, 2021

    2021

  15. [23]

    Conformal prediction after efficiency- oriented model selection.arXiv preprint arXiv:2408.07066, 2024

    Ruiting Liang, Wanrong Zhu, and Rina Foygel Barber. Conformal prediction after efficiency- oriented model selection.arXiv preprint arXiv:2408.07066, 2024

    2024 arXiv

  16. [24]

    Conformity score averaging for classification

    Rui Luo and Zhixin Zhou. Conformity score averaging for classification. InForty-second International Conference on Machine Learning, 2025

    2025

  17. [25]

    Valid selection among conformal sets.arXiv preprint arXiv:2506.20173, 2025

    Mahmoud Hegazy, Liviu Aolaritei, Michael I Jordan, and Aymeric Dieuleveut. Valid selection among conformal sets.arXiv preprint arXiv:2506.20173, 2025

    2025 arXiv

  18. [26]

    Majority vote ensembles of conformal predictors.Machine Learning, 108 (3):475–488, 2019

    Giovanni Cherubin. Majority vote ensembles of conformal predictors.Machine Learning, 108 (3):475–488, 2019

    2019

  19. [27]

    Multi split conformal prediction.Statistics & Probability Letters, 184:109395, 2021

    Aldo Solari and Vera Djordjilovi´c. Multi split conformal prediction.Statistics & Probability Letters, 184:109395, 2021

    2021

  20. [28]

    Conformal predictions for information fusion: A comparative study of p-value combination methods

    Vineeth N Balasubramanian, Shayok Chakraborty, and Sethuraman Panchanathan. Conformal predictions for information fusion: A comparative study of p-value combination methods. Annals of Mathematics and Artificial Intelligence, 74(1):45–65, 2015

    2015

  21. [29]

    Exact tests via multiple data splitting.Statistics & Probability Letters, 166:108865, 2020

    Cyrus J DiCiccio, Thomas J DiCiccio, and Joseph P Romano. Exact tests via multiple data splitting.Statistics & Probability Letters, 166:108865, 2020

    2020

  22. [30]

    lehne h o ab, wenn k unter n gegebenen tests zur ablehnung führen

    Bernhard Rüger. Das maximale signifikanzniveau des tests:“lehne h o ab, wenn k unter n gegebenen tests zur ablehnung führen”.Metrika, 25:171–178, 1978

    1978

  23. [31]

    Tests of the overall hypothesis for arbitrary dependence structures.Biometri- cal Journal, 25(5):423–430, 1983

    Gerhard Hommel. Tests of the overall hypothesis for arbitrary dependence structures.Biometri- cal Journal, 25(5):423–430, 1983

    1983

  24. [32]

    Random variables with maximum sums.Advances in Applied Probability, 14(3):623–632, 1982

    Ludger Rüschendorf. Random variables with maximum sums.Advances in Applied Probability, 14(3):623–632, 1982. 12

    1982

  25. [33]

    Conformal prediction under ambiguous ground truth.arXiv preprint arXiv:2307.09302, 2023

    David Stutz, Abhijit Guha Roy, Tatiana Matejovicova, Patricia Strachan, Ali Taylan Cemgil, and Arnaud Doucet. Conformal prediction under ambiguous ground truth.arXiv preprint arXiv:2307.09302, 2023

    2023 arXiv

  26. [34]

    Yaowu Liu and Jun Xie. Cauchy combination test: a powerful test with analytic p-value calcu- lation under arbitrary dependency structures.Journal of the American Statistical Association, 115(529):393–402, 2020

    2020

  27. [35]

    Multi-split conformal prediction via cauchy aggregation.Stat, 12(1):e522, 2022

    Xiaoyang Wu, Yuyang Huo, and Changliang Zou. Multi-split conformal prediction via cauchy aggregation.Stat, 12(1):e522, 2022

    2022

  28. [36]

    Distribution-free prediction bands for non-parametric regression

    Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71–96, 2014

    2014

  29. [37]

    Conditional validity of inductive conformal predictors

    Vladimir V ovk. Conditional validity of inductive conformal predictors. InAsian conference on machine learning, pages 475–490. PMLR, 2012

    2012

  30. [38]

    Conformalized quantile regression

    Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019

    2019

  31. [39]

    Classification with valid and adaptive coverage.Advances in Neural Information Processing Systems, 33:3581–3591, 2020

    Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage.Advances in Neural Information Processing Systems, 33:3581–3591, 2020

    2020

  32. [40]

    A data-driven software tool for enabling cooperative information sharing among police departments.European Journal of Operational Research, 141(3):660–678, 2002

    Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments.European Journal of Operational Research, 141(3):660–678, 2002

    2002

  33. [41]

    Conformal prediction with conditional guarantees.arXiv preprint arXiv:2305.12616, 2023

    Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees.arXiv preprint arXiv:2305.12616, 2023

    2023 arXiv

  34. [42]

    A comparison of some conformal quantile regression methods.Stat, 9(1):e261, 2020

    Matteo Sesia and Emmanuel J Candès. A comparison of some conformal quantile regression methods.Stat, 9(1):e261, 2020

    2020

  35. [43]

    Conformal- ized fairness via quantile regression.Advances in Neural Information Processing Systems, 35: 11561–11572, 2022

    Meichen Liu, Lei Ding, Dengdeng Yu, Wulong Liu, Linglong Kong, and Bei Jiang. Conformal- ized fairness via quantile regression.Advances in Neural Information Processing Systems, 35: 11561–11572, 2022

    2022

  36. [44]

    Ensemble conformalized quantile regression for probabilistic time series forecasting.IEEE Transactions on Neural Networks and Learning Systems, 35(7):9014–9025, 2022

    Vilde Jensen, Filippo Maria Bianchi, and Stian Normann Anfinsen. Ensemble conformalized quantile regression for probabilistic time series forecasting.IEEE Transactions on Neural Networks and Learning Systems, 35(7):9014–9025, 2022

    2022

  37. [45]

    Integrating uncertainty awareness into conformalized quantile regression

    Raphael Rossellini, Rina Foygel Barber, and Rebecca Willett. Integrating uncertainty awareness into conformalized quantile regression. InInternational Conference on Artificial Intelligence and Statistics, pages 1540–1548. PMLR, 2024

    2024

  38. [46]

    Conformal prediction with localization.arXiv preprint arXiv:1908.08558, 2019

    Leying Guan. Conformal prediction with localization.arXiv preprint arXiv:1908.08558, 2019

    1908 arXiv

  39. [47]

    Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate.arXiv preprint arXiv:2112.14397, 2021

    Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate.arXiv preprint arXiv:2112.14397, 2021

    2021 arXiv

  40. [48]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024

    2024 arXiv

  41. [49]

    portion: Python data structure and operations for intervals, 2018

    Alexandre Decan. portion: Python data structure and operations for intervals, 2018. URL https://github.com/AlexandreDecan/portion

    2018

  42. [50]

    The uci machine learning repository

    Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The uci machine learning repository. https://archive.ics.uci.edu, 2010. 13 A Proofs and additional theoretical details A.1 Proof of Proposition 4.1 Following the notation of V ovk and Wang [19], we define the merging funct...

    2010

  43. [51]

    and works well in practice. 14 A.3 Proof of Proposition 5.2 LetG be the event that the worst-case distance between the true CDFFPall and the empirical CDF bFPall is bounded by some maximum allowable deviation; that is, G= sup x∈R bFPall(x)−F Pall(x) ≤ε whereε= s log(2/δ) 2|Sme...

  44. [52]

    Ifδ≥1/m, sinceF Pall(δ)≤1, FPall(δ)/δ≤1/δ≤m

  45. [53]

    By feasibility ofm, FPall(δ) =F Pall(α/m)≤α=mδ, or FPall(δ)/δ≤m

    Ifδ∈(0,1/m), let us selectα=mδ∈(0,1). By feasibility ofm, FPall(δ) =F Pall(α/m)≤α=mδ, or FPall(δ)/δ≤m. The two cases above establishm≥c for everym∈S , or thatS⊂(c,∞) . Thus,c is a lower bound ofSandm ∗≥c. 16 A.5 Computingbm ∗ from an empirical CDF For random variable Pall =p a...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.