{"id":"3c57332a-5acd-4867-a569-2715bf1ff4b0","arxiv_id":"2505.11785","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Weighted averaging of conformal p-values with a learned scaling correction yields valid combined prediction sets for data-dependent weights, improving worst-slice coverage.","lead":"This paper introduces a way to combine several machine learning prediction sets into one, using weighted averages of p-values and a scaling correction that keeps the result statistically valid even when the weights depend on the input. The method targets mixture-of-experts and ensemble settings, improving coverage on hard data slices.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prop 5.2's proof applies DKW to the marginal distribution of P_all, but the merging-set samples are only i.i.d. conditional on S_cal; the stated finite-sample guarantee is not established as written.","rationale":"The paper's main contribution is the finite-sample validity of weighted aggregation with data-dependent weights, encapsulated in Prop 5.2. The reader's weakest assumption concerned the small merging set making the DKW tolerance eps large, which is a practical limitation. My review found a more fundamental issue in the proof of Prop 5.2 itself: the S_merge samples are not i.i.d. from the marginal F_Pall used in the DKW event, because they share the same calibration set S_cal. The theorem is likely true and can be repaired by a conditional-DKW argument, so I do not recommend rejection. However, the proof as written does not establish the central claim, and the practical gap between the proved bound and the experimental coverage remains. The verdict should stay CONDITIONAL, with the condition being a corrected proof of Prop 5.2 and ideally code/data for reproducibility. This is a good-faith technical critique, not an attack on the authors' integrity.","tokens_in":27814,"tokens_out":22221,"duration_ms":234612,"concrete_test":"Independently re-derive Prop 5.2 by conditioning on S_cal: apply DKW to the conditional law F_{S_cal}, prove P{bm* P_all <= alpha | S_cal, S_merge} <= alpha + eps on a conditional DKW event, then marginalize over S_cal and S_merge. If the re-derivation succeeds, the theorem stands and only the proof needs correction. If it fails, construct a counterexample with K=1 and |S_cal|=1 where the unconditional empirical process deviates from F_Pall by more than eps while the conditional process does not; such a case would show the published DKW event is not covered by the inequality as stated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central finite-sample guarantee (Prop 5.2, Appendix A.3) defines the DKW event G = {sup |bF_Pall - F_Pall| <= eps} and asserts P(G) >= 1 - delta. This step is not justified. The merging-set values P_i^all = p_all(X_i,Y_i; W^(i)) are computed from p-value functions bp_k calibrated on S_cal. Conditional on S_cal they are i.i.d. from the realized law F_{S_cal}; unconditionally they are dependent, with marginal F_Pall a mixture over S_cal. DKW controls sup |bF - F_{S_cal}| conditionally, not sup |bF - F_Pall|, and these two quantities can differ by a non-vanishing amount (e.g., K=1 with |S_cal|=1 yields p-values 0.5 or 1 with probabilities set by the single calibration score). Equations (14)-(16) use F_Pall for the conditional miscoverage probability while bF estimates F_{S_cal}, so the key inequality F_Pall(alpha/bm*) <= bF(alpha/bm*) + eps on the DKW event is unsupported. The theorem is likely repairable by conditioning on S_cal throughout and only then marginalizing, but the published proof of the central claim has a real gap. Separately, even after repair, at |S_merge|=40 the DKW eps is about 0.19, making the proved coverage bound roughly 1 - (alpha + 0.29); the paper's adaptive-coverage claims therefore rely on empirical behavior beyond the theorem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for aggregating K conformal prediction sets by taking a weighted average of the associated p-value functions. For data-independent weights, Proposition 4.1 applies a result of Vovk and Wang to show that the aggregated set has coverage at least 1 - min(1/v, 2)α, where v is the largest weight. For data-dependent weights, the paper defines a correction factor m* as the minimal scaling that makes the weighted average a p-variable, and proposes to estimate m* from a merging set using an empirical CDF (Proposition 5.2), yielding a finite-sample coverage guarantee of 1 - (α + ε + δ). The method is applied to mixture-of-experts models, where routing weights are used to aggregate expert p-value functions, and experiments on synthetic and UCI data report improved worst-slice coverage relative to split conformal and adaptive baselines. The central theoretical contribution is the finite-sample guarantee for data-dependent weights; the paper also provides detailed proofs in Appendices A.1-A.5 and extensive ablations in Appendices B-E.","tokens_in":28122,"tokens_out":10597,"duration_ms":110707,"significance":"If the finite-sample guarantee is established, the paper makes a useful contribution: it extends asymmetric prediction-set aggregation from fixed weights to data-dependent weights, connects the Vovk-Wang averaging result to conformal prediction, and demonstrates a practically relevant application to mixture-of-experts with improved local validity. The treatment of the data-independent case is clean and the experimental study is thorough, including comparisons to conservative CQR, localized conformal prediction, and demographic subgroup analyses. The paper also gives detailed derivations of the equivalence in (7) and of the empirical correction factor, which are helpful. However, the central finite-sample result (Proposition 5.2) currently has a genuine proof gap in the application of the DKW inequality, and the assumptions connecting the marginal quantity m* to the empirical estimator bm* are not stated precisely. The gap is repairable, but it affects the paper's main theoretical claim and therefore requires a major revision.","major_comments":[{"comment":"The DKW step in the proof of Proposition 5.2 is not justified as written. Conditional on the calibration set S_cal, the merging samples P_i^all are i.i.d. from the conditional law F_{S_cal} of P_all for an exchangeable point; unconditionally they are dependent, and the marginal CDF F_Pall is a mixture over S_cal. DKW controls sup_x |\\hat F(x) - F_{S_cal}(x)|, not sup_x |\\hat F(x) - F_Pall(x)|, and these two quantities can differ by an amount that does not vanish with |S_merge| (for example, K=1 with |S_cal|=1). Equation (14) therefore cannot use F_Pall for a probability that is conditional on S_merge, and the key inequality (15) is unsupported. The theorem is repairable by conditioning on S_cal throughout and defining the target CDF as the conditional law, but the published proof of the central finite-sample claim has a real gap.","section":"Appendix A.3, Eqs. (14)-(16)"},{"comment":"The object m* in Proposition 5.1 is defined through the marginal CDF F_Pall, whereas the empirical bm* in Proposition 5.2 estimates the scaling appropriate to the conditional law of P_all given S_cal. These are different quantities in general, so the two propositions are not connected as stated. A repaired statement should either define m* conditionally on S_cal and then marginalize over S_cal, or prove that bm* estimates the marginal m*; the first route is the natural one and preserves the coverage claim.","section":"Section 5, Propositions 5.1 and 5.2"},{"comment":"The DKW argument also requires the weights W^(i) used on the merging set to be independent across i and independent of the test point conditional on S_cal, for example when they are computed from a training split disjoint from S_merge. As stated, W may depend on {(X_i,Y_i)}_{i in [n]} union {X_{n+1}}, which includes S_merge, and then the merging samples are not i.i.d. This assumption needs to be made explicit; the MoE application satisfies it only because the routing network is trained on S_train, which should be stated in the theorem or in Section 6.","section":"Section 5, Proposition 5.2 assumptions"},{"comment":"The main UCI experiments set |S_merge|=40, so with delta=0.1 the DKW tolerance is epsilon approx 0.19 and the guarantee proved in Proposition 5.2 is only about 1-(alpha+0.29). The paper's practical conclusions about adaptive coverage at this merging size therefore rest on empirical behavior that is not captured by the finite-sample theorem. I recommend either using a larger merging set in the headline experiments, reporting the bound alongside the empirical coverage, or explicitly qualifying the practical claims as heuristic.","section":"Section 6 and Appendix E.1"}],"minor_comments":[{"comment":"The text says P1,...,PK are in U, 'the set of all uniformly distributed random variables', which conflicts with the definition of a p-variable in (1) and with the fact that split conformal p-values are generally conservative rather than exactly uniform; please clarify that U denotes p-variables or state the precise-p-variable assumption.","section":"Section 5, first paragraph"},{"comment":"Algorithm 1 presents the coverage guarantee for the m* correction, but the experiments use the m† and m‡ corrections from equations (10) and (11); the algorithm should specify which correction factor is being computed.","section":"Algorithm 1"},{"comment":"The equality m* = sup_{delta>0} F_Pall(delta)/delta requires that F_Pall has no mass at 0; this holds for conformal p-values, but the condition should be stated before the equivalence is used.","section":"Equation (7) and Appendix A.4"},{"comment":"The legend contains an incomplete label 'WA targeted (0, ]'; the intended significance-level range should be written out.","section":"Figure 3 legend"},{"comment":"The Dvoretzky-Kiefer-Wolfowitz inequality is used without a citation; please add the standard reference for the DKW bound in the proof of Proposition 5.2.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The proof gap in Proposition 5.2 is real but repairable by conditioning on the calibration set. If repaired, the paper is likely acceptable, although the practical strength of the finite-sample guarantee at |S_merge|=40 should be addressed. The experimental evaluation is otherwise solid and the application to mixture-of-experts is timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The data-independent part (Prop 4.1) is a direct application of Vovk and Wang's averaging theorem, and the paper is upfront about that. The genuinely new piece is the extension to data-dependent weights: scale the weighted average of p-values by a correction factor estimated from a merging set, with a finite-sample guarantee. That extension is useful and the MoE application is natural. Experiments support the claim that weighted aggregation improves worst-slice coverage, and the matched-marginal-coverage comparison against CQR suggests the gain is not solely from being more conservative.\n\nThe soft spot is the proof of Prop 5.2. The DKW step compares the empirical CDF of the merging samples to the marginal CDF F_Pall. But the merging samples are i.i.d. only conditional on the calibration set; their unconditional law is a mixture over S_cal. DKW controls sup|bF - F_{S_cal}|, not sup|bF - F_Pall|, and the two can differ by a non-vanishing amount. So equation (15) is unsupported. The theorem is likely repairable by conditioning on S_cal first, using the conditional law, and then marginalizing; the skeleton of that argument is already in the proof. But as written, the main finite-sample guarantee rests on an unjustified inequality. A referee should ask for a corrected proof.\n\nTwo lesser issues: with |S_merge|=40, the proved bound is about 1-(α+0.29) for δ=0.1, so the practical coverage claims rely on the empirical distribution being better behaved than the worst-case bound. That is not fatal — split conformal's own finite-sample bounds have slack — but it deserves a sentence. And no code or data accompanies the paper, which makes the experiments harder to trust at face value.\n\nOverall: the paper is worth engaging with. The data-dependent weighting construction is clever, the experiments are honest about the coverage/efficiency trade-off, and the main gap is a fixable technical one, not a conceptual dead end. I would send it to peer review with a request for a corrected proof of Prop 5.2 and a discussion of the merging-set-size issue. It is a solid contribution to conformal aggregation for anyone working on MoE or ensemble uncertainty.","headline":"Useful extension of weighted p-value aggregation to data-dependent weights, but the main finite-sample proof has a gap that needs fixing before the claims as written are taken at face value.","tokens_in":28673,"tokens_out":4791,"would_cite":true,"duration_ms":47911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighted p-value aggregation interpolates combined prediction-set coverage between the classical 1−2α worst case and the single-model 1−α guarantee, and a learned scaling factor preserves finite-sample validity when the weights depend on…","keywords":["conformal prediction","weighted p-values","prediction set aggregation","data-dependent weights","mixture-of-experts","finite-sample coverage guarantee","worst-slice coverage","local validity"],"falsifier":"Run the scaled weighted-aggregation procedure at $\\alpha=0.1$ with an exchangeable merging set of 40 points and $\\delta=0.1$ over many trials; Proposition 5.2 caps expected miscoverage at about 0.39, so observing miscoverage consistently above that level on any dataset would falsify the guarantee's assumptions, whereas much lower miscoverage would show the bound is pessimistic. Separately, the practical adaptivity claim is falsifiable by measuring worst-slice coverage on a large held-out test set: if the weighted-aggregation set undercovers the worst slab at least as much as split conformal on any of the reported datasets, the central empirical claim fails.","tokens_in":27599,"feed_emoji":"🎯","tokens_out":16354,"duration_ms":133538,"temperature":0.7,"pith_summary":"This paper asks how to combine several conformal prediction sets, each valid at level $1-\\alpha$, into one aggregate set without losing the whole guarantee. The standard symmetric answer costs a factor of two, giving $1-2\\alpha$ worst-case coverage. The paper shows that weighting the underlying p-values rather than the sets interpolates the guarantee between $1-2\\alpha$ and $1-\\alpha$, depending on how concentrated the weights are. For the harder case where the weights themselves are learned from data, as in mixture-of-experts routing, it introduces a scaling correction computed from an auxiliary merging set that restores a finite-sample coverage guarantee. Experiments on real regression and classification data show that the resulting sets keep coverage on hard data slices and demographic subgroups where plain split conformal undercovers.","feed_headline":"Weighted p-values push combined prediction sets from 1−2α to 1−α coverage","feed_subtitle":"The result is prediction sets that adapt to each input while keeping a finite-sample coverage guarantee.","key_machinery":"The load-bearing object is the weighted-average p-value function $\\bar p(x,y)=\\sum_k w_k \\hat p_k(x,y)$ built from per-model conformal p-values, together with the p-value averaging theorem the framework is built on: for fixed weights, scaling the weighted average by $A_v=\\min(1/v,2)$ turns it into a valid p-variable. For data-dependent weights this scaling no longer applies, and the argument is carried by the correction factor $m^*$, the smallest scalar making $m^* P_{\\mathrm{all}}$ satisfy the p-variable inequality, computed as the supremum ratio of the distribution function of $P_{\\mathrm{all}}$ to its argument. Its empirical version $\\hat m^*$ is the maximum such ratio over an auxiliary merging set, and the Dvoretzky-Kiefer-Wolfowitz inequality converts the estimation error into the finite-sample slack $\\epsilon+\\delta$ in Proposition 5.2. The mechanism matters because a linear scaling preserves the relative proportions of the learned weights, which a rank-based CDF transformation would destroy, while still repairing the validity condition that the weighted average alone violates.","core_discovery":"The central claim is that a prediction set formed by thresholding the weighted average $\\bar p(x,y)=\\sum_k v_k \\hat p_k(x,y)$ of per-model conformal p-values at level $\\alpha$ has coverage at least $1-\\min(1/v,2)\\alpha$, where $v$ is the largest weight. When one set dominates, with $v>1/2$, this beats the classical $1-2\\alpha$ aggregation bound, and at $v=1$ it recovers the individual set's $1-\\alpha$ guarantee. For data-dependent weights $W(x)$, the same weighted average is no longer a valid p-variable, but multiplying it by the smallest scalar $m^* = \\sup_\\delta F_{P_{\\mathrm{all}}}(\\delta)/\\delta$ that forces $P\\{m^* P_{\\mathrm{all}} \\le \\alpha\\}\\le \\alpha$ for all $\\alpha$ restores validity, giving a $1-\\alpha$ prediction set in the infinite-sample limit. Estimating $m^*$ from the empirical CDF of $P_{\\mathrm{all}}$ on a merging set yields the finite-sample coverage guarantee $1-(\\alpha+\\epsilon+\\delta)$ with worst-case statistical tolerance $\\epsilon=\\sqrt{\\log(2/\\delta)/(2|S_{\\mathrm{merge}}|)}$. Applied to mixture-of-experts, where router weights depend on the input, the method delivers prediction sets whose coverage follows the dominant expert locally, with improved worst-slice and per-group coverage relative to split conformal and CQR baselines.","pith_inferences":["The same linear-repair idea would extend to data-dependent weights in other aggregation rules, such as majority vote or rank-based averaging, though those would sacrifice the weight-proportion semantics the authors rely on.","Because the guarantee marginalizes over the merging set, the adaptive-coverage behavior is an empirical property rather than a distribution-free one; a natural stress test is measuring worst-slice coverage as $|S_{\\mathrm{merge}}|$ shrinks well below the 40-point range used in the main experiments, where the proved bound is weak ($\\epsilon\\approx 0.19$ at $\\delta=0.1$).","Since the linear scaling preserves the router's weight proportions, the aggregate prediction set can be read as expressing the router's own judgment about which expert is responsible for a test point, which suggests using the correction factor itself as a diagnostic for router confidence."],"forward_implications":["With data-independent weights, any weighted aggregation of conformal sets has coverage at least $1-\\min(1/v,2)\\alpha$, so a single dominant set with $v\\ge 1/2$ already improves on the classical $1-2\\alpha$ guarantee.","Because the aggregation operates on p-values after calibration, it can be layered on top of locally adaptive score functions such as CQR, so the two sources of adaptivity compose rather than compete.","In the mixture-of-experts setting, the finite-sample guarantee makes learned router weights a safe input to set aggregation, yielding sets whose coverage tracks the expert the router most trusts at each input.","The merging-set size controls a concrete trade-off: larger $|S_{\\mathrm{merge}}|$ shrinks the worst-case tolerance $\\epsilon$, and the paper's ablations show empirical coverage approaching the nominal level, with roughly 160 points sufficing for about 3% overcoverage.","Across the paper's datasets, weighted aggregation narrows the gap between marginal and worst-slice coverage compared with split conformal, and in demographic subgroup analyses it meets coverage for groups that split conformal undercovers."],"supporting_citations":[{"why":"Supplies the p-value averaging theorem: for fixed weights, scaling the weighted average by min(1/v,2) yields a valid p-variable, the backbone of Proposition 4.1 and the template for the m* correction.","marker":"[19]"},{"why":"Provides the p-value presentation of conformal prediction on which the paper's weighted-average construction (equations 2 and 5) rests.","marker":"[13]"},{"why":"Defines split conformal prediction, the base procedure whose per-set 1−α guarantees the paper aggregates.","marker":"[14]"},{"why":"Gives the weighted majority-vote predecessor that also achieves only the 1−2α bound, establishing the baseline the paper improves and to which it is dual for fixed weights.","marker":"[17]"},{"why":"Supplies conformalized quantile regression, the main locally adaptive baseline and the score type used in the hybrid weighted-aggregation variant.","marker":"[38]"},{"why":"Defines worst-slice coverage, the primary metric behind the paper's local-validity claims.","marker":"[39]"},{"why":"Introduces adaptive mixtures of local experts, the application whose learned routing weights motivate the data-dependent-weight extension.","marker":"[20]"},{"why":"Provides the symmetric jackknife+/CV+ aggregation methods the paper generalizes, and the data-split budget convention used in the experiments.","marker":"[16]"}],"fun_headline_variants":["Weighted conformal sets interpolate coverage from 1−2α to 1−α","Data-driven weights restore 1−α coverage in conformal aggregation","Weighted p-values adapt conformal coverage per input","Tighter coverage for conformal sets via weighted p-values","Mixture-of-experts gets finite-sample 1−α coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The formal guarantee for data-dependent weights is only $1-(\\alpha+\\epsilon+\\delta)$ where $\\epsilon$ is a worst-case statistical error that shrinks as the merging set grows, and with the merging sets used in the main experiments (about 40 points) the proved bound is roughly $1-(\\alpha+0.29)$, so the paper's adaptive-coverage results depend on the unstated assumption that the real distribution of the aggregated p-values is better behaved than that worst-case bound.","fun_headline_variants_meta":{"raw":{"variants":["Weighted conformal sets interpolate coverage from 1−2α to 1−α","Data-driven weights restore 1−α coverage in conformal aggregation","Weighted p-values adapt conformal coverage per input","Tighter coverage for conformal sets via weighted p-values","Mixture-of-experts gets finite-sample 1−α coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3643,"prompt_tokens":1084,"completion_tokens":2559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2467}},"tokens_in":700,"tokens_out":2559,"duration_ms":19211,"temperature":1.0,"reasoning_tokens":2467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:48:10.633693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the scaled weighted-aggregation procedure at $\\alpha=0.1$ with an exchangeable merging set of 40 points and $\\delta=0.1$ over many trials; Proposition 5.2 caps expected miscoverage at about 0.39, so observing miscoverage consistently above that level on any dataset would falsify the guarantee's assumptions, whereas much lower miscoverage would show the bound is pessimistic. Separately, the practical adaptivity claim is falsifiable by measuring worst-slice coverage on a large held-out test set: if the weighted-aggregation set undercovers the worst slab at least as much as split conformal on any of the reported datasets, the central empirical claim fails.","supporting_citations":[{"cited_title":"Combining p-values via averaging.Biometrika, 107(4): 791–808, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the p-value averaging theorem: for fixed weights, scaling the weighted average by min(1/v,2) yields a valid p-variable, the backbone of Proposition 4.1 and the template for the m* correction."},{"cited_title":"Predictive inference with the jackknife+.Annals of Statistics, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the symmetric jackknife+/CV+ aggregation methods the paper generalizes, and the data-split budget convention used in the experiments."}],"review_version":1}