{"id":"1ccff51c-197e-425f-8d65-8b6fea1e1463","arxiv_id":"1908.05571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A median-based combination of prediction intervals from per-source conformal predictors gives conservative intervals on a UCI regression benchmark without pooling the data.","lead":"This paper proposes a method to combine prediction ranges produced by machine learning models trained on separate data sources that cannot be pooled. It offers a lightweight privacy-preserving way to get one overall prediction range in regression, with experiments showing mostly valid ranges and modest gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Median combination of conformal intervals has no validity proof; for even K or dependent miss indicators it can under-cover, so the 'conservatively valid' claim rests only on one UCI experiment.","rationale":"The reader and I converge on the same soft spot. The paper's central claim is that NDCP supplies conservatively valid intervals; the only mechanism for validity is the median combination in Algorithm 1, and no proof is given that it preserves coverage. The failure is not merely a missing lemma: for even K or dependent miss indicators, median coverage can fall below 1-alpha. Because the experiments cover only K=2,4,6 on one dataset and Experiment 3 violates the exchangeability assumption, the empirical support does not close the gap. I therefore keep the reader's CONDITIONAL verdict: the method is plausible and the tables are encouraging, but the paper should either prove or explicitly scope the validity claim. No change in verdict.","tokens_in":14429,"tokens_out":10950,"duration_ms":118555,"concrete_test":"Run a simulation with K=3 at the 95% level: define three pairwise-disjoint events E12, E13, E23, each of probability 0.025, and when Eij occurs make sources i and j both miss below the label while the remaining source covers. Each source then has marginal miss probability 0.05. Measure the coverage of Algorithm 1's median interval over 100,000 test points. If coverage is about 0.925 rather than at least 0.95, the median-of-intervals combination is not unconditionally valid and the paper must add assumptions or restrict its claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 combines intervals by taking the median of the K lower bounds and the median of the K upper bounds. Section 2 establishes validity only for a single conformal predictor; no theorem transfers this property to the median interval. The transfer is not automatic. For K=3, median coverage is guaranteed only when at least two of the three intervals cover. If the miss indicators are dependent, P(at least two cover) can be as low as 0.925 even though each source has marginal coverage 0.95 (take three pairwise-disjoint two-source miss events, each of probability 0.025). For even K, the only values tested in the paper (2, 4, 6), the situation is worse: for K=2 the averaged interval generally requires both intervals to cover, which under independence gives about 0.90 coverage. The paper supplies no assumption under which the needed positive dependence holds; the sources are disjoint partitions of a single dataset, not independent. Experiment 3 additionally breaks exchangeability, so even per-source conformal validity is unavailable there. Thus the headline claim 'conservatively valid' rests entirely on Tables 1-3 for one UCI dataset, without error bars or a proof. This is a load-bearing gap, not a minor omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Non-Disclosed Conformal Prediction (NDCP), a method for combining prediction intervals from K independently trained conformal predictors when the underlying training data cannot be pooled across sources. For each source, an inductive or cross-conformal predictor produces an interval for a test object, and the final interval is formed by taking the median of the lower bounds and the median of the upper bounds. The method is evaluated on the UCI Concrete Compressive Strength data set with support vector regression, using 2, 4, and 6 sources under equal, unequal, and non-IID source splits, with 100 repetitions per setting. The reported results compare validity and median interval width for NDCP, the individual sources, a hypothetical 'Ideal NDCP', and a pooled-data predictor. The paper claims that NDCP produces conservatively valid prediction intervals and improves efficiency over at least some individual sources, while remaining less efficient than full pooling.","tokens_in":14684,"tokens_out":7308,"duration_ms":76715,"significance":"If the validity claim were established, NDCP would be a practically attractive, lightweight method for privacy-preserving prediction with intervals, since it requires exchanging only test objects and intervals rather than training data. The paper has clear strengths: the experiments use held-out test labels; SVR hyperparameters are tuned only on training folds through cross-validation; the 'Ideal NDCP' quantity is transparently labeled as a post hoc object and is not used to support the central claim; and the non-IID experiment addresses an important practical concern. However, the central assertion of conservative validity currently rests on empirical tables from a single UCI data set rather than on a theorem, and the interval-median operation does not obviously inherit the conformal validity guarantee. The efficiency claim, as stated in the abstract, is also stronger than the 'at least one source' conclusion reached in the discussion. The contribution is therefore plausible but not yet established at the level claimed.","major_comments":[{"comment":"The paper provides no proof that the elementwise median of K individually valid conformal intervals is itself a valid prediction interval at the chosen significance level. Section 2 establishes validity only for a single conformal predictor, and the median operation does not automatically transfer that guarantee. The failure is not merely hypothetical: for K=2, where the median of two bounds is their average, take y=0 and let each source independently output the interval [-1,1] with probability 0.95 and the interval [-100,-99] with probability 0.05. Each interval has marginal 95% coverage, but the merged interval covers only when both sources cover, giving coverage 0.9025 at the 95% level. The paper tests only even K (2, 4, 6) and supplies no assumptions under which miss indicators are positively dependent; since the sources are disjoint partitions of one data set, such dependence cannot simply be assumed. This is a load-bearing gap in the manuscript's central claim of conservative validity.","section":"Section 3, Algorithm 1"},{"comment":"Experiment 3 deliberately breaks the exchangeability assumption by partitioning the training data so that sources have different label distributions and none is identically distributed to the test set. Under these conditions the per-source conformal validity theorem from Section 2 does not apply, so the NDCP validity values in Table 3 (for example 0.953, 0.963, 0.965 for ICP with 2, 4, and 6 sources) have no theoretical backing and should not be presented as supporting the claim of conservatively valid intervals. The authors acknowledge the violation in the text, but they still use this experiment as evidence for NDCP validity. The non-IID setting can be retained as a stress test only if the validity claim is explicitly scoped as empirical rather than guaranteed.","section":"Section 4.2.3, Table 3"},{"comment":"The validity and efficiency results are reported as point summaries over 100 repetitions without standard errors, confidence intervals, or significance tests. Since the central claim is that NDCP is conservatively valid, the relevant comparison is whether observed validity is compatible with the nominal 95% level; for example, many entries near 0.96-0.97 could be consistent with 0.95 given Monte Carlo variability, while a value such as 0.9025 would not be. The efficiency comparisons also need uncertainty quantification before the abstract's claim of improved efficiency can be assessed. Without this, the empirical tables support only a qualitative impression, not the precise statements in the abstract and conclusions.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The abstract states that efficiency is improved 'as compared to predicting using a single arbitrarily chosen source', but the discussion (Section 5) and the tables support only improvement over at least one individual source; for example, in Table 1 with 2 sources and CCP, Source2 has width 26.748 while NDCP has width 26.813. The abstract should be aligned with the weaker claim that the experiments actually demonstrate.","section":"Abstract and Section 5"},{"comment":"The sentence 'NDCP using ICP in all cases has a lower efficiency when compared to the individual data sources, except for 2 data sources but here NDCP has lower efficiency than one of the data sources' is confusing and internally contradictory; 'lower efficiency' means a larger interval, which is a worse outcome, and the comparison to individual sources in Table 1 does not uniformly support the sentence. This should be rephrased to describe interval widths clearly.","section":"Section 4.2.1"},{"comment":"The median for an even number K is not defined in the algorithm; the paper should specify whether the median is the average of the two middle values, since all experimental settings use even K.","section":"Algorithm 1"},{"comment":"The method description says the number of examples in each data source is not disclosed, but the paper does not discuss how this interacts with the median combination or with the theoretical claims; if source sizes are unknown, the user also cannot know n for NDCP, yet Section 4.1 reports n=927 for NDCP as the sum over sources. This point should be clarified.","section":"Section 3"},{"comment":"Several typographical issues should be corrected: 'information efficient' in Section 2, 'yield improve results' in Section 5, and 'approches' in the Abadi reference.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper is on the borderline between major revision and rejection. The core gap is that conservative validity is asserted without a theorem, and the only even-K settings tested are exactly those where median combination is least theoretically safe. I would be willing to accept a substantially revised version that either adds a validity proof under explicit assumptions or honestly reframes the contribution as an empirical study and removes the word 'conservatively valid' from the abstract. As written, the central claim is not supported by the presented evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper tries something sensible — train a conformal predictor per data source, then merge the resulting intervals by taking the median lower and upper bound — and the experiments on the concrete strength data are honestly reported, with NDCP generally beating at least one individual source and staying near or above nominal coverage. The framing of non-disclosed multi-source regression is real and relevant. What is missing is any argument that the median interval is actually valid. The conformal guarantee applies to each source separately, not to the median of K intervals. For K=2, the median is the average, and if the two intervals each cover with probability 0.95, the average interval covers only when both cover. Under independence that is about 0.90. For K=3, coverage can be as low as 0.925 if misses are negatively dependent. The paper supplies no condition under which the needed positive dependence holds, and in Experiment 3 the sources are non-IID, so even per-source exchangeability is gone. So the headline claim 'conservatively valid' is not established; it rests on one UCI dataset and 100 runs, with no error bars, no code, and no hyperparameter ranges. That is a load-bearing gap.\n\nWhat is genuinely good: the problem setting is well described, the algorithm is simple and easy to implement, the experiments cover equal, unequal, and non-IID splits, and the authors are honest that pooled data is better, that two sources are a weak case, and that the 'Ideal NDCP' is purely hypothetical. The citation pattern is appropriate — the median combination is attributed to Park and Budescu, and the conformal machinery is standard.\n\nWhere I'd push: either prove a validity bound for the median under explicit assumptions about the sources (e.g., conditional coverage per source, or positive dependence), or substantially expand the empirical study across multiple datasets and report error bars on validity. As it stands, the paper is a useful empirical note, not a theoretically grounded method. But the question is legitimate and the experiments are not sloppy, so I would send it to peer review with the expectation of major revision.","headline":"Median-combined conformal intervals are a plausible heuristic, but the paper's central validity claim is unproven and not generally true.","tokens_in":15183,"tokens_out":2920,"would_cite":false,"duration_ms":28431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G15","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A privacy-preserving method trains conformal predictors separately on each data source and combines their prediction intervals by taking medians of the endpoints, yielding conservatively valid regression intervals without pooling data.","keywords":["conformal prediction","prediction intervals","regression","multiple data sources","non-disclosed data","support vector regression","cross-conformal prediction","inductive conformal prediction"],"falsifier":"Construct three sources whose per-source conformal intervals each have exactly 95% coverage but whose endpoint medians systematically exclude true labels, for example two tight intervals shifted to the same wrong region and one wide interval; measure the median interval's coverage over many test points. If coverage falls below 95%, the median-validity premise fails. The paper's non-IID experiment approaches this but does not include such an adversarial arrangement.","tokens_in":14246,"feed_emoji":"📊","tokens_out":6724,"duration_ms":58586,"temperature":0.7,"pith_summary":"This paper tries to establish that regression prediction intervals can be combined across separate, non-disclosed data sources without pooling the data, and that the combined interval remains valid and useful. The proposed method, Non-Disclosed Conformal Prediction (NDCP), trains a conformal predictor on each source independently, sends only the resulting interval for a new query to a central location, and merges the intervals by taking the medians of their lower and upper endpoints. In experiments on a public benchmark regression data set with two, four, and six sources, the merged intervals were conservative at the nominal 5% level and more efficient than using at least one arbitrarily chosen source, though strictly less efficient than training on all pooled data. If this holds, NDCP offers a light-weight, privacy-preserving way to get calibrated predictive uncertainty when data cannot be centralized.","feed_headline":"Median-combined prediction intervals stay valid without data pooling","feed_subtitle":"Per-source conformal intervals merged by median stay valid and beat any single source, though pooled data remains best.","key_machinery":"The load-bearing mechanism is Algorithm 1: run a conformal predictor on each of K data sources, transmit only the prediction interval for the new object, and at the aggregation location combine intervals endpoint-wise by taking the median of the K lower bounds and the median of the K upper bounds. The per-source predictors are inductive conformal predictors or cross-conformal predictors, using absolute residual or normalized residual nonconformity scores. The median combination rule follows the suggestion of Park and Budescu (2015). The paper's core claim is that this simple median operation on valid intervals preserves validity and improves efficiency over individual sources.","core_discovery":"The central claim is that the elementwise median combination of per-source conformal intervals yields conservatively valid prediction intervals in multi-source regression settings. For each source, an inductive or cross-conformal predictor produces a prediction interval for the test object; only these intervals are transmitted. The aggregator returns the interval formed by the median of all lower bounds and the median of all upper bounds. Across equal-size, unequal-size, and non-IID source partitions, observed coverage stayed at or above nominal levels and median interval width was generally below that of individual sources and in some cases best among all sources, while pooled training remained most efficient.","pith_inferences":["Editorial extension: a sharper merging rule, such as a trimmed mean of endpoints or per-source weighting by calibration size, is a natural next step that the paper leaves open; the 'Ideal NDCP' numbers indicate the potential gain.","Editorial extension: the paper does not provide a formal theorem that the median of valid conformal intervals is valid; proving such a statement for exchangeable-but-heterogeneous sources, or finding a counterexample, would settle the theoretical status of NDCP.","Editorial extension: since only intervals are disclosed, NDCP could in principle be run on top of any privacy-preserving channel that hides even the number of sources, matching the paper's non-disclosure framing."],"forward_implications":["NDCP lets each data owner keep its raw data private, sharing only intervals for the queried object, so predictive uncertainty can be obtained from distributed data without federated training or secure multi-party computation.","No single source needs to be chosen in advance: on the tested partitions NDCP matched or beat at least one individual source in efficiency and improved validity in the non-IID setting.","Because observed coverage is typically above nominal, there is slack in the median combination; an improved merging rule could narrow intervals toward the pooled-data performance shown by the 'Ideal NDCP' oracle.","The method's advantage grows with the number of sources and is weakest with only two sources, suggesting deployment should favor settings with several independent data holders."],"supporting_citations":[{"why":"Supplies the conformal prediction validity guarantee under exchangeability that NDCP builds on.","marker":"Vovk et al., 2005"},{"why":"Introduces inductive conformal prediction and the normalized nonconformity score used for per-source intervals.","marker":"Papadopoulos et al., 2002"},{"why":"Defines cross-conformal predictors, one of the two base predictors in the experiments.","marker":"Vovk, 2015"},{"why":"Warns that combined conformal p-values need not be uniformly distributed, motivating the empirical validity check for NDCP.","marker":"Linusson et al., 2017"},{"why":"Supplies the median-of-bounds interval aggregation rule that NDCP adopts.","marker":"Park and Budescu, 2015"},{"why":"Provides aggregated conformal prediction, the ensemble context NDCP extends to non-disclosed sources.","marker":"Carlsson et al., 2014"},{"why":"Gives the cross-conformal ridge-regression variant used as a base model.","marker":"Papadopoulos, 2015"},{"why":"Supplies the benchmark regression data used in all experiments.","marker":"Lichman et al., 2013"}],"fun_headline_variants":["Per-source conformal intervals: median combination beats single source","Median of per-source intervals gives valid coverage without data sharing","Combine conformal intervals by median: robust without pooling data","Non-disclosed conformal prediction: median combination is valid and efficient","Merge per-source intervals with median: valid prediction without data pooling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes, without proof, that taking the elementwise median of several individually valid prediction intervals yields a valid interval at the chosen significance level.","fun_headline_variants_meta":{"raw":{"variants":["Per-source conformal intervals: median combination beats single source","Median of per-source intervals gives valid coverage without data sharing","Combine conformal intervals by median: robust without pooling data","Non-disclosed conformal prediction: median combination is valid and efficient","Merge per-source intervals with median: valid prediction without data pooling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3360,"prompt_tokens":816,"completion_tokens":2544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":2460}},"tokens_in":432,"tokens_out":2544,"duration_ms":17276,"temperature":1.0,"reasoning_tokens":2460,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:09:21.913668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct three sources whose per-source conformal intervals each have exactly 95% coverage but whose endpoint medians systematically exclude true labels, for example two tight intervals shifted to the same wrong region and one wide interval; measure the median interval's coverage over many test points. If coverage falls below 95%, the median-validity premise fails. The paper's non-IID experiment approaches this but does not include such an adversarial arrangement.","supporting_citations":[{"cited_title":"Inductive confidence machines for regression","cited_arxiv_id":null,"evidence_quote":"Introduces inductive conformal prediction and the normalized nonconformity score used for per-source intervals."},{"cited_title":"Cross-conformal predictors","cited_arxiv_id":null,"evidence_quote":"Defines cross-conformal predictors, one of the two base predictors in the experiments."},{"cited_title":"o m, Ulf Johansson, and Tuve L \\","cited_arxiv_id":null,"evidence_quote":"Warns that combined conformal p-values need not be uniformly distributed, motivating the empirical validity check for NDCP."},{"cited_title":"Aggregating multiple probability intervals to improve calibration","cited_arxiv_id":null,"evidence_quote":"Supplies the median-of-bounds interval aggregation rule that NDCP adopts."},{"cited_title":"Aggregated conformal prediction","cited_arxiv_id":null,"evidence_quote":"Provides aggregated conformal prediction, the ensemble context NDCP extends to non-disclosed sources."},{"cited_title":"Cross-conformal prediction with ridge regression","cited_arxiv_id":null,"evidence_quote":"Gives the cross-conformal ridge-regression variant used as a base model."},{"cited_title":"Uci machine learning repository, 2013","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark regression data used in all experiments."}],"review_version":1}