REVIEW 3 major objections 5 minor 15 references
Combining Prediction Intervals on Multi-Source Non-Disclosed Regression Datasets
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A privacy-preserving method trains conformal predictors separately on each data source and combines their prediction intervals by taking medians of the endpoints, yielding conservatively valid regression intervals without pooling data.
desk verdict Median-combined conformal intervals are a plausible heuristic, but the paper's central validity claim is unproven and not generally true. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is Algorithm 1: run a conformal predictor on each of K data sources, transmit only the prediction interval for the new object, and at the aggregation location combine intervals endpoint-wise by taking the median of the K lower bounds and the median of the K upper bounds. The per-source predictors are inductive conformal predictors or cross-conformal predictors, using absolute residual or normalized residual nonconformity scores. The median combination rule follows the suggestion of Park and Budescu (2015). The paper's core claim is that this simple median operation on valid intervals preserves validity and improves efficiency over individual sources.
What would settle it
Construct three sources whose per-source conformal intervals each have exactly 95% coverage but whose endpoint medians systematically exclude true labels, for example two tight intervals shifted to the same wrong region and one wide interval; measure the median interval's coverage over many test points. If coverage falls below 95%, the median-validity premise fails. The paper's non-IID experiment approaches this but does not include such an adversarial arrangement.
Extended reading notes
Core claim
The central claim is that the elementwise median combination of per-source conformal intervals yields conservatively valid prediction intervals in multi-source regression settings. For each source, an inductive or cross-conformal predictor produces a prediction interval for the test object; only these intervals are transmitted. The aggregator returns the interval formed by the median of all lower bounds and the median of all upper bounds. Across equal-size, unequal-size, and non-IID source partitions, observed coverage stayed at or above nominal levels and median interval width was generally below that of individual sources and in some cases best among all sources, while pooled training remained most efficient.
Load-bearing premise
The method assumes, without proof, that taking the elementwise median of several individually valid prediction intervals yields a valid interval at the chosen significance level.
Editorial extensions
If this is right
- NDCP lets each data owner keep its raw data private, sharing only intervals for the queried object, so predictive uncertainty can be obtained from distributed data without federated training or secure multi-party computation.
- No single source needs to be chosen in advance: on the tested partitions NDCP matched or beat at least one individual source in efficiency and improved validity in the non-IID setting.
- Because observed coverage is typically above nominal, there is slack in the median combination; an improved merging rule could narrow intervals toward the pooled-data performance shown by the 'Ideal NDCP' oracle.
- The method's advantage grows with the number of sources and is weakest with only two sources, suggesting deployment should favor settings with several independent data holders.
Reading between the lines
- Editorial extension: a sharper merging rule, such as a trimmed mean of endpoints or per-source weighting by calibration size, is a natural next step that the paper leaves open; the 'Ideal NDCP' numbers indicate the potential gain.
- Editorial extension: the paper does not provide a formal theorem that the median of valid conformal intervals is valid; proving such a statement for exchangeable-but-heterogeneous sources, or finding a counterexample, would settle the theoretical status of NDCP.
- Editorial extension: since only intervals are disclosed, NDCP could in principle be run on top of any privacy-preserving channel that hides even the number of sources, matching the paper's non-disclosure framing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Non-Disclosed Conformal Prediction (NDCP), a method for combining prediction intervals from K independently trained conformal predictors when the underlying training data cannot be pooled across sources. For each source, an inductive or cross-conformal predictor produces an interval for a test object, and the final interval is formed by taking the median of the lower bounds and the median of the upper bounds. The method is evaluated on the UCI Concrete Compressive Strength data set with support vector regression, using 2, 4, and 6 sources under equal, unequal, and non-IID source splits, with 100 repetitions per setting. The reported results compare validity and median interval width for NDCP, the individual sources, a hypothetical 'Ideal NDCP', and a pooled-data predictor. The paper claims that NDCP produces conservatively valid prediction intervals and improves efficiency over at least some individual sources, while remaining less efficient than full pooling.
Significance. If the validity claim were established, NDCP would be a practically attractive, lightweight method for privacy-preserving prediction with intervals, since it requires exchanging only test objects and intervals rather than training data. The paper has clear strengths: the experiments use held-out test labels; SVR hyperparameters are tuned only on training folds through cross-validation; the 'Ideal NDCP' quantity is transparently labeled as a post hoc object and is not used to support the central claim; and the non-IID experiment addresses an important practical concern. However, the central assertion of conservative validity currently rests on empirical tables from a single UCI data set rather than on a theorem, and the interval-median operation does not obviously inherit the conformal validity guarantee. The efficiency claim, as stated in the abstract, is also stronger than the 'at least one source' conclusion reached in the discussion. The contribution is therefore plausible but not yet established at the level claimed.
major comments (3)
- [Section 3, Algorithm 1] The paper provides no proof that the elementwise median of K individually valid conformal intervals is itself a valid prediction interval at the chosen significance level. Section 2 establishes validity only for a single conformal predictor, and the median operation does not automatically transfer that guarantee. The failure is not merely hypothetical: for K=2, where the median of two bounds is their average, take y=0 and let each source independently output the interval [-1,1] with probability 0.95 and the interval [-100,-99] with probability 0.05. Each interval has marginal 95% coverage, but the merged interval covers only when both sources cover, giving coverage 0.9025 at the 95% level. The paper tests only even K (2, 4, 6) and supplies no assumptions under which miss indicators are positively dependent; since the sources are disjoint partitions of one data set, such dependence cannot simply be assumed. This is a load-bearing gap in the manuscript's central claim of conservative validity.
- [Section 4.2.3, Table 3] Experiment 3 deliberately breaks the exchangeability assumption by partitioning the training data so that sources have different label distributions and none is identically distributed to the test set. Under these conditions the per-source conformal validity theorem from Section 2 does not apply, so the NDCP validity values in Table 3 (for example 0.953, 0.963, 0.965 for ICP with 2, 4, and 6 sources) have no theoretical backing and should not be presented as supporting the claim of conservatively valid intervals. The authors acknowledge the violation in the text, but they still use this experiment as evidence for NDCP validity. The non-IID setting can be retained as a stress test only if the validity claim is explicitly scoped as empirical rather than guaranteed.
- [Tables 1-3] The validity and efficiency results are reported as point summaries over 100 repetitions without standard errors, confidence intervals, or significance tests. Since the central claim is that NDCP is conservatively valid, the relevant comparison is whether observed validity is compatible with the nominal 95% level; for example, many entries near 0.96-0.97 could be consistent with 0.95 given Monte Carlo variability, while a value such as 0.9025 would not be. The efficiency comparisons also need uncertainty quantification before the abstract's claim of improved efficiency can be assessed. Without this, the empirical tables support only a qualitative impression, not the precise statements in the abstract and conclusions.
minor comments (5)
- [Abstract and Section 5] The abstract states that efficiency is improved 'as compared to predicting using a single arbitrarily chosen source', but the discussion (Section 5) and the tables support only improvement over at least one individual source; for example, in Table 1 with 2 sources and CCP, Source2 has width 26.748 while NDCP has width 26.813. The abstract should be aligned with the weaker claim that the experiments actually demonstrate.
- [Section 4.2.1] The sentence 'NDCP using ICP in all cases has a lower efficiency when compared to the individual data sources, except for 2 data sources but here NDCP has lower efficiency than one of the data sources' is confusing and internally contradictory; 'lower efficiency' means a larger interval, which is a worse outcome, and the comparison to individual sources in Table 1 does not uniformly support the sentence. This should be rephrased to describe interval widths clearly.
- [Algorithm 1] The median for an even number K is not defined in the algorithm; the paper should specify whether the median is the average of the two middle values, since all experimental settings use even K.
- [Section 3] The method description says the number of examples in each data source is not disclosed, but the paper does not discuss how this interacts with the median combination or with the theoretical claims; if source sizes are unknown, the user also cannot know n for NDCP, yet Section 4.1 reports n=927 for NDCP as the sum over sources. This point should be clarified.
- [Global] Several typographical issues should be corrected: 'information efficient' in Section 2, 'yield improve results' in Section 5, and 'approches' in the Abadi reference.
Circularity Check
No significant circularity: the NDCP validity and efficiency claims are evaluated on held-out test data, and the only post hoc quantity (Ideal NDCP) is explicitly hypothetical and not used as evidence for the central claim.
full rationale
The paper's central claim is empirical: that the proposed median-combination of per-source conformal intervals yields conservatively valid intervals and improved efficiency over at least one single source. Algorithm 1 combines intervals by taking the medians of the lower and upper bounds; this is presented as 'a simple and relatively naive approach', with the median combination attributed to Park and Budescu (2015). There is no derivation in which an output quantity is shown to be identical, by construction, to an input quantity. The validity and efficiency numbers in Tables 1-3 come from 100 repetitions on a held-out test set, while SVR hyperparameters were tuned only on training folds through cross-validation, so the reported results are not fitted to the target outcome. The 'Ideal NDCP' row is explicitly post hoc ('it is only possible to do after the true labels have been revealed') and is clearly labeled hypothetical, so it does not serve as support for the main claim. The paper's unproven assumption that the median of individually valid conformal intervals is itself valid is a correctness or rigor gap, not circularity: the paper does not claim this validity follows by construction from per-source validity. Self-citations (e.g., Carlsson et al. 2014, Vovk 2015) appear as background references for existing conformal aggregation methods and are not load-bearing for the empirical evaluation of NDCP. Consequently, no circular step can be identified by the standard of quoting equations that reduce to their own inputs.
Assumptions & free parameters
free parameters (1)
- SVR hyperparameters C, epsilon, gamma =
not reported; selected by grid search and 10-fold cross-validation
assumptions (3)
- domain assumption Observations are exchangeable within and across data sources.
- standard math Inductive and cross-conformal predictors produce valid intervals for each source.
- ad hoc to paper The median of the lower bounds and the median of the upper bounds of K valid intervals form a valid interval.
Cite this review
Pith. "Pith review of Combining Prediction Intervals on Multi-Source Non-Disclosed Regression Datasets." pith.science (2026). https://pith.science/paper/35BL3AV4
@misc{pith2026190805571,
author = {Pith},
title = {Pith review of: Combining Prediction Intervals on Multi-Source Non-Disclosed Regression Datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/35BL3AV4}},
note = {Machine review of arXiv:1908.05571}
}
read the original abstract
Conformal Prediction is a framework that produces prediction intervals based on the output from a machine learning algorithm. In this paper we explore the case when training data is made up of multiple parts available in different sources that cannot be pooled. We here consider the regression case and propose a method where a conformal predictor is trained on each data source independently, and where the prediction intervals are then combined into a single interval. We call the approach Non-Disclosed Conformal Prediction (NDCP), and we evaluate it on a regression dataset from the UCI machine learning repository using support vector regression as the underlying machine learning algorithm, with varying number of data sources and sizes. The results show that the proposed method produces conservatively valid prediction intervals, and while we cannot retain the same efficiency as when all data is used, efficiency is improved through the proposed approach as compared to predicting using a single arbitrarily chosen source.
Figures
Reference graph
Works this paper leans on
-
[1]
On the protection of private information in machine learning systems: Two recent approches
Mart \' n Abadi, Ulfar Erlingsson, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Nicolas Papernot, Kunal Talwar, and Li Zhang. On the protection of private information in machine learning systems: Two recent approches. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 1--6. IEEE, 2017
work page 2017
-
[2]
Conformal predictions for information fusion
Vineeth N Balasubramanian, Shayok Chakraborty, and Sethuraman Panchanathan. Conformal predictions for information fusion. Annals of Mathematics and Artificial Intelligence, 74 0 (1-2): 0 45--65, 2015
work page 2015
-
[3]
Leo Breiman. Random forests. Mach. Learn., 45 0 (1): 0 5--32, October 2001. ISSN 0885-6125. doi:10.1023/A:1010933404324. URL https://doi.org/10.1023/A:1010933404324
-
[4]
Aggregated conformal prediction
Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 231--240. Springer, 2014
work page 2014
-
[5]
Uci machine learning repository, 2013
Moshe Lichman et al. Uci machine learning repository, 2013
work page 2013
-
[6]
o m, Ulf Johansson, and Tuve L \
Henrik Linusson, Ulf Norinder, Henrik Bostr \"o m, Ulf Johansson, and Tuve L \"o fstr \"o m. On the calibration of aggregated conformal predictors. In Alex Gammerman, Vladimir Vovk, Zhiyuan Luo, and Harris Papadopoulos, editors, Proceedings of the Sixth Workshop on Conformal and Probabilistic Prediction and Applications, volume 60 of Proceedings of Machin...
work page 2017
-
[7]
Cross-conformal prediction with ridge regression
Harris Papadopoulos. Cross-conformal prediction with ridge regression. In International Symposium on Statistical Learning and Data Sciences, pages 260--270. Springer, 2015
work page 2015
-
[8]
Inductive confidence machines for regression
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning, pages 345--356. Springer, 2002
work page 2002
Show all 15 references
-
[9]
A Marauder's Map of Security and Privacy in Machine Learning
Nicolas Papernot . A Marauder's Map of Security and Privacy in Machine Learning . arXiv e-prints, art. arXiv:1811.01134, Nov 2018
2018 arXiv
-
[10]
Aggregating multiple probability intervals to improve calibration
Saemi Park and David V Budescu. Aggregating multiple probability intervals to improve calibration. Judgment and Decision Making, 10 0 (2): 0 130, 2015
2015
-
[11]
Shokri and V
R. Shokri and V. Shmatikov. Privacy-preserving deep learning. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 909--910, Sept 2015. doi:10.1109/ALLERTON.2015.7447103
2015
-
[12]
Combination of inductive mondrian conformal predictors
Paolo Toccaceli and Alexander Gammerman. Combination of inductive mondrian conformal predictors. Machine Learning, pages 1--22, 2018
2018
-
[13]
Transductive conformal predictors
Vladimir Vovk. Transductive conformal predictors. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 348--360. Springer, 2013
2013
-
[14]
Cross-conformal predictors
Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74 0 (1-2): 0 9--28, 2015
2015
-
[15]
Algorithmic learning in a random world
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005
2005
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.