REVIEW 4 major objections 5 minor 32 references
Conformal Prediction for Uncertainty Estimation in Drug-Target Interaction Prediction
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Clustering calibration by residual-score distributions gives the tightest and most subgroup-fair intervals for drug–target affinity prediction.
desk verdict A useful empirical comparison undermined by test-set hyperparameter selection and an undefined out-of-sample cluster mapping for CCP-NC. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-level cluster-conditioned calibration procedure CCP-NC. For every drug and every protein in the calibration set it builds an empirical CDF of that entity's nonconformity scores, extracts the 10th through 90th percentiles as a fixed-length vector, and runs k-means on the drug vectors and the protein vectors separately; the calibration set is split into one part used for clustering and a disjoint part used to estimate cluster-specific conformal quantiles. At test time the method looks up the drug cluster and the protein cluster, pools their nonconformity scores when both are available, uses one side when only one is, and falls back to the global quantile otherwise. This two-sided conditioning is what lets the intervals reflect uncertainty that originates on either side of the drug–target pair rather than averaging it away.
What would settle it
Take the new drug–protein split and a small labeled hold-out pool for the unseen drugs and proteins. Run CCP-NC twice: once assigning test entities to clusters using their true labels, and once assigning them by a label-free proxy such as nearest cluster centroid in feature space. If subgroup coverage at the 90% or 95% level falls well below nominal in the label-free version while the label-based version holds coverage, the claimed subgroup-level guarantee for unseen entities fails in the form the paper states it.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the way you group calibration data determines whether conformal intervals stay valid and narrow when drugs and proteins are sparse or entirely unseen. The authors construct a cluster-conditioned conformal predictor in which each drug and each protein is embedded by the empirical distribution of its absolute residuals (the 10th through 90th percentiles of its nonconformity-score ECDF), drugs and proteins are clustered separately with k-means, and each test interaction inherits a drug-cluster and a protein-cluster quantile that are combined according to which clusters are observable. In their KIBA experiments this nonconformity-based clustering produces the tightest intervals and lowest mean absolute coverage gaps per drug and per protein under the random split and under the new drug–protein split, while group-conditioned (Mondrian) conformal prediction performs best on interval width and often on subgroup coverage when exactly one entity is familiar. They conclude that cluster-conditioned conformal predictors, especially residual-driven ones, are a better tool for uncertainty quantification in drug–target interaction prediction.
Load-bearing premise
The load-bearing assumption is that a test drug or protein can be assigned to the cluster it would have received had its true interaction labels been available, but computing that cluster embedding requires exactly those labels, and the paper does not describe a label-free assignment rule.
Editorial extensions
If this is right
- In a random split of KIBA, CCP-NC gives the narrowest intervals and the smallest drug- and protein-level coverage gaps at both 90% and 95% coverage.
- In the fully unseen drug–protein split, CCP-NC remains the most reliable method, whereas group-conditioned CP degrades because its per-entity quantiles cannot be populated for novel drugs and proteins.
- When one entity (drug or protein) is represented in calibration, GCP can produce narrower intervals, so the best method depends on which side of the interaction is familiar.
- Because the clustering parameters and calibration ratio are selected by minimizing the subgroup coverage gap, the method comes with a practical tuning recipe rather than a theory-only recommendation.
Reading between the lines
- Not in the paper: the same two-sided residual-clustering recipe should transfer to other pair-based prediction tasks (drug–drug, protein–protein, or gene–environment interactions) where each sample belongs to two structured sides with their own residual distributions.
- Not in the paper: the comparison is on a single dataset with one gradient-boosting model; a natural next check is whether the CCP-NC advantage holds on larger drug–target benchmarks and with graph-neural-network or transformer predictors.
- Not in the paper: CCP-NN is implemented with raw-feature similarities, so it does not test whether a residual-aware local neighborhood would match the clustering method; that comparison would isolate the value of the cluster structure itself.
- Not in the paper: a deployable version needs a label-free rule for assigning an unseen drug or protein to a cluster, because the paper's ECDF embedding uses true labels; defining and testing such a rule is the open step between this result and practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a comparative study of conformal prediction methods for drug-target interaction (DTI) regression. It introduces three cluster-conditioned conformal predictors (CCP-NC, CCP-FC, and CCP-NN) and benchmarks them against marginal conformal prediction (MCP) and group-conditioned conformal prediction (GCP) on the KIBA dataset under four data-splitting strategies: Random, Drug, Protein, and New Drug-Protein. The primary claim, stated in the abstract and revisited in Section 5, is that the nonconformity-based variant CCP-NC yields the tightest prediction intervals and the most reliable subgroup-level coverage, especially in random and fully unseen drug-protein splits. Evaluation is based on empirical coverage, average interval width, and the Mean Absolute Coverage Gap (MACG) defined in Eq. (14).
Significance. Cluster-conditioned CP is a promising direction for interaction datasets where individual entity groups are too sparse for reliable Mondrian quantiles, and the paper's split design covers practically relevant generalization regimes. The introduction of the MACG criterion and the systematic comparison across four splits are also useful contributions. However, the significance as presented is weakened by two load-bearing flaws: (i) the CCP-NC procedure does not specify how test drugs/proteins that never appear in calibration are assigned to clusters, which makes the headline results for Drug, Protein, and New Drug-Protein splits non-computable as written; and (ii) the CCP hyperparameters are selected on the test-set MACG, so the reported advantages over untuned baselines are not an unbiased comparison. If these issues are addressed, the study could provide a valuable benchmark for uncertainty quantification in DTI prediction; in its current form, the central claim is not supported.
major comments (4)
- [§3.4.3, Eqs. (8)–(10)] The cluster labels κ*(d_i) and κ*(t_j) used at test time are undefined for the Drug, Protein, and New Drug-Protein splits. Clusters in CCP-NC are learned from ECDF embeddings of nonconformity scores computed on calibration interactions; a held-out drug or protein has no calibration interactions, so its ECDF embedding cannot be computed and no assignment rule is provided. The fallback sentence in §3.4.3 ("If a drug or protein in the test set belongs to a cluster not represented in the quantile estimation subset, we default to the global threshold") presupposes that a cluster assignment already exists; it does not specify how an unseen entity is embedded or assigned. Because Eq. (10) branches on the availability of d_i and t_j in D_cal and then refers to κ*(d_i) and κ*(t_j), none of its cases can be evaluated for unseen entities. The reported CCP-NC results in Figures 5–7 for these three splits are therefore not reproducible from the manuscript, and the abstract's central claim for fully unseen drug-protein splits is unsupported.
- [§4, Eq. (7)] The hyperparameters (γ, nclusters) for CCP-NC and CCP-FC are selected by minimizing the exact test-set metric used for the final comparison. Section 4 states that, "for each setting, we evaluated performance using the combined MACG across both drug and protein subgroups, as defined in Equation 7, and selected the configuration that minimized this value," and the MACG values reported in Figure 7 are computed on the test set. MCP and GCP have no equivalent tuning, so the comparison in Figures 6 and 7 is not apples-to-apples, and the reported advantages of CCP-NC and CCP-FC may simply reflect selection on the evaluation data. The authors should select hyperparameters on a held-out validation set or via nested cross-validation, and then evaluate the selected configuration once on the test set.
- [§3.4.3, Eq. (11)] The claimed cluster-level coverage statement ("CCP aims to ensure valid coverage at the cluster level, i.e., P(y ∈ C(x) | κ(x)=k) ≳ 1−α") is not backed by a finite-sample argument. Cluster assignments are estimated from calibration nonconformity scores, and the cluster-specific quantiles are computed from a second portion of the same calibration set; the standard split-conformal exchangeability argument does not automatically extend to this data-dependent partitioning. The paper uses the hedge "approximately exchangeable" and does not provide a formal guarantee or a simulation check. Since the reliability of the intervals is one of the paper's selling points, the authors should either state a precise set of assumptions under which cluster-level coverage holds or present a targeted simulation/calibration study demonstrating that it does in practice.
- [§3.4.2, Eq. (6)] The GCP implementation deviates from Mondrian CP in the case where both the test drug and protein are in the calibration set: Eq. (6) pools the nonconformity scores of the drug group and the protein group. This union is not a Mondrian cell, and it is unclear which group-conditional coverage statement is being evaluated for such test points. Because GCP is the main baseline in the Drug and Protein splits, the authors should either use a standard group definition (e.g., drug identity, protein identity, or the pair) consistently or explain and justify the merging rule.
minor comments (5)
- [Figure 5] The caption says "six CP methods" but the legend lists five methods (MCP, GCP, CCP-NC, CCP-FC, CCP-NN); please correct the mismatch.
- [§3.1] The text mentions a Box-Cox transformation of binding affinities, but no details are given and the nonconformity score in Eq. (1) is the raw absolute residual; clarify where and how the transformation is applied.
- [§3.4.5] The description of protein features as "binary feature profiles (e.g., amino acid composition)" is imprecise, since amino acid composition is a real-valued frequency vector; please specify the actual binary representation used.
- [General] The paper does not include a data/code availability statement; since the results depend on hyperparameter choices and random splits, releasing code and seeds would substantially improve reproducibility.
- [Figure 8] Figure 8 is hard to read: the subplot captions are small and legend strings such as "Gamma_Label" and "Compared CP Methods" appear to be leftover code artifacts; please redraw with clear labels.
Circularity Check
CCP-NC assigns unseen test drugs/proteins to clusters only via their own nonconformity scores (hence true labels), and its reported MACG advantage is selected by minimizing the very test metric it is then claimed to beat.
-
self definitional
[Section 3.4.3, Eqs. 8-10; cluster construction paragraph and Eq. 1; Section 3.3 entity-disjoint splits]
"First, for each drug di, we construct an empirical cumulative distribution function (ECDF) over its nonconformity scores and extract the 10th to 90th percentiles to form a fixed-length embedding vector. Using these embeddings, we apply k-means clustering with Euclidean distance to assign each drug to a unique drug cluster. ... For a test interaction (di, tj), we denote the cluster assignments by κ∗(di) and κ∗(tj), respectively."
The nonconformity scores used for these ECDF embeddings are si = |yi − fθ(xi)|, so a drug's cluster embedding is a function of its true labels. In the Drug, Protein, and New Drug–Protein splits, a held-out test drug or protein has no calibration interactions, so no score distribution exists to build an ECDF. The paper provides no feature-based surrogate for CCP-NC (unlike CCP-FC). Therefore κ∗(di) or κ∗(tj) can be assigned only by using the test label the interval is supposed to predict; if such labels were used, the reported coverage and width for unseen entities are circular by construction, and if they were not used, the reported results cannot be reproduced from the stated method.
-
fitted input called prediction
[Section 3.4.3, Eq. 7; Section 4 'best-performing configurations' paragraph]
"The optimal pair (γ, nclusters) is selected by minimizing the average MACG across drug and protein subgroups. ... The results presented above for CCP-NC and CCP-FC correspond to their best-performing configurations, selected from a grid ... and selected the configuration that minimized this value."
MACG, defined in Section 3.5, is the evaluation metric used to compare methods on test subgroups, and Figure 7 reports MACG computed on the test set. The text does not introduce a separate validation split: the configuration is chosen by minimizing the combined test MACG, and that minimum is then reported as the method's subgroup-coverage performance. Thus the reported CCP-NC/CCP-FC advantage in MACG is partly the result of fitting the method's hyperparameters to the exact outcome used for evaluation; MCP and GCP baselines are not tuned on the same objective, so the comparison is not independent.
full rationale
The paper's framework is not globally circular: on the Random Split, test drugs and proteins can appear in calibration, making CCP-NC's ECDF-based clustering computable without test labels, and CCP-FC's feature-based clustering is well-defined for unseen entities. The self-citations to Dewolf et al. [11] and Iliadis et al. [24] are not load-bearing for the central derivation. However, the strongest abstract claim—that nonconformity-based clustering gives the tightest intervals and best subgroup coverage 'especially in random and fully unseen drug–protein splits'—is compromised on the unseen-entity splits. Section 3.4.3 defines cluster membership through ECDFs of residuals, which require labels; no feature-based assignment is given for an unseen test drug or protein. If the experiments assigned unseen entities to clusters at all, the only route available under the paper's own equations is through the test labels, making the interval construction self-referential. Independently, Eq. 7 selects (γ, nclusters) by minimizing the same MACG metric on which CCP is later declared superior, and the results section confirms that reported numbers are the best-performing configurations from that selection. These two issues make a substantial part of the paper's central claim reduce by construction rather than by independent evaluation, though the framework retains independent content on random splits and for the feature-based variant.
Assumptions & free parameters
free parameters (4)
- nclusters =
Grid 1,5,10,...,50; selected per split and coverage level
- gamma =
0.25, 0.5, 0.75; selected per split and coverage level
- boxcox_lambda =
Not reported
- nn_k =
20
assumptions (4)
- domain assumption Exchangeability of calibration and test samples within each split
- domain assumption Approximate exchangeability of nonconformity scores within each cluster
- domain assumption Homoscedasticity after Box-Cox transformation
- domain assumption k-means with Euclidean distance yields meaningful subgroups
Cite this review
Pith. "Pith review of Conformal Prediction for Uncertainty Estimation in Drug-Target Interaction Prediction." pith.science (2026). https://pith.science/paper/BMF5L2WB
@misc{pith2026250518890,
author = {Pith},
title = {Pith review of: Conformal Prediction for Uncertainty Estimation in Drug-Target Interaction Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/BMF5L2WB}},
note = {Machine review of arXiv:2505.18890}
}
read the original abstract
Accurate drug-target interaction (DTI) prediction with machine learning models is essential for drug discovery. Such models should also provide a credible representation of their uncertainty, but applying classical marginal conformal prediction (CP) in DTI prediction often overlooks variability across drug and protein subgroups. In this work, we analyze three cluster-conditioned CP methods for DTI prediction, and compare them with marginal and group-conditioned CP. Clusterings are obtained via nonconformity scores, feature similarity, and nearest neighbors, respectively. Experiments on the KIBA dataset using four data-splitting strategies show that nonconformity-based clustering yields the tightest intervals and most reliable subgroup coverage, especially in random and fully unseen drug-protein splits. Group-conditioned CP works well when one entity is familiar, but residual-driven clustering provides robust uncertainty estimates even in sparse or novel scenarios. These results highlight the potential of cluster-based CP for improving DTI prediction under uncertainty.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Deepdta: deep drug–target binding affinity prediction
Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug–target binding affinity prediction. Bioinformatics, 34(17):i821–i829, 2018
2018
-
[2]
Springer, 2005
Vladimir V ovk, Alexander Gammerman, and Glenn Shafer.Algorithmic learning in a random world, volume 29. Springer, 2005
2005
-
[3]
A gentle introduction to conformal prediction and distribution-free uncertainty quantification
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021
arXiv 2021
-
[4]
Class-conditional conformal prediction with many classes
Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-conditional conformal prediction with many classes. Advances in neural information processing systems, 36:64555–64576, 2023
work page 2023
-
[5]
Mondrian confidence machine.Technical Report, 2003
Vladimir V ovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman. Mondrian confidence machine.Technical Report, 2003
2003
-
[6]
Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis
Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wennerberg, and Tero Aittokallio. Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis. Journal of chemical information and modeling, 54(3):735–743, 2014
2014
-
[7]
C. Saunders, A. Gammerman, and V . V ovk. Transduction with confidence and credibility. InProceedings of the 16th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’99, page 722–726, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc
work page 1999
-
[8]
Inductive confidence machines for regression
Harris Papadopoulos, Kostas Proedrou, V olodya V ovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19–23, 2002 proceedings 13, pages 345–356. Springer, 2002. 12 A PREPRINT - AUGUST 12, 2025 1 5 10 15 20 25 30 35 40 45 50 0.090 0.100 ...
work page 2002
Show all 32 references
-
[9]
A review of nonconformity measures for conformal prediction in regression
Yuko Kato, David MJ Tax, and Marco Loog. A review of nonconformity measures for conformal prediction in regression. Conformal and Probabilistic Prediction with Applications, pages 369–383, 2023
2023
-
[10]
Constructing normalized nonconformity measures based on maximizing predictive efficiency
Anthony Bellotti. Constructing normalized nonconformity measures based on maximizing predictive efficiency. In Conformal and Probabilistic Prediction and Applications, pages 41–54. PMLR, 2020
2020
-
[11]
Conditional validity of heteroskedastic conformal regression
Nicolas Dewolf, Bernard De Baets, and Willem Waegeman. Conditional validity of heteroskedastic conformal regression. arXiv preprint arXiv:2309.08313, 2023
2023 arXiv
-
[12]
Conditional validity of inductive conformal predictors
Vladimir V ovk. Conditional validity of inductive conformal predictors. InProceedings of the Asian Conference on Machine Learning, volume 25 of Proceedings of Machine Learning Research, pages 475–490, Singapore, Nov
-
[13]
Distribution-free prediction bands for non-parametric regression
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71–96, 07 2013
2013
-
[14]
The limits of distribution-free conditional predictive inference
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455–482, 2021
2021
-
[15]
Conformal prediction with conditional guarantees
Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, page qkaf008, 2025
2025
-
[16]
Batch multivalid conformal prediction
Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Batch multivalid conformal prediction. arXiv preprint arXiv:2209.15145, 2022
2022 arXiv
-
[17]
Conformal prediction under covariate shift
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019. 13 A PREPRINT - AUGUST 12, 2025
2019
-
[18]
Conformal prediction with local weights: randomization enables local guarantees
Rohan Hore and Rina Foygel Barber. Conformal prediction with local weights: randomization enables local guarantees. arXiv preprint arXiv:2310.07850, 2023
2023 arXiv
-
[19]
Localized conformal prediction: A generalized inference framework for conformal prediction
Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110(1):33–50, 2023
2023
-
[20]
Clustered conformal prediction for the housing market
Anders Hjort, Jonathan P Williams, and Johan Pensar. Clustered conformal prediction for the housing market. Proceedings of Machine Learning Research, 230:1–21, 2024
2024
-
[21]
Normalized nonconformity measures for automated valuation models
Zhe Lim and Anthony Bellotti. Normalized nonconformity measures for automated valuation models. Expert Systems with Applications, 180:115165, 2021
2021
-
[22]
Functional protein mining with conformal guarantees
Ron S Boger, Seyone Chithrananda, Anastasios N Angelopoulos, Peter H Yoon, Michael I Jordan, and Jennifer A Doudna. Functional protein mining with conformal guarantees. Nature Communications, 16(1):85, 2025
2025
-
[23]
Codrug: Conformal drug property prediction with density estimation under covariate shift
Siddhartha Laghuvarapu, Zhen Lin, and Jimeng Sun. Codrug: Conformal drug property prediction with density estimation under covariate shift. Advances in Neural Information Processing Systems, 36:37728–37747, 2023
2023
-
[24]
A comparison of embedding aggregation strategies in drug–target interaction prediction
Dimitrios Iliadis, Bernard De Baets, Tapio Pahikkala, and Willem Waegeman. A comparison of embedding aggregation strategies in drug–target interaction prediction. BMC bioinformatics, 25(1):59, 2024
2024
-
[25]
Dynamic applicability domain (dad): compound–target binding affinity estimates with local conformal prediction
Davor Oršoli´c and Tomislav Šmuc. Dynamic applicability domain (dad): compound–target binding affinity estimates with local conformal prediction. Bioinformatics, 39(8):btad465, 2023
2023
-
[26]
Smiles, a chemical language and information system
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988
1988
-
[27]
Extended-connectivity fingerprints
David Rogers and Mathew Hahn. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742–754, 2010
2010
-
[28]
Mordred: a molecular descriptor calculator
Hirotomo Moriwaki, Yu-Shi Tian, Norihito Kawashita, and Tatsuya Takagi. Mordred: a molecular descriptor calculator. Journal of cheminformatics, 10(1):4, 2018
2018
-
[29]
Prottrans: towards cracking the language of life’s code through self-supervised learning
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: towards cracking the language of life’s code through self-supervised learning. IEEE Transactions on Patte...
2021
-
[30]
Greedy function approximation: a gradient boosting machine
Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001
2001
-
[31]
Regression conformal prediction with nearest neighbours
Harris Papadopoulos, Vladimir V ovk, and Alex Gammerman. Regression conformal prediction with nearest neighbours. Journal of Artificial Intelligence Research, 40:815–840, 2011
2011
-
[32]
An analysis of transformations
George EP Box and David R Cox. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 26(2):211–243, 1964. 14
1964
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.