{"id":"746a0c2a-5e42-4f8e-9de5-45c4fc352bfe","arxiv_id":"1908.03569","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A review of conformal prediction methods for estimating prediction reliability in computational drug discovery, with a worked hERG example and a survey of applications.","lead":"This review explains how conformal prediction (CP) can attach statistically guaranteed confidence intervals or label sets to machine-learning predictions in drug discovery. It summarizes CP variants, lists published applications, and discusses the exchangeability assumption that must hold for the guarantees to apply.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The classification tutorial conflates a confidence filter with conformal prediction: the majority-vote 'P value' does not yield the claimed 80% correctness guarantee.","rationale":"The reader's weakest assumption was the exchangeability condition, which the paper itself explicitly acknowledges in its limitations section (pp. 31-32). That is a real scope limitation, but the authors already state it, so it is not the most damaging issue for a review whose purpose is to teach CP correctly. The more load-bearing problem is in the classification tutorial: the paper describes a one-sided quantile filter on the majority-class vote fraction as if it were an Inductive Conformal Predictor and then attributes to CP a guarantee that the theorem does not give. Conformal classification validity applies to prediction sets built from per-label p-values; it does not ensure that a single label whose confidence percentile is high is correct with probability at least the confidence level. A confidently wrong but exchangeable model exposes this gap immediately. Since the hERG example and Figure 2 are presented as the main illustration of classification CP, a reader implementing the described protocol may believe they have a valid conformal predictor when they only have a calibrated confidence cutoff. This is an internal correctness issue in the exposition, not a disagreement with the consensus on CP theory, and it warrants a correction before the review is used as a guide. The concrete simulation test would settle whether the claimed guarantee fails for the described procedure; if it does, the relevant passages and Table 3 need to be revised or clarified.","tokens_in":25995,"tokens_out":20580,"duration_ms":221534,"concrete_test":"Simulate an exchangeable binary-classification problem with a model that always outputs top-class probability 0.9 while the true prevalence of that class is only 0.1. Implement the paper's ICP recipe: calibration scores are majority-class vote fractions; mark a test point reliable if its score exceeds the (1-epsilon) quantile of calibration scores with epsilon = 0.2; measure accuracy among reliable points. If accuracy is roughly 0.1 instead of at least 0.8, the described reliability filter is not a valid conformal predictor. For contrast, run a standard ICP with per-class p-values and verify that the prediction sets achieve empirical coverage of at least 0.8.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest promise, that CP produces valid confidence regions with a guaranteed coverage lower bound, is correct for a conformal predictor in the sense of Vovk et al. However, the worked Inductive Conformal Prediction for Classification (pp. 15-18, Figure 2 and Table 3) does not implement that definition. It computes a single nonconformity score from the RF vote fraction of the majority class, calibrates a percentile of that score, and calls a test prediction reliable when its score ranks above epsilon. This is a quantile filter on model confidence, not a per-label conformal p-value. CP validity applies to the set of labels whose individual p-values exceed epsilon; it does not imply that among predictions whose majority-vote score is high, at least 1-epsilon are correct. A miscalibrated but exchangeable model can be confidently wrong on most such 'reliable' instances, for example with constant top-class probability 0.9 while the true prevalence of that class is 0.1. Therefore the statement in the text that 'the mathematical validity of CP guarantees that at least 80% of the predictions considered reliable will be correct' is not a consequence of CP theory. The later Mondrian CP section uses class-conditioned scores, but the generic ICP section misleads practitioners about what validity actually guarantees. This is a more immediate correctness problem than the acknowledged exchangeability caveat, and it should be fixed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This review introduces conformal prediction (CP) to the computational drug discovery community, covering the core concepts of validity, efficiency, nonconformity measures, and confidence levels; the main CP modalities (inductive, Mondrian, aggregated, cross-conformal, and deep-learning variants); a broad survey of applications in virtual screening and activity modelling; open-source implementations; and current limitations, especially the exchangeability assumption. The authors include a worked hERG classification example and a regression example to illustrate ICP and MCP, and they provide a large table summarizing 25+ CP studies in drug discovery.","tokens_in":26278,"tokens_out":6824,"duration_ms":66940,"significance":"The manuscript is a useful and timely review of a technique that is rapidly gaining adoption for uncertainty quantification in drug discovery. Its strengths include the comprehensive survey of CP applications (Table 2), the practical descriptions of CP variants and their corresponding software, and the candid discussion of exchangeability failure in iterative screening. The core mathematical promise of CP, namely coverage validity at a user-specified confidence level, is correctly stated in the introduction and in the regression sections. However, the classification tutorial incorrectly presents a model-confidence filter as conformal prediction and claims a precision guarantee that CP does not provide. Because the paper aims to teach practitioners how to apply CP correctly, this is a load-bearing error that should be fixed before the review can serve as a reliable guide.","major_comments":[{"comment":"The worked example does not implement conformal prediction as defined by Vovk et al. It computes a single nonconformity score per test instance (the RF majority-vote fraction), thresholds it at the 80th percentile of the calibration scores, and labels a prediction 'reliable' when its score exceeds the threshold. The statement that 'the mathematical validity of CP guarantees that at least 80% of the predictions considered reliable will be correct' is not a consequence of conformal prediction theory. Conformal validity is a coverage guarantee for the set of labels whose individual p-values exceed the significance level; it is not a precision guarantee for a confidence-filtered subset. Even under exchangeability, a model that always outputs 0.9 for class A when the true prevalence of A is 0.1 will pass the threshold almost everywhere while being correct on only 10% of the 'reliable' predictions. To be a valid conformal predictor, the example must compute per-class nonconformity scores and per-label p-values (e.g., comparing the vote fraction for each class with the calibration distribution for that class), or it should be explicitly presented as a heuristic confidence filter that does not inherit CP's validity guarantee.","section":"Inductive Conformal Prediction for Classification (pp. 15-18, Figure 2, Table 3)"},{"comment":"The statement that 'at least 1-ε of the predictions for the minority class will be correct' conflates class-conditional coverage with precision. Mondrian CP guarantees that for test objects whose true class is k, the prediction set contains k with probability at least 1-ε; it does not guarantee that among objects assigned the single label k, at least 1-ε truly belong to k, since the latter depends on the prior class frequencies. Table 4 reports class-conditional coverage, so the prose should be revised to state the coverage guarantee precisely rather than implying a precision guarantee on single-label predictions.","section":"Mondrian Conformal Prediction (pp. 20-21)"}],"minor_comments":[{"comment":"The definition of classification efficiency in Table 1 (\"the fraction of single-class predictions that are correct\") is inconsistent with the definition in the Mondrian section (\"the single-label prediction rate\"); these are different quantities and should be aligned.","section":"Table 1 and p. 21"},{"comment":"The caption of Figure 5 repeats panel labels (b) and (c), misspells 'sorted' and 'corresponds', and references non-existent panels; the panel labels and caption text should be corrected.","section":"Figure 5 caption"},{"comment":"The typesetting of Equations 1 and 2 is garbled (e.g., '∝0=', '>?@ABC6@D6 E6FB?@='), making the formulas hard to read; the notation for predicted values and standard deviations should be defined consistently for calibration and test instances.","section":"Equations 1 and 2"},{"comment":"The claim that \"CP does not introduce more assumptions than those generally made when modelling bioactivity data\" is in tension with the later acknowledgement that exchangeability is not usually verified in practice and is breached in iterative screening; consider softening the earlier claim or pointing forward to the limitations section.","section":"p. 5-6 and pp. 31-32"},{"comment":"The statement that CP requires \"no parameterization ... except for the selection of a non-conformity measure\" overlooks the user-specified confidence level, which is also a parameter; this should be clarified.","section":"p. 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a review rather than a primary research article, and the main issue is pedagogical: the classification tutorial misstates what CP validity guarantees. The authors' self-citations to their prior CP papers are appropriate given the topic. The manuscript fits the journal's scope and, once the classification example and the associated precision claims are corrected, would be a valuable resource for practitioners."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a broadly useful review, but the classification tutorial is not actually conformal prediction as the term is defined in the literature. The hERG example computes a single non-conformity score from the RF majority vote, calibrates a percentile, and flags test instances above that percentile as reliable. That is a quantile filter on model confidence. Conformal prediction validity applies to the set of labels whose individual p-values exceed the significance level. It does not guarantee that among predictions whose vote fraction is high, at least 80% are correct. So the sentence 'the mathematical validity of CP guarantees that at least 80% of the predictions considered reliable will be correct' is wrong, and the confusion carries into Figure 2 and Table 3. The later Mondrian section is closer to correct, but the generic ICP part will mislead practitioners.\n\nThe rest of the review holds up reasonably well. The chronological table of applications is a good compilation, the coverage of ICP, MCP, ACP, CCP, and the deep learning variants (Deep Confidence, test-time dropout) is useful, and the authors are candid about the exchangeability assumption and about the low efficiency of regression intervals. The regression equations are standard. The limitation about iterative screening breaking exchangeability is a real point, and the authors cite their own work for it.\n\nMinor issues: Table 1 defines classification efficiency as the fraction of correct single-class predictions, while the MCP section defines it as the single-label prediction rate; these are different. And no code is shipped for the hERG illustration, making the exact numbers hard to reproduce.\n\nWho should read it: someone wanting a map of the CP literature in drug discovery and a sense of which variants have been applied. It is not the place to learn how to actually construct a conformal predictor for classification. With the classification section fixed, it would be a solid review. As is, I would send it to peer review and insist on that correction before acceptance.","headline":"Useful review of conformal prediction in drug discovery, but the classification tutorial confuses a confidence filter with CP and overclaims the validity guarantee.","tokens_in":26748,"tokens_out":2576,"would_cite":true,"duration_ms":24325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conformal prediction gives any drug-discovery model confidence regions whose coverage is guaranteed at a chosen level, provided the data are exchangeable.","keywords":["conformal prediction","drug discovery","virtual screening","QSAR","prediction uncertainty","applicability domain","exchangeability","Mondrian conformal prediction"],"falsifier":"Build an inductive conformal predictor at $\\text{CL}=0.8$ on a drug-discovery dataset, hold out a truly exchangeable test set, and count how often the true activity falls inside the predicted interval; if the empirical coverage is materially below 80% on a large test set, the practical claim of validity fails, signalling either a broken non-conformity implementation or an unnoticed breach of exchangeability. The authors' own iterative-screening experiment, in which selected molecules are no longer exchangeable with the training set, is the concrete setting where this failure is expected and observed.","tokens_in":25823,"feed_emoji":"💊","tokens_out":9132,"duration_ms":93068,"temperature":0.7,"pith_summary":"Conformal Prediction (CP) is a wrapper: given any machine-learning model, it turns a bare prediction into a confidence region whose coverage is guaranteed at a user-chosen level, provided the calibration and future data are exchangeable. This review argues that CP is the practical answer to the long-standing 'applicability domain' problem in drug discovery—at confidence level 80%, the true activity or label falls inside the predicted region in at least 80% of cases, with negligible extra compute and no retraining of the underlying algorithm. The authors work through an hERG potassium-channel dataset to show how an inductive conformal predictor can be globally valid yet locally unreliable for the minority active class, and they present Mondrian calibration as the fix that restores class-wise validity. They also survey aggregated and cross-conformal variants that use all labelled data, plus deep-learning recipes based on snapshot ensembles and test-time dropout. The review is candid that the guarantee is not assumption-free: when iterative screening breaks exchangeability, conformal predictors become 'useless,' so users must check the assumption rather than take coverage on faith.","feed_headline":"Guaranteed confidence regions for drug predictions, with one caveat","feed_subtitle":"At 80% confidence a valid predictor misses at most 20% of true values — if the data stays exchangeable.","key_machinery":"The load-bearing mechanism is the non-conformity score, a scalar that measures how unusual a new instance is relative to the training distribution—for example, the fraction of random-forest trees voting for the predicted class, or the absolute prediction residual scaled by the ensemble standard deviation. These scores are computed for a held-out calibration set and sorted; a test instance receives a P-value equal to the fraction of calibration scores at least as large as its own, and the prediction is accepted at significance level $\\epsilon = 1 - \\text{CL}$ when the P-value is at least $\\epsilon$. The quantile comparison is what converts an arbitrary model output into a region with guaranteed coverage under exchangeability, and the choice of non-conformity measure controls efficiency, meaning the tightness of the intervals or the rate of single-class predictions. Mondrian conformal prediction refines the machinery by maintaining a separate sorted score list per class, which is what restores class-wise validity on imbalanced data.","core_discovery":"The review's central claim is that conformal prediction converts any model used in computational drug discovery—random forests, SVMs, deep networks, matrix-factorization multitask models—into a predictor that reports a confidence region with a formal coverage guarantee: at confidence level CL, the true value (regression) or true class label (classification) is contained in the predicted region in at least CL of cases. The guarantee is distribution-free and does not depend on the choice of underlying algorithm or non-conformity measure; it follows from ranking a test instance's non-conformity score against the scores of a held-out calibration set. This is what distinguishes CP from earlier applicability-domain heuristics, which only correlate distance or ensemble variance with error but do not bound the error rate. The hERG case study in the review demonstrates the distinction between global and local validity: the model is well-calibrated overall, yet most active compounds get flagged as unreliable, motivating Mondrian (class-wise) conformal prediction as the standard treatment for imbalanced screening data.","pith_inferences":["If exchangeability is routinely violated in prospective use, the durable contribution of the review may be negative: it clarifies that distribution-free coverage is not assumption-free, and that the practical value of CP hinges on developing cheap exchangeability diagnostics or online re-calibration methods.","The efficiency bottleneck—intervals spanning multiple pIC50 units—is where future progress will matter most; the review's own benchmark suggests that non-conformity measures exploiting model-internal variance (bagged or dropout variance) beat simple distance-based measures, pointing to a general recipe of using learner-specific confidence signals.","The same quantile-calibration idea extends naturally to clinical decision support, where the guarantee would be attractive for patient-level risk predictions; but the exchangeability caveat is even more severe there, since patient cohorts are selected, not sampled, so the review's caveat transfers with added force.","A testable extension suggested by the review's comparative tables: use the calibration slope (observed vs nominal coverage) as a dataset-quality screen—datasets with inconsistent labels or duplicated records should show flatter calibration curves, turning conformal prediction into a diagnostic tool rather than only a prediction tool."],"forward_implications":["At a chosen confidence level such as 80%, a conformal predictor flags reliable predictions with a bounded false-positive rate, so a medicinal chemist can prioritize compounds knowing the error rate is capped rather than merely correlated with model confidence.","On imbalanced bioactivity sets, ordinary conformal validity is only global; the minority (active) class can receive mostly unreliable predictions, so Mondrian calibration is needed to guarantee per-class coverage.","Because CP adds negligible computational cost and no architectural changes, the same wrapper can be applied to random forests, SVMs, deep regression networks, and multitask matrix-factorization models, giving all of them comparable uncertainty statements.","If exchangeability is breached—as in iterative screening where model-selected molecules are tested next—the guarantee no longer holds and the predictors can become useless, so practitioners should diagnose exchangeability or re-calibrate at each screening round."],"supporting_citations":[{"why":"Supplies the mathematical proof that conformal predictors are valid under the randomness and exchangeability assumptions.","marker":"[36]"},{"why":"Provides the tutorial treatment of conformal prediction that the review follows for definitions of validity, efficiency, and the conformal algorithm.","marker":"[37]"},{"why":"Introduces conformal prediction to QSAR modelling and supplies the inductive and Mondrian workflows on which the review's tutorial section is built.","marker":"[32]"},{"why":"The authors' iterative-screening study showing that breaching exchangeability yields useless conformal predictors; the load-bearing evidence for the caveat.","marker":"[41]"},{"why":"Benchmarks six non-conformity measures for regression QSAR, supporting the recommendation to use ensemble-variance scaling for tighter intervals.","marker":"[40]"},{"why":"Establishes class-wise (conditional) validity of inductive Mondrian predictors, the basis for the imbalanced-data remedy.","marker":"[84]"},{"why":"Introduces Aggregated Conformal Prediction, one of the all-data training modalities reviewed.","marker":"[98]"},{"why":"Defines cross-conformal predictors, the other all-data modality that rotates the calibration set across k folds.","marker":"[96]"},{"why":"Shows how to generate conformal intervals for regression neural networks, the seed for the deep-learning CP methods reviewed.","marker":"[95]"}],"fun_headline_variants":["Conformal prediction: valid confidence for drug models","CP makes drug prediction errors trustworthy","Distribution-free confidence intervals for drug discovery","Conformal CP: reliability by design for screening","CP: valid confidence for drug AI, if data stays exchangeable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is exchangeability: the calibration set and the molecules to be predicted must come from the same underlying distribution without any ordering or selection effect — an assumption the review admits is 'not usually verified in practice' and is openly breached when iterative virtual screening selects which molecules to test next, in which case the coverage guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction: valid confidence for drug models","CP makes drug prediction errors trustworthy","Distribution-free confidence intervals for drug discovery","Conformal CP: reliability by design for screening","CP: valid confidence for drug AI, if data stays exchangeable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000726,"raw_usage":{"total_tokens":3258,"prompt_tokens":956,"completion_tokens":2302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2232}},"tokens_in":572,"tokens_out":2302,"duration_ms":16290,"temperature":1.0,"reasoning_tokens":2232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:14:21.788532+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an inductive conformal predictor at $\\text{CL}=0.8$ on a drug-discovery dataset, hold out a truly exchangeable test set, and count how often the true activity falls inside the predicted interval; if the empirical coverage is materially below 80% on a large test set, the practical claim of validity fails, signalling either a broken non-conformity implementation or an unnoticed breach of exchangeability. The authors' own iterative-screening experiment, in which selected molecules are no longer exchangeable with the training set, is the concrete setting where this failure is expected and observed.","supporting_citations":[{"cited_title":"C., Bender, A","cited_arxiv_id":null,"evidence_quote":"The authors' iterative-screening study showing that breaching exchangeability yields useless conformal predictors; the load-bearing evidence for the caveat."},{"cited_title":"& Svensson, F","cited_arxiv_id":null,"evidence_quote":"Establishes class-wise (conditional) validity of inductive Mondrian predictors, the basis for the imbalanced-data remedy."},{"cited_title":"& Norinder, U","cited_arxiv_id":null,"evidence_quote":"Introduces Aggregated Conformal Prediction, one of the all-data training modalities reviewed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines cross-conformal predictors, the other all-data modality that rotates the calibration set across k folds."},{"cited_title":"& Haralambous, H","cited_arxiv_id":null,"evidence_quote":"Shows how to generate conformal intervals for regression neural networks, the seed for the deep-learning CP methods reviewed."}],"review_version":1}