REVIEW 2 major objections 5 minor 19 references
A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It
T0 review · 2 major / 5 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read On imbalanced virtual-screening data, standard conformal prediction hits its global coverage target while leaving the minority class badly under-covered; class-conditional calibration restores per-class reliability.
desk verdict Solid applied paper: known marginal-vs-Mondrian gap is severe and invisible on imbalanced MoleculeNet tasks, multi-architecture evidence is clean, and the conservation identity predicts the measured shortfalls. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The conservation identity of coverage: minority shortfall equals majority surplus times the imbalance ratio (π_maj/π_min). It follows directly from overall coverage being the prevalence-weighted average of per-class coverages, predicts measured BBBP gaps to within about one point, and orders severity across the four datasets.
What would settle it
On a large held-out imbalanced molecular endpoint run with the same split, score, and target, if marginal minority coverage stayed near 90 percent without majority over-coverage, or if Mondrian failed to restore minority coverage while overall coverage remained near target, the central claim would be false.
Extended reading notes
Core claim
On imbalanced molecular property datasets, standard marginal conformal prediction satisfies its global coverage guarantee while systematically under-covering the minority class—measured as low as 64.8 percent on BBBP and 4.2 percent on ClinTox at a 90 percent target—because overall coverage is the class-prevalence-weighted average of per-class coverages. The minority shortfall therefore equals the majority surplus amplified by the imbalance ratio. The collapse appears across three unrelated architectures and two nonconformity scores, is hidden by ordinary aggregate metrics, localizes to generic scaffolds present in both classes, and is repaired by class-conditional Mondrian calibration.
Load-bearing premise
The argument treats measured coverages as readable under the usual exchangeability of calibration and test compounds; when that is deliberately broken by scaffold splits the numbers get noisier and Mondrian slightly over-covers.
Editorial extensions
If this is right
- On any imbalanced screening task, report coverage per class, not only overall.
- Default to Mondrian (class-conditional) conformal calibration; the cost is modestly larger sets on the hard class.
- Overall accuracy, overall coverage, and selective accuracy on accepted compounds can all look healthy while minority reliability collapses.
- Confident minority errors concentrate on generic scaffolds (e.g., bare benzene and pyridine) that occur in both classes and carry little class signal.
- A one-number minority coverage gap can flag the failure before a campaign trusts the sets.
Reading between the lines
- The same amplification should appear wherever calibrated abstention meets rare labels—rare-disease diagnosis, fraud, adverse-event prediction—even though the paper only measures chemistry.
- A pre-calibration safety check that estimates the minority coverage gap from prevalence and baseline minority calibration alone could warn teams before any conformal wrapper is applied.
- Probability recalibration methods that leave a single pooled quantile unchanged will not close the gap; the fix has to act on the calibration partition.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper shows that on imbalanced MoleculeNet tasks (BACE, BBBP, Tox21 SR-ARE, held-out ClinTox), standard marginal split conformal prediction meets its global 90% coverage target while substantially under-covering the minority class (down to 64.8% on BBBP and 4.2% on ClinTox). The shortfall is reproduced across a random forest on ECFP fingerprints, a graph convolutional network, and a frozen ChemBERTa model (p < 0.001 each), and across both LAC and APS nonconformity scores. An elementary conservation identity (Eq. 7) equates the minority shortfall to the majority surplus amplified by the imbalance ratio π_maj/π_min; it predicts the BBBP gap to within about one point and orders severity across datasets. Aggregate accuracy, overall coverage, and selective-prediction summaries remain healthy, so the failure is easy to miss. Class-conditional (Mondrian) calibration restores minority coverage to target on every dataset and architecture at a modest cost in set size. Failures concentrate on generic scaffolds (benzene, pyridine); a simple cost model shows that correct abstention flips minority utility from net-negative to net-positive.
Significance. If the result holds, it is a practically important warning for the growing use of conformal prediction in virtual screening and other imbalanced decision settings. The multi-architecture, multi-score, multi-seed design with bootstrap CIs, Wilcoxon tests, a held-out ClinTox task, and scaffold-split robustness is stronger than a typical empirical note. The conservation identity is elementary but useful: it turns a known theoretical gap into a quantitative, pre-checkable prediction of severity from class prevalences alone. Code and fixed seeds are promised, which supports reproducibility. The contribution is correctly framed as empirical and explanatory rather than as a new method; that framing is appropriate and strengthens the paper.
major comments (2)
- Section 4.3 / Table 3 (ClinTox row): the near-total collapse (4.2% minority coverage) is load-bearing for the severity claim, yet the minority test set is only ~28 compounds per split. The paper notes coarseness and slight Mondrian overshoot, but the main text still presents 4.2% and MCG ≈ 86 as headline numbers without reporting per-seed ranges or binomial uncertainty. Adding those (or a leave-one-out style sensitivity) would make the strongest claim fully defensible rather than directionally clear but imprecise.
- Section 5, Eq. (7) and the BBBP prediction (63.6% predicted vs 64.8% measured): the identity is presented as predicting the gap 'to within one point,' yet the text suppresses finite-sample quantile corrections. Because the paper's explanatory claim rests on this numerical agreement, either restore the finite-sample form of the identity or report the residual after applying the same (n+1) correction used in Eq. (2), so readers can judge how much of the one-point residual is accounting versus approximation.
minor comments (5)
- Table 5 reports LAC marginal minority coverage as 64.1% while Table 3/4 and the abstract use 64.8%; the discrepancy is small but should be reconciled or explained (different seeds/rounding).
- Figure 1 reliability diagrams and Figure 6 score histograms would benefit from explicit class labels and a shared x-axis scale so the majority-mass vs minority-tail contrast is immediately readable.
- Section 3.5 / Eq. (5): the cost model is illustrative and the sensitivity sweep (Figure 4) is welcome; still, state once that B_tp, C_fp, C_lab are not estimated from real campaign data so the utility numbers are not over-interpreted as economics.
- Limitations correctly note the single α = 0.10; a one-sentence statement that the conservation identity is α-independent (only the realized surplus depends on α) would clarify that the mechanism is not tied to that choice.
- Minor typography: 'Whatmakesthefailuredangerous' and similar run-together words in the Introduction PDF text should be fixed for readability.
Circularity Check
No significant circularity; the conservation identity is an elementary algebraic rearrangement of weighted coverage, openly presented as definitional, while the core claims are independent multi-seed empirical measurements.
full rationale
The paper's load-bearing results are empirical: measured minority coverage under marginal conformal prediction falls well below the 90% target (64.8% BBBP, 38.9% Tox21, 4.2% ClinTox) while overall coverage hits target, the collapse reproduces across three architectures and two scores at p<0.001, Mondrian restores it, and aggregate metrics hide it. These are obtained from repeated random/scaffold splits on public MoleculeNet data with fixed seeds; nothing is fitted then re-predicted. The sole candidate for circularity is the Section 5 conservation identity (Eq. 7), which rearranges the definitional identity overall coverage = π_maj cov_maj + π_min cov_min into minority shortfall = (π_maj/π_min) imes majority surplus. The paper itself states that it "follows directly from the definition of weighted coverage" and is "not a new theoretical result but as an explanatory tool," and the numerical check (predicted 63.6% vs measured 64.8%) is simply consistency of the measured coverages under the finite-sample overall of 90.2%. Mondrian is the standard literature remedy (Vovk), MCG/CII are trivial derived summaries, and the cost model plugs measured rates into an illustrative utility. No self-citation is load-bearing, no uniqueness theorem is imported, no free parameter is fitted and renamed a prediction, and no ansatz is smuggled. The derivation chain is therefore self-contained and non-circular.
Assumptions & free parameters
free parameters (2)
- target error rate α =
0.10
- decision-cost constants (B_tp, C_fp=C_fn, C_lab) =
1 / 5 / 0.5
assumptions (4)
- standard math Exchangeability of calibration and test points yields the finite-sample marginal coverage guarantee for split conformal prediction.
- standard math Class-conditional (Mondrian) quantiles deliver per-class coverage ≥1−α when each class has its own calibration scores.
- domain assumption Base models handle the majority class better than the minority (lower nonconformity scores), which is the normal situation under imbalance.
- domain assumption MoleculeNet binary tasks (BBBP, BACE, Tox21 SR-ARE, ClinTox CT-TOX) after RDKit parsing are representative of imbalanced virtual-screening problems.
invented entities (1)
-
minority coverage gap (MCG) and coverage imbalance index (CII)
independent evidence
Cite this review
Pith. "Pith review of A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It." pith.science (2026). https://pith.science/paper/S7PBS5MZ
@misc{pith2026260706605,
author = {Pith},
title = {Pith review of: A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It},
year = {2026},
howpublished = {\url{https://pith.science/paper/S7PBS5MZ}},
note = {Machine review of arXiv:2607.06605}
}
read the original abstract
Conformal prediction is being adopted in drug discovery to put an honest number on model reliability: pick an error rate alpha, and the method returns prediction sets containing the true label with probability at least 1 - alpha. We show this guarantee can be dangerous on imbalanced datasets. Across four datasets, standard (marginal) conformal prediction hits its global 90% coverage target while leaving the minority class badly exposed: realized minority coverage falls to 64.8% on blood-brain-barrier penetration and to 4.2% on clinical-trial toxicity, where the rare class is nearly abandoned. The failure is not tied to one model: a random forest, a graph network, and a frozen chemical language model all reproduce it (p < 0.001 in every case), with severity tracking baseline calibration on rare labels rather than architecture. A conservation identity explains the effect: the minority's shortfall equals the majority's surplus amplified by the imbalance ratio, predicting the measured gap to within one point and ordering severity across datasets. The failure survives realistic scaffold splits and a second conformal score, while aggregate accuracy and overall coverage stay reassuringly high, which is exactly why it is easy to miss. Class-conditional (Mondrian) conformal prediction closes the gap on every dataset, restoring minority coverage to target for a modest increase in prediction-set size. We localize the failures to generic molecular scaffolds - plain benzene and pyridine cores occurring in both classes - propose a one-number diagnostic, and show with a cost model that abstaining on affected compounds flips a screening campaign from net-negative to net-positive utility. Our contribution is demonstrating on real chemistry how severe and invisible this known conformal-theory gap becomes under imbalance, and laying out a practical protocol restoring per-class reliability.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Vovk, V.; Gammerman, A.; Shafer, G.Algorithmic Learning in a Random World; Springer: New York, 2005
work page 2005
-
[2]
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
Angelopoulos, A. N.; Bates, S. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.Foundations and Trends in Machine Learning 2023, 16, 494–591. arXiv:2107.07511
work page Pith review arXiv 2023
-
[3]
Sadinle, M.; Lei, J.; Wasserman, L. Least Ambiguous Set-Valued Classifiers With Bounded Error Levels.Journal of the American Statistical Association2019, 114, 223–234
-
[4]
Classification with Valid and Adaptive Coverage
Romano, Y.; Sesia, M.; Candès, E. Classification with Valid and Adaptive Coverage. InAd- vances in Neural Information Processing Systems (NeurIPS), 2020. 15
work page 2020
-
[5]
Conditional Validity of Inductive Conformal Predictors.Machine Learning2013, 92, 349–376
Vovk, V. Conditional Validity of Inductive Conformal Predictors.Machine Learning2013, 92, 349–376
-
[6]
N.; Gomes, J.; Geniesse, C.; Pappu, A
Wu, Z.; Ramsundar, B.; Feinberg, E. N.; Gomes, J.; Geniesse, C.; Pappu, A. S.; Leswing, K.; Pande, V. MoleculeNet: A Benchmark for Molecular Machine Learning.Chemical Science 2018, 9, 513–530
work page 2018
-
[7]
Extended-Connectivity Fingerprints.Journal of Chemical Information and Modeling2010, 50, 742–754
Rogers, D.; Hahn, M. Extended-Connectivity Fingerprints.Journal of Chemical Information and Modeling2010, 50, 742–754
-
[8]
Random Forests.Machine Learning2001, 45, 5–32
Breiman, L. Random Forests.Machine Learning2001, 45, 5–32
Show all 19 references
-
[9]
N.; Welling, M
Kipf, T. N.; Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. InInternational Conference on Learning Representations (ICLR), 2017
2017
-
[10]
Fey, M.; Lenssen, J. E. Fast Graph Representation Learning with PyTorch Geometric. arXiv:1903.02428, 2019
1903 arXiv
-
[11]
ChemBERTa: Large-Scale Self-Supervised Pre- training for Molecular Property Prediction.arXiv:2010.09885, 2020
Chithrananda, S.; Grand, G.; Ramsundar, B. ChemBERTa: Large-Scale Self-Supervised Pre- training for Molecular Property Prediction.arXiv:2010.09885, 2020
2010 arXiv
-
[12]
Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning (ICML), 2017
2017
-
[13]
Introducing Conformal Prediction in Predic- tive Modeling
Norinder, U.; Carlsson, L.; Boyer, S.; Eklund, M. Introducing Conformal Prediction in Predic- tive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination. Journal of Chemical Information and Modeling2014, 54, 1596–1603
-
[14]
The Application of Conformal Prediction to the Drug Discovery Process.Annals of Mathematics and Artificial Intelligence2015, 74, 117–132
Eklund, M.; Norinder, U.; Boyer, S.; Carlsson, L. The Application of Conformal Prediction to the Drug Discovery Process.Annals of Mathematics and Artificial Intelligence2015, 74, 117–132
-
[15]
F.; Teixeira, A
Martins, I. F.; Teixeira, A. L.; Pinheiro, L.; Falcao, A. O. A Bayesian Approach to in Silico Blood–Brain Barrier Penetration Modeling.Journal of Chemical Information and Modeling 2012, 52, 1686–1697
2012
-
[16]
Subramanian, G.; Ramsundar, B.; Pande, V.; Denny, R. A. Computational Modeling ofβ- Secretase 1 (BACE-1) Inhibitors Using Ligand-Based Approaches.Journal of Chemical Infor- mation and Modeling2016, 56, 1936–1949
1936
-
[17]
A.; Rossoshek, A.; Simeonov, A
Huang, R.; Xia, M.; Nguyen, D.-T.; Zhao, T.; Sakamuru, S.; Zhao, J.; Shahane, S. A.; Rossoshek, A.; Simeonov, A. Tox21Challenge to Build Predictive Models of Nuclear Recep- tor and Stress Response Pathways as Mediated by Exposure to Environmental Chemicals and Drugs.Frontiers ...
-
[18]
RDKit: Open-Source Cheminformatics.https://www.rdkit.org
Landrum, G. RDKit: Open-Source Cheminformatics.https://www.rdkit.org
-
[19]
W.; Murcko, M
Bemis, G. W.; Murcko, M. A. The Properties of Known Drugs. 1. Molecular Frameworks. Journal of Medicinal Chemistry1996, 39, 2887–2893. 16
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.