REVIEW 2 major objections 5 minor 55 references
Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap
T0 review · 2 major / 5 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read Single-cell perturbations are distinguishable as populations but not separable cell by cell, so scoring averaged classifier probabilities recovers near-perfect identity where per-cell accuracy plateaus.
desk verdict Clean diagnosis of class-overlap in single-cell perturbation scoring, with a minimal fix (CDS) that beats PDS on sample efficiency; scoped carefully and worth engaging. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Classifier Discrimination Score (CDS): average a trained classifier’s softmax probability vectors over all cells of a perturbation to form a population profile, then rank candidate perturbations by that profile. It reuses an ordinary cell-level classifier without retraining, runs in linear time, and differs from PDS only by averaging in a learned discriminative probability space rather than raw gene-expression space.
What would settle it
On the same Tahoe or VCC splits, if raw-pseudobulk PDS matched or beat CDS rank-1 accuracy across cell fractions, or if CDS collapsed once the classifier was trained on a disjoint set of perturbations from those being scored, the reliability claim would fail.
Extended reading notes
Core claim
Perturbation classes in single-cell data are distinguishable as distributions yet not separable cell by cell. Cell-level accuracy is therefore capped by overlap and plateaus near macro-F1 0.2–0.3 regardless of model capacity, even while pairwise energy-distance tests reject almost every pair. Averaging a classifier’s per-cell probability vectors into a population profile (CDS) recovers rank-1 identification between 0.976 and 1.000 on Tahoe-100M from the same models, and does so more reliably than raw-pseudobulk PDS on both Tahoe-100M and the Virtual Cell Challenge, with the gap widening as evaluation cells become scarce.
Load-bearing premise
The paper treats recovering the identity of perturbations already seen in classifier training, from real held-out cells of those same classes, as the right test of a discrimination metric that will later rank generative models whose outputs may leave that closed set.
Editorial extensions
If this is right
- Per-cell accuracy and raw-pseudobulk scores can misrank ground truth and therefore misrank the models scored against them.
- Discrimination metrics for perturbation and virtual-cell predictors should score populations rather than individual cells.
- When evaluation cells are scarce, averaging through a trained classifier is more sample-efficient than averaging raw expression.
- Near-perfect CDS recovery shows the signal for known perturbations lives in the population even when no single cell carries it.
- Benchmarks that rely on PDS leave usable discriminative signal unmeasured.
Reading between the lines
- The same population-level probability average could serve as a scoring head inside generative training, not only at evaluation time.
- If perturbed populations are mixtures of responders and non-responders, CDS may implicitly weight responders via the learned map; comparing it to explicit mixture models would test that.
- Scoring truly novel or combinatorial perturbations would require open-set or multi-label extensions, since CDS ranks only classes seen in training.
- The gap between distinguishability and separability may appear in other high-dimensional biological assays, so population scoring could improve evaluation beyond chemical and genetic perturbations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that single-cell perturbation data violates the usual separability assumption of classification: perturbations can be statistically distinguishable as distributions while heavily overlapping cell-by-cell, so per-cell accuracy and macro-F1 largely measure that overlap rather than model quality. On Tahoe-100M (A549, two doses, 380 classes) and the Virtual Cell Challenge (301 genetic perturbations), linear, MLP, and Transformer classifiers all plateau at cell-level macro-F1 ~0.18–0.31 even though energy-distance permutation tests reject almost every pair. The proposed fix is the Classifier Discrimination Score (CDS): average a trained classifier’s per-cell softmax vectors over a perturbation’s cells and take the argmax of that average. The same models then recover rank-1 perturbation identity of 0.976–1.000 on Tahoe and substantially higher accuracy on VCC. CDS is shown to be more sample-efficient and more reliable than the pseudobulk Perturbation Discrimination Score (PDS) used in recent benchmarks, with the gap widening as the number of evaluation cells shrinks. The authors conclude that per-cell accuracy and raw-pseudobulk scores should be used with caution when ranking perturbation models.
Significance. If the empirical pattern holds, the paper supplies a concrete, low-cost correction to a metric that is already used to rank generative perturbation models (PDS in the Virtual Cell Challenge and related benchmarks). The diagnosis of distinguishable-but-not-separable classes is cleanly illustrated with a one-dimensional Gaussian example, energy-distance tests, and capacity-plateau experiments across three architectures and two datasets. CDS requires no retraining, is linear in the number of cells, and is a natural many-class extension of classifier two-sample tests. The work therefore has immediate practical value for evaluation design in single-cell perturbation modeling and is carefully scoped: it validates a discrimination metric on real labeled cells rather than claiming zero-shot prediction of unseen perturbations.
major comments (2)
- The central claim is that a metric that misranks ground-truth real cells will misrank generative models scored against it (§1, §6, Discussion). All reported CDS/PDS comparisons, however, use real held-out cells whose labels are already known to the classifier (Tables 1–2, Figures 2–3). The Discussion acknowledges that generative predictions may lie outside the training support, yet no experiment is provided that applies CDS (or PDS) to synthetic cells produced by a forward model. Without that bridge, the practical recommendation to replace PDS remains an extrapolation. A minimal addition would be to score a small set of published forward-model predictions (or simple baselines) under both metrics and show that the ranking of models changes when CDS is used.
- §5 and Appendix B: CDS inherits the inductive bias of the underlying classifier. The paper shows that linear, MLP and Transformer all reach near-perfect CDS once pooled, but does not quantify how sensitive the ranking is to classifier misspecification (e.g., under-trained models, wrong gene panel, or class-weighting choices). Because the metric is intended for model comparison, a short ablation that deliberately degrades the classifier (early stopping at low validation F1, random gene subsets) and reports the resulting drop in rank-1 would strengthen the claim that CDS is robust rather than merely “as good as the best classifier one can train.”
minor comments (5)
- Figure 2 y-axis starts at 0.84 and Figure 5 at 0.75; the caption should state this explicitly so readers do not misread absolute performance.
- Appendix E table header “VCC datasaest results” contains a typo (“datasaest”).
- The energy-distance formula in §4 is standard, but the permutation-test details (number of permutations, exact p-value formula) appear only in Appendix C; a one-sentence pointer in the main text would help.
- Notation for the CDS profile (¯π_p) is introduced cleanly, yet the relationship to the feature-map view ϕ(p) = (1/n) Σ ψ(x_i) could be stated once as an equation for easier citation.
- References to Liu et al. (2025) on PDS scale-sensitivity are well placed; a brief note on whether CDS inherits any analogous scale sensitivity (softmax outputs are already normalized) would be useful.
Circularity Check
No significant circularity; CDS is a post-hoc readout of a standard classifier, evaluated on held-out real cells of known classes by design.
full rationale
The paper trains ordinary supervised classifiers (linear, MLP, Transformer) with class-weighted cross-entropy on labeled single-cell data, then changes only the evaluation readout: average the per-cell softmax vectors over a perturbation's held-out cells and take argmax (Section 5). This recovers near-perfect rank-1 identification of the same known perturbations (Table 1, Figures 2-3). The authors explicitly scope the claim as metric validation for discrimination of already-seen classes, not zero-shot or generative prediction (Discussion: 'CDS is scored on the perturbations its classifier was trained to recognize, and that is by design'). Distinguishability is established independently via energy-distance permutation tests on the raw data (Appendix C). CDS vs. PDS is a head-to-head retrieval comparison on stratified real-cell splits, differing only in the feature map being averaged (raw expression vs. learned probabilities). No equation equates a claimed prediction to a fitted input by construction; no self-citations appear in the reference list; no uniqueness theorem or ansatz is imported. The derivation is self-contained empirical evidence on external atlases.
Assumptions & free parameters
free parameters (2)
- HVG panel size d
- Classifier architecture hyperparameters (layer widths, learning rates, patience)
assumptions (3)
- domain assumption Cells of a perturbation are i.i.d. draws from a distribution Dp over gene-expression space.
- standard math Energy distance is zero iff two distributions are identical, and a permutation test yields valid p-values under exchangeability.
- domain assumption Averaging unsystematic cell-level prediction errors cancels noise while preserving the consistent population-level signal learned by the classifier.
invented entities (1)
-
Classifier Discrimination Score (CDS)
independent evidence
Cite this review
Pith. "Pith review of Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap." pith.science (2026). https://pith.science/paper/ECDASA7Z
@misc{pith2026260704595,
author = {Pith},
title = {Pith review of: Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap},
year = {2026},
howpublished = {\url{https://pith.science/paper/ECDASA7Z}},
note = {Machine review of arXiv:2607.04595}
}
read the original abstract
Most classification problems assume the classes are roughly separable, so that an individual sample can usually be assigned to one class. Single-cell perturbation data violates this assumption: two perturbations can produce different populations of cells while overlapping so much that an individual cell could belong to either. Per-cell accuracy then measures this overlap rather than model quality. We see this on Tahoe-100M and the Virtual Cell Challenge, where a linear classifier, an MLP, and a Transformer all plateau near macro-F1 0.2-0.3 even though almost every pair of perturbations is statistically distinguishable. The fix is to score perturbations across the whole population rather than cell by cell. We average a classifier's per-cell probability vectors over all cells of a perturbation to form a population profile, then rank candidate perturbations by this profile; we call the resulting score the Classifier Discrimination Score (CDS). Taking the top-ranked class recovers the winning perturbation. It needs no retraining, costs linear time in the number of cells, and recovers near-perfect identification from the same weak models. CDS differs from the pseudobulk-based Perturbation Discrimination Score (PDS) used in recent benchmarks only in where the average is taken, raw gene expression for PDS versus a learned discriminative space for CDS, and identifies the true perturbation more reliably on both datasets, with the gap widening as cells grow scarce. Because a metric that misranks the ground truth will misrank the models scored against it, per-cell accuracy and raw-pseudobulk scores should be used with caution when comparing perturbation models.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
cell , volume=
Perturb-Seq: dissecting molecular circuits with scalable single-cell RNA profiling of pooled genetic screens , author=. cell , volume=. 2016 , publisher=
2016
-
[2]
Cell , volume=
A multiplexed single-cell CRISPR screening platform enables systematic dissection of the unfolded protein response , author=. Cell , volume=. 2016 , publisher=
2016
-
[3]
Cell , volume=
Dissecting immune circuits by linking CRISPR-pooled screens with single-cell RNA-seq , author=. Cell , volume=. 2016 , publisher=
2016
-
[4]
Cell , volume=
Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq , author=. Cell , volume=. 2022 , publisher=
2022
-
[5]
Molecular systems biology , volume=
Current best practices in single-cell RNA-seq analysis: a tutorial , author=. Molecular systems biology , volume=. 2019 , publisher=
2019
-
[6]
Nature methods , volume=
Learning single-cell perturbation responses using neural optimal transport , author=. Nature methods , volume=. 2023 , publisher=
2023
-
[7]
BioRxiv , pages=
Tahoe-100m: A giga-scale single-cell perturbation atlas for context-dependent gene function and cellular modeling , author=. BioRxiv , pages=. 2025 , publisher=
2025
-
[8]
Science , volume=
Massively multiplex chemical transcriptomics at single-cell resolution , author=. Science , volume=. 2020 , publisher=
2020
Show all 55 references
-
[9]
Cell , volume=
Virtual Cell Challenge: Toward a Turing test for the virtual cell , author=. Cell , volume=. 2025 , publisher=
2025
-
[10]
Science , volume=
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes , author=. Science , volume=. 2019 , publisher=
2019
-
[11]
Nature Methods , volume=
scPerturb: harmonized single-cell perturbation data , author=. Nature Methods , volume=. 2024 , publisher=
2024
-
[12]
Nature methods , volume=
scGen predicts single-cell perturbation responses , author=. Nature methods , volume=. 2019 , publisher=
2019
-
[13]
Molecular systems biology , volume=
Predicting cellular responses to complex perturbations in high-throughput screens , author=. Molecular systems biology , volume=. 2023 , publisher=
2023
-
[14]
Nature Biotechnology , volume=
Predicting transcriptional outcomes of novel multigene perturbations with GEARS , author=. Nature Biotechnology , volume=. 2024 , publisher=
2024
-
[15]
arXiv preprint arXiv:2408.10609 , year=
Perturbench: Benchmarking machine learning models for cellular perturbation analysis , author=. arXiv preprint arXiv:2408.10609 , year=
-
[16]
Nature Methods , volume=
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines , author=. Nature Methods , volume=. 2025 , publisher=
2025
-
[17]
Nature Methods , volume=
Comparison of transformations for single-cell RNA-seq data , author=. Nature Methods , volume=. 2023 , publisher=
2023
-
[18]
BioRxiv , pages=
Depth normalization for single-cell genomics count data , author=. BioRxiv , pages=. 2022 , publisher=
2022
-
[19]
Nature biotechnology , volume=
Spatial reconstruction of single-cell gene expression data , author=. Nature biotechnology , volume=. 2015 , publisher=
2015
-
[20]
Cell , volume=
Comprehensive integration of single-cell data , author=. Cell , volume=. 2019 , doi=
2019
-
[21]
Normalization and variance stabilization of single-cell
Hafemeister, Christoph and Satija, Rahul , journal=. Normalization and variance stabilization of single-cell. 2019 , doi=
2019
-
[22]
Analytic
Lause, Jan and Berens, Philipp and Kobak, Dmitry , journal=. Analytic. 2021 , doi=
2021
-
[23]
The journal of machine learning research , volume=
A kernel two-sample test , author=. The journal of machine learning research , volume=. 2012 , publisher=
2012
-
[24]
Journal of statistical planning and inference , volume=
Energy statistics: A class of statistics based on distances , author=. Journal of statistical planning and inference , volume=. 2013 , publisher=
2013
-
[25]
wiley interdisciplinary reviews: Computational statistics , volume=
Energy distance , author=. wiley interdisciplinary reviews: Computational statistics , volume=. 2016 , publisher=
2016
-
[26]
2005 , publisher=
Permutation, parametric and bootstrap tests of hypotheses , author=. 2005 , publisher=
2005
-
[27]
Statistical Science , pages=
Permutation methods: a basis for exact inference , author=. Statistical Science , pages=. 2004 , publisher=
2004
-
[28]
Nature communications , volume=
Muscat detects subpopulation-specific state transitions from multi-sample multi-condition single-cell transcriptomics data , author=. Nature communications , volume=. 2020 , publisher=
2020
-
[29]
Nature communications , volume=
Confronting false discoveries in single-cell differential expression , author=. Nature communications , volume=. 2021 , publisher=
2021
-
[30]
Journal of the American statistical association , volume=
Hierarchical grouping to optimize an objective function , author=. Journal of the American statistical association , volume=. 1963 , publisher=
1963
-
[31]
Journal of classification , volume=
Hierarchical clustering via joint between-within distances: Extending Ward's minimum variance method , author=. Journal of classification , volume=. 2005 , publisher=
2005
-
[32]
Information systems research , volume=
Research commentary—too big to fail: large samples and the p-value problem , author=. Information systems research , volume=. 2013 , publisher=
2013
-
[33]
2013 , publisher=
A probabilistic theory of pattern recognition , author=. 2013 , publisher=
2013
-
[34]
2009 , publisher=
The elements of statistical learning: data mining, inference, and prediction , author=. 2009 , publisher=
2009
-
[35]
Advances in neural information processing systems , volume=
Attention is all you need , author=. Advances in neural information processing systems , volume=
-
[36]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Class-balanced loss based on effective number of samples , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[37]
arXiv preprint arXiv:2511.16954 , year=
Effects of Distance Metrics and Scaling on the Perturbation Discrimination Score , author=. arXiv preprint arXiv:2511.16954 , year=
-
[38]
bioRxiv , year=
MORPH Predicts the Single-Cell Outcome of Genetic Perturbations Across Conditions and Data Modalities , author=. bioRxiv , year=
-
[39]
Roohani and Alexander Dobin and Noam Teyssier and others , title =
Yusuf H. Roohani and Alexander Dobin and Noam Teyssier and others , title =. 2025 , month = aug, howpublished =
2025
-
[40]
bioRxiv , pages=
HEIMDALL: Disentangling tokenizer design for robust transfer in single-cell foundation models , author=. bioRxiv , pages=. 2025 , publisher=
2025
-
[41]
bioRxiv , pages=
Combinatorial prediction of therapeutic perturbations using causally-inspired neural networks , author=. bioRxiv , pages=
-
[42]
Nature Biotechnology , pages=
Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation , author=. Nature Biotechnology , pages=. 2025 , publisher=
2025
-
[43]
BioRxiv , pages=
Predicting cellular responses to perturbation across diverse contexts with State , author=. BioRxiv , pages=. 2025 , publisher=
2025
-
[44]
Cell , volume=
Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets , author=. Cell , volume=. 2015 , publisher=
2015
-
[45]
Nature communications , volume=
Massively parallel digital transcriptional profiling of single cells , author=. Nature communications , volume=. 2017 , publisher=
2017
-
[46]
Nature reviews genetics , volume=
RNA-Seq: a revolutionary tool for transcriptomics , author=. Nature reviews genetics , volume=. 2009 , publisher=
2009
-
[47]
Science , volume=
Stochastic gene expression in a single cell , author=. Science , volume=. 2002 , publisher=
2002
-
[48]
Nature methods , volume=
Bayesian approach to single-cell differential expression analysis , author=. Nature methods , volume=. 2014 , publisher=
2014
-
[49]
Genome biology , volume=
A benchmark of batch-effect correction methods for single-cell RNA sequencing data , author=. Genome biology , volume=. 2020 , publisher=
2020
-
[50]
2009 IEEE conference on computer vision and pattern recognition , pages=
Imagenet: A large-scale hierarchical image database , author=. 2009 IEEE conference on computer vision and pattern recognition , pages=. 2009 , organization=
2009
-
[51]
Proceedings of the IEEE , volume=
Gradient-based learning applied to document recognition , author=. Proceedings of the IEEE , volume=. 2002 , publisher=
2002
-
[52]
International conference on machine learning , pages=
Learning deep kernels for non-parametric two-sample tests , author=. International conference on machine learning , pages=. 2020 , organization=
2020
-
[53]
International Conference on Learning Representations (ICLR) , year=
Revisiting Classifier Two-Sample Tests , author=. International Conference on Learning Representations (ICLR) , year=
-
[54]
International Conference on Machine Learning (ICML) , pages=
Attention-based deep multiple instance learning , author=. International Conference on Machine Learning (ICML) , pages=. 2018 , organization=
2018
-
[55]
Nature Genetics , volume=
Characterizing the molecular regulation of inhibitory immune checkpoints with multimodal single-cell screens , author=. Nature Genetics , volume=. 2021 , publisher=
2021
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.