Issues arising from benchmarking single-cell RNA sequencing imputation methods

Jingyi Jessica Li; Wei Vivian Li

REVIEW 4 major objections 4 minor 1 cited by

Issues arising from benchmarking single-cell RNA sequencing imputation methods

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reanalysis of the SAVER benchmarking study finds that its semi-synthetic data and non-biological cluster labels produced a conclusion that reverses when real data and biologically defined cell types are used.

desk verdict A credible critique of SAVER's benchmark that exposes real circularity, but the ranking reversal is conditional on a single clustering pipeline and one dataset. read the letter →

arxiv 1908.07084 v1 pith:HNCX3A47 submitted 2019-08-19 stat.AP q-bio.GNq-bio.QM

Wei Vivian Li , Jingyi Jessica Li This is my paper · ORCID

classification stat.APq-bio.GNq-bio.QM

keywords single-cellRNAsequencingimputationSAVERscImputeMAGICbenchmarkingsemi-syntheticdatacell-typeclustering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges a widely cited benchmarking result that the imputation method SAVER outperforms scImpute and MAGIC on single-cell RNA sequencing data. It argues that the original benchmark was built on semi-synthetic data simulated from the same Poisson-Gamma model that SAVER itself uses, and on cell clusters defined by an algorithm rather than by biological cell types. Re-running the comparison on the original real dataset, with hierarchical clustering scored against biologically defined cell types and subtypes, the paper finds that scImpute leads to comparable or higher adjusted Rand index, Jaccard index, normalized mutual information, and purity than SAVER for both K=9 and K=47. The point matters because imputation methods are meant to help recover biological cell types, and a benchmark that favors one method's model may mislead users.

What carries the argument

The load-bearing object is the reanalysis protocol: hierarchical clustering of cells into $K=9$ and $K=47$ groups on the first ten principal components of the real or imputed expression matrix, followed by comparison of those groups with the biologically defined cell types and subtypes using four external agreement measures (adjusted Rand index, Jaccard index, normalized mutual information, and purity), with 100 bootstrap resamples of cells. This protocol replaces the original benchmark's two contested elements: semi-synthetic data generated under the Poisson-Gamma model embedded in SAVER, and cluster labels produced by the Seurat algorithm that do not match biological cell types. The swap of data and labels is what flips the ranking.

What would settle it

A concrete check would be to rerun the same four agreement metrics on the same real data using a different clustering algorithm, such as Seurat, or using independently validated cell type labels; if SAVER then outranks scImpute on a majority of the metrics, the paper's central comparison would be undermined.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the earlier benchmark's conclusion does not survive a change of evaluation data and reference labels. When SAVER, scImpute, and MAGIC are compared on the original real scRNA-seq data, using hierarchical clustering on the first ten principal components and scoring against nine major cell types and 47 subtypes defined by biological marker genes, scImpute achieves comparable or higher values than SAVER on all four clustering agreement metrics, at both resolutions. The paper therefore claims that the earlier statement that SAVER achieves a higher Jaccard index than the other methods is contradicted by real-data evidence.

Load-bearing premise

The reanalysis assumes that the biologically labeled cell types reported in the original real dataset, as recovered by hierarchical clustering on the first ten principal components, are the correct ground truth for judging imputation performance.

Editorial extensions

If this is right

The published claim that SAVER consistently achieves a higher Jaccard index than scImpute and MAGIC is not supported when the comparison uses real data and biologically defined cell types.
Benchmarks that validate an imputation method on semi-synthetic data generated from that method's own statistical model should be interpreted cautiously, since the simulation can encode the method's assumptions.
For cell-type discovery, evaluating imputed data by how well clustering recovers biologically defined cell types is a more meaningful standard than matching algorithmically generated clusters.
Version choice matters in method comparisons: the earlier benchmark used an archived scImpute version lacking the subpopulation identification step introduced in version 0.0.3, which can affect clustering accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same logic would apply to other method comparisons that rely on semi-synthetic data simulated under one contender's model; re-evaluating on real data with independent ground truth could change other published rankings.
A direct test of generalizability would repeat this protocol on several real scRNA-seq datasets, with multiple clustering algorithms and resolutions, to see whether scImpute's advantage over SAVER is specific to this dataset or a broader pattern.
Cluster-recovery metrics measure only one aspect of imputation quality; gene-level accuracy, differential-expression preservation, or downstream biological pathway reconstruction could rank the methods differently.
The reliance on 100 bootstrap resamples of cells and a single clustering method means the reported margins should be read as indicative rather than as fixed effect sizes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is that Li and Li have identified a genuine problem in the SAVER benchmark: Huang et al. simulated their semi-synthetic data from the same Poisson-Gamma model that SAVER itself uses. That is circular in an obvious way, and showing that the simulated data have different zero rates, expression variability, and low correlation with real data (R2 = 0.14) is a real service. The cluster-label mismatch in Table 1 is also compelling—it is not just a theoretical worry, but a concrete demonstration that Huang's evaluation clusters do not match biological cell types. I would want any benchmarking paper to be sent back if its synthetic data were this unrepresentative and its labels this divorced from biology.

The reanalysis is where I get cautious. Figure 1c is the load-bearing evidence that scImpute beats or ties SAVER on real data, but it rests on a single downstream pipeline: hierarchical clustering on the first ten PCs. The stress-test note is right that this pipeline is not validated in the text (e.g., by reporting ARI for the un-imputed Zeisel data relative to the known nine cell types), and the authors do not check whether the ranking survives under Seurat, k-means, or Louvain. They criticize Huang for relying on Seurat, but replacing it with one unvalidated protocol is not a robustness demonstration. The problem is compounded by changing both the data (synthetic to real) and the clustering method at once, and by using scImpute 0.0.3 instead of the 0.0.2 version Huang tested. That version change is legitimate—the authors argue the newer version is the one from the actual paper—but it means the reversal is not a controlled comparison.

There is also a generality gap. The semi-synthetic critique is illustrated with only the Zeisel dataset, yet the text claims all four simulated datasets underrepresent zeros and heterogeneity. It is entirely plausible, but the evidence shown is for one dataset. The clustering reanalysis is also only on Zeisel, so the claim that scImpute is comparable or better than SAVER is data-limited. No statistical tests accompany the ARI/AMI/purity boxplots; bootstrapping gives visual confidence, but a formal test would help.

The authors are the scImpute developers, and the paper does not include an explicit competing-interests statement. That is not disqualifying—it is normal in methods papers—but it should have been disclosed, and it raises the bar for robustness analyses.

Is the paper still worth engaging? Yes. The circularity critique is sharp and the cluster-label mismatch is likely to be reproducible. The ranking reversal is plausible but not proven. A serious referee should ask for all four datasets, a validation of the clustering pipeline, alternative clusterers, and a clear statement on the version choice. I would send this to peer review; it is exactly the kind of post-publication critique that should be vetted.

My bottom line: the central methodological warning is solid, and the specific result on scImpute vs SAVER should be taken as conditional until the robustness checks are done.

Referee Report

4 major / 4 minor

Summary. This manuscript, authored by the developers of scImpute, re-examines the benchmarking of single-cell RNA-seq imputation methods by Huang et al., who concluded that SAVER outperforms scImpute and MAGIC. The authors argue that Huang et al.'s evaluation relied on semi-synthetic data generated from the same Poisson-Gamma model used by SAVER, and that the cluster labels used for benchmarking were derived from the Seurat algorithm rather than from biological marker-based cell types. Based on the Zeisel et al. dataset, they show that the semi-synthetic data have lower zero fractions and reduced gene expression variability compared to real data, and that the Seurat clusters do not correspond well to known cell types. They then re-evaluate imputation performance using hierarchical clustering of the first ten principal components with K=9 and K=47 on the real Zeisel data, reporting that scImpute achieves comparable or higher adjusted Rand index, Jaccard index, normalized mutual information, and purity than SAVER, contradicting Huang et al.'s conclusion. They also provide tSNE visualizations and argue that imputation performance should be assessed on real data with biological ground truth.

Significance. The paper identifies a plausible and important pitfall in benchmarking scRNA-seq imputation methods: evaluating methods on semi-synthetic data generated from one method's own statistical model can bias the comparison in favor of that method. This is a legitimate concern that the field should take seriously. If the reanalysis were robust, the paper would provide a valuable cautionary example. However, the current evidence is too limited to establish the authors' central claims: the semi-synthetic data critique is demonstrated on only one dataset, and the main finding of a ranking reversal rests on a single, unvalidated clustering protocol. The paper also changes the version of scImpute between the original benchmark and the reanalysis, introducing a confound. These limitations mean the paper is not yet convincing as a definitive refutation, though the underlying principle is sound.

major comments (4)

[Figure 1a-b and accompanying text] The claim that Huang et al.'s semi-synthetic data underrepresent zero proportions and gene expression heterogeneity is supported by data from only the Zeisel dataset. The text asserts that this holds for all four semi-synthetic datasets used by Huang et al., but no corresponding analyses are shown for the other three datasets. Without such evidence, the generality of the critique is not established, and the conclusion that 'the semi-synthetic datasets may have significantly different properties' is an overstatement based on a single example.
[Figure 1c and the paragraph describing the reanalysis] The central finding that 'data imputed by scImpute leads to comparable or higher adjusted Rand index, Jaccard index, normalized mutual information, and purity' is derived from a single clustering pipeline: hierarchical clustering on the first ten principal components with K=9 and K=47. The paper criticizes Huang et al. for relying on Seurat clusters, but it does not validate that the chosen pipeline recovers the known cell types from the un-imputed data, nor does it test whether the ranking of imputation methods is stable under alternative clustering algorithms such as Seurat, k-means, or Louvain community detection. If the ranking reverses under another widely used pipeline, the stated conclusion would be an artifact of the clustering protocol rather than a robust property of the imputed data. This is a load-bearing gap given that the paper's title and abstract promise a re-evaluation of benchmarking.
[Methods paragraph on software versions] The reanalysis uses scImpute version 0.0.3, whereas Huang et al. benchmarked scImpute version 0.0.2. The authors acknowledge this and state that version 0.0.3 'includes an important step for the identification of cell sub-populations' that could improve accuracy and robustness. This change confounds the comparison: the different results could be due to the version update rather than to the use of real versus semi-synthetic data. To support the claim that Huang et al.'s conclusion is flawed, the authors should report results obtained with scImpute v0.0.2 under the same reanalysis pipeline, or explicitly demonstrate that the version change does not alter the ranking reversal.
[Figure 1c (statistical assessment)] The boxplots in Figure 1c are based on 100 bootstrap resamples of cells, but no statistical tests or effect sizes are reported. The phrase 'comparable or higher' is vague: for K=9 and K=47, some measures may show overlapping distributions across methods, making the claim of superiority or equivalence unsupported. The authors should provide formal comparisons (e.g., confidence intervals for pairwise differences or permutation tests) to substantiate their conclusion that scImpute is comparable or better than SAVER.

minor comments (4)

[Table 1 caption and text] There is a typo in the Table 1 caption: 'cell-type parker genes' should be 'cell-type marker genes'.
[Text after Figure 1] The sentence 'our tSNE visualization suggests that based on the real data, the all the three imputation methods lead to clear separation patterns' contains a grammatical error: 'the all the three' should be 'all three'.
[Paragraph on semi-synthetic data properties] The manuscript reports an average coefficient of determination of R2 = 0.14 between gene expression in real and semi-synthetic data, but does not specify how this value was computed (e.g., across which genes, whether on log scale, and whether it is a mean over genes or a pooled value). This prevents readers from assessing the claim.
[Clustering methods description] The hierarchical clustering used for the reanalysis is not fully specified: the distance metric, linkage criterion, and how the first ten principal components were computed (e.g., scaling, normalization) are not described. This lack of detail makes the analysis difficult to reproduce or compare with alternative choices.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reanalysis is an empirical benchmark; the scImpute version and KCluster choices are fairness concerns, not reductions of the conclusion to its inputs.

full rationale

The paper is an empirical reanalysis of Huang et al.'s benchmark, not a derivation. Its two substantive findings—that the Poisson-Gamma semi-synthetic data poorly reproduce real scRNA-seq statistics (e.g., R2 = 0.14, lower zero fractions, smaller standard deviations in Fig. 1a–b) and that Huang et al.'s Seurat-derived cluster labels do not match Zeisel et al.'s marker-based cell types (Table 1)—are established by direct comparison to external data and do not assume the conclusion. The ranking reversal claim (Fig. 1c) is an observed outcome of running scImpute v0.0.3, MAGIC v1.0.0, and SAVER v1.0.0 on Zeisel data and clustering on the first ten PCs; the ARI/Jaccard/NMI/purity values are not algebraically forced by setting KCluster=9 or by the evaluation K, and the paper does not define performance in terms of its own method's output. The self-citation to the scImpute paper [3] supports the version change, but the version change is a software artifact and the semi-synthetic data critique is independent of it. Concerns that hierarchical clustering on PCs is unvalidated, or that KCluster=9 gives scImpute an unfair advantage, are benchmarking-fairness and robustness issues, not circularity: none of the headline conclusions is equivalent by construction to an input or to a self-citation chain. Hence no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about the validity of Zeisel cell-type labels and the evaluation criterion, plus an unshown generalization from the Zeisel dataset to all four semi-synthetic datasets. The analysis also uses hand-chosen parameters (PC count, K, scImpute KCluster) without sensitivity analysis. No new entities are introduced.

free parameters (3)

Number of principal components = 10
Clustering is performed on the first ten principal components of the expression matrix; no sensitivity analysis is reported.
Number of clusters K = 9 and 47
K is set to the number of major cell types and subtypes reported by Zeisel et al.; it is a hand-chosen evaluation setting.
scImpute KCluster = 9
scImpute is run with KCluster=9, matching the number of known cell types; this input can affect imputation results.

assumptions (4)

domain assumption Zeisel et al.'s marker-based cell type labels are a valid ground truth for scRNA-seq cell identity.
The reanalysis scores clustering against these labels; if the labels are wrong, the comparison is invalid.
domain assumption Cluster agreement with known cell types is the appropriate criterion for imputation quality.
The paper equates better cluster recovery with better imputation, without validating other metrics such as gene-gene correlation recovery.
domain assumption Hierarchical clustering on the first ten principal components is a suitable evaluation pipeline.
The authors use this pipeline for K=9 and K=47 but do not test alternatives; this is the same kind of limitation they criticize in Huang et al.
ad hoc to paper The semi-synthetic data critique based on the Zeisel dataset generalizes to all four Huang et al. datasets.
The text claims all four semi-synthetic datasets are unrepresentative, yet Figures 1a-b only show Zeisel; the broad claim requires this unshown generalization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Issues arising from benchmarking single-cell RNA sequencing imputation methods." pith.science (2026). https://pith.science/paper/HNCX3A47

@misc{pith2026190807084,
  author       = {Pith},
  title        = {Pith review of: Issues arising from benchmarking single-cell RNA sequencing imputation methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HNCX3A47}},
  note         = {Machine review of arXiv:1908.07084}
}

read the original abstract

On June 25th, 2018, Huang et al. published a computational method SAVER on Nature Methods for imputing dropout gene expression levels in single cell RNA sequencing (scRNA-seq) data. Huang et al. performed a set of comprehensive benchmarking analyses, including comparison with the data from RNA fluorescence in situ hybridization, to demonstrate that SAVER outperformed two existing scRNA-seq imputation methods, scImpute and MAGIC. However, their computational analyses were based on semi-synthetic data that the authors had generated following the Poisson-Gamma model used in the SAVER method. We have therefore re-examined Huang et al.'s study. We find that the semi-synthetic data have very different properties from those of real scRNA-seq data and that the cell clusters used for benchmarking are inconsistent with the cell types labeled by biologists. We show that a reanalysis based on real scRNA-seq data and grounded on biological knowledge of cell types leads to different results and conclusions from those of Huang et al.

Figures

Figures reproduced from arXiv: 1908.07084 by the authors.

**Figure 1.** Re-evaluation of Huang et al. a: Comparison between the original scRNA-seq data from Zeisel et al. for 19, 912 genes and 3, 005 cells 5 and the semi-synthetic data for 3, 529 genes and 1, 800 cells that Huang et al. derived from the original data. For genes with the same mean expression level, the semi-synthetic data exhibit a smaller standard deviation (sd) in gene expression (top panel) and a smaller fraction of z… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Reply to "Issues arising from benchmarking single-cell RNA sequencing imputation methods"
stat.AP 2019-09 conditional novelty 4.0 of 10

The authors redo their downsampling benchmark with a revised procedure that matches real data characteristics and report that SAVER's original conclusions still hold.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Saver: gene expression recovery for single-cell rna sequencing

Mo Huang, Jingshu Wang, Eduardo Torre, Hannah Dueck, Sydney Shaffer, Roberto Bonasio, John I Murray, Arjun Raj, Mingyao Li, and Nancy R Zhang. Saver: gene expression recovery for single-cell rna sequencing. Nature Methods, page 1, 2018

work page 2018
[2]

Zifa: Dimensionality reduction for zero-inﬂated single-cell gene expression analysis

Emma Pierson and Christopher Y au. Zifa: Dimensionality reduction for zero-inﬂated single-cell gene expression analysis. Genome biology, 16(1):241, 2015

work page 2015
[3]

An accurate and robust imputation method scimpute for single-cell rna-seq data

Wei Vivian Li and Jingyi Jessica Li. An accurate and robust imputation method scimpute for single-cell rna-seq data. Nature communications, 9(1):997, 2018

work page 2018
[4]

Recovering gene interactions from single-cell data using data diffusion

David Van Dijk, Roshan Sharma, Juoas Nainys, Kristina Yim, Pooja Kathail, Ambrose Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. 2018

work page 2018
[5]

Cell types in the mouse cortex and hippocampus revealed by single-cell rna-seq

Amit Zeisel, Ana B Mu˜noz-Manchado, Simone Codeluppi, Peter L¨onnerberg, Gioele La Manno, Anna Jur ´eus, Sueli Marques, Hermany Munguba, Liqun He, Christer Betsholtz, et al. Cell types in the mouse cortex and hippocampus revealed by single-cell rna-seq. Science, 347(6226):1138–1142, 2015

work page 2015
[6]

Spatial reconstruction of single-cell gene expression data

Rahul Satija, Jeffrey A Farrell, David Gennert, Alexander F Schier, and Aviv Regev. Spatial reconstruction of single-cell gene expression data. Nature biotechnology, 33(5):495, 2015

work page 2015
[7]

Comparing partitions

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classiﬁcation, 2(1):193– 218, 1985

work page 1985
[8]

A study of the comparability of external criteria for hierarchical cluster analysis

Glenn W Milligan and Martha C Cooper. A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21(4):441–458, 1986

work page 1986

Show all 9 references

[9]

Data Mining: Practical machine learning tools and techniques

Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2016. 6

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Saver: gene expression recovery for single-cell rna sequencing

Mo Huang, Jingshu Wang, Eduardo Torre, Hannah Dueck, Sydney Shaffer, Roberto Bonasio, John I Murray, Arjun Raj, Mingyao Li, and Nancy R Zhang. Saver: gene expression recovery for single-cell rna sequencing. Nature Methods, page 1, 2018

work page 2018

[2] [2]

Zifa: Dimensionality reduction for zero-inﬂated single-cell gene expression analysis

Emma Pierson and Christopher Y au. Zifa: Dimensionality reduction for zero-inﬂated single-cell gene expression analysis. Genome biology, 16(1):241, 2015

work page 2015

[3] [3]

An accurate and robust imputation method scimpute for single-cell rna-seq data

Wei Vivian Li and Jingyi Jessica Li. An accurate and robust imputation method scimpute for single-cell rna-seq data. Nature communications, 9(1):997, 2018

work page 2018

[4] [4]

Recovering gene interactions from single-cell data using data diffusion

David Van Dijk, Roshan Sharma, Juoas Nainys, Kristina Yim, Pooja Kathail, Ambrose Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. 2018

work page 2018

[5] [5]

Cell types in the mouse cortex and hippocampus revealed by single-cell rna-seq

Amit Zeisel, Ana B Mu˜noz-Manchado, Simone Codeluppi, Peter L¨onnerberg, Gioele La Manno, Anna Jur ´eus, Sueli Marques, Hermany Munguba, Liqun He, Christer Betsholtz, et al. Cell types in the mouse cortex and hippocampus revealed by single-cell rna-seq. Science, 347(6226):1138–1142, 2015

work page 2015

[6] [6]

Spatial reconstruction of single-cell gene expression data

Rahul Satija, Jeffrey A Farrell, David Gennert, Alexander F Schier, and Aviv Regev. Spatial reconstruction of single-cell gene expression data. Nature biotechnology, 33(5):495, 2015

work page 2015

[7] [7]

Comparing partitions

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classiﬁcation, 2(1):193– 218, 1985

work page 1985

[8] [8]

A study of the comparability of external criteria for hierarchical cluster analysis

Glenn W Milligan and Martha C Cooper. A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21(4):441–458, 1986

work page 1986