Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Ahmadreza Argha; Amin Beheshti; Fatemeh Vafaee; Hamid Alinejad-Rokny; Lucy Chhuo; Marjan BaghGolshani; Mark Grosser; Min Yang; Nigel Lovell; Nona Farbehi

REVIEW 4 major objections 6 minor 3 references

No single foundation model dominates across single-cell and spatial tasks, a harmonised benchmark finds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:38 UTC pith:LOMKUR7Q

load-bearing objection Broad benchmark with a real headline observation—no foundation model dominates—but the strongest version of the claim is undercut by protocol tuning, missing baselines, and single-seed metrics. the 4 major comments →

arxiv 2607.17227 v1 pith:LOMKUR7Q submitted 2026-07-19 q-bio.GN q-bio.CB

Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Sally Chen , Roxana Zahedi , Lucy Chhuo , Ricky Nguyen , Marjan BaghGolshani , Amin Beheshti , Mark Grosser , Min Yang

show 6 more authors

Nona Farbehi Nigel Lovell Ahmadreza Argha Fatemeh Vafaee Youqiong Ye Hamid Alinejad-Rokny

This is my paper

classification q-bio.GN q-bio.CB

keywords foundation modelssingle-cell transcriptomicsspatial transcriptomicsPerturb-seqbenchmarkingzero-shot clusteringcell type annotationperturbation prediction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether single-cell and spatial 'foundation models' truly generalise across modalities and analytical tasks by running six models through one harmonised benchmark spanning scRNA-seq, spatial transcriptomics and Perturb-seq. It claims that 'foundation' is a conditional property, not a consequence of model scale or pretraining: rankings shift with task, modality, preprocessing, tokenisation and metric. Expression-trained cell-level transformers resolve many cell-identity tasks best, spatial and graph-aware models better preserve tissue architecture, and text-derived gene embeddings are competitive for selected perturbation-response metrics. If the paper is right, model selection must be task- and modality-specific, and single-number leaderboard evaluations are misleading.

Core claim

On the paper's own terms, the central discovery is that generalisation in single-cell and spatial foundation models is context-dependent rather than universal. Across five downstream tasks, no architecture consistently beats the others: the strongest cell-identity clustering in scRNA-seq came from expression-trained cell-level transformers such as scGPT-spatial and CellPLM, while tissue-domain recovery favoured spatial and graph-based models such as Novae, with scELMo leading selected settings, and language-derived gene embeddings (GenePT-w, scELMo) led response-focused perturbation metrics. The paper argues this pattern shows that current models capture useful but partial biological represe

What carries the argument

A harmonised benchmarking workflow that combines each model's own recommended preprocessing and tokenisation with shared downstream tasks, shared metrics, fixed random seeds, and—for perturbation prediction—shared evaluated gene sets. The load-bearing design choice is Leiden clustering with the resolution swept until the cluster count matches the number of reference labels, used for both zero-shot and continually pretrained clustering. This workflow is what lets the paper attribute ranking differences to model design rather than inconsistent evaluation, and it is also the step on which cross-model comparability rests.

Load-bearing premise

The benchmark assumes that a single harmonised pipeline—model-specific preprocessing plus Leiden resolution swept to match label counts—compares models fairly; if these choices are not interchangeable across models, the observed ranking shifts could be evaluation artefacts rather than genuine differences in biological generalisation.

What would settle it

A concrete check: re-run zero-shot clustering on the three scRNA-seq datasets using a different principled resolution rule for every model, such as a fixed resolution across all models or each model's native clustering method with default parameters. If the rank order of ARI or NMI changes substantially—for example, a model that was weakest becomes strongest—the harmonised pipeline, not the biology, is driving the reported conditional generalisation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Model selection for cell-identity tasks should favour expression-trained cell-level transformers over text-derived or spatial-domain models in many scRNA-seq settings.
Spatial-domain discovery benefits from graph-based or spatially aware models; using a cell-identity model for tissue segmentation can produce fragmented or over-segmented domains.
Global post-perturbation expression reconstruction is not sufficient: models must be judged on delta-based, DE-gene metrics to test whether they capture actual perturbation response.
Continual pretraining is not a default improvement; it can sharpen fine-grained immune states and tumour domains but degrade other settings, so adaptation protocols must be evaluated per architecture and dataset.
Any single leaderboard score misleads; rankings must be reported per modality, biological granularity and evaluation metric.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A direct extension would be systematic leave-one-domain-out evaluation (leave-one-tissue-out, leave-one-platform-out, leave-one-species-out), since the paper argues domain shift is the central biological problem yet its datasets are varied rather than fully crossed.
The result predicts that task-specialised models encoding a specific biological hypothesis—such as immune-context, developmental programme or tissue niche—will outperform generic foundation models on those tasks, pointing toward problem-specific rather than universal models.
Because the conclusion depends on the harmonised pipeline, a testable extension is to vary resolution rules and preprocessing choices per model and check whether the observed ranking inversions persist; stability across those choices would strengthen the claim that the rankings reflect biology rather than evaluation artefacts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Broad benchmark with a real headline observation—no foundation model dominates—but the strongest version of the claim is undercut by protocol tuning, missing baselines, and single-seed metrics.

read the letter

Colleague,

Here is the paper I owe you: a benchmark of six single-cell/spatial foundation models across scRNA-seq, spatial transcriptomics, and Perturb-seq, with zero-shot clustering, continual pretraining, supervised annotation, marker-gene concordance, and perturbation prediction. The headline—no model dominates, and rankings shift with dataset, task, and metric—is supported by the tables and is worth taking seriously. The breadth is genuinely new: no prior benchmark in the cited literature covers this exact model panel across these modalities with a shared evaluation pipeline. They also ship code, which puts them ahead of many benchmarks in the field.

What the paper does well: it treats "foundation" as a hypothesis rather than a given, it checks biological interpretability via marker-gene overlap, and it separates global expression reconstruction from perturbation-response prediction. The practical message—evaluate task-specific, don't trust leaderboards—is solid and actionable.

The soft spots are mostly in the methodology, not the conclusion. "Harmonised" is doing a lot of work: each model uses its own preprocessing and tokenisation, and the paper never varies those factors for a fixed model. The attributions in the abstract and discussion—that ranking shifts are driven by "preprocessing, tokenisation, biological prior"—are not tested by ablations; those factors are fully confounded with model identity. Second, zero-shot clustering sweeps Leiden resolution until cluster count matches the reference label count. That makes models comparable in name but not in operating point: a model with different intrinsic granularity is being evaluated at a different point on the ARI curve, and a small change in resolution can reorder models. Third, there are no simple baselines (e.g., PCA+Leiden), no confidence intervals or bootstrap, and no multi-seed variability, even though the Discussion itself recommends exactly these. The perturbation prediction comparison also mixes native workflows with a shared GEARS decoder, which helps but doesn't fully standardise the decoders.

My read: these issues weaken the strong version of the claim—that ranking shifts are caused by the listed design factors—but the qualitative observation that no single model dominates across tasks and modalities is likely robust. It would take more than a resolution sweep to explain away scGPT-spatial being best on ATAA and scELMo best on BMMC, or GenePT winning on Norman perturbation response. Still, the paper as written oversells the causal attributions.

This deserves a serious referee. A revision with simple baselines, error bars, and a preprocessing ablation would turn a useful benchmark into a really solid one. I'd take it to a reading group—there's a lot to argue about, which is exactly what makes it useful.

Recommendation: send it to peer review, with a request for those additions.

Referee Report

4 major / 6 minor

Summary. This manuscript benchmarks six foundation models (Nicheformer, CellPLM, scGPT-spatial, GenePT-w, scELMo, Novae) across scRNA-seq, spatial transcriptomics and Perturb-seq, using zero-shot clustering, continual pretraining, supervised annotation, marker-gene concordance and perturbation prediction. The central claim is that no model dominates and that ‘foundation’ is a conditional property: rankings shift with modality, dataset, biological granularity, metric and preprocessing/tokenisation. The empirical basis is mainly Tables 2–5, with interpretation framed around the conditional-generalisation thesis in the Discussion.

Significance. If the cross-model comparisons were clean, this would be a valuable and timely resource: the paper covers a broad and well-chosen model panel, uses external datasets, fixes random seeds, applies shared test sets for supervised annotation, uses shared evaluated gene sets for perturbation prediction, and provides a public code repository. It also explicitly argues for biological, perturbation-grounded evaluation rather than leaderboard ranking, which is an important message for the field. However, the central ranking evidence is not yet shielded from evaluation-protocol confounds. The paper’s own Discussion recommends simple baselines and uncertainty intervals, but the analyses do not include them, and several methodological choices—especially resolution sweeping and model-specific preprocessing—directly affect the reported ranking shifts. The significance of the claims is therefore conditional on additional sensitivity analyses and reframing.

major comments (4)

[Methods — Zero-shot cell type clustering; Table 2] The central ranking evidence in Table 2 is not protected against clustering-protocol artifacts. Leiden resolution is ‘swept until the number of clusters matched the number of reference labels’, and for Novae the number of assigned domains is set to the number of labels. Matching the cluster count does not make the operating point comparable across models: each model is evaluated at a different point on the ARI/NMI-versus-resolution curve, and small resolution changes can reorder models. The abstract’s attribution of ranking shifts to ‘preprocessing, tokenisation’ is also not supported by the design, since these factors are not varied independently of model identity. The same concern applies to the continual-pretraining clustering results in Table 3. I request sensitivity analyses with a fixed resolution and/or multiple resolutions, simple baselines (e.g., PCA + Leiden), and bootstrap con
[Methods — Perturbation Prediction; Table 5] The perturbation-prediction comparison is architecture- and decoder-confounded. CellPLM and scGPT-spatial are evaluated with their native end-to-end models, whereas GenePT-w and scELMo are evaluated by feeding their gene embeddings into a shared GEARS decoder. The shared gene set is defined as an intersection of model vocabularies/embedding spaces, but the downstream prediction machinery is not shared across all four models. Consequently, the claim that ‘language-derived gene embeddings were competitive for selected perturbation-response metrics’ conflates gene-representation quality with decoder compatibility and training protocol. In addition, the Discussion cites Ahlmann-Eltze et al. and Systema as showing that deep perturbation models do not yet outperform simple linear baselines, yet Table 5 includes no such baselines. Global Pearson correlations are all ≈0.98–0.99, so the response-
[Methods — Cell type annotation; Table 4] Supervised annotation results in Table 4 are also difficult to interpret as representation comparisons because each model uses a different supervised head and training protocol: full fine-tuning for Nicheformer, CellPLM and scGPT-spatial; kNN classifiers for GenePT-w and scELMo; and a frozen encoder with a one-epoch MLP for Novae. The Novae row on SEAAD (accuracy 0.30, macro-F1 0.03, ROC-AUC 0.50) is presented as a representation-scale mismatch, but it is equally explained by the very weak one-epoch MLP protocol. The claim that ‘supervised performance depends … on whether the model’s pretraining objective and representation scale match the biological resolution’ would be better supported by a common classifier (e.g., logistic regression or kNN on embeddings) applied to all models, with the native protocols reported as secondary information.
[Discussion; Tables 2–5] The paper’s own Discussion recommends that future benchmarks report ‘simple baselines, leave-domain-out splits, uncertainty intervals’. None of these are provided for the central tables. All results appear to be single-seed point estimates, and many ranking differences are small (e.g., Table 3, BMMC NMI: scGPT-spatial 0.66 vs CellPLM 0.66; Table 5, Replogle Pearson Delta DE: CellPLM 0.4604 vs scELMo 0.4579). With no bootstrap or repeated-seed variance, claims that ‘rankings shifted’ and ‘no model dominated’ are over-precise and may be affected by noise. The authors should add confidence intervals, repeated seeds, or an explicit noise-aware analysis, and at minimum soften the abstract and Discussion claims until such evidence is available.

minor comments (6)

[Abstract] The phrase ‘rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice’ overstates what the design can show: preprocessing and tokenisation are model-specific and never varied independently. Consider wording such as ‘rankings differed across models, whose preprocessing and tokenisation differ’.
[Results — first paragraph] Minor grammatical error: ‘using harmonised evaluation framework’ should be ‘using a harmonised evaluation framework’.
[Methods — Zero-shot cell type clustering] The statement that fixing random seeds at 42 makes results reproducible is not a substitute for variance estimation; please clarify that reproducibility and statistical certainty are distinct.
[Figure 4 caption] The cell-state labels mix formatting: ‘AS DC’ appears as ‘AS-DC’ elsewhere. Please standardise across figure, text and tables.
[Data and Tool Availability] A versioned release of the code repository (e.g., a DOI or release tag) would improve reproducibility, since the current link points to an unversioned GitHub repository.
[Supplementary Table S2] The k-means sensitivity check is reported only for GenePT-w. It would be informative to report k-means results for the other models as well, given the resolution-sweeping concern in the main clustering analysis.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark with external labels; resolution matching is not a fitted prediction.

full rationale

This paper is an empirical benchmark rather than a mathematical derivation, and its central claims—that 'foundation' remains a conditional property and that no model dominates across tasks—are supported by direct measurements of model outputs against externally curated labels and held-out data. Zero-shot clustering embeddings are produced by pretrained checkpoints without target-label training; the only label-derived quantity is the number of clusters, introduced when 'Leiden resolution was swept until the number of clusters matched the number of reference labels' (Methods, 'Zero-shot cell type clustering'). Fixing the cluster count does not determine the partition, so ARI/NMI/silhouette values are not forced by construction. Supervised annotation uses stratified held-out test sets and section-wise splits for SEAAD, and perturbation prediction uses standard simulation splits where test perturbations are unseen, with all models evaluated on shared gene sets. Marker-gene concordance compares top-20 DEG overlaps against ground-truth annotations and is therefore an external biological check. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The Discussion's attribution of ranking shifts to 'preprocessing, tokenisation, biological prior' is a causal-interpretation limitation because these factors vary jointly with model identity, and the absence of simple baselines and uncertainty intervals weakens the ranking evidence—but these are correctness concerns, not circularity. The derivation chain is self-contained with respect to its empirical inputs.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

No new entities, forces, or latent variables are postulated; all models and metrics are pre-existing. The ledger instead captures the hand-chosen evaluation hyperparameters and domain assumptions that the central ranking claims depend on, chiefly resolution tuning, model-specific preprocessing, and the two-epoch continual-pretraining protocol.

free parameters (7)

Leiden clustering resolution (per model–dataset) = Not reported; swept until cluster count equals reference label count
Tuned for each model and dataset in zero-shot and continual-pretraining clustering; determines cluster granularity and can affect ARI/NMI comparisons.
Novae number of assigned domains = Set equal to reference label count per spatial dataset
Domain count is a free parameter forced to match annotations, potentially favoring or penalizing models depending on embedding geometry.
scGPT-spatial PCA components and kNN neighbors (spatial) = 20 PCA components; n_neighbors=50 (zero-shot) and k=15 (continual)
Chosen by authors; affects spatial graph construction and clustering input.
CellPLM continual-pretraining hyperparameters = 2 epochs, Adam lr 1e-4, weight decay 1e-6
Reconstructed pipeline; these choices were not from an official continual-pretraining tutorial.
scELMo adapter training hyperparameters = 500 epochs, contrastive loss weight lambda=100
Hand-chosen; heavy contrastive weighting may dominate the classification objective.
Perturbation-prediction shared gene set = Intersection of dataset genes and all evaluated model vocabularies (per dataset)
Defines the evaluated genes; changing the intersection changes metric values and can alter model rankings.
Top-k DEG cutoff for marker-gene concordance = 20 genes per cluster/domain
Fixed threshold for Wilcoxon marker sets; Jaccard overlap depends on k.

axioms (6)

domain assumption Reference cell-type and spatial-domain annotations are accurate, complete ground truth for evaluating biological structure.
All clustering/annotation metrics are computed against these labels; if labels are coarse or noisy (e.g., MHPC continuous lamination), rankings may penalize valid structure. Paper itself notes this for MHPC.
domain assumption Model-specific preprocessing and tokenisation pipelines are compatible enough to support 'harmonised' cross-model comparison.
Each model uses its own normalization, gene vocabulary, tokenization, and graph construction; differences in rankings are attributed to model design, but preprocessing is confounded.
ad hoc to paper Tuning Leiden/domain count to match the number of reference labels yields comparable and unbiased clusterings.
Methods section: 'Leiden resolution was swept until the number of clusters matched the number of reference labels.' This injects label-count information and may favor models whose embeddings are sensitive to resolution.
ad hoc to paper Two epochs of continual pretraining is a sufficient and representative adaptation protocol for all architectures.
Methods: all models trained for two self-supervised epochs; no per-model convergence check; longer/shorter adaptation could change conclusions.
domain assumption Top-20 Wilcoxon DEG overlap/Jaccard is a valid measure of biological marker-programme preservation.
DEG analysis compares 20-gene sets; small/heterogeneous domains show low concordance regardless of embedding quality.
ad hoc to paper The shared gene-set intersection and GEARS decoder provide a fair common perturbation-prediction testbed for all models.
Perturbation methods: evaluated genes restricted to intersection of model vocabularies; GenePT/scELMo are tested only as gene-feature inputs to GEARS, not as end-to-end predictors.

pith-pipeline@v1.3.0-alltime-deepseek · 24828 in / 13619 out tokens · 121040 ms · 2026-08-01T18:38:09.071454+00:00 · methodology

0 comments

read the original abstract

Single-cell and spatial foundation models promise transferable biological representations, yet their generality remains largely untested across modalities, biological domains and analytical tasks. We benchmarked six representative models, Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo and Novae, using a harmonised framework spanning scRNA-seq, spatial transcriptomics and Perturb-seq. We evaluated zero-shot and continually pretrained clustering, supervised annotation, marker-gene concordance and perturbation prediction. Model performance was strongly conditional: expression-trained cell-level transformers best resolved many cell-identity tasks, spatial and graph-aware models better preserved tissue architecture, and language-derived gene embeddings were competitive for selected perturbation-response metrics. No model dominated across tasks, and rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice. This benchmark provides practical guidance for model selection and argues that future models should be judged by biological generalisation, interpretability and perturbation-grounded validity, not by scale or leaderboard performance alone.

Figures

Figures reproduced from arXiv: 2607.17227 by Ahmadreza Argha, Amin Beheshti, Fatemeh Vafaee, Hamid Alinejad-Rokny, Lucy Chhuo, Marjan BaghGolshani, Mark Grosser, Min Yang, Nigel Lovell, Nona Farbehi, Ricky Nguyen, Roxana Zahedi, Sally Chen, Youqiong Ye.

**Figure 2.** Figure 2: Zero-shot foundation-model embeddings recover biological structure unevenly across single-cell and spatial modalities. a, UMAP visualisations of zero- [PITH_FULL_IMAGE:figures/full_fig_p036_2.png] view at source ↗

**Figure 3.** Figure 3: Continual pretraining selectively refines cell [PITH_FULL_IMAGE:figures/full_fig_p038_3.png] view at source ↗

**Figure 4.** Figure 4: Supervised annotation of closely related dendritic [PITH_FULL_IMAGE:figures/full_fig_p039_4.png] view at source ↗

**Figure 6.** Figure 6: Perturbation prediction separates global expression reconstruction [PITH_FULL_IMAGE:figures/full_fig_p041_6.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

3 extracted references

[1]

1 Jovic, D. et al. Single‐cell RNA sequencing technologies and applications: A brief overview. Clinical and Translational Medicine 12, e694 (2022). https://doi.org/10.1002/ctm2.694 2 Hwang, B., Lee, J. H. & Bang, D. Single-cell RNA sequencing technologies and bioinformatics pipelines. Exp Mol Med 50, 1–14 (2018). https://doi.org/10.1038/s12276-018-0071-8 ...

arXiv 2022
[2]

Models are compared by tokenisation strategy, pretraining data, modality, architecture, spatial awareness, reported tasks, pretraining objective and publication

Overview of foundation models included in the benchmark. Models are compared by tokenisation strategy, pretraining data, modality, architecture, spatial awareness, reported tasks, pretraining objective and publication. Model Tokenisation Strategy Pretraining Data Modality Architecture Spatial Awareness Reported Tasks Pretraining Objective Publication Nich...

2025
[5]

Models were evaluated using MSE, MSE DE, Pearson, Pearson DE, Pearson Delta and Pearson Delta DE metrics

Perturbation prediction benchmarking results across three Perturb -seq datasets. Models were evaluated using MSE, MSE DE, Pearson, Pearson DE, Pearson Delta and Pearson Delta DE metrics. Model Dataset MSE MSE DE Pearson Pearson DE Pearson Delta Pearson Delta DE CellPLM Norman 0.0064 0.4034 0.9826 0.8724 0.4727 0.3982 Adamson 0.0091 0.2778 0.9876 0.9236 0....

1913

[1] [1]

1 Jovic, D. et al. Single‐cell RNA sequencing technologies and applications: A brief overview. Clinical and Translational Medicine 12, e694 (2022). https://doi.org/10.1002/ctm2.694 2 Hwang, B., Lee, J. H. & Bang, D. Single-cell RNA sequencing technologies and bioinformatics pipelines. Exp Mol Med 50, 1–14 (2018). https://doi.org/10.1038/s12276-018-0071-8 ...

arXiv 2022

[2] [2]

Models are compared by tokenisation strategy, pretraining data, modality, architecture, spatial awareness, reported tasks, pretraining objective and publication

Overview of foundation models included in the benchmark. Models are compared by tokenisation strategy, pretraining data, modality, architecture, spatial awareness, reported tasks, pretraining objective and publication. Model Tokenisation Strategy Pretraining Data Modality Architecture Spatial Awareness Reported Tasks Pretraining Objective Publication Nich...

2025

[3] [5]

Models were evaluated using MSE, MSE DE, Pearson, Pearson DE, Pearson Delta and Pearson Delta DE metrics

Perturbation prediction benchmarking results across three Perturb -seq datasets. Models were evaluated using MSE, MSE DE, Pearson, Pearson DE, Pearson Delta and Pearson Delta DE metrics. Model Dataset MSE MSE DE Pearson Pearson DE Pearson Delta Pearson Delta DE CellPLM Norman 0.0064 0.4034 0.9826 0.8724 0.4727 0.3982 Adamson 0.0091 0.2778 0.9876 0.9236 0....

1913