Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a bidirectional state space model, GeneMamba, can process single-cell gene expression sequences in linear time and deliver competitive or better performance than transformer-based single-cell foundation models on…

desk verdict Bidirectional Mamba for single-cell is a plausible idea, but the pretraining objective leaks the next token through the reverse pass and the paper's own tables undercut its performance claims. read the letter →

arxiv 2504.16956 v4 pith:MXQKB2FP submitted 2025-04-22 cs.CL cs.LGq-bio.GN

Cong Qi , Hanzhang Fang , Siqi Jiang , Xun Song , Tianxing Hu , Wei Zhi This is my paper · ORCID
classification cs.CLcs.LGq-bio.GN
keywords single-cellRNAsequencingstatespacemodelBi-Mambafoundationcelltypeannotationbatchintegrationgene-genecorrelationrank-basedtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GeneMamba asks whether a state space model can replace the transformer as the backbone for single-cell RNA sequencing analysis. The paper's answer is yes: by tokenizing each cell as a rank-ordered list of its most expressed genes and reading that list with a bidirectional Mamba encoder, the model processes sequences in linear time and, after pretraining on roughly 30 million human cells, matches or beats transformer-based single-cell foundation models on cell type annotation, multi-batch integration, and gene-gene correlation. The payoff is practical: single-cell foundation models have been limited by the quadratic cost of self-attention, and a linear-time backbone makes it feasible to train on tens of millions of cells or to use longer gene sequences. The authors further claim that the bidirectional state space design and the pathway-aware training loss give the model interpretable gene embeddings, as evidenced by gene rank reconstruction and perturbation predictions.

What carries the argument

The load-bearing object is the Bi-Mamba block, a selective state space layer that scans a gene token sequence in both the forward and reverse directions with shared weights and fuses the two scans through a learnable sigmoid gate; the state recurrence gives linear-time complexity in sequence length, and the bidirectional scan gives each gene both upstream and downstream context. Around this encoder, the paper places two supporting mechanisms: the rank-based gene tokenizer (Eq. 2), which orders a cell's genes by normalized expression and keeps the top 2,048 or 4,096 as tokens, and the pretraining objective (Eq. 15), which sums next-gene prediction loss with an InfoNCE pathway loss that pulls genes sharing a biological pathway together in embedding space.

What would settle it

A matched-cost benchmark would settle the efficiency-performance claim: train GeneMamba and a transformer baseline with the same parameter count and the same FLOPs on 4,096-token sequences and compare cell-annotation Macro-F1 and batch-integration Avg_bio; if the transformer ties or wins at equal cost, the paper's central claim collapses. A second decisive check is whether a held-out set of genes that never appear in the top 2,048 ranks can be reconstructed above chance, which would test whether the rank-based pretraining objective learns transferable biology or only memorizes the training distribution.

Watch

Extended reading notes

Core claim

The central claim is that a bidirectional selective state space model can replace quadratic-complexity transformers as the backbone of single-cell foundation models without sacrificing predictive quality. GeneMamba tokenizes each cell as a rank-ordered list of its top expressed genes, processes that list with a Bi-Mamba stack that reads forward and backward through shared-weight state space layers, and is pretrained on roughly 30 million human cells with a next-gene prediction loss plus a pathway-aware contrastive loss. In the paper's experiments, this combination delivers the best or near-best scores among compared models: highest Macro-F1 on hPancreas (0.7710) and Myeloid (0.3650), best accuracy on hPancreas (0.9713) and Myeloid_b (0.9603), top Avg_bio on Immune (0.8131) and BMMC (0.7628) for batch integration, and a Levenshtein distance of 6 versus 23 for GeneFormer in gene-rank reconstruction on PBMC12k, with BLEU 0.987 versus 0.968. The authors conclude that GeneMamba is a practical and powerful alternative to transformer-based single-cell models.

Load-bearing premise

The argument rests on the premise that ranking a cell's genes by expression and keeping only the top 2,048 or 4,096 still leaves enough biological information for cell annotation, batch integration, and gene-pair analysis; if the discarded low-expression genes carry the signal, the reported gains could reflect the truncation rather than the Bi-Mamba architecture.

Editorial extensions

If this is right

  • Longer gene sequences become affordable: at 4,096 tokens the paper reports GeneMamba training at 0.2393 seconds per sample versus 0.6025 for a transformer, so analyses that cover more of the transcriptome become practical.
  • Pretraining scale can grow: linear-time processing removes the quadratic wall that has limited transformer-based single-cell foundation models, making corpora of tens of millions of cells a routine target.
  • A simple MLP head on GeneMamba embeddings reaches the best accuracy on hPancreas (0.9713) and best Macro-F1 on Myeloid (0.3650) among the models compared, which would simplify cell type annotation pipelines.
  • Batch integration with GeneMamba embeddings yields the highest biological-conservation scores on the Immune and BMMC benchmarks while still removing batch effects near the level of a dedicated integration method.
  • The joint next-gene and pathway-contrastive pretraining organizes gene embeddings so that same-pathway pairs are more separable than with transformer baselines, supporting downstream gene-gene correlation and regulatory analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the pathway-loss weight (set to gamma = 0.1); varying this weight would show how much of the gene-pair separation comes from the contrastive loss rather than from the bidirectional architecture itself.
  • The paper does not test inputs beyond 4,096 tokens; if the linear-time claim transfers, full-length transcriptomes and spatial transcriptomics sequences are natural next targets.
  • The batch-integration protocol uses embeddings from a model already fine-tuned on cell type labels, so an unsupervised integration benchmark would isolate how much the pretrained representation itself contributes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. GeneMamba proposes a bidirectional Mamba (Bi-Mamba) state-space backbone for single-cell RNA-seq, using rank-based gene tokenization, pretraining on roughly 30 million human cells from CELLxGENE, a next-gene prediction loss, a pathway-aware contrastive loss, and a third objective whose identity is not actually specified. The authors evaluate the model on multi-batch integration, cell type annotation, gene rank reconstruction, gene correlation analysis, and perturbation prediction, and claim linear-time scalability and performance that makes GeneMamba a practical and powerful alternative to transformer-based single-cell foundation models. The central claim is that a bidirectional state-space architecture with rank-based tokenization can serve as an effective and scalable foundation model for transcriptomics.

Significance. If the substantial concerns about the pretraining objective and evaluation protocol were resolved, the contribution could be valuable: a linear-time bidirectional SSM backbone for large-scale single-cell pretraining, a reproducible rank-based tokenization pipeline, broad benchmarking against GeneFormer, scGPT, scFoundation, and Harmony, and useful scalability measurements. The paper makes code and model links available and reports pretraining on a large corpus. However, the current evidence does not support the claimed superiority: several key metrics in Tables 1 and 2 favor the transformer baselines, the causal validity of the next-gene loss is not established, and the interpretability evidence in Section 4.3 is substantially circular. The paper therefore needs major revision rather than acceptance in its present form.

major comments (5)
  1. [§3.2 and §3.3, Eqs. (6)–(11)] The paper does not specify a causal mask for the reversed pass. In Bi-Mamba, the combined output o_{j-1} is formed from the forward state h_{j-1} and the reversed state \tilde{h}_{j-1}, and the reversed recurrence has already incorporated g_j, ..., g_n by the time position j-1 is processed. If the language-model head for L_lang in Eq. (11) uses o_{j-1}, the loss can be minimized by copying the true next gene from the future rather than by learning gene-gene structure. If, instead, only the forward hidden state is used, the paper must state this explicitly and justify why the bidirectional representation is still central to the pretraining objective. This is load-bearing because Section 4.3's rank-reconstruction experiment measures exactly this next-gene task, so the high BLEU and Spearman values in Table 3 could reflect target leakage. Please state which hidden state feeds the LM head, describe the masking convention for the reversed pass, and, if the implemented model is leaky, re-run the pretraining and the affected downstream experiments.
  2. [§3.3, Eq. (15)] The text says 'The final pretraning loss is the weighted sum of the three loss,' but Eq. (15) defines only L = L_lang + gamma * L_pathway. No third term appears anywhere in Section 3.3 or the experiments. This is not purely a typographical issue: if a third biologically informed loss exists, the pretraining objective and its hyperparameters are under-specified; if it does not exist, the sentence and the word 'three' must be corrected.
  3. [Tables 1 and 2; Abstract and §1] The abstract and introduction claim strong or superior performance and 'substantial improvements' over transformer-based methods, but the reported tables do not support those claims. On MS in Table 2, GeneMamba obtains accuracy 0.6825 and Macro-F1 0.5342, well below scGPT's 0.8471 and 0.6630. On COVID-19 Avg_bio in Table 1, GeneMamba scores 0.5537 versus scGPT's 0.6476, and it also trails scGPT on PBMC12k Avg_bio (0.8344 versus 0.9018) and on Perirhinal Cortex Avg_bio (0.9062 versus 0.9552). No confidence intervals, standard deviations, or significance tests are reported anywhere. The defensible claim is that GeneMamba is competitive on some datasets and datasets-specific in its advantages; please recalibrate the wording and add measures of variability.
  4. [§4.3, Table 3, and Figure 4] The gene rank reconstruction experiment directly evaluates the same conditional next-gene objective used in pretraining, so high overlap between input and output ranks is partly a check that the training objective was learned, not an independent test of biological interpretability. The comparison with GeneMamba_U and GeneFormer is informative, but the paper should acknowledge this circularity and report held-out cells, sequence lengths, number of replicates, and standard deviations. In addition, Figure 4's caption says the results are on the pancreas dataset, while Section 4.3 and Table 3 report PBMC12k; these references must be reconciled.
  5. [Appendix B.2 and 'Finetuning Details'] The evaluation protocol is internally inconsistent. Appendix B.2 states that a manual stratified 0.9/0.1 train/test split is applied to the downstream datasets, while the 'Finetuning Details' paragraph states that 'The train/test split is predefined by the dataset provider, so we do not perform any additional splitting.' These statements cannot both be true. Since all reported accuracies and integration metrics depend on the exact partition, and since the COVID-19 example shows that the split was modified to avoid missing rare classes, please specify the precise split for every dataset and confirm that each baseline was evaluated on exactly the same partitions.
minor comments (6)
  1. [Eq. (1)] The normalization formula uses t-digest in the denominator without a clear per-gene index; please define the median over cells for each gene j explicitly and fix the notation.
  2. [Abstract and §4, Pretraining Dataset Construction] The abstract says the model enables processing of over 50 million cells, while the pretraining corpus after filtering contains 29,849,897 cells; please distinguish raw downloaded cells from the final training set in both places.
  3. [§4.3 and Figure 4] The Venn diagram and density plot are described as being on the pancreas dataset in the figure caption, whereas the surrounding text and Table 3 refer to PBMC12k; please align the caption with the reported experiment.
  4. [Appendix C and Figure 15] The text on embedding alignment compares GeneMamba with scGPT and scFoundation, but the Figure 15 caption lists Gene2Vec, GeneMamba, and Random; also, 'eucilean distance' should be 'Euclidean distance.' Please make the reference models consistent and provide quantitative values for the claimed separations.
  5. [Appendix B.3] The Levenshtein distance is defined over single-character edits, but the sequences being compared are gene tokens; please define the metric over tokens rather than characters.
  6. [Table 5 and §5] The column header 'Training Time (h/million)' is ambiguous, and it is not reconciled with the main text's statement that pretraining took approximately three weeks on four A100 GPUs; please report total pretraining time and per-sample time separately.

Circularity Check

3 steps flagged · score 6.0 of 10

Bi-Mamba's next-gene loss may leak the target through the reverse pass, and the two interpretability experiments re-test the training losses.

  1. self definitional [Section 3.2 (Bi-Mamba Eqs. 6-9) and Section 3.3 (Pretraining Objective Eq. 11)]
    "Both the original sequence S and the reversed sequence Srev are processed independently using identical Mamba layers with shared weights ... o(l)t = z(l)t · h(l)t + (1 − z(l)t ) · ˜h(l)t ... Llang = − 1 M PM j=1 log P(g j | g 1, g 2, ..., g j−1)"

    At position j−1, the reverse pass over Srev has already consumed tokens sn,...,sj, so the combined state o_{j−1} contains the target gene g_j. Unless the LM head is restricted to the forward state h_t or the reverse pass is causally masked, Llang can be minimized by copying the target from the future instead of learning gene-gene structure. The paper states neither restriction, so the 'next-gene' pretraining objective is not a causal prediction; its high likelihood can reflect target leakage rather than learned context.

  2. fitted input called prediction [Section 4.3, Gene Rank Reconstruction]
    "The analysis begins with the random selection of sample cells, from which input gene tokens are extracted. GeneMamba then generates output tokens, which are ranked according to their predicted likelihoods."

    The experiment measures whether the model's predicted token likelihoods reproduce the input gene ranks. Those likelihoods are exactly the conditional probabilities P(g_j | preceding tokens) optimized by Eq. 11 during pretraining on cells from the same rank-based distribution. Reporting high token overlap and Spearman correlation as 'reconstruction ability' and 'explainability' is therefore a check that the model fits its own training objective, not an independent test of biological interpretability.

1 more flagged steps
  1. fitted input called prediction [Appendix C, Gene Correlation Analysis (Figure 7)]
    "GeneMamba's ability to differentiate between 'positive' and 'negative' gene pairs was assessed ... Nearly 30,000 gene pairs, evenly distributed as positive (1) or negative (0), were analyzed for similarity scores using cosine similarity and Pearson correlation. GeneMamba demonstrated a significant separation between the mean similarity scores of positive and negative pairs."

    The positive/negative labels are same-pathway versus different-pathway pairs, and the pretraining loss Lpathway (Eq. 14) explicitly maximizes cosine similarity of same-pathway pairs and minimizes it for different-pathway pairs via InfoNCE. Measuring the separation of these distributions after training is therefore evaluating the exact objective the model was trained to satisfy; the separation is forced by the loss rather than discovered by the model.

full rationale

The external benchmarks are genuinely external: cell type annotation is evaluated on held-out splits and multi-batch integration is scored with batch/biology metrics, and there is no load-bearing self-citation chain. However, the pretraining objective is not clearly causal under the paper's own equations: Bi-Mamba combines forward and reversed hidden states (Eqs. 6-9), and Eq. 11 conditions the next-gene prediction only on previous tokens without stating that the LM head is restricted to the forward state or that the reverse pass is masked, so the target gene can enter the conditioning state by construction. Two interpretability results are direct re-tests of the training losses rather than independent evidence: the rank reconstruction experiment (Section 4.3) evaluates the next-gene probabilities optimized by Eq. 11, and the gene-pair separation analysis (Appendix C) evaluates the pathway contrastive objective of Eq. 14. The multi-batch protocol also fine-tunes on the same cell-type labels later used in Avg_bio, which is an additional label-leakage concern. Because the main external task numbers do not reduce to the fitted objectives, the score is 6 rather than 8.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or biological entities. The free parameters are standard hyperparameters, but they are numerous and not all reported. The axioms are domain assumptions common in single-cell foundation model papers, but the rank-based tokenization and pathway label reliability are load-bearing for the reported results.

free parameters (4)
  • gamma = 0.1
    Weight for the pathway loss in Eq. (15), chosen based on validation results from sample datasets (Section 4, Pretraining Dataset Construction).
  • temperature tau = not reported
    Temperature scaling in the InfoNCE gene pathway loss (Eq. 13), a free hyperparameter that affects the contrastive objective.
  • top_k = 2048 or 4096
    Number of top ranked genes selected per cell as input sequence, affecting all pretraining and downstream results (Section 4).
  • model size = 24 layers, hidden 512, vocab 25426
    Architecture specifications with no ablation in the main text; Appendix F reports variants but not systematic ablations of these choices.
assumptions (3)
  • domain assumption Rank-based gene tokenization preserves the biological signal needed for cell type, batch, and gene correlation tasks.
    The entire input representation depends on ranking genes by normalized expression (Eq. 2). If ranking discards information needed for a task, all downstream claims are affected.
  • domain assumption Bidirectional state space models capture long-range gene dependencies better than transformers for this data.
    The motivation for Bi-Mamba (Section 3.2) assumes that SSM-based bidirectional context is sufficient and advantageous for gene-gene interactions, without a formal derivation.
  • domain assumption Pathway labels used for the contrastive loss are biologically accurate and useful for representation learning.
    The gene pathway loss (Eq. 14) relies on external pathway definitions. If those labels are noisy or irrelevant, the pretraining objective may not provide the claimed biological grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data." pith.science (2026). https://pith.science/paper/MXQKB2FP

@misc{pith2026250416956,
  author       = {Pith},
  title        = {Pith review of: GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXQKB2FP}},
  note         = {Machine review of arXiv:2504.16956}
}
read the original abstract

Single-cell RNA sequencing (scRNA-seq) enables high-resolution analysis of cellular heterogeneity, but its complexity, which is marked by high dimensionality, sparsity, and batch effects, which poses major computational challenges. Transformer-based models have made significant advances in this domain but are often limited by their quadratic complexity and suboptimal handling of long-range dependencies. In this work, we introduce GeneMamba, a scalable and efficient foundation model for single-cell transcriptomics built on state space modeling. Leveraging the Bi-Mamba architecture, GeneMamba captures bidirectional gene context with linear-time complexity, offering substantial computational gains over transformer baselines. The model is pretrained on nearly 30 million cells and incorporates biologically informed objectives, including pathway-aware contrastive loss and rank-based gene encoding. We evaluate GeneMamba across diverse tasks, including multi-batch integration, cell type annotation, and gene-gene correlation, demonstrating strong performance, interpretability, and robustness. These results position GeneMamba as a practical and powerful alternative to transformer-based methods, advancing the development of biologically grounded, scalable tools for large-scale single-cell data analysis.

Figures

Figures reproduced from arXiv: 2504.16956 by the authors.

Figure 1
Figure 1. The GeneMamba architecture and its downstream task applications. The framework begins [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The schematic overview of BiMamba Block. The BiMamba Block processes input se￾quences bidirectionally, capturing forward and re￾verse context through shared convolutional layers (Conv) and structured state machines (SSM). A gat￾ing mechanism integrates the outputs, followed by linear projection and nonlinearity layers, generat￾ing a context-aware representation for downstream tasks. State Space Models (SSMs) provide… view at source ↗
Figure 3
Figure 3. Results of multi-batch integration. Benchmark of the fine-tuned GeneMamba on the PBMC [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Gene rank reconstruct results on PBMC12k dataset. (a) Venn diagrams show￾ing overlapping between input and output tokens in the pancreas dataset by three models: Gene￾Mamba_U (unidirectional Mamba module as back￾bone), GeneFormer, GeneMamba (BiMamba mod￾ule as backbone…
Figure 5
Figure 5. Figure 5: Cell type distribution in the original and modified Myeloid datasets [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Gene Rank Reconstruct framework of GeneMamba [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Gene Correlation Analysis. (a) Distribution of consine similarity score for positive and negative gene pairs using Gene2Vec embeddings. (b) Distribution of pearson correlation score for positive and negative gene pairs using Gene2Vec embeddings. (c) Gene-gene topology …
Figure 8
Figure 8. Figure 8: Mean Squared Error of DE analysis Embedding Alignment To quantify discrimination power, we compared GeneMamba’s embeddings with those of scGPT and scFoundation. The "distance" between positive and negative pair distributions was measured using Euclidean distance, KL di…
Figure 9
Figure 9. Figure 9: ETS2 + CEBPE perturbation result [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Magnitude scores computed for all test perturbing combinations on the Norman dataset. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Results of cell type annotation. Bar plot of the classification metrics across various datasets of GeneMamba [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Results of multi-batch integration. Benchmark of the fine-tuned GeneMamba on the Covid19 dataset for the multi-batch integration task. The UMAP plot of learned cell embeddings is colored by cell types. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Results of multi-batch integration. Benchmark of the fine-tuned GeneMamba on the Perirhinal Cortex dataset for the multi-batch integration task. The UMAP plot of learned cell embeddings is colored by cell types. a b c d [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Cell type annotation results of GeneMamba. a. Confusion matrix of dataset hPancreas. b. Confusion matrix of dataset MS. c. Confusion matrix of dataset Myeloid. d. Confusion matrix of dataset Myeloid_b. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Gene-gene pairs correlation analysis. The Euclidean Distance, KL Divergence, and JS Divergence across three embedding methods (Gene2Vec, GeneMamba, and Random). A lower value indicates higher similarity. The embeddings generated by GeneMamba are more similar to Gene2V…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. bioMoR: Biology-Guided Mixture-of-Recursions for Effective Genomic Learning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A biology-aware Mixture-of-Recursions model, which routes gene and pathway tokens to different recursion depths using a graph of biological interactions, reports consistent accuracy and efficiency gains over biology-a...

Reference graph

Works this paper leans on

53 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    scmulan: a multitask generative pre-trained lan- guage model for single-cell analysis

    Haiyang Bian, Yixin Chen, Xiaomin Dong, Chen Li, Minsheng Hao, Sijie Chen, Jinyi Hu, Maosong Sun, Lei Wei, and Xuegong Zhang. scmulan: a multitask generative pre-trained lan- guage model for single-cell analysis. InInternational Conference on Research in Computational Molecular Biology, pages 479–482. Springer, 2024

    work page 2024

  2. [2]

    A deep dive into single-cell rna sequencing foundation models.bioRxiv, pages 2023–10, 2023

    Rebecca Boiarsky, Nalini Singh, Alejandro Buendia, Gad Getz, and David Sontag. A deep dive into single-cell rna sequencing foundation models.bioRxiv, pages 2023–10, 2023

    work page 2023

  3. [3]

    Transformer for one stop interpretable cell type annotation.Nature Communications, 14(1):223, 2023

    Jiawei Chen, Hao Xu, Wanyu Tao, Zhaoxiong Chen, Yuxuan Zhao, and Jing-Dong J Han. Transformer for one stop interpretable cell type annotation.Nature Communications, 14(1):223, 2023

    2023

  4. [4]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

  5. [5]

    scgpt: toward building a foundation model for single-cell multi-omics using generative ai

    Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scgpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, pages 1–11, 2024

    work page 2024

  6. [6]

    White-Box Diffusion Transformer for single-cell RNA-seq generation

    Zhuorui Cui, Shengze Dong, and Ding Liu. White-box diffusion transformer for single-cell rna-seq generation.arXiv preprint arXiv:2411.06785, 2024

    work page Pith review arXiv 2024

  7. [7]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    2022

  8. [8]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volume 235 ofProceeding...

    2024

Show all 53 references
  1. [9]

    Recent advances in trajectory inference from single-cell omics data.Current Opinion in Systems Biology, 27:100344, 2021

    Louise Deconinck, Robrecht Cannoodt, Wouter Saelens, Bart Deplancke, and Yvan Saeys. Recent advances in trajectory inference from single-cell omics data.Current Opinion in Systems Biology, 27:100344, 2021. 10

    2021

  2. [10]

    scrdit: Generating single-cell rna-seq data by diffusion transformers and accelerating sampling.arXiv preprint arXiv:2404.06153, 2024

    Shengze Dong, Zhuorui Cui, Ding Liu, and Jinzhi Lei. scrdit: Generating single-cell rna-seq data by diffusion transformers and accelerating sampling.arXiv preprint arXiv:2404.06153, 2024

    2024 arXiv

  3. [11]

    Gene2vec: distributed representation of genes based on co-expression.BMC genomics, 20:7–15, 2019

    Jingcheng Du, Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vec: distributed representation of genes based on co-expression.BMC genomics, 20:7–15, 2019

    2019

  4. [12]

    scgraphformer: unveiling cellular heterogeneity and interactions in scrna-seq data using a scalable graph transformer network.Communications Biology, 7(1):1463, 2024

    Xingyu Fan, Jiacheng Liu, Yaodong Yang, Chunbin Gu, Yuqiang Han, Bian Wu, Yirong Jiang, Guangyong Chen, and Pheng-Ann Heng. scgraphformer: unveiling cellular heterogeneity and interactions in scrna-seq data using a scalable graph transformer network.Communications Biology, 7(1...

    2024

  5. [13]

    Pathway analysis: state of the art.Frontiers in physiology, 6:383, 2015

    Miguel A García-Campos, Jesús Espinal-Enríquez, and Enrique Hernández-Lemus. Pathway analysis: state of the art.Frontiers in physiology, 6:383, 2015

    2015

  6. [14]

    xtrimogene: an efficient and scalable representation learner for single-cell rna-seq data.Advances in Neural Information Processing Systems, 36, 2024

    Jing Gong, Minsheng Hao, Xingyi Cheng, Xin Zeng, Chiming Liu, Jianzhu Ma, Xuegong Zhang, Taifeng Wang, and Le Song. xtrimogene: an efficient and scalable representation learner for single-cell rna-seq data.Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  8. [16]

    Integrating pathway knowledge with deep neural networks to reduce the dimen- sionality in single-cell rna-seq data.BioData Mining, 15:1–21, 2022

    Pelin Gundogdu, Carlos Loucera, Inmaculada Alamo-Alvarez, Joaquin Dopazo, and Isabel Nepomuceno. Integrating pathway knowledge with deep neural networks to reduce the dimen- sionality in single-cell rna-seq data.BioData Mining, 15:1–21, 2022

    2022

  9. [17]

    Large-scale foundation model on single-cell transcriptomics.Nature Methods, pages 1–11, 2024

    Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Xuegong Zhang, and Le Song. Large-scale foundation model on single-cell transcriptomics.Nature Methods, pages 1–11, 2024

    2024

  10. [18]

    sctranssort: Transformers for intelligent annotation of cell types by gene embeddings.Biomolecules, 13(4):611, 2023

    Linfang Jiao, Gan Wang, Huanhuan Dai, Xue Li, Shuang Wang, and Tao Song. sctranssort: Transformers for intelligent annotation of cell types by gene embeddings.Biomolecules, 13(4):611, 2023

    2023

  11. [19]

    Assessing the limits of zero-shot foundation models in single-cell biology.bioRxiv, pages 2023–10, 2023

    Kasia Z Kedzierska, Lorin Crawford, Ava P Amini, and Alex X Lu. Assessing the limits of zero-shot foundation models in single-cell biology.bioRxiv, pages 2023–10, 2023

    2023

  12. [20]

    Fast, sensitive and accurate integration of single-cell data with harmony.Nature methods, 16(12):1289–1296, 2019

    Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony.Nature methods, 16(12):1289–1296, 2019

    2019

  13. [21]

    Single-cell rna sequencing in cancer research: New insights and applications.Cancer Research, 84(10):1234–1245, 2024

    Jing Li and Yu Wang. Single-cell rna sequencing in cancer research: New insights and applications.Cancer Research, 84(10):1234–1245, 2024

    2024

  14. [22]

    Bi-mamba4ts: Bidirectional mamba for time series forecasting.arXiv preprint arXiv:2404.15772, 2024

    Aobo Liang, Xingguo Jiang, Yan Sun, and Chang Lu. Bi-mamba4ts: Bidirectional mamba for time series forecasting.arXiv preprint arXiv:2404.15772, 2024

    2024 arXiv

  15. [23]

    Bidirectional gated mamba for sequential recommendation.arXiv preprint arXiv:2408.11451, 2024

    Ziwei Liu, Qidong Liu, Yejing Wang, Wanyu Wang, Pengyue Jia, Maolin Wang, Zitao Liu, Yi Chang, and Xiangyu Zhao. Bidirectional gated mamba for sequential recommendation.arXiv preprint arXiv:2408.11451, 2024

    2024 arXiv

  16. [24]

    Exploring genetic interaction manifolds constructed from rich single-cell phenotypes.Science, 365(6455):786–793, 2019

    Thomas M Norman, Max A Horlbeck, Joseph M Replogle, Alex Y Ge, Albert Xu, Marco Jost, Luke A Gilbert, and Jonathan S Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes.Science, 365(6455):786–793, 2019

    2019

  17. [25]

    Single-cell rna-seq data augmentation using generative fourier transformer

    Nima Nouri. Single-cell rna-seq data augmentation using generative fourier transformer. Communications Biology, 8(1):113, 2025

    2025

  18. [26]

    schyena: Foundation model for full-length single-cell rna-seq analysis in brain.arXiv preprint arXiv:2310.02713, 2023

    Gyutaek Oh, Baekgyu Choi, Inkyung Jung, and Jong Chul Ye. schyena: Foundation model for full-length single-cell rna-seq analysis in brain.arXiv preprint arXiv:2310.02713, 2023

    2023 arXiv

  19. [27]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 11

    2018 arXiv

  20. [28]

    Machine learning and statistical methods for clustering single-cell rna-sequencing data.Briefings in bioinformatics, 21(4):1209–1223, 2020

    Raphael Petegrosso, Zhuliu Li, and Rui Kuang. Machine learning and statistical methods for clustering single-cell rna-sequencing data.Briefings in bioinformatics, 21(4):1209–1223, 2020

    2020

  21. [29]

    Integration of single-cell rna-seq datasets: a review of computational methods.Molecules and cells, 46(2):106–119, 2023

    Yeonjae Ryu, Geun Hee Han, Eunsoo Jung, and Daehee Hwang. Integration of single-cell rna-seq datasets: a review of computational methods.Molecules and cells, 46(2):106–119, 2023

    2023

  22. [30]

    Regenne: genetic pathway-based deep neural network using canonical correlation regularizer for disease prediction.Bioinformatics, 39(11):btad679, 2023

    Divya Sharma and Wei Xu. Regenne: genetic pathway-based deep neural network using canonical correlation regularizer for disease prediction.Bioinformatics, 39(11):btad679, 2023

    2023

  23. [31]

    Generative pretraining from large-scale transcriptomes for single-cell deciphering.Iscience, 26(5), 2023

    Hongru Shen, Jilei Liu, Jiani Hu, Xilin Shen, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang, et al. Generative pretraining from large-scale transcriptomes for single-cell deciphering.Iscience, 26(5), 2023

    2023

  24. [32]

    A universal approach for integrating super large-scale single- cell transcriptomes by exploring gene rankings.Briefings in Bioinformatics, 23(2):bbab573, 2022

    Hongru Shen, Xilin Shen, Mengyao Feng, Dan Wu, Chao Zhang, Yichen Yang, Meng Yang, Jiani Hu, Jilei Liu, Wei Wang, et al. A universal approach for integrating super large-scale single- cell transcriptomes by exploring gene rankings.Briefings in Bioinformatics, 23(2):bbab573, 2022

    2022

  25. [33]

    Bidirectional mamba with dual-branch feature extraction for hyperspectral image classification.Sensors, 24(21):6899, 2024

    Ming Sun, Jie Zhang, Xiaoou He, and Yihe Zhong. Bidirectional mamba with dual-branch feature extraction for hyperspectral image classification.Sensors, 24(21):6899, 2024

    2024

  26. [34]

    Transformers in single-cell omics: a review and new perspectives.Nature methods, 21(8):1430–1443, 2024

    Artur Szałata, Karin Hrovatin, Sören Becker, Alejandro Tejada-Lapuerta, Haotian Cui, Bo Wang, and Fabian J Theis. Transformers in single-cell omics: a review and new perspectives.Nature methods, 21(8):1430–1443, 2024

    2024

  27. [35]

    Transfer learning enables predictions in network biology.Nature, 618(7965):616–624, 2023

    Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al. Transfer learning enables predictions in network biology.Nature, 618(7965):616–624, 2023

    2023

  28. [36]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  29. [37]

    Cellplm: pre-training of cell language model beyond single cells.bioRxiv, pages 2023–10, 2023

    Hongzhi Wen, Wenzhuo Tang, Xinnan Dai, Jiayuan Ding, Wei Jin, Yuying Xie, and Jiliang Tang. Cellplm: pre-training of cell language model beyond single cells.bioRxiv, pages 2023–10, 2023

    2023

  30. [38]

    Single cells are spatial tokens: Transformers for spatial transcriptomic data imputation.arXiv preprint arXiv:2302.03038, 2023

    Hongzhi Wen, Wenzhuo Tang, Wei Jin, Jiayuan Ding, Renming Liu, Xinnan Dai, Feng Shi, Lulu Shang, Hui Liu, and Yuying Xie. Single cells are spatial tokens: Transformers for spatial transcriptomic data imputation.arXiv preprint arXiv:2302.03038, 2023

    2023 arXiv

  31. [39]

    A comprehensive review of computational methods for scrna-seq.Briefings in Bioinformatics, 25(6):789–799, 2024

    Daniel White and Olivia Harris. A comprehensive review of computational methods for scrna-seq.Briefings in Bioinformatics, 25(6):789–799, 2024

    2024

  32. [40]

    scclip: Multi-modal single-cell contrastive learning integration pre-training

    Lei Xiong, Tianlong Chen, and Manolis Kellis. scclip: Multi-modal single-cell contrastive learning integration pre-training. InNeurIPS 2023 AI for Science Workshop, 2023

    2023

  33. [41]

    Stgrns: an interpretable transformer- based method for inferring gene regulatory networks from single-cell transcriptomic data

    Jing Xu, Aidi Zhang, Fang Liu, and Xiujun Zhang. Stgrns: an interpretable transformer- based method for inferring gene regulatory networks from single-cell transcriptomic data. Bioinformatics, 39(4):btad165, 2023

    2023

  34. [42]

    scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data.Nature Machine Intelligence, 4(10):852–866, 2022

    Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data.Nature Machine Intelligence, 4(10):852–866, 2022

    2022

  35. [43]

    Genecompass: deciphering universal gene regulatory mechanisms with a knowledge-informed cross-species foundation model.Cell Research, pages 1–16, 2024

    Xiaodong Yang, Guole Liu, Guihai Feng, Dechao Bu, Pengfei Wang, Jie Jiang, Shubai Chen, Qinmeng Yang, Hefan Miao, Yiyang Zhang, et al. Genecompass: deciphering universal gene regulatory mechanisms with a knowledge-informed cross-species foundation model.Cell Research, pages 1–16, 2024

    2024

  36. [44]

    sctca: a hybrid transformer-cnn architecture for imputation and denoising of scdna-seq data.Briefings in Bioinformatics, 25(6):bbae577, 2024

    Zhenhua Yu, Furui Liu, and Yang Li. sctca: a hybrid transformer-cnn architecture for imputation and denoising of scdna-seq data.Briefings in Bioinformatics, 25(6):bbae577, 2024. 12

    2024

  37. [45]

    Innovative super-resolution in spatial transcriptomics: a transformer model exploiting histology images and spatial gene expression.Briefings in Bioinformatics, 25(2):bbae052, 2024

    Chongyue Zhao, Zhongli Xu, Xinjun Wang, Shiyue Tao, William A MacDonald, Kun He, Amanda C Poholek, Kong Chen, Heng Huang, and Wei Chen. Innovative super-resolution in spatial transcriptomics: a transformer model exploiting histology images and spatial gene expression.Briefings...

    2024

  38. [46]

    Large-scale cell representation learning via divide-and-conquer contrastive learning.arXiv preprint arXiv:2306.04371, 2023

    Suyuan Zhao, Jiahuan Zhang, and Zaiqing Nie. Large-scale cell representation learning via divide-and-conquer contrastive learning.arXiv preprint arXiv:2306.04371, 2023

    2023 arXiv

  39. [47]

    Langcell: Language- cell pre-training for cell identity understanding.arXiv preprint arXiv:2405.06708, 2024

    Suyuan Zhao, Jiahuan Zhang, Yushuai Wu, Yizhen Luo, and Zaiqing Nie. Langcell: Language- cell pre-training for cell identity understanding.arXiv preprint arXiv:2405.06708, 2024

    2024 arXiv

  40. [48]

    in some cases, data from the same cell exists in different datasets, therefore cells can be duplicated throughout CELLxGENE Discover and by extension the Census,

    Alexander Zien, Robert Küffner, Ralf Zimmer, and Thomas Lengauer. Analysis of gene expression data with pathway scores. InIsmb, volume 8, pages 407–417, 2000. 13 Appendix A Dataset Construction and Setup We constructed our pretraining dataset manually, which is sourced from th...

    2000

  41. [49]

    By adjusting for chance agreement, ARIcell captures how well the integration maintains the clustering structure: ARIcell = Index observed−Index expected Max index−Index expected

    Adjusted Rand Index (ARIcell):We use ARIcell to quantify the agreement between the true biological labels and the clusters predicted after integration. By adjusting for chance agreement, ARIcell captures how well the integration maintains the clustering structure: ARIcell = In...

  42. [50]

    This score ranges from 0 (no alignment) to 1 (perfect alignment), and we use it to assess the consistency of clustering

    Normalized Mutual Information ( NMI cell):To evaluate how much information is shared between the true biological labels (Y ) and the predicted cluster labels (C), we compute NMI cell using: NMI cell = 2·I(Y;C) H(Y) +H(C) , whereI(Y;C) is the mutual information, and H(Y) andH(C...

  43. [51]

    The silhouette score (ASWC) evaluates whether cells are closer to their own cluster than to other clusters

    Average Silhouette Width (ASWcell):We calculate ASWcell to measure how well-separated clusters are based on cell type labels. The silhouette score (ASWC) evaluates whether cells are closer to their own cluster than to other clusters. We normalize the silhouette score using: AS...

  44. [52]

    First, we calculate the silhouette score based on batch labels (ASWB), which measures how batch-specific artifacts affect the integrated space

    Average Silhouette Width for Batch Labels ( ASWbatch):To evaluate the extent of batch effect removal, we computeASWbatch. First, we calculate the silhouette score based on batch labels (ASWB), which measures how batch-specific artifacts affect the integrated space. Then, we ad...

  45. [53]

    positive

    Graph Connectivity ( GraphConn):To measure the connectivity of cells within the same biological type, we construct a k-nearest neighbors (kNN) graph for each cell type. Then, we identify the largest connected component (LCC) within each graph and calculate the connectivity sco...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.