{"id":"95630095-0be1-4a45-b6a4-be2314c1ffdb","arxiv_id":"2504.16956","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Bi-Mamba foundation model with rank-based gene tokenization and a pathway contrastive loss reports competitive single-cell benchmarks while claiming linear-time efficiency over transformers.","lead":"GeneMamba is a state space model pretrained on roughly 30 million single-cell transcriptomes that aims to match transformer-based single-cell models with lower compute. It reports competitive results on some cell type annotation and batch integration benchmarks, but underperforms baselines on others and contains several internal inconsistencies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The next-gene pretraining loss may read the target through the reverse Bi-Mamba pass, making both the learning signal and the rank-reconstruction evidence untrustworthy.","rationale":"The central claim has two parts: linear-time scalability and strong downstream performance. The scalability part is structurally reasonable because the backbone is Mamba, so I do not attack it. The 'strong performance' part depends on a valid pretraining objective and fair evaluation. I focused on the pretraining objective because a flaw there would contaminate every downstream embedding, including the rank-reconstruction evidence, which the paper presents as a contribution. The manuscript's own equations create a real ambiguity: a bidirectional encoder that combines forward and reverse states cannot be trained with an ordinary next-token NLL unless the loss is masked or the head uses only the forward state. No such masking is described. This is an internal consistency concern, not a disagreement with the field's consensus. The reader's verdict remains CONDITIONAL: the code check I propose is narrow and can be run on the released repository. If leakage is confirmed, the correct verdict would move toward REJECT because the pretraining signal, not just one evaluation, is compromised. I also note the post hoc Myeloid_b evaluation and the MS underperformance as secondary problems, but they do not affect my recommendation relative to the reader.","tokens_in":18441,"tokens_out":12224,"duration_ms":124154,"concrete_test":"Inspect the released training code to see whether the LM head for L_lang consumes o_t (the gated forward+reverse output) or only a causally masked forward state. A decisive probe: train the published model for one epoch on a small corpus with gene order shuffled, and compare full Bi-Mamba training against an ablation in which the reverse branch is zeroed out before the LM head. If the full model reaches far higher next-gene accuracy (or far lower L_lang) than the ablated model, the loss is exploiting future tokens. If the code shows an explicit causal mask or forward-only head, the concern is vacated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines Bi-Mamba by running the sequence forward and reversed, then combining the two hidden states into o_t via a gating mechanism (Eqs. 6-9). Section 3.3 then defines the pretraining loss L_lang as the negative log-likelihood of the true next gene g_j given only the preceding tokens {g_1,...,g_{j-1}} (Eq. 11). The paper never states that the LM head is restricted to the forward-state h_t, or that the reverse pass is causally masked. In the natural implementation of Eq. 7 on a genuinely reversed sequence, the combined representation at position j-1 has already seen g_j through the reverse context, so L_lang can be minimized by copying the target from the future instead of learning gene-gene structure. If instead Eq. 7 is read literally (same s_t in both directions), the 'reverse' pass is not reversed and the claimed bidirectional advantage is untested. Either reading leaves the pretraining objective and the architecture mutually inconsistent. Because Section 4.3's rank-reconstruction experiment measures exactly this next-gene task, its high token overlap and Spearman values may reflect target leakage rather than biological interpretability. This is load-bearing because the 'effective foundation model' claim rests on the pretraining signal being a valid causal prediction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GeneMamba proposes a bidirectional Mamba (Bi-Mamba) state-space backbone for single-cell RNA-seq, using rank-based gene tokenization, pretraining on roughly 30 million human cells from CELLxGENE, a next-gene prediction loss, a pathway-aware contrastive loss, and a third objective whose identity is not actually specified. The authors evaluate the model on multi-batch integration, cell type annotation, gene rank reconstruction, gene correlation analysis, and perturbation prediction, and claim linear-time scalability and performance that makes GeneMamba a practical and powerful alternative to transformer-based single-cell foundation models. The central claim is that a bidirectional state-space architecture with rank-based tokenization can serve as an effective and scalable foundation model for transcriptomics.","tokens_in":18676,"tokens_out":7494,"duration_ms":74450,"significance":"If the substantial concerns about the pretraining objective and evaluation protocol were resolved, the contribution could be valuable: a linear-time bidirectional SSM backbone for large-scale single-cell pretraining, a reproducible rank-based tokenization pipeline, broad benchmarking against GeneFormer, scGPT, scFoundation, and Harmony, and useful scalability measurements. The paper makes code and model links available and reports pretraining on a large corpus. However, the current evidence does not support the claimed superiority: several key metrics in Tables 1 and 2 favor the transformer baselines, the causal validity of the next-gene loss is not established, and the interpretability evidence in Section 4.3 is substantially circular. The paper therefore needs major revision rather than acceptance in its present form.","major_comments":[{"comment":"The paper does not specify a causal mask for the reversed pass. In Bi-Mamba, the combined output o_{j-1} is formed from the forward state h_{j-1} and the reversed state \\tilde{h}_{j-1}, and the reversed recurrence has already incorporated g_j, ..., g_n by the time position j-1 is processed. If the language-model head for L_lang in Eq. (11) uses o_{j-1}, the loss can be minimized by copying the true next gene from the future rather than by learning gene-gene structure. If, instead, only the forward hidden state is used, the paper must state this explicitly and justify why the bidirectional representation is still central to the pretraining objective. This is load-bearing because Section 4.3's rank-reconstruction experiment measures exactly this next-gene task, so the high BLEU and Spearman values in Table 3 could reflect target leakage. Please state which hidden state feeds the LM head, describe the masking convention for the reversed pass, and, if the implemented model is leaky, re-run the pretraining and the affected downstream experiments.","section":"§3.2 and §3.3, Eqs. (6)–(11)"},{"comment":"The text says 'The final pretraning loss is the weighted sum of the three loss,' but Eq. (15) defines only L = L_lang + gamma * L_pathway. No third term appears anywhere in Section 3.3 or the experiments. This is not purely a typographical issue: if a third biologically informed loss exists, the pretraining objective and its hyperparameters are under-specified; if it does not exist, the sentence and the word 'three' must be corrected.","section":"§3.3, Eq. (15)"},{"comment":"The abstract and introduction claim strong or superior performance and 'substantial improvements' over transformer-based methods, but the reported tables do not support those claims. On MS in Table 2, GeneMamba obtains accuracy 0.6825 and Macro-F1 0.5342, well below scGPT's 0.8471 and 0.6630. On COVID-19 Avg_bio in Table 1, GeneMamba scores 0.5537 versus scGPT's 0.6476, and it also trails scGPT on PBMC12k Avg_bio (0.8344 versus 0.9018) and on Perirhinal Cortex Avg_bio (0.9062 versus 0.9552). No confidence intervals, standard deviations, or significance tests are reported anywhere. The defensible claim is that GeneMamba is competitive on some datasets and datasets-specific in its advantages; please recalibrate the wording and add measures of variability.","section":"Tables 1 and 2; Abstract and §1"},{"comment":"The gene rank reconstruction experiment directly evaluates the same conditional next-gene objective used in pretraining, so high overlap between input and output ranks is partly a check that the training objective was learned, not an independent test of biological interpretability. The comparison with GeneMamba_U and GeneFormer is informative, but the paper should acknowledge this circularity and report held-out cells, sequence lengths, number of replicates, and standard deviations. In addition, Figure 4's caption says the results are on the pancreas dataset, while Section 4.3 and Table 3 report PBMC12k; these references must be reconciled.","section":"§4.3, Table 3, and Figure 4"},{"comment":"The evaluation protocol is internally inconsistent. Appendix B.2 states that a manual stratified 0.9/0.1 train/test split is applied to the downstream datasets, while the 'Finetuning Details' paragraph states that 'The train/test split is predefined by the dataset provider, so we do not perform any additional splitting.' These statements cannot both be true. Since all reported accuracies and integration metrics depend on the exact partition, and since the COVID-19 example shows that the split was modified to avoid missing rare classes, please specify the precise split for every dataset and confirm that each baseline was evaluated on exactly the same partitions.","section":"Appendix B.2 and 'Finetuning Details'"}],"minor_comments":[{"comment":"The normalization formula uses t-digest in the denominator without a clear per-gene index; please define the median over cells for each gene j explicitly and fix the notation.","section":"Eq. (1)"},{"comment":"The abstract says the model enables processing of over 50 million cells, while the pretraining corpus after filtering contains 29,849,897 cells; please distinguish raw downloaded cells from the final training set in both places.","section":"Abstract and §4, Pretraining Dataset Construction"},{"comment":"The Venn diagram and density plot are described as being on the pancreas dataset in the figure caption, whereas the surrounding text and Table 3 refer to PBMC12k; please align the caption with the reported experiment.","section":"§4.3 and Figure 4"},{"comment":"The text on embedding alignment compares GeneMamba with scGPT and scFoundation, but the Figure 15 caption lists Gene2Vec, GeneMamba, and Random; also, 'eucilean distance' should be 'Euclidean distance.' Please make the reference models consistent and provide quantitative values for the claimed separations.","section":"Appendix C and Figure 15"},{"comment":"The Levenshtein distance is defined over single-character edits, but the sequences being compared are gene tokens; please define the metric over tokens rather than characters.","section":"Appendix B.3"},{"comment":"The column header 'Training Time (h/million)' is ambiguous, and it is not reconciled with the main text's statement that pretraining took approximately three weeks on four A100 GPUs; please report total pretraining time and per-sample time separately.","section":"Table 5 and §5"}],"recommendation":"major_revision","confidential_remarks":"The pivotal question for the next round is whether the released training code applies causal masking to the reversed Bi-Mamba pass when computing L_lang. Since the paper provides a GitHub link, I recommend verifying the implementation directly: if the LM head reads the combined bidirectional state, the pretraining objective and the Section 4.3 reconstruction evidence cannot be interpreted as claimed. Also, the paper lists both an anonymous code URL and a named GitHub/HuggingFace repository; please ensure the version under review corresponds to a single, unambiguous code snapshot."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the core idea is worth a look—Bi-Mamba for single-cell RNA-seq with rank-based tokenization—but the pretraining objective as written leaks the next token through the reverse pass, and the reported performance doesn't back the 'superior' framing. If the leak is real (and I think it is), the strongest evidence for interpretability is just the model copying its input.\n\nWhat's new: applying Bi-Mamba to single-cell is sensible; it's a linear-time alternative to transformer backbones, and they pretrain on roughly 30M cells, which is serious. The pathway InfoNCE loss is a reasonable addition. They cover a decent range of tasks: batch integration, annotation, perturbation, gene correlation. The perturbation analysis in Appendix D is a nice extra, even if compressed.\n\nThe soft spots. First, Eq. (15) says 'weighted sum of three losses' but only L_lang and L_pathway are defined. That's minor but sloppy. The bigger issue is Section 3.2: the reverse pass processes the full reversed sequence with no causal masking mentioned, so at position j-1 the combined representation includes the reverse state that has already seen g_j. The next-gene loss (Eq. 11) then becomes a copying task. The rank reconstruction experiment (Section 4.3) directly evaluates this same objective, so those high overlap numbers are expected, not evidence of biological insight. Second, the tables contradict the abstract's 'strong performance' claim: MS accuracy 0.68 vs scGPT 0.85; COVID-19 Avg_bio 0.55 vs scGPT 0.65. On Myeloid_b, GeneMamba's Macro-F1 is lower than all three baselines. The Myeloid_b dataset itself is post hoc constructed by dropping rare types—that's cherry-picking. There are no error bars, no seeds, and the code link is anonymous, so reproducibility is unverifiable.\n\nNone of this makes the paper worthless. The architecture is plausible, and the efficiency argument is real. But the claims need rewriting, the leak needs to be addressed (e.g., restrict the LM head to the forward state or mask the reverse pass), and the experiments need error bars. I'd send it to peer review because the problem is relevant and the resources are substantial; reviewers should dig into the architecture-loss consistency. If you read only one thing, read Section 3.2 and Eq. 11 side by side.\n\nRecommendation: solid conditional; needs major revision before it could be accepted. I wouldn't cite it as-is, but I'd bring it to reading group to discuss the leak and the single-cell evaluation practices.","headline":"Bidirectional Mamba for single-cell is a plausible idea, but the pretraining objective leaks the next token through the reverse pass and the paper's own tables undercut its performance claims.","tokens_in":19241,"tokens_out":4607,"would_cite":false,"duration_ms":42366,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a bidirectional state space model, GeneMamba, can process single-cell gene expression sequences in linear time and deliver competitive or better performance than transformer-based single-cell foundation models on…","keywords":["single-cell RNA sequencing","state space model","Bi-Mamba","foundation model","cell type annotation","batch integration","gene-gene correlation","rank-based tokenization"],"falsifier":"A matched-cost benchmark would settle the efficiency-performance claim: train GeneMamba and a transformer baseline with the same parameter count and the same FLOPs on 4,096-token sequences and compare cell-annotation Macro-F1 and batch-integration Avg_bio; if the transformer ties or wins at equal cost, the paper's central claim collapses. A second decisive check is whether a held-out set of genes that never appear in the top 2,048 ranks can be reconstructed above chance, which would test whether the rank-based pretraining objective learns transferable biology or only memorizes the training distribution.","tokens_in":18212,"feed_emoji":"🧬","tokens_out":11904,"duration_ms":102668,"temperature":0.7,"pith_summary":"GeneMamba asks whether a state space model can replace the transformer as the backbone for single-cell RNA sequencing analysis. The paper's answer is yes: by tokenizing each cell as a rank-ordered list of its most expressed genes and reading that list with a bidirectional Mamba encoder, the model processes sequences in linear time and, after pretraining on roughly 30 million human cells, matches or beats transformer-based single-cell foundation models on cell type annotation, multi-batch integration, and gene-gene correlation. The payoff is practical: single-cell foundation models have been limited by the quadratic cost of self-attention, and a linear-time backbone makes it feasible to train on tens of millions of cells or to use longer gene sequences. The authors further claim that the bidirectional state space design and the pathway-aware training loss give the model interpretable gene embeddings, as evidenced by gene rank reconstruction and perturbation predictions.","feed_headline":"GeneMamba scales to 30M cells with linear-time Bi-Mamba","feed_subtitle":"A bidirectional state space model matches transformer quality on cell annotation and integration at lower cost.","key_machinery":"The load-bearing object is the Bi-Mamba block, a selective state space layer that scans a gene token sequence in both the forward and reverse directions with shared weights and fuses the two scans through a learnable sigmoid gate; the state recurrence gives linear-time complexity in sequence length, and the bidirectional scan gives each gene both upstream and downstream context. Around this encoder, the paper places two supporting mechanisms: the rank-based gene tokenizer (Eq. 2), which orders a cell's genes by normalized expression and keeps the top 2,048 or 4,096 as tokens, and the pretraining objective (Eq. 15), which sums next-gene prediction loss with an InfoNCE pathway loss that pulls genes sharing a biological pathway together in embedding space.","core_discovery":"The central claim is that a bidirectional selective state space model can replace quadratic-complexity transformers as the backbone of single-cell foundation models without sacrificing predictive quality. GeneMamba tokenizes each cell as a rank-ordered list of its top expressed genes, processes that list with a Bi-Mamba stack that reads forward and backward through shared-weight state space layers, and is pretrained on roughly 30 million human cells with a next-gene prediction loss plus a pathway-aware contrastive loss. In the paper's experiments, this combination delivers the best or near-best scores among compared models: highest Macro-F1 on hPancreas (0.7710) and Myeloid (0.3650), best accuracy on hPancreas (0.9713) and Myeloid_b (0.9603), top Avg_bio on Immune (0.8131) and BMMC (0.7628) for batch integration, and a Levenshtein distance of 6 versus 23 for GeneFormer in gene-rank reconstruction on PBMC12k, with BLEU 0.987 versus 0.968. The authors conclude that GeneMamba is a practical and powerful alternative to transformer-based single-cell models.","pith_inferences":["The paper does not ablate the pathway-loss weight (set to gamma = 0.1); varying this weight would show how much of the gene-pair separation comes from the contrastive loss rather than from the bidirectional architecture itself.","The paper does not test inputs beyond 4,096 tokens; if the linear-time claim transfers, full-length transcriptomes and spatial transcriptomics sequences are natural next targets.","The batch-integration protocol uses embeddings from a model already fine-tuned on cell type labels, so an unsupervised integration benchmark would isolate how much the pretrained representation itself contributes."],"forward_implications":["Longer gene sequences become affordable: at 4,096 tokens the paper reports GeneMamba training at 0.2393 seconds per sample versus 0.6025 for a transformer, so analyses that cover more of the transcriptome become practical.","Pretraining scale can grow: linear-time processing removes the quadratic wall that has limited transformer-based single-cell foundation models, making corpora of tens of millions of cells a routine target.","A simple MLP head on GeneMamba embeddings reaches the best accuracy on hPancreas (0.9713) and best Macro-F1 on Myeloid (0.3650) among the models compared, which would simplify cell type annotation pipelines.","Batch integration with GeneMamba embeddings yields the highest biological-conservation scores on the Immune and BMMC benchmarks while still removing batch effects near the level of a dedicated integration method.","The joint next-gene and pathway-contrastive pretraining organizes gene embeddings so that same-pathway pairs are more separable than with transformer baselines, supporting downstream gene-gene correlation and regulatory analysis."],"supporting_citations":[{"why":"It supplies the selective state space model that GeneMamba extends into a bidirectional encoder, and it is the load-bearing efficiency mechanism.","marker":"[15]"},{"why":"It provides the bidirectional Mamba design pattern, including forward/reverse processing with a gating mechanism, that the Bi-Mamba block adapts to gene sequences.","marker":"[22]"},{"why":"It establishes rank-based gene tokenization for single-cell transformers and serves as the main transformer baseline for annotation and gene-rank reconstruction.","marker":"[35]"},{"why":"It is a transformer-based single-cell foundation model used as a baseline for integration, annotation, and perturbation experiments.","marker":"[5]"},{"why":"It is a large-scale transformer foundation model for single-cell transcriptomics whose value-projection tokenization is contrasted with the rank-based encoding.","marker":"[17]"},{"why":"It is the classical batch-integration method used as a baseline, reported to remove batch effects strongly but preserve less biological variation.","marker":"[20]"},{"why":"It supplies the Gene2Vec reference embeddings and the positive/negative gene-pair sets used to evaluate gene correlation and topology.","marker":"[11]"},{"why":"It introduced rank-based tokenization for single-cell transformers, supporting the paper's choice of gene ranking over binning.","marker":"[32]"},{"why":"It defines the InfoNCE contrastive objective that the pathway-aware loss is built on, pulling co-pathway gene embeddings together.","marker":"[27]"}],"fun_headline_variants":["GeneMamba: linear-time single-cell foundation model on 30M cells","Bi-Mamba beats transformers on single-cell at linear cost","State space model for single-cell: GeneMamba's 30M-cell pretraining","Efficient single-cell AI: GeneMamba trades quadratic for linear time","GeneMamba: bidirectional state space for scRNA-seq at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the premise that ranking a cell's genes by expression and keeping only the top 2,048 or 4,096 still leaves enough biological information for cell annotation, batch integration, and gene-pair analysis; if the discarded low-expression genes carry the signal, the reported gains could reflect the truncation rather than the Bi-Mamba architecture.","fun_headline_variants_meta":{"raw":{"variants":["GeneMamba: linear-time single-cell foundation model on 30M cells","Bi-Mamba beats transformers on single-cell at linear cost","State space model for single-cell: GeneMamba's 30M-cell pretraining","Efficient single-cell AI: GeneMamba trades quadratic for linear time","GeneMamba: bidirectional state space for scRNA-seq at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3017,"prompt_tokens":983,"completion_tokens":2034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":599,"tokens_out":2034,"duration_ms":12427,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:07:27.425027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched-cost benchmark would settle the efficiency-performance claim: train GeneMamba and a transformer baseline with the same parameter count and the same FLOPs on 4,096-token sequences and compare cell-annotation Macro-F1 and batch-integration Avg_bio; if the transformer ties or wins at equal cost, the paper's central claim collapses. A second decisive check is whether a held-out set of genes that never appear in the top 2,048 ranks can be reconstructed above chance, which would test whether the rank-based pretraining objective learns transferable biology or only memorizes the training distribution.","supporting_citations":[{"cited_title":"scgpt: toward building a foundation model for single-cell multi-omics using generative ai","cited_arxiv_id":null,"evidence_quote":"It is a transformer-based single-cell foundation model used as a baseline for integration, annotation, and perturbation experiments."},{"cited_title":"Large-scale foundation model on single-cell transcriptomics.Nature Methods, pages 1–11, 2024","cited_arxiv_id":null,"evidence_quote":"It is a large-scale transformer foundation model for single-cell transcriptomics whose value-projection tokenization is contrasted with the rank-based encoding."},{"cited_title":"Fast, sensitive and accurate integration of single-cell data with harmony.Nature methods, 16(12):1289–1296, 2019","cited_arxiv_id":null,"evidence_quote":"It is the classical batch-integration method used as a baseline, reported to remove batch effects strongly but preserve less biological variation."},{"cited_title":"Gene2vec: distributed representation of genes based on co-expression.BMC genomics, 20:7–15, 2019","cited_arxiv_id":null,"evidence_quote":"It supplies the Gene2Vec reference embeddings and the positive/negative gene-pair sets used to evaluate gene correlation and topology."},{"cited_title":"A universal approach for integrating super large-scale single- cell transcriptomes by exploring gene rankings.Briefings in Bioinformatics, 23(2):bbab573, 2022","cited_arxiv_id":null,"evidence_quote":"It introduced rank-based tokenization for single-cell transformers, supporting the paper's choice of gene ranking over binning."}],"review_version":1}