{"id":"004190eb-79a9-4375-abad-de1d2125d7e7","arxiv_id":"2501.08187","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal language model, InstructCell, follows natural-language commands to annotate cell types, predict drug sensitivity, and generate realistic single-cell expression profiles.","lead":"InstructCell is a new AI system that lets researchers analyze single-cell gene expression data by typing plain-language instructions, and it can also generate artificial cell profiles. It matches or beats several existing single-cell foundation models on cell typing, drug response, and pseudo-cell generation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random cell-level splits leak patient/sample identity in the DSP task and mask poor cross-dataset generalization, so the headline performance claims are not yet supported for unseen experimental conditions.","rationale":"The reader correctly identifies the random-split evaluation as the weakest assumption. My analysis sharpens this concern by specifying the mechanism: for the DSP task, cell-level random splitting causes patient/sample leakage, which can artificially inflate the drug-sensitivity results. This is a concrete, testable flaw. For CTA and CPCG, the random-split protocol remains a limitation for cross-dataset generalization, but it is not necessarily invalid for within-dataset model comparison. The paper makes a credible engineering contribution with a clean architecture (Q-Former cell encoder, T5 backbone, ZINB CVAE decoder) and a large synthetic instruction dataset; the instruction-following robustness shown in Fig. 5 is a genuine positive result. However, the central claim of adapting to diverse experimental conditions requires held-out-dataset evaluation and sample-level splitting for DSP. Since the reader already recommended CONDITIONAL acceptance, and my concern reinforces the need for additional evaluation without invalidating the method's potential, the verdict remains CONDITIONAL (no change). I do not see an internal inconsistency in the architecture, and the marker-gene analysis and template-diversity ablations provide useful independent support. The missing LangCell/scFoundation comparison and lack of error bars are secondary concerns; the evaluation protocol is the load-bearing issue that should be settled first.","tokens_in":27829,"tokens_out":6682,"duration_ms":69353,"concrete_test":"Re-run the DSP evaluation with a sample-level or patient-level split, e.g., leave-one-patient-out or grouped 8:1:1 split by sample ID, and compare accuracy and F1 against the current cell-level random-split results. If performance drops by more than 10 percentage points, the reported DSP results are inflated by patient leakage. Additionally, perform a held-out-dataset test for CTA by training on four of the five datasets and evaluating on the fifth, then report zero-shot accuracy; this directly tests whether the model generalizes to unseen experimental conditions rather than memorizing dataset-specific structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that InstructCell 'consistently meets or exceeds' single-cell foundation models across CTA, CPCG, and DSP while 'adapting to diverse experimental conditions'. The evaluation protocol in Methods divides each dataset into 8:1:1 random splits at the cell level. For the DSP datasets (GSE117872, GSE149383, GSE110894), each dataset consists of cells from a small number of patients or samples, and all cells from a given sample share the same drug-sensitivity label. A random cell-level split therefore places cells from the same patient in both training and test sets. The model can memorize patient-specific expression signatures or batch effects rather than learn a generalizable drug-response rule. The reported DSP accuracy exceeding 0.95 on confusion matrices (Fig. 4c) is implausibly high for cross-patient drug response prediction, strongly suggesting leakage. The same random-split protocol applies to CTA and CPCG: test cells originate from the same studies, tissues, and protocols as training cells, so the evaluation does not test adaptation to a genuinely new dataset or experimental condition. The robustness analysis in Fig. 5 only varies instruction phrasing on the same held-out cells; it does not vary the biological source. Thus, the core generalization claim—that InstructCell adapts to diverse experimental conditions—is not supported by the current evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"InstructCell is a multi-modal instruction-tuned model for scRNA-seq that combines a Q-Former cell encoder, a T5-base language model, and a ZINB-based conditional VAE cell decoder. The authors synthesize a large GPT-4o-generated instruction template set, train instruct and chat variants on 11 public datasets, and evaluate the model on cell type annotation (CTA), conditional pseudo-cell generation (CPCG), and drug sensitivity prediction (DSP). The paper compares against scGPT, scBERT, Geneformer, scDiffusion, scGAN, and Cell2Sentence, validates marker-gene findings against CellMarker2.0, and includes ablations of the Q-Former, query count, multi-task tuning, pre-trained weights, and template diversity. The headline claims are that InstructCell meets or exceeds single-cell foundation models and adapts to diverse experimental conditions while preserving discrete gene-expression information.","tokens_in":28120,"tokens_out":8299,"duration_ms":77832,"significance":"If the evaluation supported them, the claims would be significant: InstructCell would provide a lightweight, instruction-following alternative to large single-cell foundation models, unifying understanding and generation tasks in one multi-modal framework. The paper has clear strengths: the models and code are publicly released; the architecture math in Eqs. (1)-(13) is standard and largely sound; comparisons are made against externally published baselines; marker-gene results are checked against an external database; and the template-diversity study is a useful analysis. The weakness is evaluational rather than conceptual: the random cell-level split protocol does not test the core generalization claim, and most headline comparisons lack uncertainty estimates. These gaps are fixable with additional experiments, which is why I recommend major revision rather than rejection.","major_comments":[{"comment":"The central claim that InstructCell 'adapts to diverse experimental conditions' is not supported by the current evaluation protocol. The Methods state that 'each dataset is divided into training, validation, and test sets in an 8:1:1 ratio' and that multi-task tuning 'merge[s] all training splits into a single mixed training set.' For DSP, the datasets GSE117872, GSE149383, and GSE110894 are composed of a small number of patients/samples, and all cells from one sample share the same drug-sensitivity label; a random cell-level split places cells from the same sample in both training and test sets, allowing the model to memorize sample-specific expression signatures. The near-perfect accuracies in Fig. 4(c) (>0.95) are consistent with such leakage. The same random-split design is used for CTA and CPCG, so the model is never tested on an unseen tissue, batch, donor, or study; the 'unseen template' analysis in Fig. 5(a) varies only instruction phrasing on the same held-out cells. I request a held-out-patient or leave-one-dataset-out evaluation (with per-fold metrics) for DSP, and a leave-one-dataset-out or cross-tissue protocol for CTA and CPCG, before the generalization claim can be accepted.","section":"Methods, Experimental setup; Fig. 9; Figs. 4-5"},{"comment":"The main model-vs-baseline comparisons are reported as single values without error bars, confidence intervals, or significance tests. Fig. 3(a) and Fig. 4(a) show colored bars for CTA and DSP, and the ablations in Fig. 7(b-e) compare single average values. Because the headline claim is that InstructCell 'consistently meets or exceeds' several baselines, differences that could be within run-to-run noise need to be quantified. The authors should report mean and standard deviation over at least 3-5 random seeds or split instantiations, and include a paired significance test (e.g., McNemar's test or a bootstrap) for the classification comparisons, with analogous confidence estimates for the CPCG metrics.","section":"Results, Figs. 3(a), 4(a), 7(b-e)"},{"comment":"The second strong claim—that InstructCell 'preserves the discrete nature of gene expression profiles'—is not actually tested by the CPCG evaluation. All CPCG metrics (MMD, ΔsKNN, pKNN) are computed after normalizing counts to 10,000, applying a log1p transform, reducing to 50 principal components, and embedding with UMAP (Methods, Metrics, Eqs. 14-17). These transformations remove the count-level information that the discrete-modality design is intended to protect; a model producing smoothed continuous values could receive similar scores. I ask the authors to add count-level fidelity checks (e.g., raw-count histograms, zero-inflation statistics, NB/ZINB likelihood, or comparisons of the generated count distributions against real data) or to moderate the claim accordingly.","section":"Results, 'InstructCell enables conditional pseudo-cell generation'; Methods, 'Metrics'"}],"minor_comments":[{"comment":"The caption says 'Evaluation of InstructCell's CTA performance across human oral, lung, and mouse bone datasets' and refers to 'predicted cell types'; both should refer to drug sensitivity prediction and drug-sensitivity labels.","section":"Fig. 4 caption"},{"comment":"The text says the CPCG experiments use '9 tissues—bladder, blood, liver, lung, spleen, thymus, and vasculature' but lists only seven tissues; the count and list should be reconciled with the dataset table in Fig. 9.","section":"Results, 'InstructCell enables conditional pseudo-cell generation'"},{"comment":"The sentence that InstructCell works 'despite not relying on large-scale unlabeled pre-training' is imprecise because the backbone is a pre-trained T5-base model, and Fig. 7(e) shows that removing pre-trained weights degrades classification performance; the authors should qualify what kind of pre-training they mean.","section":"Results, 'InstructCell boosts the performance of cell type annotation'"},{"comment":"The T5 paper is cited twice as references [25] and [70]; this duplicate should be removed, and reference [49] should give the full author list and venue for 'Attention is all you need'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a candidate for a solid systems paper if the evaluation is reworked along the lines above. The main risk is not conceptual circularity: the baselines and marker-gene validation are external, and the architecture is well within standard practice. The risk is that the reported DSP accuracies are inflated by sample-level leakage, which would overstate the utility of the method for real users. A leave-one-patient-out or leave-one-dataset-out result set would resolve this concern. I would also encourage the editor to require the authors to deposit the exact data-split metadata so the revised experiments can be checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a solid engineering contribution with a real new capability, but the evaluation protocol does not support the abstract's claim that InstructCell \"consistently meets or exceeds\" single-cell foundation models while \"adapting to diverse experimental conditions.\" The main experiments use random 8:1:1 cell-level splits within each dataset, so test cells come from the same patients, batches, and tissues as training cells. For the drug sensitivity task that is close to leakage: GSE117872 and the other DSP datasets have few patients or samples, and cells from the same patient share the same drug label. The reported >0.95 confusion-matrix accuracies are implausibly high for cross-patient drug response prediction and strongly suggest the model memorizes patient-specific signatures. The stress-test note lands.\n\nWhat is genuinely new and good: InstructCell is, to my knowledge, the first system that treats raw scRNA-seq count profiles as a distinct modality inside an interleaved instruction-following architecture, using a Q-Former encoder, a T5 backbone, and a scVI-style ZINB CVAE decoder. The synthetic instruction dataset covering three tasks across 299k cells is a real resource, and the code and model weights are released. The CVAE math is standard and mostly correct, and the marker-gene analysis is checked against an external database (CellMarker2.0). Baselines are reproduced from public repositories, which is more than many papers in this area do.\n\nSoft spots, in proportion: the central generalization claim is unsupported, and that is the main issue. There are also no error bars or significance tests in the headline comparisons against scGPT, scBERT, and Geneformer. The paper cites LangCell and scFoundation in related work but never compares against them, which is a gap for a paper claiming to meet or exceed existing single-cell foundation models. The robustness analysis only varies instruction phrasing on the same held-out cells; it does not test a truly unseen dataset. The abstract overstates what the evidence shows. These are fixable: add held-out-dataset or patient-level evaluation, report variance across seeds, soften the claims, and add the missing comparisons.\n\nWho this is for: researchers building instruction-tuned multimodal cell models, and biologists who want a natural-language interface to standard single-cell tasks. It deserves a serious referee because the architecture is novel, the resource is useful, and the work is reproducible. A reviewer should ask for a revised evaluation before acceptance, not for a desk reject.","headline":"Useful instruction-following architecture and dataset, but random cell-level splits leave the headline generalization claims unsupported.","tokens_in":28644,"tokens_out":1692,"would_cite":true,"duration_ms":19633,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-modal copilot lets researchers drive single-cell annotation, generation, and drug prediction with natural-language instructions and matches foundation-model performance.","keywords":["single-cell RNA-seq","instruction following","multimodal language model","cell type annotation","pseudo-cell generation","drug sensitivity prediction","Q-Former","zero-inflated negative binomial"],"falsifier":"Train InstructCell on one set of tissues or sequencing batches and test it on tissues or batches that never appear in training; if cell type annotation and drug sensitivity accuracy fall to near chance, the claim of adapting to diverse experimental conditions is disproved.","tokens_in":27655,"feed_emoji":"🧬","tokens_out":7501,"duration_ms":72573,"temperature":0.7,"pith_summary":"InstructCell is a system that lets a researcher work with single-cell RNA-sequencing data by typing natural-language instructions instead of writing custom code or running specialized pipelines. The paper argues that one instruction-tuned model can perform three core tasks—identifying cell types, generating synthetic cells that match a described condition, and predicting whether a cell is sensitive or resistant to a drug—and that on these tasks it matches or beats large single-cell foundation models that required enormous unlabeled pretraining. The practical case is accessibility: a biologist who can phrase a question in English can get an answer or a cell profile back, and the model keeps gene expression as exact integer counts rather than losing precision by converting to text. The paper also shows that the model copes with instructions phrased in styles it never saw during training, and that its sensitivity analysis surfaces known marker genes.","feed_headline":"Natural-language commands match single-cell foundation models","feed_subtitle":"InstructCell answers cell-type queries, generates synthetic cells, and predicts drug response from plain English.","key_machinery":"The load-bearing machinery is a three-part multimodal cell-language architecture. A Q-Former-style cell encoder with eight learnable query vectors reads the raw gene-expression vector and produces a small set of cell embeddings; a pretrained text-to-text language model processes these embeddings together with the text tokens, so the same transformer stack reasons over both modalities; and a conditional variational autoencoder, using a zero-inflated negative binomial output distribution, reconstructs or generates exact integer expression counts when the model emits a special signal token. The special tokens that mark where cell data enters the token stream and the signal token that marks the handoff from text to cell generation are what let one set of weights answer with text, with a vector, or with an interleaved mix.","core_discovery":"The central claim is that the discrete, numerical language of single-cell transcription and the flexible language of human instructions can be coupled in one end-to-end model, and that coupling is enough to make instruction following competitive with dedicated pretrained foundation models. InstructCell interleaves text and cell profiles in one token sequence: a cell profile is marked by special tokens, encoded by a learned query-based transformer, and fused with text in a pre-trained language-model backbone; for generation tasks the model emits a signal token whose hidden state conditions a count-based decoder that outputs a new gene-expression profile. The paper reports that on cell type annotation and drug sensitivity prediction InstructCell matches or exceeds the foundation-model baselines despite no large-scale single-cell pretraining, and on conditional pseudo-cell generation it produces distributions closer to real cells than the generative baselines it is compared with. It further claims this works across human and mouse data from multiple tissues and remains stable when instruction wording changes.","pith_inferences":["The paper's random-split evaluation leaves untested how the model would fare on a wholly unseen tissue or sequencing batch, so a natural stress test is to hold out entire batches or organs before concluding that it adapts to diverse experimental conditions.","The instruction-template synthesis recipe, which varies personality, motivation, and proficiency traits when prompting a large language model, could plausibly transfer to other molecular modalities such as chromatin accessibility or spatial transcriptomics with minimal architectural change.","The marker-gene results hint that saliency over this architecture could double as a hypothesis generator for novel cell-type markers, but the paper only checks agreement with known markers, so prospective validation in the wet lab would be needed."],"forward_implications":["A single instruction-tuned model can replace task-specific pipelines for cell type annotation, conditional pseudo-cell generation, and drug sensitivity prediction.","Because cell profiles stay in their native count modality, outputs preserve low-expression genes and exact integer values, which matters for downstream count-based analyses.","The model generalizes to instruction templates and multiple-choice formats it has never seen, so users can phrase queries freely without retraining.","Multi-task instruction tuning beats single-task tuning on every task, implying that the shared cell-language representation is a genuine asset rather than a compromise."],"supporting_citations":[{"why":"Supplies the masked-language-model foundation baseline for cell type annotation comparison.","marker":"[12]"},{"why":"Supplies the transfer-learning foundation baseline the paper compares against for classification tasks.","marker":"[13]"},{"why":"Supplies the generative pretraining baseline used for annotation, generation, and drug prediction comparisons.","marker":"[14]"},{"why":"Is the text-based gene-name representation baseline that InstructCell is designed to improve on.","marker":"[16]"},{"why":"Is the diffusion-based generation baseline for conditional pseudo-cell generation.","marker":"[32]"},{"why":"Is the adversarial generation baseline for conditional pseudo-cell generation.","marker":"[33]"},{"why":"Provides the query-based transformer cell encoder that the model's cell encoder is based on.","marker":"[22]"},{"why":"Provides the conditional variational autoencoder formulation used for cell reconstruction.","marker":"[30]"},{"why":"Supplies the zero-inflated negative binomial count model and inference objective used by the cell decoder.","marker":"[59]"},{"why":"Is the large language model used to synthesize the instruction and response templates in the multimodal dataset.","marker":"[20]"}],"fun_headline_variants":["AI copilot turns natural language into single-cell insights","Talk to your cells: AI copilot for single-cell analysis","InstructCell: plain English controls single-cell AI","Single-cell analysis via natural language instructions","InstructCell: matches or exceeds foundation models with language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a random 8:1:1 split of each dataset, where test cells come from the same tissues and batches as training cells, tells us how the model will perform on genuinely new biological samples.","fun_headline_variants_meta":{"raw":{"variants":["AI copilot turns natural language into single-cell insights","Talk to your cells: AI copilot for single-cell analysis","InstructCell: plain English controls single-cell AI","Single-cell analysis via natural language instructions","InstructCell: matches or exceeds foundation models with language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2107,"prompt_tokens":938,"completion_tokens":1169,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1093}},"tokens_in":554,"tokens_out":1169,"duration_ms":10527,"temperature":1.0,"reasoning_tokens":1093,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:31.255862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train InstructCell on one set of tissues or sequencing batches and test it on tissues or batches that never appear in training; if cell type annotation and drug sensitivity accuracy fall to near chance, the claim of adapting to diverse experimental conditions is disproved.","supporting_citations":[],"review_version":1}