{"id":"09155290-7bda-4a08-812d-0a7b5e89c6cd","arxiv_id":"2505.08844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid LLM-plus-database pipeline improves automated cell type annotation accuracy in single-cell RNA-seq across nine datasets.","lead":"CellTypeAgent is an AI system that uses a large language model to suggest candidate cell types, then checks them against a gene expression database to pick the final answer. The authors report better accuracy than existing methods on nine single-cell RNA sequencing datasets, although the evaluation may be affected by overlap between the database and the ground truth.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data leakage via CellxGene overlap with benchmark ground truth is the main unresolved threat to the central claim.","rationale":"I read the paper in good faith. The method is a clean two-stage pipeline: an LLM proposes ranked candidates and CellxGene expression ranks select among them. The headline claim is that this consistently outperforms GPTCelltype and database-only baselines. That claim is plausible and the code is available. However, the evaluation depends on the independence of the verification database from the benchmark annotations. This is exactly the reader's weakest_assumption, and the manuscript text itself (Section 1, Appendix D) provides strong grounds for worry: CellxGene aggregates 1,634 datasets from 257 studies, while the benchmark datasets are public atlases from [8-14]. Several of those benchmark studies are large multi-tissue atlases that are standard constituents of such aggregated repositories. The authors provide no exclusion test, no statement that the benchmark cells are absent from CellxGene, and no versioning or temporal separation. If the ground-truth cells or their annotations are in CellxGene, the verification step scores the ground truth highly by construction, so the accuracy gain over LLM-only annotation is not evidence of trustworthy grounding; it is evidence of memorization or lookup. This is a correctness risk, not a novelty dispute. The suggested test is concrete and executable: inspect the CellxGene census for the benchmark studies and rerun on an exclusion-filtered snapshot. I agree with the reader's conditional verdict and would keep it conditional (not reject) because the method itself may still be useful; the evaluation must be fixed before the performance claim is accepted.","tokens_in":5429,"tokens_out":2192,"duration_ms":18069,"concrete_test":"Query the CZ CELLxGene Discover census for the nine benchmark datasets' source studies (Tabula Sapiens, Han 2020 human landscape, Han 2018 mouse atlas, Eraslan 2022, etc.) and check whether these datasets are included in the CellxGene aggregation. Concretely, list CellxGene dataset IDs and confirm whether each benchmark study's single-cell data appears. Then rerun CellTypeAgent with a leakage-controlled CellxGene snapshot that excludes any datasets from the benchmark studies. If the accuracy advantage over GPTCelltype largely disappears (or the per-dataset advantage flips), the central claim fails; if the advantage persists, the leakage concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CellTypeAgent outperforms GPTCelltype and database-only baselines across nine datasets. The verification stage selects the candidate with the highest average expression in CellxGene. This only validates rather than reads the answer if the CellxGene aggregated profiles were built from the same manual annotations or the same donor cells used as ground truth in the benchmark. The paper states in Appendix D that benchmark marker genes and annotations come from the original studies [8-14] and that CellxGene is a repository of 1,634 datasets including human and mouse atlases. Several of the cited benchmark datasets (Tabula Sapiens, human cell landscape, mouse cell atlas, Eraslan cross-tissue reference) are large public atlases that are highly likely to be constituent datasets of CellxGene. If the same annotated cells from these atlases are aggregated into CellxGene and the benchmark queries the database for expression of the very marker genes derived from those cells, then the verification step is not independent: the database trivially assigns high scores to the ground-truth cell types. In the extreme case, the method reduces to nearest-neighbor lookup into the answer key. The paper provides no check that the benchmark datasets are excluded from CellxGene, no de-duplication step, and no temporal cutoff. The reader's weakest assumption is therefore correct and load-bearing. A second related concern is that the agreement metric and the absence of error bars make the per-dataset margin hard to assess, but leakage is the primary threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CellTypeAgent, a two-stage cell type annotation method for single-cell RNA-seq data. In the first stage, an LLM is prompted with tissue type and marker genes to propose an ordered list of candidate cell types; in the second stage, candidates are rescored using expression value and expressed-ratio statistics queried from the CELLxGENE Discover database, and the top-scoring candidate is returned as the final annotation. Optional LitSense literature retrieval and NCBI gene summaries are also tested and reported to degrade performance. The method is evaluated on nine datasets from prior studies, with manual annotations as ground truth, and compared with GPTCelltype, CELLxGENE-only, and PanglaoDB baselines. The authors report consistent improvement across datasets, analyze the effect of candidate number and marker gene number, handle mixed cell types, and show that the open-source Deepseek-R1 with verification improves by 5.1%.","tokens_in":5689,"tokens_out":5079,"duration_ms":53960,"significance":"The proposed combination is simple and practically appealing, and the release of code and the inclusion of open-source LLM results are strengths. If the accuracy improvements hold, the method offers a low-cost way to ground LLM predictions in empirical expression data and would be a useful baseline for future annotation tools. The main empirical result, however, currently depends on an unexamined independence assumption between the verification database and the benchmark ground truth; resolving this determines whether the central claim is supported.","major_comments":[{"comment":"The verification stage scores candidates using CELLxGENE expression statistics, while the benchmark ground truth consists of manual annotations from references [8-14], including large public atlases such as Tabula Sapiens [10], the human cell landscape [8], the mouse cell atlas [9], and the Eraslan cross-tissue reference [14]. CELLxGENE Discover aggregates 1,634 datasets and very plausibly includes these or overlapping atlases. If the same annotated cells are stored in CELLxGENE, then the verification step in §A.2 is not an independent check: it selects the candidate whose stored expression profile agrees with the marker genes derived from the ground-truth annotation, and the comparison to GPTCelltype and database-only baselines in Figure 2a becomes circular. The manuscript reports no exclusion of the benchmark datasets from CELLxGENE, no de-duplication, and no temporal cutoff. Please quantify the overlap and either re-run the evaluation with benchmark atlases excluded or restrict the claims to annotation settings where the database does not contain the test atlases.","section":"§1 (Main), §A.2, Appendix D"},{"comment":"The central claim of consistent superiority is based on single-point agreement scores for each dataset, with no error bars, confidence intervals, or significance tests. Given the small number of datasets and the coarseness of the 0/0.5/1 agreement metric, the reported gains (including the 5.1% Deepseek-R1 improvement) may not be distinguishable from sampling or annotation variation. Please provide per-cluster or per-sample variation estimates and paired statistical comparisons, or state explicitly that the comparison is descriptive.","section":"Figure 2a"},{"comment":"The abstract and introduction claim that CellTypeAgent 'mitigates hallucinations', but no hallucination metric is defined or measured; the reported evidence is accuracy relative to manual annotations. Please either define and directly assess hallucination (for example, predictions that are not valid cell types or that contradict expression evidence) or soften the claim to improved annotation accuracy.","section":"Abstract, §1"}],"minor_comments":[{"comment":"The definition rank(v)=|unique(V)|−i−1 is ambiguous because i is not tied to a sorting convention; the subsequent score equation also omits a cell-type subscript in the tissue-summed term (1/|T|)∑_τ rank(e_gc(∫)), so a worked example would improve reproducibility.","section":"Appendix A.2"},{"comment":"The implementation of the CELLxGENE-only and PanglaoDB baselines is not described; it is unclear how candidates were generated and how thresholds for expressed ratio and scaled expression values were set.","section":"§1, Figure 2a, Appendix D"},{"comment":"Panels (c)-(g) do not report the number of samples or repetitions underlying each average, making it difficult to assess stability; please state these numbers in the caption.","section":"Figure 2"},{"comment":"The symbol ∫ for species is unconventional and easily mistaken for an integral; a standard abbreviation such as 'sp.' would improve readability.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is the potential leakage between CELLxGENE and the benchmark ground truth. This is appropriate for major revision rather than rejection because it is addressable with an overlap analysis and a leave-one-atlas-out evaluation. I would also ask for statistical rigor before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi X,\n\nQuick take on arXiv:2505.08844. CellTypeAgent pairs an LLM candidate generator with a CellxGene expression-based scoring function to pick the final cell type. The idea is clean, the code is public, and the evaluation spans nine datasets and several base models. The finding that database verification adds more when the base LLM is weaker (Deepseek-R1 +5.1%) is genuinely useful.\n\nThe problem is that the headline claim—consistently outperforms GPTCelltype and database-only baselines in all nine datasets—is probably an artifact of leakage. The benchmark ground truth comes from Tabula Sapiens, the human cell landscape, the mouse cell atlas, and similar public atlases. CellxGene aggregates those same datasets. When the agent queries CellxGene for expression of marker genes derived from those very cells, the verification step effectively reads the answer key. The paper does not check for overlap, does not exclude benchmark datasets, and has no temporal cutoff. In the extreme case the method is nearest-neighbor lookup into its own test set. That is load-bearing.\n\nThere are secondary issues: no error bars or significance tests, the agreement metric is coarse, and the GPT-4 version differs from GPTCelltype's original numbers (they disclose this, but it makes the comparison messy). The hallucination claim is inferred from accuracy, not measured directly. These are minor by comparison.\n\nWhat the paper does well: the pipeline is simple enough to reimplement, the ablation on candidate count and marker-gene number is sensible, and the mixed-cell-type evaluation is a nice addition. The literature-search negative result (LitSense/NCBI gene summaries hurt performance) is worth reporting.\n\nWho is this for? Anyone building LLM-based annotation tools for scRNA-seq. The idea itself is reasonable; the evaluation just does not support the claim as written. A serious referee should see it, with a clear request: show that results hold when benchmark datasets are removed from CellxGene, or provide a de-duplicated holdout. If that check fails, the paper is a negative result about leakage. If it passes, it is a useful incremental tool.\n\nI would not cite it in its current form. I'd bring it to reading group as a cautionary example of evaluation leakage.\n\nBest,\n[Your name]","headline":"A clean two-stage LLM + CellxGene annotation pipeline whose headline result is likely compromised by leakage from benchmark atlases inside the verification database.","tokens_in":6213,"tokens_out":1989,"would_cite":false,"duration_ms":19522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining an LLM's candidate cell types with gene-expression verification from the CellxGene database yields more accurate cell type annotation than either step alone, across nine datasets and 303 cell types.","keywords":["cell type annotation","single-cell RNA sequencing","large language models","LLM agent","hallucination mitigation","CellxGene","marker genes"],"falsifier":"For each benchmark dataset, remove from CellxGene every cell-type label and expression profile matching the manual ground truth before running CellTypeAgent; if accuracy on those held-out cell types falls to roughly the LLM-only baseline, the verification step is retrieving known answers. A complementary test runs the method on cell types that CellxGene does not contain at all and checks whether the database step still helps.","tokens_in":5238,"feed_emoji":"🧬","tokens_out":6803,"duration_ms":54506,"temperature":0.7,"pith_summary":"The paper proposes CellTypeAgent, a two-step annotation tool for single-cell RNA sequencing. First a large language model proposes the three most likely cell types for a cluster from its marker genes; then the agent scores those candidates against quantitative gene-expression profiles in the CellxGene database and picks the highest-scoring one. The claim is that this hybrid beats LLM-only annotation (GPTCelltype), database-only lookup, and the PanglaoDB baseline on all nine datasets tested, while reducing the hallucinations that pure LLM annotation suffers. The authors also show the database step helps most when the base LLM is weak, lifting the open-source Deepseek-R1 by 5.1 percent to near the top. If right, this makes automated cell type annotation more reliable without requiring a stronger proprietary model.","feed_headline":"Gene-database check lifts LLM cell-type annotation past every baseline","feed_subtitle":"A two-step agent—LLM guesses, then CellxGene expression check—beats LLM-only and database-only baselines on nine scRNA-seq datasets.","key_machinery":"The load-bearing mechanism is the two-stage selection score. Stage 1 assigns each LLM candidate an initial rank score $r_c = n - i$. Stage 2 pulls scaled expression $e_{gc}$ and expressed ratio $\\rho_{gc}$ for each marker gene in each candidate cell type from CellxGene and computes a combined score: candidate rank plus the rank of summed expression, plus the rank of summed expressed ratio, plus a tissue-averaged expression rank; the cell type with the maximum score is the final annotation. This score is what lets the database arbitrate among the LLM's candidates rather than replace the LLM.","core_discovery":"CellTypeAgent's central claim is that the trustworthiness of LLM-based cell type annotation is improved not by adding more text knowledge but by grounding the LLM's shortlist in a quantitative expression database. On nine real datasets covering 303 cell types from 36 tissues, the agent's agreement with manual annotations is consistently higher than the GPTCelltype baseline, CellxGene alone, and PanglaoDB. The paper also reports that open-source Deepseek-R1, which trails GPT-4o when used alone, is brought within reach of the best base model by the database-verification step, and that adding literature snippets or gene summaries to the prompt makes performance worse.","pith_inferences":["The same candidate-and-verify scoring could transfer to other biomedical annotation tasks that pair a generative model with a quantitative reference database, such as phenotype or disease-label assignment.","The strong gain for Deepseek-R1 suggests verification can substitute for model scale; a direct test would run smaller open-source models through the same two-step pipeline and measure how far the database step closes the gap.","Accuracy is bounded by CellxGene coverage, so the method should be stress-tested on cell types or tissues absent from the database, where the verification step has nothing to check.","If the benchmark atlases are themselves deposited in CellxGene, part of the reported gain could be retrieval rather than validation; a held-out design that removes benchmark cell types from the database before scoring would settle this."],"forward_implications":["Stronger base LLMs give higher accuracy, with the o1-preview model the best in the paper's tests, so the method scales with model quality.","Database verification provides the largest boost for weaker models: Deepseek-R1 improves by 5.1 percent and nearly matches the best base models, making open-source, privacy-preserving annotation competitive.","Including more marker genes generally improves accuracy, and using the top three LLM candidates is slightly better than one or five.","Adding literature snippets from LitSense or gene summaries from NCBI harms accuracy and raises cost, so the plain two-step design is preferable.","When prompted that a sample may be a mixture, CellTypeAgent can name multiple cell types, though mixed-sample accuracy is lower than for pure cell types."],"supporting_citations":[{"why":"Supplies the CellxGene expression and expressed-ratio data used to score and select the final cell type.","marker":"[7]"},{"why":"Provides the GPTCelltype LLM-only baseline, the agreement-score metric, and the evaluation datasets reused here.","marker":"[2]"},{"why":"PanglaoDB serves as the second external database-only baseline for comparison.","marker":"[15]"},{"why":"Deepseek-R1 is the open-source base model whose 5.1 percent gain under verification supports the claim that the database step helps weak models most.","marker":"[17]"},{"why":"LitSense is the literature-retrieval tool tested in the optional-knowledge experiments that perform worse than the plain pipeline.","marker":"[18]"},{"why":"One of the nine benchmark datasets whose manual annotations and marker genes define the accuracy comparison.","marker":"[8]"}],"fun_headline_variants":["LLM plus database check beats all baselines in cell typing","Grounding LLM in expression data curbs hallucinations in cell annotation","Two-step agent: LLM guess, CellxGene verify, tops every baseline","Open-source LLM with database check nears GPT-4o performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CellxGene database contains accurate expression profiles for every true cell type in the nine test datasets, and the manual annotations used as ground truth are not themselves already in CellxGene in a way that lets the verification step retrieve the answer rather than validate it.","fun_headline_variants_meta":{"raw":{"variants":["LLM plus database check beats all baselines in cell typing","Grounding LLM in expression data curbs hallucinations in cell annotation","Two-step agent: LLM guess, CellxGene verify, tops every baseline","Open-source LLM with database check nears GPT-4o performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1679,"prompt_tokens":751,"completion_tokens":928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":367,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":367,"tokens_out":928,"duration_ms":9802,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:49:31.842489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each benchmark dataset, remove from CellxGene every cell-type label and expression profile matching the manual ground truth before running CellTypeAgent; if accuracy on those held-out cell types falls to roughly the LLM-only baseline, the verification step is retrieving known answers. A complementary test runs the method on cell types that CellxGene does not contain at all and checks whether the database step still helps.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CellxGene expression and expressed-ratio data used to score and select the final cell type."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPTCelltype LLM-only baseline, the agreement-score metric, and the evaluation datasets reused here."},{"cited_title":"& Bj¨ orkegren, J","cited_arxiv_id":null,"evidence_quote":"PanglaoDB serves as the second external database-only baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LitSense is the literature-retrieval tool tested in the optional-knowledge experiments that perform worse than the plain pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the nine benchmark datasets whose manual annotations and marker genes define the accuracy comparison."}],"review_version":1}