{"id":"016550fb-18a8-464f-9edc-2ff33b66fe62","arxiv_id":"2412.18156","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SCREADER improves scRNA-seq cell-type annotation by passing GPT-embedded gene descriptions and rank-ordered expression through a frozen Llama-13b with an instruction prompt.","lead":"SCREADER feeds gene embeddings built from functional text descriptions and ranked expression levels through a frozen Llama-13b language model to classify single-cell RNA-seq cell types, reporting large gains over GenePT on human and mouse developmental datasets. The paper is a preliminary empirical study whose internal inconsistencies and missing evaluation details weaken its conclusions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy gain over GenePT is not established because the reported 'GenePT' baseline is a nonstandard reimplementation (learned up-projection before averaging), so the gap in Fig. 2 may be a comparison artifact.","rationale":"The reader's verdict identifies the weakest point correctly: the baseline is not the original GenePT. My reading of the manuscript text confirms this. Section V explicitly says the GenePT group keeps the two MLPs, up-projects cell embeddings to 4096 dimensions, and only then averages gene embeddings; the original GenePT averages 1536-dimensional GPT gene embeddings directly. Because the central claim is comparative, a weakened or nonstandard baseline is load-bearing. The concern is not merely about 'not matching the original paper' in a cosmetic sense: averaging after a learned nonlinear projection is a different model, and the baseline's quality directly determines the reported margin. The paper also lacks error bars and contains multiple internal contradictions (training-free vs. trained projection/head; 5120 vs. 4096; ranked sequence vs. expression multiplication/stacking; 'scInterpreter' naming in figures and text), which further erode confidence that the reported results correspond to the described method. I agree with the reader's REJECT verdict: as written, the evidence for SCREADER's superiority over GenePT is not reliable. If the authors provide a faithful GenePT comparison and reconcile the method description with the actual code, CONDITIONAL acceptance might be warranted, but the current manuscript does not support the central claim.","tokens_in":10564,"tokens_out":3217,"duration_ms":30985,"concrete_test":"Recompute the comparison using the published GenePT: build cell embeddings by expression-weighted averaging of the 1536-dimensional text-embedding-ada-002 gene embeddings (no learned up-projection), train the same classifier used in GenePT on the identical HUMAN-10k and MOUSE-13k train/test splits, and report accuracy, precision, recall, F1, and error bars across folds. If this faithful baseline matches or exceeds SCREADER's reported numbers, the claimed gains are artifacts; if SCREADER still wins, the authors should also release exact code and dimensions to resolve the 4096/5120 and ranked-sequence inconsistencies.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim — that SCREADER beats GenePT by large margins in accuracy, precision, recall, and F1 (Fig. 2) — depends entirely on the GenePT baseline being a faithful implementation of the published method. Section V (Training Workflow) describes the baseline as: 'we removed the frozen Llama model as the GenePT group, the two MLPs remained and were trained normally. The initialized cell embeddings are first passed through an up-projection MLP, increasing the dimensionality to 4096. Subsequently, the embeddings of each gene within the cell are averaged and then directly fed into a classification MLP.' This is not the GenePT method of Chen and Zou: published GenePT averages the 1536-dimensional GPT-embedding gene vectors (with expression weighting in the original), then applies a classifier; it does not learn a per-cell nonlinear up-projection before averaging. Averaging after a trainable nonlinear transform changes the feature geometry and adds substantial learnable capacity, so the baseline can be much weaker than the actual GenePT. The paper never reports the published GenePT's performance on these datasets, so the observed gaps cannot be attributed to Llama's 'common knowledge' as claimed in Section VI. Additional internal inconsistencies compound the problem: Section IV defines a ranked top-2048 sequence with positional encodings and claims 'training is unnecessary,' while Section V multiplies gene embeddings by expression values and stacks them; Section IV says projection to 5120 dimensions while Section V says 4096; and Figure 1/Section IV refer to 'scInterpreter' rather than SCREADER. These discrepancies make it impossible to confirm that the evaluated pipeline is the one described. Without a faithful GenePT baseline or an ablation that isolates the Llama component on equal footing, the central comparative claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SCREADER, a pipeline that embeds NCBI gene descriptions with GPT-3.5, builds cell representations from expression-ranked genes, projects them with an MLP, and passes them through a frozen Llama-13b with an instruction prompt. The authors evaluate cell-type annotation on two in-house datasets (HUMAN-10k and MOUSE-13k) against a baseline they call GenePT, reporting large gains in accuracy, precision, recall, and F1 (Figure 2) as well as UMAP visualizations. The core claim is that the frozen LLM contributes biological 'common knowledge' that improves cell representation.","tokens_in":10843,"tokens_out":7381,"duration_ms":65838,"significance":"If the reported results were established, the idea of using a frozen general-purpose LLM as an interpreter over text-derived gene embeddings would be a useful contribution to cross-species scRNA-seq analysis, and the paper is commendable for stating a clear problem and releasing code. However, the central empirical claim is currently not established: the GenePT baseline as described is not the published method, the method section contains internal contradictions about the input representation and training, and no variance or ablation information is reported. The contribution is therefore promising in principle but requires a full re-evaluation before it can be assessed.","major_comments":[{"comment":"The baseline described as GenePT is not the method of Chen and Zou (Ref. [16]). The manuscript's baseline first passes cell-initialized embeddings through a trainable five-layer MLP that up-projects them to 4096 dimensions, then averages the projected gene embeddings and applies a classification MLP. Published GenePT averages gene-level GPT-3.5 embeddings, with expression weighting, and then applies a classifier; it does not insert a learned nonlinear projection before the averaging operation. Averaging after a trained nonlinear transformation changes the feature geometry and adds considerable trainable capacity, so the large performance gaps in Figure 2 cannot be attributed to Llama-13b's 'common knowledge' as claimed in Section VI. A faithful implementation of GenePT must be compared before the headline result is supported.","section":"Section V, Training Workflow; Figure 2"},{"comment":"The method description is internally contradictory. Section IV defines the cell representation as a sequence of top-2048 gene embeddings concatenated with positional encodings and states that 'training is unnecessary,' while Section V instead says that gene embeddings are multiplied by their expression values and stacked, and the optimization objective in Section IV trains the projection MLP, the classification head, and the class-token embedding. In addition, Section IV says the projection is to 5120 dimensions to match Llama-13b, whereas Section V says the input is projected to 4096 dimensions. These contradictions leave the exact pipeline ambiguous and must be resolved for the experiments to be reproducible.","section":"Section IV (Cell-level Representation, Eq. (2)) vs. Section V (Cell Embedding Initialization, Training Workflow)"},{"comment":"No variance, sample-size, or ablation information is provided for the main quantitative claim. Figure 2 reports single bars for accuracy, precision, recall, and F1 without error bars or repeated-run statistics, and no ablation removes the Llama component while keeping the input representation identical. Consequently, even setting aside the baseline issue, the claim of a 'huge margin' over GenePT is not statistically supported.","section":"Section VI, Figure 2"}],"minor_comments":[{"comment":"The abstract uses 'interoperability' where 'interpretability' is presumably intended, and Section I contains the typo 'high-throughout data' instead of 'high-throughput data.'","section":"Abstract and Section I"},{"comment":"The caption and text refer to 'scInterpreter' rather than 'SCREADER,' which appears to be leftover from another paper and should be corrected.","section":"Section IV and Figure 1 caption"},{"comment":"The phrase '10-fold validation split ratio' is unclear; the authors should state explicitly whether 10-fold cross-validation or a single split was used, and the sentence fragment 'sFinally' should be fixed.","section":"Section V, Dataset Preparation and Training Workflow"},{"comment":"No accession numbers or preprocessing scripts are provided for HUMAN-10k and MOUSE-13k, and the golden labels are generated by the authors' own Seurat clustering and marker-gene annotation, so the datasets and labels are not independently verifiable.","section":"Section V, Dataset Preparation"},{"comment":"The confusion matrices have no axis labels or numeric values, which makes the discussion of specific cell types such as Parietal Endoderm difficult to verify.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The main quantitative evidence is compromised by the nonstandard GenePT baseline and by internal inconsistencies in the method description. I chose major_revision rather than reject because these problems are fixable within the scope of an experimental paper: rerun the comparison with a faithful GenePT implementation, add error bars and ablations, and clarify the input-construction and projection details. If the authors cannot close the gap under a faithful baseline, the paper should not be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nThe core idea here—initialize gene vectors from GPT embeddings of NCBI text, then pass a ranked gene sequence through a frozen Llama-13b with a trainable CLS head—is sensible, and the cross-species framing is fine. Credit where it's due: using a frozen LLM as an aggregator over text-derived gene embeddings is a cheap, plausible hybrid, and the training setup (10-fold split, fixed epochs, LR, batch size) is concrete.\n\nThe problems are in the evaluation and the write-up. The GenePT baseline in Section V is not GenePT. They remove Llama from their own pipeline but keep the up-projection MLP (1536 to 4096) and then average the projected gene embeddings. Published GenePT averages the 1536-d GPT embeddings directly and then classifies. Averaging after a learnable nonlinear projection is a different model; it can be weaker or stronger, but it isn't the published method. The paper never reports the actual GenePT numbers, so the large accuracy gains in Figure 2 cannot be attributed to Llama's \"common knowledge\" as claimed in Section VI.\n\nThe method description also contradicts itself. Section IV says \"training is unnecessary,\" then trains the projection MLP, classification head, and CLS embedding. Section IV projects to 5120 dimensions; Section V says 4096. Section IV defines a ranked top-2048 sequence with positional encodings; Section V multiplies gene embeddings by expression values and stacks them. And Figure 1 and Section IV refer to \"scInterpreter\" instead of SCREADER, with no citation. That last point needs to be checked—if SCREADER is a renamed version of prior work by these authors, that's a disclosure problem. As written, the internal inconsistencies make it impossible to confirm what was actually evaluated.\n\nNo error bars, no ablations, one baseline, two datasets. The confusion matrices and UMAPs are presented descriptively, but without quantitative support they don't carry the claim.\n\nMy recommendation: this is a plausible idea, but the evaluation doesn't support the headline result. I would not send it to peer review in its current form. If the authors fix the baseline, add error bars and ablations, and clarify the scInterpreter issue, it might deserve a round of refereeing. For now, desk reject with an invitation to resubmit after major revision.","headline":"Plausible hybrid LLM+gene-embedding idea, but the reported win over GenePT is not established because the baseline is a nonstandard reimplementation and the method description contradicts itself.","tokens_in":11440,"tokens_out":4163,"would_cite":false,"duration_ms":36609,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that sending expression-weighted gene embeddings through a frozen Llama-13b with an instruction prompt substantially improves cell-type annotation compared with the GenePT baseline on human and mouse data.","keywords":["large language models","single-cell RNA sequencing","cell type annotation","gene embedding","prompt learning","cross-species analysis","foundation model","gene expression"],"falsifier":"Re-run the same train/test splits using the original GenePT recipe—averaging the 1536-dimensional GPT embeddings without the up-projection MLP—and check whether SCREADER's accuracy, precision, recall, and F1 margins persist; if the margins collapse or reverse, the gain is an artifact of the modified baseline.","tokens_in":10345,"feed_emoji":"🧬","tokens_out":11752,"duration_ms":91665,"temperature":0.7,"pith_summary":"This paper proposes a hybrid pipeline, SCREADER, that builds a cell representation from GPT-3.5-generated gene embeddings plus expression rankings, then passes it through a frozen Llama-13b with an instruction prompt. The central claim is that adding this frozen LLM reader yields substantially higher accuracy, precision, recall, and F1 on cell-type annotation than using the same gene embeddings without the LLM, as the paper's comparison against GenePT shows. A sympathetic reader would care because, if true, general-purpose language models could supply biological knowledge that improves single-cell interpretation across species—including species with scarce training data—without training a new foundation model.","feed_headline":"Frozen Llama-13b reader outperforms GenePT in cell-type annotation","feed_subtitle":"SCREADER pairs GPT-embedded gene descriptions with a frozen Llama-13b reader to beat GenePT on two cell atlases.","key_machinery":"The load-bearing mechanism is the pairing of (1) gene-level embeddings produced by GPT-3.5 from functional gene descriptions and (2) a frozen Llama-13b that receives a concatenation of a text instruction, the projected cell-embedding sequence, and a class token, with only an MLP projection and a classification head trainable. The paper calls the overall pipeline 'LLM as Single-Cell RNA Data Reader' (SCREADER). It is the frozen LLM's contextual aggregation—what the paper terms the model's 'common knowledge'—that is claimed to supply the extra supervisory signal; no gene or cell foundation model is trained.","core_discovery":"SCREADER's central discovery is that a frozen Llama-13b, prompted with a task instruction and a class token, can act as an interpreter that turns a sequence of expression-weighted gene embeddings into a cell representation that separates cell types far more cleanly than the embeddings alone. The gene embeddings come from GPT-3.5 applied to functional descriptions from a public gene database; each cell is encoded by its top 2048 differentially expressed genes, with each gene's embedding scaled by its expression level. After an MLP projects the sequence to Llama's input size, the frozen LLM aggregates it, and a trainable classification head reads out the cell type from the class token. The paper reports that on HUMAN-10k (61 cell types) and MOUSE-13k (37 cell types), SCREADER outperforms its GenePT comparison across accuracy, precision, recall, and F1, with a larger margin on human data, which have a higher fraction of genes with gene-description coverage.","pith_inferences":["The paper leaves open whether the gain comes from the LLM's semantic knowledge or simply from the extra capacity of the learned projection layers; a perturbation test that shuffles gene descriptions before embedding would separate these two accounts.","Because the methods section describes a positional-encoding variant while the experiments section describes multiplying embeddings by expression values, testing both encoding schemes against a common GenePT baseline would clarify which cell-embedding construction is responsible for the reported gains.","The approach's reliance on textual gene summaries suggests a cheap extension: for genes missing database entries (about 16% of mouse genes), generating descriptions with a conversational LLM, as the paper already does for missing entries, could be applied systematically and evaluated for its effect on performance."],"forward_implications":["Using the same initial gene embeddings, adding a frozen Llama-13b raises accuracy, precision, recall, and F1 substantially on both HUMAN-10k and MOUSE-13k cell-type annotation.","Because only the projection MLP and classification head are trained, the pipeline avoids per-species foundation-model pretraining, which the paper argues could make it practical for species with limited single-cell data.","The improvement is larger on the human dataset (97.5% of genes with gene descriptions) than on the mouse dataset (83.6%), suggesting the method's benefit scales with coverage of textual gene knowledge.","The paper concludes that the same reader-style prompting could extend to multi-omics integration and rare-cell-type identification for precision medicine and developmental biology."],"supporting_citations":[{"why":"Defines the GenePT baseline that SCREADER is compared against; the paper argues SCREADER shares its conceptual framing but adds a frozen LLM reader.","marker":"[16]"},{"why":"Supplies the frozen Llama-13b model that acts as the LLM reader in the SCREADER pipeline.","marker":"[37]"},{"why":"Provides the preprocessing and clustering workflow used to construct the HUMAN-10k and MOUSE-13k datasets and their cell-type labels.","marker":"[38]"},{"why":"Supplies known cell-type-specific marker genes used to assign golden labels to the clusters.","marker":"[40]"},{"why":"Supplies a second marker-gene reference used in the manual cell-type annotation of clusters.","marker":"[41]"},{"why":"Provides the gene-description and taxonomy resource from which functional descriptions are drawn for gene-level embedding initialization.","marker":"[20]"}],"fun_headline_variants":["Frozen Llama-13b reader beats GenePT in cell typing","Prompted Llama-13b interprets single-cell data, beats GenePT","scReader: LLM prompt beats GenePT on cell annotation","Frozen Llama-13b reader outperforms GenePT in cell annotation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage rests on comparing SCREADER to a GenePT baseline that the paper reimplements with an added MLP and an averaging step, so if that reimplementation is not faithful to the original GenePT, the large gap might reflect a weakened baseline rather than the LLM's understanding.","fun_headline_variants_meta":{"raw":{"variants":["Frozen Llama-13b reader beats GenePT in cell typing","Prompted Llama-13b interprets single-cell data, beats GenePT","scReader: LLM prompt beats GenePT on cell annotation","Frozen Llama-13b reader outperforms GenePT in cell annotation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2645,"prompt_tokens":1005,"completion_tokens":1640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1561}},"tokens_in":621,"tokens_out":1640,"duration_ms":11311,"temperature":1.0,"reasoning_tokens":1561,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:58:42.906544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same train/test splits using the original GenePT recipe—averaging the 1536-dimensional GPT embeddings without the up-projection MLP—and check whether SCREADER's accuracy, precision, recall, and F1 margins persist; if the margins collapse or reverse, the gain is an artifact of the modified baseline.","supporting_citations":[{"cited_title":"Genept: A simple but hard-to-beat foundation model for genes and cells built from chatgpt,","cited_arxiv_id":null,"evidence_quote":"Defines the GenePT baseline that SCREADER is compared against; the paper argues SCREADER shares its conceptual framing but adds a frozen LLM reader."},{"cited_title":"Comprehensive integration of single-cell data,","cited_arxiv_id":null,"evidence_quote":"Provides the preprocessing and clustering workflow used to construct the HUMAN-10k and MOUSE-13k datasets and their cell-type labels."},{"cited_title":"Panglaodb: a web server for exploration of mouse and human single-cell rna sequencing data,","cited_arxiv_id":null,"evidence_quote":"Supplies known cell-type-specific marker genes used to assign golden labels to the clusters."},{"cited_title":"Cellmarker: a manually curated resource of cell markers in human and mouse,","cited_arxiv_id":null,"evidence_quote":"Supplies a second marker-gene reference used in the manual cell-type annotation of clusters."}],"review_version":1}