{"id":"719c0c9b-a7c7-4a7f-81de-94b9162cec5f","arxiv_id":"2608.00099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A zero-shot LLM ranking framework assigns GO terms to user-defined BioDomains, outperforming REVIGO and SapBERT on AD and FXS benchmarks.","lead":"Researchers show an AI language model can sort gene-ontology terms into broad biological theme buckets with high accuracy, beating older tools on two disease datasets. If it holds up, this gives biologists a fast, customizable way to summarize gene-enrichment results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth reliability is the load-bearing premise: AD labels may be memorized by the LLM and FXS labels are unreleased author annotations, so reported margins may overstate true zero-shot performance.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue I find: the evaluation treats two potentially unreliable ground-truth sets as authoritative. The AD labels come from a 2024 paper that is almost certainly in the training data of DeepSeek-reasoner, and the FXS labels are author-generated and unreleased. If either issue materializes, the quantitative core of the paper—the P@1/R@1 improvements in Table 2 and the ARI/NMI improvements over REVIGO in Figure 3—would not support the central claim of zero-shot expert-level categorization. I considered alternative concerns: the REVIGO comparison may be unfair because LlmBdc receives the BioDomain names while REVIGO does not, and the best configuration was selected on the same test datasets. However, these are secondary: a REVIGO comparison could be re-run under fairer conditions, and selection bias could be addressed with a held-out disease. The ground-truth issue is more fundamental because if the labels are invalid or memorized, no amount of methodological adjustment would salvage the quantitative evidence. The reader's CONDITIONAL verdict appropriately recognizes that the concern is addressable through data release and additional checks. My analysis does not move that verdict, so I mark it UNCHANGED. I also note the paper's own Limitations section acknowledges model-specific biases and limited knowledge of rare terms, but does not discuss the more acute risk of benchmark contamination—a notable omission given the 2024 source of the AD labels.","tokens_in":14853,"tokens_out":12164,"duration_ms":127196,"concrete_test":"Release both label sets and run two checks: (1) a memorization probe on AD—divide GO terms into those that appear verbatim in the Cary et al. (2024) supplement and a matched control set not present in that paper; if LlmBdc accuracy is significantly higher on the verbatim set, contamination is confirmed. (2) Independent re-annotation of a random 100-term subset of the FXS dataset by a second biologist, measuring inter-annotator agreement (e.g., Cohen's kappa). If the memorization gap is negligible and kappa is high (>0.8), the reported margins are credible; otherwise the zero-shot claim is overstated for AD, FXS, or both.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests entirely on two expert-annotated benchmarks, described in Section 4.1. The AD dataset comprises 7,120 GO terms labeled into 19 BioDomains according to Cary et al. (2024), a publication that almost certainly appears in DeepSeek-reasoner's training corpus. If the LLM memorized these GO-term-to-BioDomain mappings, the AD results—P@1 83.9%, ARI 73.3%—reflect retrieval, not zero-shot reasoning. The FXS benchmark is the only contamination-free check: its 516 labels were manually produced by the authors and are not released. However, no inter-annotator agreement or annotation protocol is reported, so the reliability of this dataset is unverified; if the annotators' scheme was idiosyncratic, the FXS figures (P@1 78.4%, ARI 66.6%) would not generalize. No contamination test or label release is mentioned anywhere in the manuscript. Since every quantitative claim in Sections 4.2 and 4.3—including the headline REVIGO ARI improvements—depends on these two label sets being both accurate and unseen by the model, this is the single most load-bearing assumption of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LlmBdc, a training-free zero-shot framework that assigns GO terms to user-defined biological domains (BioDomains) by prompting an LLM to return a ranked list of candidate domains. The framework is evaluated on two expert-annotated benchmarks: Alzheimer's disease (7,120 GO terms, 19 domains, from Cary et al. 2024) and Fragile X syndrome (516 GO terms, 21 author-annotated domains). Against six baselines, LlmBdc with DeepSeek-reasoner reports Precision@1 of 83.9% (AD) and 78.4% (FXS) and Recall@2 of 93.6% and 93.1%, respectively. Against REVIGO, the paper reports ARI improvements from 9.7% to 73.3% (AD) and from 15.7% to 66.6% (FXS), with full coverage. Ablations examine temperature, the 'Unknown' label, and prompt content. The central claim is that an LLM can reproduce expert-level GO-to-BioDomain categorization without fine-tuning, using only term names and definitions.","tokens_in":15137,"tokens_out":5817,"duration_ms":69289,"significance":"If the results hold, the framework is practically valuable: it offers a context-aware, customizable, and reproducible alternative to static GO summarization tools, with no training required and code publicly available. The paper's strengths are its clear ranking formulation, evaluation across multiple LLMs, ablation of prompt factors, and use of a Cauchy combination test for domain-level significance. The main weakness is that the evaluation's validity rests on two ground-truth label sets whose reliability and non-contamination are not established. The AD labels are from a 2024 publication likely present in the training data of the deployed LLMs, and the FXS labels are unreleased author-generated annotations without an annotation protocol. The REVIGO comparison is partly mechanical because REVIGO drops a large fraction of terms. These issues are fixable with additional experiments and data release, so the central idea remains defensible pending those checks.","major_comments":[{"comment":"The AD ground truth comes from Cary et al. (2024), a publication released before DeepSeek-reasoner and ChatGPT-4o-mini were deployed. No contamination check is reported. Since the prompt supplies the ground-truth BioDomain names as candidates, the reported P@1=83.9% and ARI=73.3% could reflect memorized retrieval rather than zero-shot reasoning. I do not claim this is an internal inconsistency, but it is a correctness risk on the central claim. Please add a contamination audit: e.g., evaluate on GO terms added after the model cutoff, run a label-renaming experiment where BioDomain names are paraphrased, and disclose model snapshot/cutoff information.","section":"§4.1, Tables 2 and 5"},{"comment":"The FXS benchmark is the only non-published ground truth, but it was generated by the authors and is not released. No annotation protocol, number of annotators, or inter-annotator agreement is reported. Consequently, the FXS results (P@1=78.4%, ARI=66.6%) are unverifiable and cannot be compared in reliability to an independent expert benchmark. Please publish the label file, describe the annotation manual and agreement statistics, or replace the set with independent third-party annotations. This is load-bearing for the generalizability claim across heterogeneous disease contexts.","section":"§4.1, FXS dataset"},{"comment":"The REVIGO comparison is not apples-to-apples. REVIGO covers only 64% of AD terms and 37% of FXS terms, while LlmBdc assigns all terms. The reported ARI and NMI are therefore computed over different term sets, and the coverage difference is entangled with clustering quality. It is also not stated how REVIGO's unlabeled clusters are aligned to the BioDomain ground truth before computing ARI/NMI. Please recompute all metrics on the common subset of covered terms, or report the coverage gap as a separate axis, before claiming headline ARI improvements over REVIGO.","section":"§4.3, Fig. 3"}],"minor_comments":[{"comment":"The name is spelled inconsistently: 'LLMBDC' appears in the abstract while 'LlmBdc' is used throughout the body and Figure 1. Please standardize.","section":"Abstract and title"},{"comment":"The text cites 'Gregory et al. [3]' for the AD dataset, but reference [3] is Gregory A. Cary et al. The citation should read 'Cary et al.'.","section":"Section 1 and reference [3]"},{"comment":"The claimed 'closely match' between expert and LlmBdc domain ranks is overstated: Autophagy has truth rank 19 and LlmBdc rank 6, with a truth p-value of 1.00. Even one such large mismatch deserves explicit discussion in the text.","section":"Table 3"},{"comment":"The phrase 'adaptive prompt generation' is not strictly accurate, since the prompt template appears fixed per experiment; 'template-based prompt construction' would better describe Eq. (2).","section":"Section 3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the paper is well suited to a bioinformatics/ML audience, but the benchmark contamination risk and the unreleased FXS labels are essential to resolve. If the authors can provide a credible contamination analysis and release or replace the FXS ground truth, I would be willing to accept a revised version. The REVIGO comparison also needs to be recomputed on a common term set before the main quantitative claim is credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives a clean, practical method: use an LLM to rank user-defined BioDomains for each GO term, zero-shot, no training. That task framing is genuinely new, and the reported Precision@1 / Recall@1 numbers on AD and FXS are much higher than the SapBERT baseline. The ablations are honest — they found that adding GO structure, gene sets, or disease descriptions actually hurts, so the recommended prompt is just term name + definition. Code is promised at an anonymous link. This is a useful tool for functional genomics summarization, likely worth adopting once validated.\n\nThe soft spot is exactly where the reader put it: the benchmark labels. The AD ground truth comes from Cary et al. 2024, which almost certainly appears in DeepSeek-reasoner's training corpus. The paper does not check for contamination, and the P@1 of 83.9% on AD could be partly retrieval. The FXS dataset is the cleaner test because it is novel and unreleased, but it was created by the authors themselves, with no inter-annotator agreement or annotation protocol reported, and the labels are not released. So the FXS numbers (P@1 78.4%, ARI 66.6%) are the real evidence, and they are still impressive — but we cannot verify them independently.\n\nTwo smaller issues. First, the REVIGO comparison is not fair: REVIGO clusters by semantic similarity, it does not assign to user-defined BioDomains. Comparing ARI between REVIGO's clusters and LlmBdc's domain assignments conflates clustering with assignment and overstates the gap. Second, the recommended configuration (DeepSeek, temperature 0.0-0.1, no Unknown label) was selected on the same two datasets used for the headline results, so there is a selection-on-test component. Also, the abstract mentions 'confidence scoring' and 'context-aware,' but the method does not output confidence scores and the best prompt is not context-rich. Those are wording issues, not math issues.\n\nThe central idea is sound, and the math is not circular — it is LLM-based ranking plus standard evaluation metrics. The paper deserves a serious referee, but it needs a contamination check, a released or independently annotated FXS label set, and a more carefully scoped comparison against REVIGO. If those are addressed, the claims could hold; as is, the headline margins are probably optimistic but the method likely works. I would send it to peer review.","headline":"Useful zero-shot GO-to-domain assignment with strong reported numbers, but the benchmarks are the weak link — the AD labels may be memorized and the FXS labels are unreleased, so the margins should be read with caution.","tokens_in":700,"tokens_out":828,"would_cite":false,"duration_ms":48994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free LLM ranking prompt assigns Gene Ontology terms to user-defined biological domains, reaching 83.9% precision and ARI 0.73 against expert labels.","keywords":["Gene Ontology clustering","biological domains","zero-shot LLM ranking","GO enrichment summarization","Alzheimer's disease","Fragile X syndrome","Cauchy combination test","LlmBdc"],"falsifier":"Generate fresh expert BioDomain labels for a set of GO terms after the LLM's training cutoff (or from an unpublished curation process), run LlmBdc with the recommended configuration, and compare Precision@1 and ARI. If performance drops toward the SapBERT baseline or ARI falls below roughly 0.5, the headline gains are largely an artifact of training-data overlap rather than zero-shot reasoning.","tokens_in":14731,"feed_emoji":"🧬","tokens_out":5549,"duration_ms":57467,"temperature":0.7,"pith_summary":"LlmBdc is a training-free framework that turns Gene Ontology (GO) term clustering into a zero-shot ranking task: a large language model is prompted to rank a user-supplied list of biological domains for each GO term, and the top-ranked domain becomes the cluster label. The paper's central claim is that this simple procedure reproduces expert manual curation. On 7,120 Alzheimer's-related GO terms it reports 83.9% precision at rank 1 and a 73.3% adjusted Rand index against expert labels, versus 9.7% for REVIGO, with analogous gains on 516 Fragile X syndrome terms. The authors argue this makes scalable, context-aware, and reproducible summarization of GO enrichment results possible without training or manual annotation. The best configuration uses only the term name and definition, a low decoding temperature, and no 'unknown' label.","feed_headline":"Zero-shot LLM maps GO terms to biological domains at 84% precision","feed_subtitle":"Training-free ranking matches expert curation on Alzheimer's and Fragile X, beating REVIGO and SapBERT.","key_machinery":"The load-bearing mechanism is the zero-shot ranking prompt: a template that presents the GO term name and definition as the query, lists the candidate BioDomains, and asks the LLM to return a top-5 ranked list, which is parsed into cluster labels. This replaces fixed similarity metrics, gene-overlap measures, and static GO hierarchy mappings with the LLM's pretrained semantic knowledge. The Cauchy combination test then aggregates per-term enrichment p-values into a single domain-level p-value, allowing statistical significance to be reported per cluster rather than per term.","core_discovery":"LlmBdc casts GO-to-BioDomain annotation as a semantic ranking problem and solves it with a frozen LLM. For each GO term, the model outputs the five most appropriate domains from the user's list; the top pick is the assignment. Against expert-annotated ground truth, the paper reports Precision@1 of 83.9% (AD) and 78.4% (FXS), and adjusted Rand index of 0.73 and 0.67, clearly above all baselines including SapBERT and REVIGO. Domain-level p-values computed with the Cauchy combination test confirm that the LLM-produced clusters retain statistically significant enrichment signals. Ablation studies show that adding GO structure, gene-set similarity information, or disease descriptions to the promp","pith_inferences":["A fair test of whether the model is truly reasoning about biology would use expert labels created after the LLM's training cutoff; the current AD labels come from a 2024 publication the model may have memorized.","The same ranking approach could be applied to other controlled vocabularies, such as disease ontologies or pathway databases, wherever a user can define target categories.","One could auto-generate the BioDomain list from a study's abstract or gene-set context, making the pipeline fully automatic from gene list to domain summary.","For rare or newly coined GO terms, consulting a knowledge graph or pathway database before prompting could patch the identified weakness of limited LLM knowledge."],"forward_implications":["GO enrichment output can be condensed into user-defined biological domains without training data or manual curation, at a scale that manual annotation cannot match.","Domain-level enrichment p-values computed from LLM clusters remain statistically significant, so interpretation can proceed at the domain level.","The method transfers across diseases with different domain lists, indicating it is not tied to a particular ontology or annotation scheme.","Because only term name and definition are needed, the framework keeps working as GO terms are added, refined, or removed.","Backend model choice is a real trade-off: reasoning-oriented LLMs give the best accuracy but higher cost and latency."],"fun_headline_variants":["Zero-shot LLM maps GO terms to BioDomains with 84% precision","Training-free LLM outperforms REVIGO and SapBERT on GO clustering","LLMBDC: frozen LLM clusters gene ontology into biology domains","Gene ontology clustering made context-aware by zero-shot LLM","LLM groups GO terms into BioDomains, beating standard tools"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The results stand on the assumption that the two expert label sets are reliable ground truth: the FXS labels are unreleased and were made by the authors, and the AD labels come from a 2024 publication that the evaluated LLM may have seen during pretraining; if either label set is unrepresentative or memorized, the reported margins overstate real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM maps GO terms to BioDomains with 84% precision","Training-free LLM outperforms REVIGO and SapBERT on GO clustering","LLMBDC: frozen LLM clusters gene ontology into biology domains","Gene ontology clustering made context-aware by zero-shot LLM","LLM groups GO terms into BioDomains, beating standard tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3218,"prompt_tokens":876,"completion_tokens":2342,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2246}},"tokens_in":620,"tokens_out":2342,"duration_ms":18147,"temperature":1.0,"reasoning_tokens":2246,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:17:04.887398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate fresh expert BioDomain labels for a set of GO terms after the LLM's training cutoff (or from an unpublished curation process), run LlmBdc with the recommended configuration, and compare Precision@1 and ARI. If performance drops toward the SapBERT baseline or ARI falls below roughly 0.5, the headline gains are largely an artifact of training-data overlap rather than zero-shot reasoning.","supporting_citations":[],"review_version":1}