{"id":"e8dd011f-dca1-4796-8313-a7f1f4fb2fee","arxiv_id":"2601.12805","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SciHorizon-GENE is a 540K-question benchmark showing that LLMs systematically fail on low-attention genes, hallucinate when annotations are absent, and give incomplete multi-answer responses.","lead":"This paper builds a large gene-focused benchmark that tests how well AI language models can turn gene facts into functional understanding. It finds that models perform unevenly and often make up answers for poorly studied genes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Absence-of-annotation ground truth is the load-bearing assumption: if 'no right answer' cases actually have known answers, the hallucination metric and the sparse-evidence reliability gap collapse.","rationale":"In good faith, the paper's contribution is a large, automatically constructed benchmark and a behavioral evaluation across 27 models. The four perspectives are reasonable and the multi-answer and literature-context analyses are less dependent on this concern. However, the central conclusion that LLM reliability degrades in 'sparse evidence settings' depends directly on how 'no right answer' is defined. The reader's weakest_assumption correctly identified construct validity, and specifically the missing-value-as-negative case; my concern narrows to that single issue because it is the one most likely to change the headline finding if wrong. The LOC-prefix proxy is also a confound, but it is secondary: even if LOC genes are mostly uncharacterized loci, the research-attention comparison still reveals systematic performance differences, though the interpretation as 'attention' would weaken. The representativeness of the 2,710-question subset is important but testable only with released data; the absence-of-annotation issue is more fundamental because it affects the metric's validity. I do not think this requires rejection: the benchmark can be salvaged by re-annotating a validation sample with current databases and human judgment, and by reporting the fraction of 'no right answer' cases that are confirmed absent. Since the reader already recommends CONDITIONAL, my read does not shift the verdict; it strengthens the condition.","tokens_in":19729,"tokens_out":3407,"duration_ms":38483,"concrete_test":"Sample roughly 200 genes from the 'no right answer' subset (e.g., genes with no GO annotations in the benchmark). Independently query current NCBI Gene, the latest Gene Ontology release, and PubMed for any authoritative functional annotation or synonym. Have two biology annotators judge whether a known answer exists. If more than a small fraction (e.g., >10%) have known answers, the hallucination metric is mislabeling correct model responses, and the sparse-evidence conclusion must be re-analyzed with a curated ground-truth set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claim—that reliability 'predictably decreases for … sparse evidence settings'—rests on the hallucination metric defined in Section 3.4 and used in Section 5.2. There, a missing attribute value (|a_ij|=0) in NCBI/GO is taken as the ground-truth answer 'no right answer', and any substantive model output is scored as a hallucination. The load-bearing assumption is that database absence equals biological absence. For poorly annotated genes, missing GO terms, synonyms, and chromosome mappings often reflect annotation lag, curation gaps, or incomplete database integration rather than true nonexistence. If an LLM supplies a correct answer from knowledge not present in the database snapshot, it is falsely penalized as hallucinating. That would invert the metric's meaning and make 'hallucination resistance' a measure of compliance with an incomplete reference, not faithfulness. The paper provides no evidence on snapshot currency, no independent check that the missing attributes were genuinely absent from authoritative sources at evaluation time, and no human baseline. Appendix G indicates the benchmark data are encrypted and scheduled for later release, so the core analysis is not currently reproducible. Without validating this assumption, the strongest claim about a systematic reliability gap in sparse-evidence settings is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SciHorizon-Gene, a large-scale benchmark for evaluating LLMs' gene-to-function reasoning. It constructs a gene-centric knowledge base from NCBI Gene, Gene Ontology, and PubMed, generating over 540K questions across three biological scenarios (nomenclature, genomic knowledge, and literature-influenced tasks). The benchmark defines four evaluation perspectives: research attention sensitivity, hallucination tendency, knowledge completeness, and literature influence. The authors evaluate 27 general-purpose and biomedical LLMs and report a consistent reliability gap: model performance declines for low-attention genes, sparse-evidence settings, multi-answer queries, and shifts in reference context. The paper claims the benchmark provides a principled, scalable, and automatically computable foundation for characterizing gene-level understanding in LLMs.","tokens_in":20028,"tokens_out":5148,"duration_ms":57770,"significance":"If the construct-validity issues are resolved, this benchmark would fill a real gap: existing biomedical LLM benchmarks mostly operate at document or clinical-questions level, not at the gene-centric granularity needed for cell-atlas interpretation and functional annotation workflows. The paper's strengths include its large scale (190K genes, 540K questions), automatic metrics inspired by CAFA, cyclic permutation to mitigate positional bias, and the breadth of models evaluated. The reported finding that biomedical-specialized models do not consistently outperform general-purpose LLMs is practically relevant. However, the benchmark's validity as a measure of hallucination and research-attention sensitivity is currently undermined by the absence-of-annotation assumption, the LOC-prefix proxy, missing human/random baselines, and lack of public data. These issues must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The hallucination metric treats missing attribute values (|a_ij|=0) as the ground truth 'no right answer' and scores any substantive output as hallucination. This equates database absence with biological absence. The paper provides no snapshot currency analysis, no manual audit of missing-value cases, and no human/random baseline. If an LLM supplies a true fact not present in the NCBI/GO snapshot, it is falsely penalized, which would invert the metric's meaning. The 'sparse-evidence reliability gap' claimed in Section 5.2 therefore rests on an unvalidated assumption. Please add an independent validation: cross-check a sample of missing attributes against current authoritative resources, and include human/random baselines for these questions.","section":"Section 3.4, Section 5.2"},{"comment":"Using the LOC gene-symbol prefix as the proxy for low research attention is confounded. LOC-designated genes are often uncharacterized loci and differ systematically from non-LOC genes in gene type, annotation completeness, and name meaningfulness. Table 5 shows large low-attention vs high-attention gaps (e.g., GPT-5 gene full name: 36.19 vs 99.17), but this could reflect lexical cues or annotation status rather than 'research attention.' The authors acknowledge the lexical-cue explanation in Section 5.1 but do not control for it. A matched design (e.g., non-LOC genes with similarly few PubMed references, or a continuous attention variable) is needed to separate attention from confounds.","section":"Section 3.4, Figure 3, Table 5"},{"comment":"The 'representative subset' of 2,710 questions is asserted but not demonstrated. The paper states only that a subset was sampled to cover all scenarios and perspectives; no stratification details, randomization procedure, or per-task sample sizes are given. Table 3 reports total benchmark counts but not the counts for the sampled subset. Because all model rankings and headline claims in Tables 2, 5, and 6 are computed on this subset, the lack of a precise sampling protocol prevents reproduction and verification of the reported results. Please provide the exact sampling scheme, seed, and a composition table for the 2,710-question subset.","section":"Section 4.2, Appendix B"},{"comment":"The paper claims 'fully reproducible evaluation without manual scoring' (Section 1) and 'completely automated metrics,' but Appendix G states that the benchmark data are encrypted and 'will be released' later on the SciHorizon platform. As of submission, the core benchmark and evaluation data are not available, so no external reproducibility check is possible. Please release the benchmark data, evaluation scripts, and model outputs in a persistent archive with clear access instructions, at least upon publication.","section":"Appendix G, Section 1"},{"comment":"There is a circularity concern: the four evaluation perspectives are motivated by the same preliminary model behaviors later reported as findings (Figure 1 and Section 3.4, then Section 5). The benchmark is thus partly shaped by the phenomena it claims to discover. This does not invalidate the benchmark as a diagnostic tool, but the paper should clarify whether the question sets and metrics were frozen before the preliminary observations, and should state this temporal ordering explicitly. An independent confirmation on held-out questions or models would strengthen the generalizability claim.","section":"Section 1, Figure 1, Section 3.4"}],"minor_comments":[{"comment":"Typos: 'Answer Completness' should be 'Answer Completeness' in Table 3, and 'Hallucination Resistence' should be 'Hallucination Resistance' in Table 5.","section":"Table 3, Table 5"},{"comment":"The expression-pattern score uses alpha and beta with equal weights (0.5) but no sensitivity analysis or justification is provided. Since alpha and beta are free parameters, please report whether the main results are robust to alternative choices (e.g., alpha=0.3/0.7).","section":"Appendix D, Eq. (4)"},{"comment":"Many entries in Table 6 are em-dashes (e.g., Gemma 3-27B, PMC-LLaMA, MedAlpaca). Please clarify whether these indicate unparseable outputs, tasks not run, or formatting failures, and specify how missing values are handled in the overall score calculations.","section":"Table 6"},{"comment":"The functional summary score is an unweighted arithmetic mean of ROUGE-L, BERTScore, and fluency. ROUGE-L and BERTScore are both lexical-semantic overlap measures and may be highly correlated; consider reporting them separately or using a weighted/combined metric with rationale.","section":"Appendix D, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The benchmark addresses an important gap and the evaluation framework is largely transparent, but the central claims currently rest on two unvalidated proxies (database absence as 'no right answer' and LOC prefix as low attention) and on a 2,710-question subset whose sampling is undocumented. The data are also not yet released. These issues are fixable within the manuscript's scope: adding validation studies, human/random baselines, a clear sampling protocol, and public release would substantially strengthen the paper. I do not think rejection is warranted, but the current version overclaims the conclusiveness of the reliability-gap findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about SciHorizon-GENE before reading it. First, it is a real contribution: a fully automated, gene-centric benchmark built from NCBI Gene, GO, and PubMed covering 190K+ genes and 540K+ questions, with 27 models scored on four behavioral axes. That scale alone makes it useful for anyone studying LLMs in genomics. Second, the paper's headline reliability claim — that failures are \"predictable\" and get worse in sparse-evidence settings — rests on an assumption I don't think is validated: that a missing value in a database snapshot means \"no right answer\" for the model.\n\nWhat the paper does well: the evaluation protocol is carefully specified. Cyclic permutation for multiple-choice, macro F1 for multi-answer, CAFA-style closure scoring for GO, ROUGE-L/BERTScore/fluency for summaries, and Pydantic-validated outputs. The finding that completeness is recall-limited (precision high, recall low across all models) is a clean, useful result. So is the observation that supplying PubMed abstracts does not improve GO prediction and can actually increase hallucinated terms. The 27-model comparison, including biomedical specialists that don't consistently beat general models, is useful for model selection. These are good, reproducible-in-principle results.\n\nThe soft spots are real, though. The biggest is the hallucination metric. Section 3.4 defines attributes with no recorded value as \"no right answer,\" and any substantive output is scored a hallucination. But absence in NCBI/GO often reflects annotation lag or curation gaps, not biological absence. If a model supplies a correct gene symbol or chromosome location that happens to be missing from the snapshot, it is falsely penalized. The paper provides no snapshot currency check, no independent verification that the missing attributes were genuinely absent at evaluation time, and no human/random baseline. Without that, the \"hallucination resistance\" numbers and the sparse-evidence reliability gap are not yet established as measures of faithfulness.\n\nSecond, the LOC-prefix proxy for low research attention is reasonable but confounded: LOC genes are mostly uncharacterized loci, so the effect could be driven by naming conventions or annotation quality, not attention. The paper's own Figure 3 shows the distribution difference, but doesn't disaggregate.\n\nThird, the four evaluation perspectives were motivated by the same preliminary observations that the paper then reports as findings (Section 1, Figure 1). That's a circularity burden, though the underlying annotations are external, so it's partial. Fourth, the 2,710-question subsample is asserted to be representative without demonstration, and Appendix G says the data are encrypted and planned for later release, so the core analysis is not currently reproducible.\n\nWho it's for: researchers building or choosing LLMs for gene-level analysis — this gives a broad, if imperfect, map. It deserves a serious referee, but a conditional one: the authors need to validate the absence-as-ground-truth assumption, add baselines, and release the artifacts. As it stands, I'd treat the benchmark as promising infrastructure whose main empirical claim is not yet proven.\n\nRecommendation: send to peer review, with a request for major revision on the validity checks.","headline":"A genuinely useful automated gene-centric benchmark, but the core hallucination metric depends on treating missing database annotations as 'no right answer' — an assumption that needs validation before the reliability-gap claim can stand.","tokens_in":20491,"tokens_out":3207,"would_cite":true,"duration_ms":35769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM gene-to-function reasoning is systematically uneven: reliability drops predictably for low-attention genes, sparse evidence, multi-answer queries, and shifts in reference context, and it introduces a 540K-question","keywords":["large language models","benchmark","gene function","hallucination","knowledge completeness","research attention","gene ontology","functional genomics"],"falsifier":"Take a random sample of genes whose attributes are missing in the benchmark, manually curate the true answers from the literature, and recompute hallucination rates; if a substantial fraction of those genes are actually annotated in the literature, the reported hallucination numbers are not measuring what they claim.","tokens_in":19649,"feed_emoji":"🧬","tokens_out":4328,"duration_ms":44576,"temperature":0.7,"pith_summary":"This paper builds a 540K-question benchmark covering 190K human genes to test whether large language models can reliably move from gene-level knowledge to functional understanding. The benchmark evaluates models along four behavioral axes: sensitivity to research attention, tendency to hallucinate when no grounded answer exists, completeness on multi-answer questions, and use of provided literature context. Across 27 models, the authors find a consistent reliability gap: performance degrades predictably for low-attention genes, sparse evidence, multi-answer queries, and shifts in reference context. The paper argues this makes gene-level failure modes systematic and measurable, and that current biomedical-specialised models do not consistently outperform general-purpose ones. If right, the benchmark gives a scalable, automatic way to characterise a core skill for AI-assisted cell-atlas interpretation.","feed_headline":"LLM gene knowledge drops sharply for neglected genes","feed_subtitle":"Benchmark shows reliability falls predictably for obscure genes and sparse evidence.","key_machinery":"The benchmark's core mechanism is a gene-centric knowledge integration: each of 190K human genes is linked to structured attributes (name, synonyms, chromosome location, gene type, related proteins, expression patterns, GO terms, functional summary) and a set of PubMed references. Question cardinality determines format—single-answer, multi-answer, generative, or 'no right answer' for missing attributes. Four evaluation perspectives operationalise the paper's behavioral hypotheses: research attention is proxied by the LOC gene-symbol prefix (low PubMed counts), hallucination is measured by selection of any substantive option when the gold label is 'no right answer', completeness by macro-aver","core_discovery":"SciHorizon-GENE is a gene-centric evaluation built from NCBI Gene, Gene Ontology, and PubMed. From curated attributes it auto-generates single-answer, multi-answer, and open-ended questions, and labels attributes with no recorded value as having 'no right answer' to test hallucination. Evaluating 27 open and closed models, the paper reports substantial heterogeneity and a systematic reliability gap: accuracy falls for LOC-prefixed (low-attention) genes, models often select a substantive option when none is correct, recall lags precision on multi-answer tasks, and supplying PubMed abstracts does not consistently improve GO prediction or functional summaries and can increase hallucination. The","pith_inferences":["Editorial inference: The hallucination metric treats a missing database value as biological absence; a manual audit of a sample of missing-attribute genes could reveal how much of the reported 'hallucination' is actually incomplete annotation.","Editorial inference: The LOC-prefix proxy may confound research attention with gene class (mostly uncharacterized loci); matching genes on PubMed reference counts would isolate whether attention or gene type drives the performance gap.","Editorial inference: The consistent recall lag on multi-answer tasks suggests a calibration or decoding issue; exploring retrieval-augmented generation or confidence-thresholded sampling for these tasks could be a testable extension.","Editorial inference: Because the gap is systematic, accuracy on low-attention genes could serve as a probe for pretraining corpus exposure; a simple extension is to plot accuracy against continuous PubMed counts rather than binary LOC status."],"forward_implications":["If the reliability gap is real, model selection for gene interpretation should be guided by gene-attention profiles rather than overall biomedical QA scores.","Biomedical pretraining is not sufficient: specialised models may need structured gene-level supervision to close the gap on low-attention genes and multi-answer tasks.","Supplying abstracts as context should not be assumed to help; systems that ground outputs in controlled vocabularies need explicit mechanisms to map narrative text to GO terms.","The benchmark enables automatic and reproducible evaluation of gene-centric reliability without manual scoring, allowing broad model comparisons.","The predictability of failures suggests targeted data augmentation or prompting interventions could improve handling of under-studied genes."],"fun_headline_variants":["LLMs stumble on obscure genes, bio benchmark reveals","Gene benchmark finds LLMs hallucinate when evidence is sparse","Adding literature fails to boost LLM gene predictions","LLM gene reasoning degrades sharply for low-attention genes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a missing value in NCBI/GO means there is no correct answer, so 'no right answer' is a true ground truth rather than an artifact of incomplete annotation.","fun_headline_variants_meta":{"raw":{"variants":["LLMs stumble on obscure genes, bio benchmark reveals","Gene benchmark finds LLMs hallucinate when evidence is sparse","Adding literature fails to boost LLM gene predictions","LLM gene reasoning degrades sharply for low-attention genes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2071,"prompt_tokens":742,"completion_tokens":1329,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1264}},"tokens_in":486,"tokens_out":1329,"duration_ms":11591,"temperature":1.0,"reasoning_tokens":1264,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:39:39.062373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of genes whose attributes are missing in the benchmark, manually curate the true answers from the literature, and recompute hallucination rates; if a substantial fraction of those genes are actually annotated in the literature, the reported hallucination numbers are not measuring what they claim.","supporting_citations":[],"review_version":2}