{"id":"81ced998-bb86-4a73-ac6a-072fe60a5ed5","arxiv_id":"2412.18154","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage LLM pipeline that selects key sentences from gene literature via GO annotations and fine-tunes Gemma-7B to generate gene summaries, reporting large ROUGE gains that may be inflated by training/evaluation overlap.","lead":"This paper builds GeneSUM, a two-stage system that filters gene-related papers with statistical signature terms and then fine-tunes a large language model to write short gene summaries. It reports large ROUGE improvements over six baselines on 8,887 human genes, but the evaluation may be circular because the model was trained on the same summaries it is scored against.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out gene split is described; the headline ROUGE gains (esp. ROUGE-2 0.1856 vs 0.0467) may reflect memorization of Entrez reference summaries used for fine-tuning.","rationale":"The reader's weakest assumption and my concern coincide: the evaluation is circular unless a held-out split exists. The strongest claim is purely empirical, and the paper's own text references only a training dataset. This is not a disagreement with consensus; it is an internal gap between the described protocol and the claimed generalization. All other issues (no error bars, no significance tests, no artifacts) compound the problem but would be secondary if a proper split existed and reproduced the gains. Therefore no change to the reader's REJECT verdict; the paper needs a described and ideally released gene-level split plus variance estimates before the central claim can be assessed.","tokens_in":10374,"tokens_out":2600,"duration_ms":24122,"concrete_test":"Partition the 8,887 genes randomly into train (80%) and test (20%) at gene level, ensuring no test gene appears in fine-tuning or signature-term extraction. Re-run GeneSUM and the six baselines, reporting ROUGE on the test-only summaries. If GeneSUM's test ROUGE-2 drops substantially (e.g., below 0.10 or near the Llama3-OpenBioLLM baseline), the reported advantage is largely memorization. Also report the overlap of 4-grams between the training references and generated summaries for test genes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on evaluating on genes not seen during training. The paper never specifies such a split. Section II-A uses \"reference summaries in the training set\" for signature-term filtering; Section II-D says key sentences are extracted \"from our training dataset\" and used for LoRA fine-tuning; Section III-A describes only the full set of 8,887 genes \"for experimentation.\" No train/test/validation partition, no held-out gene list, and no cross-validation are reported. If the 8,887 genes used for fine-tuning are the same genes whose Entrez summaries are the ROUGE references, the model can reproduce near-reference text, inflating all metrics and especially ROUGE-2 (0.1856 vs next-best 0.0467), because fine-tuning directly on the reference summaries teaches the exact bigram sequences in the targets. The case studies (COL4A2, TLR9) also show near-verbatim reuse of expert summary sentences, which is exactly the leakage signature. Since no code or data are released, the reader cannot check this from the artifact, making the assumption load-bearing rather than cosmetic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GeneSUM, a two-stage pipeline for automatic gene-summary generation. In the first stage, gene-related literature is retrieved and filtered using chi-square-based signature terms, Gene Ontology annotations are rewritten into natural-language descriptions via a prompted LLM, and a BioBERT-vectorized K-means clustering procedure with cosine similarity selects key sentences. In the second stage, Gemma-7B is fine-tuned with LoRA on prompts built from the selected key sentences to generate final summaries. Experiments on 8,887 genes are reported against six extractive and LLM baselines using ROUGE-1, ROUGE-2, and ROUGE-L, with additional ablations, vectorization-model comparisons, and two case studies (COL4A2 and TLR9). The central claim is that GeneSUM significantly outperforms all baselines across all three ROUGE metrics and better integrates gene-specific information.","tokens_in":10633,"tokens_out":4315,"duration_ms":39585,"significance":"If the empirical claim were validated, GeneSUM would be a practically useful contribution to biomedical text summarization: the idea of rewriting GO annotations into full sentences and using them as semantic anchors for key-sentence selection is sensible, and the combination of literature filtering, GO-based selection, clustering, and LoRA fine-tuning is a coherent pipeline. The paper also addresses a real bottleneck in database curation, where most genes lack structured summaries. However, the paper ships no code, data, or trained models, and the evaluation methodology as written does not establish generalization: no held-out test split is described, and the fine-tuning target appears to coincide with the ROUGE reference. The reported gains, especially the fourfold ROUGE-2 improvement over the best baseline, are therefore not currently interpretable as evidence of the method's effectiveness. The underlying idea remains worth testing under a proper evaluation protocol.","major_comments":[{"comment":"The paper never describes a held-out test split. Section II-D states that key sentences are extracted \"from our training dataset\" and used for LoRA fine-tuning, while Section III-A says that 8,887 genes with existing gene function description information were selected \"for experimentation.\" If the Entrez Gene summaries used as ROUGE references are the same summaries that appear in the fine-tuning data, the model can memorize the exact bigram sequences of the reference text, which would explain the large ROUGE-2 gap in Table I (0.1856 vs. the next-best 0.0467). The authors must specify a gene-level train/validation/test split, ensure that the signature-term extraction in Section II-A uses only the training portion, and report all metrics separately on genes never seen during fine-tuning.","section":"§II-D and §III-A"},{"comment":"The claim that \"our model significantly outperforms six baselines across three ROUGE metrics\" is not supported by any measure of variance or hypothesis test. Table I reports single point estimates, and no confidence intervals, standard errors, or paired tests are given. Even after a proper held-out split is introduced, the authors should report multiple runs or bootstrap over genes and use a paired significance test (e.g., Wilcoxon signed-rank or approximate randomization) to justify the word \"significantly.\"","section":"§III-B"},{"comment":"The baselines are not sufficiently specified to ensure a controlled comparison. It is unclear whether the Random baseline uses the same LoRA-fine-tuned model or the base model, and the prompts given to Llama2-70B, ChatGPT-3.5, BioMistral, and Llama3-OpenBioLLM are not provided. If the baselines receive different candidate-sentence sets, different context lengths, or different generation instructions, the ROUGE differences in Table I may reflect input or prompt variation rather than the proposed method.","section":"§III-A"},{"comment":"The case studies demonstrate near-verbatim reuse of expert-written reference sentences, for example \"This gene encodes one of the six subunits of type IV collagen, the major structural component of basement membranes\" for COL4A2 and \"TLRs are highly conserved from Drosophila to humans and share structural and functional similarities\" for TLR9. The authors present this as traceability, but it is also the expected signature of training-target leakage. These case studies should be repeated on held-out genes after the split is introduced, and the authors should report both ROUGE and a measure of factual overlap with the training summaries.","section":"Figures 6 and 7"}],"minor_comments":[{"comment":"The alternative hypothesis and the subsequent sentence are misstated: the null hypothesis is that term t_i is not a signature term, so rejection of the null implies that t_i is a signature term; the text as written says the opposite.","section":"§II-A, Eq. (1)"},{"comment":"There are typos in the K-means description: \"fist step\" should be \"first step\" and \"closet center\" should be \"closest center.\"","section":"§II-C"},{"comment":"Figures 4 and 5 are referenced in the text with captions, but the figures themselves are not present in the submitted manuscript. Please include the actual plots.","section":"Figures 4 and 5"},{"comment":"Reference [24] is cited for both Llama2-70B and ChatGPT-3.5, but [24] is a paper about GPT-3 and does not support those model names. Baseline 6 (Llama3-OpenBioLLM-8B) is paired with reference [26], which is the BioMistral paper; the citation is mismatched.","section":"§III-A"},{"comment":"The statement that \"many genes have around 10 GO annotations on average\" lacks a citation or supporting statistic, and the choice of the K-means range k=3..10 is justified only informally.","section":"§II-C"}],"recommendation":"major_revision","confidential_remarks":"The evaluation flaw is load-bearing and must be fixed before the paper can be considered: no held-out split is described, and the fine-tuning data appear to overlap with the ROUGE references. The fix is feasible within the paper's scope, so I am recommending major revision rather than outright rejection; however, if the authors cannot produce a disjoint gene-level split, the central claim cannot be salvaged and rejection would be appropriate. I also note the missing figures and citation mismatches, which are easy to correct but currently make the submission harder to assess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: the paper builds a plausible two-stage gene summarization pipeline, but its headline numbers are not trustworthy because the evaluation set appears to overlap with the fine-tuning set.\n\nWhat it does well: the problem is real, the pipeline is clearly described, and the GO-based sentence selection is a sensible idea. The ablations are informative in showing the relative contribution of components, and the comparison of vectorization models and base LLMs is useful.\n\nThe load-bearing flaw is the missing train/test split. Section II-A filters signature terms using reference summaries in the training set; Section II-D extracts key sentences from the training dataset for LoRA fine-tuning; Section III-A uses all 8,887 genes for experimentation with no partition described. If the same summaries serve as both training targets and ROUGE references, the model can memorize exact bigrams, which explains the fourfold ROUGE-2 jump. The case studies reinforce the leak: the TLR9 generated summary is nearly verbatim the expert summary. No code or data are released, so this cannot be checked. There are also no error bars or significance tests behind the claim of \"significantly outperforms.\" These are not minor blemishes; they undermine the central empirical claim.\n\nThe fix is straightforward: run a clean gene-level train/test split (ideally with a dev set for LoRA hyperparameters), report variance and significance, and release the artifacts. As it stands, reject, but with a clear revision path. This paper deserves referee time if the authors are willing to run the evaluation correctly.","headline":"Useful pipeline, but the headline results are not trustworthy because the evaluation set appears to overlap with the fine-tuning set.","tokens_in":11138,"tokens_out":1587,"would_cite":false,"duration_ms":14794,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GeneSUM, a two-stage LLM pipeline using Gene Ontology anchors, substantially outperforms six existing gene-summarization baselines on ROUGE metrics, generating summaries closer to expert-written Entrez Gene entries.","keywords":["gene summary","large language model","Gene Ontology","text summarization","LoRA fine-tuning","biomedical literature","ROUGE"],"falsifier":"Re-train GeneSUM on one half of the 8,887 genes and evaluate on the other half, or audit the training set for overlap with the evaluation genes; if ROUGE-2 falls from 0.1856 toward the best baseline's 0.0467, the central claim fails.","tokens_in":10201,"feed_emoji":"🧬","tokens_out":6420,"duration_ms":49965,"temperature":0.7,"pith_summary":"The paper tries to establish that gene summaries, which are normally written by expert curators and stored in databases like Entrez Gene, can be generated automatically and at scale by a two-stage large-language-model pipeline. Its central claim is that the proposed system, GeneSUM, produces summaries much closer to expert-written references than six existing extractive and generative baselines, as measured by ROUGE-1, ROUGE-2, and ROUGE-L. The authors argue that the key to this improvement is using Gene Ontology annotations as semantic anchors to select the most relevant sentences from the literature, and then fine-tuning a general LLM on those selected sentences to write the final summary. If the claim is right, automated gene summarization could keep biological knowledge bases current with the expanding literature and give researchers fast, traceable overviews of gene function.","feed_headline":"Gene Ontology anchors plus a tuned LLM beat six gene-summary baselines","feed_subtitle":"Two-stage GeneSUM lifts ROUGE-2 fourfold over the best baseline, moving gene summaries closer to expert-written entries.","key_machinery":"The load-bearing component is the GO-anchored sentence selection step. Gene Ontology terms for the target gene (molecular function, biological process, cellular component) are expanded into natural-language sentences by an LLM; both these descriptions and the filtered literature sentences are vectorized with BioBERT, clustered with K-means (k chosen by Calinski-Harabasz score), and within each cluster the sentence with maximum cosine similarity to a GO description is kept as a key sentence. These key sentences are then injected through a prompt into Gemma-7B fine-tuned with LoRA, which generates the final summary. Signature-term filtering (Pearson chi-square test against background literature) is the first-stage reducer that trims the candidate sentence pool.","core_discovery":"On its own terms, the paper's discovery is that a gene summary can be reconstructed by combining two cheap signals: the gene's structured Gene Ontology annotations and a filtered set of literature sentences. GO terms are rewritten by an LLM into short descriptive sentences, used to locate and select key sentences from the literature by cosine similarity in a BioBERT embedding space, and the chosen sentences are passed to a LoRA-fine-tuned Gemma-7B model to generate the final summary. The reported result is that this pipeline reaches ROUGE-1 0.3874, ROUGE-2 0.1856, and ROUGE-L 0.3681 against expert-written Entrez Gene summaries, with the GO-based selection step alone accounting for most of the gain over filtering without GO.","pith_inferences":["A critical test the paper does not report is a train/test split by gene; if the fine-tuning set overlaps the evaluation set, the ROUGE advantage could shrink or vanish, so the method's generalization to unseen genes remains unverified.","The same GO-anchored selection could be applied to other ontology-based summarization targets, such as disease or drug summaries, since the mechanism is not gene-specific.","The GO-rewrite step could likely be replaced by direct embeddings of GO terms, avoiding reliance on an external LLM for the expansion."],"forward_implications":["If the reported ROUGE gains hold, gene summary generation can be automated from raw PubMed literature and kept current as new papers appear, without waiting for expert curation.","The method produces summaries whose sentences can be traced back to specific PubMed Central articles, addressing hallucination concerns for this task.","The GO-anchored sentence selection finding suggests that structured ontology knowledge can materially improve LLM summarization in specialized domains.","Fine-tuning a relatively small 7B model with LoRA appears sufficient to outperform much larger general-purpose LLMs on this task."],"supporting_citations":[{"why":"Supplies the LTR baseline that GeneSUM must beat and the earlier formulation of gene summary extraction.","marker":"[3]"},{"why":"Provides the signature-term filtering method that GeneSUM's first stage builds on.","marker":"[10]"},{"why":"Motivates using GO annotations as gene-specific information for selecting gene references into function.","marker":"[12]"},{"why":"BioBERT is the domain-specific embedding model used to vectorize sentences and GO descriptions.","marker":"[15]"},{"why":"LoRA is the low-rank adaptation method used to fine-tune Gemma-7B efficiently.","marker":"[21]"},{"why":"Defines the ROUGE metrics used for all evaluations against expert-written summaries.","marker":"[23]"}],"fun_headline_variants":["LLM plus Gene Ontology lifts gene summaries fourfold on ROUGE-2","Two-stage GeneSUM: GO-filtered literature, LoRA-tuned Gemma boosts summaries","GeneSUM: Simpler recipe – GO terms + filtered sentences + fine-tuned LLM","Gene summary extraction: GO filtering and LLM tuning beat six baselines","GeneSUM pipeline: from GO annotations to expert-level summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without describing a train/test split, that the genes used to fine-tune the generator are not the same genes whose Entrez Gene summaries serve as ROUGE references; if they overlap, the headline scores could reflect memorization rather than summarization ability.","fun_headline_variants_meta":{"raw":{"variants":["LLM plus Gene Ontology lifts gene summaries fourfold on ROUGE-2","Two-stage GeneSUM: GO-filtered literature, LoRA-tuned Gemma boosts summaries","GeneSUM: Simpler recipe – GO terms + filtered sentences + fine-tuned LLM","Gene summary extraction: GO filtering and LLM tuning beat six baselines","GeneSUM pipeline: from GO annotations to expert-level summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1891,"prompt_tokens":886,"completion_tokens":1005,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":901}},"tokens_in":502,"tokens_out":1005,"duration_ms":9077,"temperature":1.0,"reasoning_tokens":901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:58:37.338063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train GeneSUM on one half of the 8,887 genes and evaluate on the other half, or audit the training set for overlap with the evaluation genes; if ROUGE-2 falls from 0.1856 toward the best baseline's 0.0467, the central claim fails.","supporting_citations":[{"cited_title":"Learning to rank-based gene summary extraction,","cited_arxiv_id":null,"evidence_quote":"Supplies the LTR baseline that GeneSUM must beat and the earlier formulation of gene summary extraction."},{"cited_title":"Towards automatic generation of gene summary,","cited_arxiv_id":null,"evidence_quote":"Provides the signature-term filtering method that GeneSUM's first stage builds on."},{"cited_title":"Measuring gene functional similarity based on group-wise comparison of go terms,","cited_arxiv_id":null,"evidence_quote":"Motivates using GO annotations as gene-specific information for selecting gene references into function."}],"review_version":1}