REVIEW 4 major objections 5 minor 36 references
GeneSUM: Large Language Model-based Gene Summary Extraction
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that GeneSUM, a two-stage LLM pipeline using Gene Ontology anchors, substantially outperforms six existing gene-summarization baselines on ROUGE metrics, generating summaries closer to expert-written Entrez Gene entries.
desk verdict Useful pipeline, but the headline results are not trustworthy because the evaluation set appears to overlap with the fine-tuning set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the GO-anchored sentence selection step. Gene Ontology terms for the target gene (molecular function, biological process, cellular component) are expanded into natural-language sentences by an LLM; both these descriptions and the filtered literature sentences are vectorized with BioBERT, clustered with K-means (k chosen by Calinski-Harabasz score), and within each cluster the sentence with maximum cosine similarity to a GO description is kept as a key sentence. These key sentences are then injected through a prompt into Gemma-7B fine-tuned with LoRA, which generates the final summary. Signature-term filtering (Pearson chi-square test against background literature) is the first-stage reducer that trims the candidate sentence pool.
What would settle it
Re-train GeneSUM on one half of the 8,887 genes and evaluate on the other half, or audit the training set for overlap with the evaluation genes; if ROUGE-2 falls from 0.1856 toward the best baseline's 0.0467, the central claim fails.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a gene summary can be reconstructed by combining two cheap signals: the gene's structured Gene Ontology annotations and a filtered set of literature sentences. GO terms are rewritten by an LLM into short descriptive sentences, used to locate and select key sentences from the literature by cosine similarity in a BioBERT embedding space, and the chosen sentences are passed to a LoRA-fine-tuned Gemma-7B model to generate the final summary. The reported result is that this pipeline reaches ROUGE-1 0.3874, ROUGE-2 0.1856, and ROUGE-L 0.3681 against expert-written Entrez Gene summaries, with the GO-based selection step alone accounting for most of the gain over filtering without GO.
Load-bearing premise
The paper assumes, without describing a train/test split, that the genes used to fine-tune the generator are not the same genes whose Entrez Gene summaries serve as ROUGE references; if they overlap, the headline scores could reflect memorization rather than summarization ability.
Editorial extensions
If this is right
- If the reported ROUGE gains hold, gene summary generation can be automated from raw PubMed literature and kept current as new papers appear, without waiting for expert curation.
- The method produces summaries whose sentences can be traced back to specific PubMed Central articles, addressing hallucination concerns for this task.
- The GO-anchored sentence selection finding suggests that structured ontology knowledge can materially improve LLM summarization in specialized domains.
- Fine-tuning a relatively small 7B model with LoRA appears sufficient to outperform much larger general-purpose LLMs on this task.
Reading between the lines
- A critical test the paper does not report is a train/test split by gene; if the fine-tuning set overlaps the evaluation set, the ROUGE advantage could shrink or vanish, so the method's generalization to unseen genes remains unverified.
- The same GO-anchored selection could be applied to other ontology-based summarization targets, such as disease or drug summaries, since the mechanism is not gene-specific.
- The GO-rewrite step could likely be replaced by direct embeddings of GO terms, avoiding reliance on an external LLM for the expansion.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GeneSUM, a two-stage pipeline for automatic gene-summary generation. In the first stage, gene-related literature is retrieved and filtered using chi-square-based signature terms, Gene Ontology annotations are rewritten into natural-language descriptions via a prompted LLM, and a BioBERT-vectorized K-means clustering procedure with cosine similarity selects key sentences. In the second stage, Gemma-7B is fine-tuned with LoRA on prompts built from the selected key sentences to generate final summaries. Experiments on 8,887 genes are reported against six extractive and LLM baselines using ROUGE-1, ROUGE-2, and ROUGE-L, with additional ablations, vectorization-model comparisons, and two case studies (COL4A2 and TLR9). The central claim is that GeneSUM significantly outperforms all baselines across all three ROUGE metrics and better integrates gene-specific information.
Significance. If the empirical claim were validated, GeneSUM would be a practically useful contribution to biomedical text summarization: the idea of rewriting GO annotations into full sentences and using them as semantic anchors for key-sentence selection is sensible, and the combination of literature filtering, GO-based selection, clustering, and LoRA fine-tuning is a coherent pipeline. The paper also addresses a real bottleneck in database curation, where most genes lack structured summaries. However, the paper ships no code, data, or trained models, and the evaluation methodology as written does not establish generalization: no held-out test split is described, and the fine-tuning target appears to coincide with the ROUGE reference. The reported gains, especially the fourfold ROUGE-2 improvement over the best baseline, are therefore not currently interpretable as evidence of the method's effectiveness. The underlying idea remains worth testing under a proper evaluation protocol.
major comments (4)
- [§II-D and §III-A] The paper never describes a held-out test split. Section II-D states that key sentences are extracted "from our training dataset" and used for LoRA fine-tuning, while Section III-A says that 8,887 genes with existing gene function description information were selected "for experimentation." If the Entrez Gene summaries used as ROUGE references are the same summaries that appear in the fine-tuning data, the model can memorize the exact bigram sequences of the reference text, which would explain the large ROUGE-2 gap in Table I (0.1856 vs. the next-best 0.0467). The authors must specify a gene-level train/validation/test split, ensure that the signature-term extraction in Section II-A uses only the training portion, and report all metrics separately on genes never seen during fine-tuning.
- [§III-B] The claim that "our model significantly outperforms six baselines across three ROUGE metrics" is not supported by any measure of variance or hypothesis test. Table I reports single point estimates, and no confidence intervals, standard errors, or paired tests are given. Even after a proper held-out split is introduced, the authors should report multiple runs or bootstrap over genes and use a paired significance test (e.g., Wilcoxon signed-rank or approximate randomization) to justify the word "significantly."
- [§III-A] The baselines are not sufficiently specified to ensure a controlled comparison. It is unclear whether the Random baseline uses the same LoRA-fine-tuned model or the base model, and the prompts given to Llama2-70B, ChatGPT-3.5, BioMistral, and Llama3-OpenBioLLM are not provided. If the baselines receive different candidate-sentence sets, different context lengths, or different generation instructions, the ROUGE differences in Table I may reflect input or prompt variation rather than the proposed method.
- [Figures 6 and 7] The case studies demonstrate near-verbatim reuse of expert-written reference sentences, for example "This gene encodes one of the six subunits of type IV collagen, the major structural component of basement membranes" for COL4A2 and "TLRs are highly conserved from Drosophila to humans and share structural and functional similarities" for TLR9. The authors present this as traceability, but it is also the expected signature of training-target leakage. These case studies should be repeated on held-out genes after the split is introduced, and the authors should report both ROUGE and a measure of factual overlap with the training summaries.
minor comments (5)
- [§II-A, Eq. (1)] The alternative hypothesis and the subsequent sentence are misstated: the null hypothesis is that term t_i is not a signature term, so rejection of the null implies that t_i is a signature term; the text as written says the opposite.
- [§II-C] There are typos in the K-means description: "fist step" should be "first step" and "closet center" should be "closest center."
- [Figures 4 and 5] Figures 4 and 5 are referenced in the text with captions, but the figures themselves are not present in the submitted manuscript. Please include the actual plots.
- [§III-A] Reference [24] is cited for both Llama2-70B and ChatGPT-3.5, but [24] is a paper about GPT-3 and does not support those model names. Baseline 6 (Llama3-OpenBioLLM-8B) is paired with reference [26], which is the BioMistral paper; the citation is mismatched.
- [§II-C] The statement that "many genes have around 10 GO annotations on average" lacks a citation or supporting statistic, and the choice of the K-means range k=3..10 is justified only informally.
Circularity Check
Headline ROUGE gains may reduce to memorization: no held-out gene split is described, and the fine-tuning targets are the same Entrez 'summary attribute' used as ROUGE references.
-
fitted input called prediction
[Section II-A, II-D, III-A]
"We use the Pearson’s chi-square test [11] to extract topic signature terms from reference summaries in the training set ... The process begins with the extraction of key sentences for each gene from our training dataset using a key sentence extraction module ... We utilized gene function description information from the summary attribute of the database as a reference from NCBI sub-database Entrez Gene. ... we selected 8,887 genes that had existing gene function description information for experimentation."
The LoRA fine-tuning targets are gene summaries derived from the Entrez Gene 'summary attribute', and Section II-A explicitly places 'reference summaries' in the training set. Section II-D then extracts key sentences from the same 'training dataset' for fine-tuning. Section III-A uses the very same Entrez 'summary attribute' as the ROUGE reference and reports results on the 8,887 genes 'for experimentation' with no described train/test/validation split, held-out gene list, or cross-validation. If the evaluation genes overlap with the fine-tuning genes, the reported ROUGE scores--especially ROUGE-2 0.1856 vs. the best baseline 0.0467--measure reproduction of training references rather than generalization.
full rationale
The central empirical claim is that GeneSUM 'significantly outperforms six baselines across three ROUGE metrics.' The only load-bearing circularity is the apparent identity between the fine-tuning data and the evaluation references. Section II-A states that signature terms are extracted from 'reference summaries in the training set'; Section II-D states that key sentences are extracted from 'our training dataset' for LoRA fine-tuning; Section III-A defines the ROUGE reference as the Entrez Gene 'summary attribute' and describes experiments on all 8,887 genes without any partition. As written, the model is fine-tuned on the same summaries against which it is scored, so the reported gains (particularly ROUGE-2, a bigram overlap metric) are consistent with memorization rather than independent prediction. The case studies of COL4A2 and TLR9 show near-verbatim reuse of expert summary sentences, which is the expected leakage signature. No code or data is released, so the reader cannot verify a hidden split. Other self-citations in the paper are not load-bearing for the main claim, and the GO-rewriting and clustering components retain independent technical content, so the circularity is partial but serious, affecting the headline performance claim.
Assumptions & free parameters
free parameters (4)
- chi-square signature threshold =
10.83
- minimum signature terms per sentence =
3
- k-means cluster count k =
selected by CH score in [3,10]
- LoRA rank r =
32
assumptions (5)
- domain assumption The Entrez Gene summary attribute is a valid ground truth for gene summaries.
- domain assumption ROUGE scores measure the quality of gene summaries.
- domain assumption BioBERT embedding cosine similarity captures semantic relevance between literature sentences and GO term descriptions.
- domain assumption ChatGPT-generated GO term descriptions are faithful expansions of the GO annotations.
- domain assumption The signature terms extracted from training reference summaries generalize to unseen genes.
Cite this review
Pith. "Pith review of GeneSUM: Large Language Model-based Gene Summary Extraction." pith.science (2026). https://pith.science/paper/ZTEQAXRF
@misc{pith2026241218154,
author = {Pith},
title = {Pith review of: GeneSUM: Large Language Model-based Gene Summary Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZTEQAXRF}},
note = {Machine review of arXiv:2412.18154}
}
read the original abstract
Emerging topics in biomedical research are continuously expanding, providing a wealth of information about genes and their function. This rapid proliferation of knowledge presents unprecedented opportunities for scientific discovery and formidable challenges for researchers striving to keep abreast of the latest advancements. One significant challenge is navigating the vast corpus of literature to extract vital gene-related information, a time-consuming and cumbersome task. To enhance the efficiency of this process, it is crucial to address several key challenges: (1) the overwhelming volume of literature, (2) the complexity of gene functions, and (3) the automated integration and generation. In response, we propose GeneSUM, a two-stage automated gene summary extractor utilizing a large language model (LLM). Our approach retrieves and eliminates redundancy of target gene literature and then fine-tunes the LLM to refine and streamline the summarization process. We conducted extensive experiments to validate the efficacy of our proposed framework. The results demonstrate that LLM significantly enhances the integration of gene-specific information, allowing more efficient decision-making in ongoing research.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
S. Federhen, “The ncbi taxonomy database,” Nucleic acids research , vol. 40, no. D1, pp. D136–D143, 2012
work page 2012
-
[2]
Ncbi taxonomy: a comprehensive update on curation, resources and tools,
C. L. Schoch, S. Ciufo, M. Domrachev, C. L. Hotton, S. Kannan, R. Khovanskaya, D. Leipe, R. Mcveigh, K. O’Neill, B. Robbertse et al., “Ncbi taxonomy: a comprehensive update on curation, resources and tools,” Database, vol. 2020, p. baaa062, 2020
work page 2020
-
[3]
Learning to rank-based gene summary extraction,
Y . Shang, H. Hao, J. Wu, and H. Lin, “Learning to rank-based gene summary extraction,” BMC bioinformatics , vol. 15, pp. 1–8, 2014
work page 2014
-
[4]
Fast abstractive summarization with reinforce-selected sentence rewriting,
Y .-C. Chen and M. Bansal, “Fast abstractive summarization with reinforce-selected sentence rewriting,” arXiv preprint arXiv:1805.11080, 2018
arXiv 2018
-
[5]
Bottom-up abstractive summarization,
S. Gehrmann, Y . Deng, and A. M. Rush, “Bottom-up abstractive summarization,” arXiv preprint arXiv:1808.10792 , 2018
arXiv 2018
-
[6]
Expert knowledge-guided length-variant hierarchical label generation for pro- posal classification,
M. Xiao, Z. Qiao, Y . Fu, Y . Du, P. Wang, and Y . Zhou, “Expert knowledge-guided length-variant hierarchical label generation for pro- posal classification,” in 2021 IEEE International Conference on Data Mining (ICDM) . IEEE, 2021, pp. 757–766
work page 2021
-
[7]
Hierarchical interdisciplinary topic detection model for research proposal classification,
M. Xiao, Z. Qiao, Y . Fu, H. Dong, Y . Du, P. Wang, H. Xiong, and Y . Zhou, “Hierarchical interdisciplinary topic detection model for research proposal classification,” IEEE Transactions on Knowledge and Data Engineering , vol. 35, no. 9, pp. 9685–9699, 2023
work page 2023
-
[8]
M. Xiao, M. Wu, Z. Qiao, Y . Fu, Z. Ning, Y . Du, and Y . Zhou, “In- terdisciplinary fairness in imbalanced research proposal topic inference: A hierarchical transformer-based method with selective interpolation,” ACM Transactions on Knowledge Discovery from Data
Show all 36 references
-
[9]
The automated acquisition of topic signatures for text summarization,
C.-Y . Lin and E. Hovy, “The automated acquisition of topic signatures for text summarization,” in COLING 2000 volume 1: The 18th interna- tional conference on computational linguistics , 2000
2000
-
[10]
Towards automatic generation of gene summary,
F. Jin, M. Huang, Z. Lu, and X. Zhu, “Towards automatic generation of gene summary,” in Proceedings of the BioNLP 2009 Workshop , 2009, pp. 97–105
2009
-
[11]
Manning and H
C. Manning and H. Schutze, F oundations of statistical natural language processing. MIT press, 1999
1999
-
[12]
Measuring gene functional similarity based on group-wise comparison of go terms,
Z. Teng, M. Guo, X. Liu, Q. Dai, C. Wang, and P. Xuan, “Measuring gene functional similarity based on group-wise comparison of go terms,” Bioinformatics, vol. 29, no. 11, pp. 1424–1432, 2013
2013
-
[13]
Gpt-ner: Named entity recognition via large language models,
S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, and G. Wang, “Gpt-ner: Named entity recognition via large language models,” arXiv preprint arXiv:2304.10428, 2023
2023 arXiv
-
[14]
Biobert based named entity recognition in electronic medical record,
X. Yu, W. Hu, S. Lu, X. Sun, and Z. Yuan, “Biobert based named entity recognition in electronic medical record,” in 2019 10th interna- tional conference on information technology in medicine and education (ITME). IEEE, 2019, pp. 49–52
2019
-
[15]
Biobert: a pre-trained biomedical language representation model for biomedical text mining,
J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020
2020
-
[16]
Extracting drug-drug interactions from texts with biobert and multiple entity-aware attentions,
Y . Zhu, L. Li, H. Lu, A. Zhou, and X. Qin, “Extracting drug-drug interactions from texts with biobert and multiple entity-aware attentions,” Journal of biomedical informatics , vol. 106, p. 103451, 2020
2020
-
[17]
K-means clustering: a half-century synthesis,
D. Steinley, “K-means clustering: a half-century synthesis,” British Journal of Mathematical and Statistical Psychology , vol. 59, no. 1, pp. 1–34, 2006
2006
-
[18]
Unsupervised k-means clustering algo- rithm,
K. P. Sinaga and M.-S. Yang, “Unsupervised k-means clustering algo- rithm,” IEEE access , vol. 8, pp. 80 716–80 727, 2020
2020
-
[19]
Semantic cosine similar- ity,
F. Rahutomo, T. Kitasuka, M. Aritsugi et al., “Semantic cosine similar- ity,” in The 7th international student conference on advanced science and technology ICAST , vol. 4, no. 1. University of Seoul South Korea, 2012, p. 1
2012
-
[20]
Learning similarity with cosine similarity ensemble,
P. Xia, L. Zhang, and F. Li, “Learning similarity with cosine similarity ensemble,” Information sciences , vol. 307, pp. 39–52, 2015
2015
-
[21]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021
2021 arXiv
-
[22]
Per- formance analysis of lora finetuning llama-2,
A. Pathak, O. Shree, M. Agarwal, S. D. Sarkar, and A. Tiwary, “Per- formance analysis of lora finetuning llama-2,” in 2023 7th International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech). IEEE, 2023, pp. 1–4
2023
-
[23]
Rouge: A package for automatic evaluation of summaries,
C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81
2004
-
[24]
Gpt-3: Its nature, scope, limits, and consequences,
L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines , vol. 30, pp. 681–694, 2020
2020
-
[25]
Can- cerllm: A large language model in cancer domain,
M. Li, A. Blaes, S. Johnson, H. Liu, H. Xu, and R. Zhang, “Can- cerllm: A large language model in cancer domain,” arXiv preprint arXiv:2406.10459, 2024
2024 arXiv
-
[26]
Biomistral: A collection of open-source pretrained large language models for medical domains,
Y . Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, and R. Du- four, “Biomistral: A collection of open-source pretrained large language models for medical domains,” arXiv preprint arXiv:2402.10373 , 2024
2024 arXiv
-
[27]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
2018 arXiv
-
[28]
Generating gene summaries from biomedical literature: A study of semi-structured summarization,
X. Ling, J. Jiang, X. He, Q. Mei, C. Zhai, and B. Schatz, “Generating gene summaries from biomedical literature: A study of semi-structured summarization,” Information Processing & Management , vol. 43, no. 6, pp. 1777–1791, 2007
2007
-
[29]
Bottlesum: Unsupervised and self-supervised sentence summarization using the information bot- tleneck principle,
P. West, A. Holtzman, J. Buys, and Y . Choi, “Bottlesum: Unsupervised and self-supervised sentence summarization using the information bot- tleneck principle,” arXiv preprint arXiv:1909.07405 , 2019
1909 arXiv
-
[30]
Needed: Introducing hierarchical transformer to eye diseases diagnosis,
X. Ye, M. Xiao, Z. Ning, W. Dai, W. Cui, Y . Du, and Y . Zhou, “Needed: Introducing hierarchical transformer to eye diseases diagnosis,” in Pro- ceedings of the 2023 SIAM International Conference on Data Mining (SDM). SIAM, 2023, pp. 667–675
2023
-
[31]
A continued pretrained llm approach for automatic medical note generation,
D. Yuan, E. Rastogi, G. Naik, J. Chintagunta, S. P. Rajagopal, F. Zhao, S. Goyal, and J. Ward, “A continued pretrained llm approach for automatic medical note generation,” arXiv preprint arXiv:2403.09057 , 2024
2024 arXiv
-
[32]
Biorag: A rag-llm framework for biological question reasoning,
C. Wang, Q. Long, X. Meng, X. Cai, C. Wu, Z. Meng, X. Wang, and Y . Zhou, “Biorag: A rag-llm framework for biological question reasoning,” arXiv preprint arXiv:2408.01107 , 2024
2024 arXiv
-
[33]
scinterpreter: Training large language models to interpret scrna-seq data for cell type annotation,
C. Li, M. Xiao, P. Wang, G. Feng, X. Li, and Y . Zhou, “scinterpreter: Training large language models to interpret scrna-seq data for cell type annotation,” arXiv preprint arXiv:2402.12405 , 2024
2024 arXiv
-
[34]
Resolving the imbalance issue in hierarchical disciplinary topic inference via llm-based data augmentation,
X. Cai, M. Xiao, Z. Ning, and Y . Zhou, “Resolving the imbalance issue in hierarchical disciplinary topic inference via llm-based data augmentation,” in 2023 IEEE International Conference on Data Mining (ICDM). IEEE, 2023, pp. 956–961
2023
-
[35]
What makes good in-context examples for gpt- 3?
J. Liu, D. Shen, Y . Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt- 3?” arXiv preprint arXiv:2101.06804, 2021
2021 arXiv
-
[36]
Enhanced gene selection in single-cell genomics: Pre-filtering synergy and reinforced optimization,
W. Zhang, Z. Meng, D. Wang, M. Wu, K. Liu, Y . Zhou, and M. Xiao, “Enhanced gene selection in single-cell genomics: Pre-filtering synergy and reinforced optimization,” arXiv preprint arXiv:2406.07418 , 2024
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.