{"id":"786d40ce-ecc7-4294-b3c3-ba9d07139930","arxiv_id":"2509.10575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Gene-R1 combines domain pre-training, GPT-o1 distilled reasoning examples, and GRPO reinforcement learning so lightweight Llama models match commercial LLMs on gene set analysis.","lead":"A new training pipeline, Gene-R1, makes small open-source language models annotate gene sets with biological function names and step-by-step reasoning. On 1,508 Gene Ontology gene sets it reports text and semantic similarity scores at or above commercial models like GPT-4, and comparable results on 106 out-of-distribution gene sets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic metric overlaps with training reward: reported gains may reflect optimization against MedCPT rather than improved biological annotation.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: the evaluation semantic metric overlaps with the training reward. I agree with the conditional verdict because the ROUGE improvements and OOD robustness provide some independent support, but the semantic similarity headline is potentially inflated by MedCPT contamination. The proposed concrete test would settle whether the contamination actually changes the conclusions. I do not see a reason to move from CONDITIONAL to ACCEPT or REJECT; the concern is substantial but addressable with additional analysis or a minor evaluation change.","tokens_in":12514,"tokens_out":1809,"duration_ms":23121,"concrete_test":"Recompute Tables 4 and 5 with SentenceBERT and SapBERT reported separately, and compute the average excluding MedCPT. If Gene-R1(8B)'s non-MedCPT similarity on GO:BP or OOD datasets no longer exceeds GPT-4/o1, the semantic-matching claim is largely an artifact of reward optimization. A stronger test: retrain Gene-R1 with the MedCPT term removed from the GRPO reward (e.g., using only LSC or SentenceBERT) and check whether the held-out non-MedCPT evaluation gap persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Gene-R1 matches commercial LLMs on semantic similarity—rests on an evaluation metric that is not independent of the training objective. In the TA module (Section 3), the GRPO reward is a soft match combining MedCPT similarity and LSC lexical overlap against the gold label. The RA module also filters GPT-o1-generated training instances by requiring a similarity score >0.7 to the ground-truth label. Section 4.2 then defines the evaluation similarity as the average of MedCPT, SentenceBERT, and SapBERT scores. Thus Gene-R1 is directly trained to maximize MedCPT similarity, and the reported average includes that same encoder. This is a train/eval overlap: the gap in 'Score (avg.)' in Table 4 may be driven primarily by MedCPT, with SentenceBERT/SapBERT contributing little. The out-of-distribution 'n.s.' comparisons in Table 5 are similarly affected. The ROUGE-L gains (0.277 vs 0.184 on GO:BP) are independent evidence and are impressive, but the semantic-similarity component of the 'matching commercial LLMs' conclusion is insecure without per-encoder breakdowns or a held-out evaluation encoder. The absence of code, data, and seed-level error bars compounds this, but the core methodological issue is the reward-evaluation encoder overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Gene-R1, a three-stage fine-tuning pipeline that adapts lightweight open-source LLMs (Llama 3.2 1B/3B and Llama 3.1 8B) to gene set analysis (GSA). Stage 1 (knowledge warm-up) continues pre-training on gene-centric relational sentences from public biological databases. Stage 2 (reasoning activation) supervised fine-tunes the model on GPT-o1-generated step-by-step reasoning chains for GO gene sets, retaining only examples whose proposed function is >0.7 similar to the gold label. Stage 3 (task alignment) applies GRPO with a soft-match reward combining MedCPT semantic similarity and longest-common-subsequence lexical overlap, together with an exact format match. The authors evaluate on 1,508 in-distribution GO gene sets (GO:BP, GO:MF, GO:CC) and 106 out-of-distribution gene sets (NeST, MsigDB), reporting ROUGE and semantic similarity averaged over MedCPT, SentenceBERT, and SapBERT. The headline claim is that Gene-R1(8B) substantially improves over base Llama models and matches commercial LLMs such as GPT-4 in-distribution and on out-of-distribution gene sets.","tokens_in":12835,"tokens_out":5546,"duration_ms":64289,"significance":"If the reported gains are robust, the paper makes a useful practical contribution: it provides a concrete recipe for distilling reasoning capability into small open-source models for a biomedical annotation task, shows a clear benefit of domain-knowledge warm-up, and demonstrates a potential path to local, privacy-preserving deployment. The breadth of evaluation across three model sizes and five datasets is a strength, and the inclusion of an out-of-distribution test is valuable. However, the central quantitative claim is currently weakened by the overlap between the training reward and the evaluation encoders, and by the absence of code, data, and seed-level variance. The underlying idea is promising, but the evidence as presented needs substantial strengthening before the 'matching commercial LLMs' conclusion can be accepted.","major_comments":[{"comment":"The semantic evaluation is not independent of the training reward. In the RA module, GPT-o1 outputs are retained only when a similarity score to the gold label exceeds 0.7; in the TA module, the GRPO reward is a soft match combining MedCPT and LSC. Section 4.2 then defines the evaluation similarity as the average of MedCPT, SentenceBERT, and SapBERT, and ROUGE-L is itself based on longest common subsequence. Thus Gene-R1 is directly optimized against the same family of similarity functions used in evaluation. The reported 'Score (avg.)' in Table 4 and the 'n.s.' comparisons in Table 5 may be driven primarily by MedCPT, with SentenceBERT and SapBERT contributing little or nothing. The authors should report per-encoder scores for MedCPT, SentenceBERT, and SapBERT separately, run significance tests per encoder, and ideally add a held-out evaluation encoder that was not used in RA/TA. Withou","section":"Section 3; Section 4.2; Tables 4-5"},{"comment":"The text overstates the consistency of the results. Table 4 shows that Gene-R1(1B) on GO:BP has a semantic similarity of 0.617, below GPT-4o's 0.629, and Gene-R1(3B) on GO:CC is 0.578, below several baselines (e.g., o1 at 0.609). Moreover, Table 6 shows Gene-R1(3B) produces correct output format in only 42.4% of GO:BP and 42.9% of GO:CC cases. The abstract and Section 5.1 claim that Gene-R1 'consistently outperforms all baselines' and 'closes the performance gap' with commercial LLMs; this is defensible only for the 8B variant in-distribution. Please qualify the claims by model size, report per-variant significance tests, and directly address the 3B format-instability issue.","section":"Section 5.1; Table 4"},{"comment":"No code, training data, or evaluation scripts are provided, and all results are reported as single numbers without error bars or seed-level variance. The paired t-test p-values in Tables 4 and 5 are computed over items within one run, not over independent training seeds, so they do not establish stability of the pipeline. Since the paper's contribution is a training recipe, this is a major reproducibility gap. The authors should release the RA/TA datasets, the filtering and reward code, and evaluation scripts, and report mean and standard deviation over at least three seeds for the main comparisons.","section":"Section 4.2; Section 4.3; Tables 4-5"},{"comment":"The similarity threshold of 0.7 used to filter GPT-o1-generated reasoning examples is not defined: which encoder computes this similarity? This is important both because it determines the quality of the SFT corpus and because it is directly related to the evaluator-overlap issue. In addition, no sensitivity analysis is provided for this threshold, nor for the relative weights of MedCPT and LSC in the soft-match reward. At minimum, specify the encoder and report how performance varies as the threshold and reward weights are changed.","section":"Section 3; Section 5.3"}],"minor_comments":[{"comment":"There are numerous typos and formatting errors: 'realtional', 'modue', 'acorss', 'solation', 'constract', 'bechmark', 'Alation experiments', 'incoporating', 'diffuclt', and 'specificized'. The paper needs a careful proofreading pass.","section":"General"},{"comment":"The evaluation section states that similarity is averaged over three encoders to 'mitigate potential bias inherent to a single semantic encoder.' That rationale is undermined by the fact that one of the three encoders (MedCPT) is used directly in the training reward. Please move MedCPT out of the primary evaluation average or treat it as a development metric.","section":"Section 4.2"},{"comment":"The 3B variant shows a striking performance drop on GO:CC (ROUGE-L 0.143 vs 0.222 for the 1B variant), which is not explained in the text. If this is due to format instability (Table 6), the reader should be told explicitly.","section":"Section 5.1; Table 4"},{"comment":"The backbone is Llama 3.1 and Llama 3.2, but the paper cites Touvron et al. (2023), which introduces Llama 1. Please cite the appropriate Llama 3 model card (e.g., Grattafiori et al., 2024) or the relevant model release.","section":"References; Section 4.3"},{"comment":"The limitations paragraph is honest but brief. It acknowledges manual data collection and hallucination risk on unseen genes, but it does not mention the evaluator-reward overlap or the absence of public code/data. Please add these as explicit limitations.","section":"Section 6; Limitations"},{"comment":"The text references results in Figure 3 without providing axis labels or error bars in the figure description. Please ensure all subfigures are labeled and that the figure caption states what is plotted and with what error bars.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a machine-learning-for-biology venue and addresses a relevant problem. The main risk is the overlap between the training reward (MedCPT + LSC) and the evaluation metrics (MedCPT average, ROUGE-L). I would not reject on this basis, because the authors could remedy it by reporting per-encoder results, adding a held-out encoder, and running seed-level experiments. However, the abstract and Section 5.1 currently overclaim consistency across all model sizes, and the absence of code and data is a significant barrier for a pipeline paper. I recommend major revision and would be willing to look at a revised version that addresses the per-encoder breakdown and releases the assets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about Gene-R1 because it is the first serious attempt to bring step-by-step reasoning training to gene set analysis, and it works better than I expected. The three-stage recipe—domain pre-training on gene-centric relational data, SFT on GPT-o1-distilled reasoning chains, then GRPO with a soft-match reward—is assembled from known components, but the application is new and the engineering is competent. The results on the 8B Llama model are strong: ROUGE-L on GO:BP goes from 0.184 (GPT-4) to 0.277, and on the 106 out-of-distribution gene sets the model is statistically indistinguishable from GPT-4 and Llama3-70B. Those gains are not trivial. The ablation study is also honest, showing each module contributes, and the paper includes an error analysis and a limitations section, which I respect.\n\nThe real soft spot is the evaluation metric. The training loop uses MedCPT similarity both to filter the RA data (threshold >0.7) and as part of the GRPO reward (MedCPT + LSC). The reported evaluation metric averages MedCPT, SentenceBERT, and SapBERT. So the semantic-similarity component of the 'matching commercial LLMs' claim is at least partly optimized against the same encoder used in evaluation. That is a train/eval overlap, and it means the average similarity scores in Tables 4 and 5 may be inflated. The ROUGE results are independent and remain impressive, but the paper needs per-encoder breakdowns or a held-out evaluation encoder before I would trust the semantic-similarity conclusion.\n\nThere are also smaller issues. The text says Gene-R1 'consistently outperforms all baselines,' but Table 4 shows the 1B variant below GPT-4o on GO:BP semantic similarity, and the 3B variant below o1 on GO:CC. The 3B model also fails to produce the correct output format on over half of GO:BP sets, yet this is buried in the error analysis. No code, data, or seeds are released, and there are no run-level error bars. All of these are fixable, and I would not call any of them fatal.\n\nThis paper deserves a serious referee. The central idea is sound, the 8B results are meaningful, and the OOD robustness is a real finding. But the authors should be asked to release artifacts, add a held-out semantic encoder or per-encoder results, and soften the 'consistently outperforms' language. I would not cite the current version in a paper that depends on the semantic-similarity result, but I would certainly follow the revision.","headline":"Gene-R1 is a genuinely useful three-stage fine-tuning pipeline for lightweight LLMs on gene set analysis, and the 8B model's ROUGE gains are real; but the semantic-similarity headline is muddied by using MedCPT both as a training reward and inside the evaluation metric.","tokens_in":13325,"tokens_out":2243,"would_cite":true,"duration_ms":26837,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gene-R1 is a three-stage data-augmented fine-tuning framework that lets lightweight open-source LLMs match commercial reasoning models on gene set functional annotation.","keywords":["gene set analysis","functional annotation","large language models","reasoning","data augmentation","reinforcement learning","GRPO","Gene Ontology"],"falsifier":"Re-run the in-distribution and out-of-distribution evaluations using a semantic similarity encoder that was never used in training (for example, a freshly fine-tuned bioencoder or a blind LLM judge), and compare Gene-R1 with GPT-4 on both ROUGE and semantic similarity. If the similarity advantage disappears while the ROUGE gains persist, the claim that task-specific reasoning closes the semantic gap is not supported.","tokens_in":12352,"feed_emoji":"🧬","tokens_out":4817,"duration_ms":49060,"temperature":0.7,"pith_summary":"The paper aims to show that a lightweight, openly available LLM can perform gene set analysis—naming the biological function of a gene list and explaining it—as well as commercial reasoning models like GPT-4 and o1. It introduces Gene-R1, a three-stage fine-tuning recipe: warm up the model on gene-centric relational knowledge, activate step-by-step reasoning on traces distilled from GPT-o1, then align the output to gold-standard labels with reinforcement learning. On 1,508 in-distribution GO gene sets, Gene-R1 with an 8-billion-parameter Llama backbone outperforms all baselines on ROUGE and semantic similarity; on 106 out-of-distribution sets it is statistically indistinguishable from GPT-4 and Llama3.3-70B. The practical significance is that institutions with sensitive or proprietary gene-expression data could deploy local, low-cost models without sacrificing annotation quality.","feed_headline":"Lightweight open LLM ties GPT-4 on gene-set analysis","feed_subtitle":"Three-stage training adds domain knowledge, reasoning traces, and RL to close the gap on 1,614 gene sets.","key_machinery":"The load-bearing mechanism is the three-stage fine-tuning sequence: (1) knowledge warm-up, where the backbone is pre-trained on gene-centric relational sentences from databases so gene symbols become meaningful rather than arbitrary strings; (2) reasoning activation, where the warmed-up model is fine-tuned on GPT-o1-generated step-by-step reasoning traces, retaining only those with similarity greater than 0.7 to the gold label; and (3) task alignment, where GRPO reinforcement learning optimizes a dual reward—soft-match semantic/lexical similarity to the gold annotation and exact-match structural formatting. The soft-match reward integrates MedCPT for semantic similarity and longest common su","core_discovery":"Gene-R1's central claim is that a carefully ordered data-augmentation pipeline—domain-knowledge pre-training, supervised reasoning distillation filtered by similarity to gold labels, and GRPO with a soft-match reward combining MedCPT semantic similarity with longest-common-subsequence lexical overlap—can close the performance gap between open-weight models and proprietary reasoning LLMs on gene set functional annotation. The strongest in-distribution numbers occur on GO:BP, where Gene-R1 (8B) reaches ROUGE-L 0.277 and average semantic similarity 0.647 versus GPT-4's 0.184 and 0.614, with improvements reported as significant at p<0.05. On 106 out-of-distribution gene sets from NeST and MSigDB","pith_inferences":["An implication the authors leave implicit is that the evaluation's semantic similarity may partially overlap with the training reward: the soft-match reward in Section 3 uses MedCPT and LSC, while the evaluation in Section 4.2 averages MedCPT, SentenceBERT, and SapBERT; an independent encoder would isolate how much of the reported semantic gain is genuine reasoning rather than reward overfitting.","A natural extension the paper does not test is shifting the same three-stage pipeline to other label spaces such as disease ontology or phenotype ontology; the authors themselves list these as open questions.","The cascade knowledge-injection order (knowledge warm-up followed by reasoning activation) yields only modest, mostly non-significant gains over a merged single-stage injection, suggesting that the explicit warm-up phase may be less critical than the reinforcement-learning alignment phase.","Because the reasoning traces are distilled from GPT-o1 and filtered by similarity, the ceiling of Gene-R1 is partly bounded by the teacher's reasoning quality and coverage; replacing those traces with independently validated reasoning annotations could raise performance further."],"forward_implications":["Gene-R1 (8B) raises ROUGE-L to 0.277 on GO:BP versus GPT-4's 0.184 and average semantic similarity to 0.647 versus 0.614, with improvements significant at p<0.05.","On 106 out-of-distribution gene sets, Gene-R1's similarity scores show no significant difference from GPT-4 and Llama3.3-70B, supporting generalization across gene sources.","Ablations show that each module contributes, and that the task-alignment stage with soft-match GRPO rewards yields the largest gains, outperforming online DPO and exact-match-only GRPO.","Performance gains hold across 1B, 3B, and 8B Llama backbones, with over 15% similarity-score improvement over base models on every in-distribution dataset.","Because the final model is small and open, it can be deployed locally, reducing the cost and privacy concerns associated with sending gene-expression data to commercial APIs."],"fun_headline_variants":["Open-source 8B model matches GPT-4 on gene set analysis","Open LLM with reasoning training ties GPT-4 on gene sets","Gene-R1: open 8B rivals GPT-4 in gene set analysis","Reasoning recipe lets open 8B match GPT-4 on gene tasks","Lightweight open LLM achieves GPT-4 parity in gene analysis"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that optimizing MedCPT/LSC similarity during task-alignment training (Section 3) teaches biological reasoning rather than overfitting the same semantic encoders used in evaluation (Section 4.2); if that premise gives way, the reported semantic gap-closing is inflated, although the ROUGE gains remain partly independent support.","fun_headline_variants_meta":{"raw":{"variants":["Open-source 8B model matches GPT-4 on gene set analysis","Open LLM with reasoning training ties GPT-4 on gene sets","Gene-R1: open 8B rivals GPT-4 in gene set analysis","Reasoning recipe lets open 8B match GPT-4 on gene tasks","Lightweight open LLM achieves GPT-4 parity in gene analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3689,"prompt_tokens":711,"completion_tokens":2978,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":2881}},"tokens_in":455,"tokens_out":2978,"duration_ms":21762,"temperature":1.0,"reasoning_tokens":2881,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:44:54.055772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the in-distribution and out-of-distribution evaluations using a semantic similarity encoder that was never used in training (for example, a freshly fine-tuned bioencoder or a blind LLM judge), and compare Gene-R1 with GPT-4 on both ROUGE and semantic similarity. If the similarity advantage disappears while the ROUGE gains persist, the claim that task-specific reasoning closes the semantic gap is not supported.","supporting_citations":[],"review_version":1}