{"id":"14fcd10f-1060-4d12-9ce2-5f6743f9d4c0","arxiv_id":"2506.08647","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using LLM-generated summaries as input improves instruction-tuned generative relation extraction in the low-resource microbiome domain, though fine-tuned BERT models remain more accurate.","lead":"This paper tests whether asking a large language model to first summarize a scientific passage helps a second model extract relationships between microbes, diseases, and other biomedical concepts. The summaries did help the generative models, but standard BERT-style classifiers still got higher scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of summarization is confounded: the summarization-based pipeline also adds instruction tuning, removes the class list from the prompt, and uses a different extractor setup, so no condition isolates summarization's contribution.","rationale":"The reader's weakest_assumption identifies exactly the same confound: the two compared conditions differ by more than the presence of summaries. This is the single most load-bearing concern because the paper's headline claim is about summarization, yet the experimental design cannot distinguish summarization from instruction tuning or prompt changes. The reader's CONDITIONAL verdict is appropriate: the claim is plausible but under-supported, and a specific ablation would settle it. I agree with the reader's analysis and do not think the verdict should change. I also note that the summary-quality evaluation (Section 5.2) only measures similarity to the original text, not whether summarization is necessary for RE accuracy; that is consistent with the confound rather than evidence for the claim. The paper is otherwise a reasonable empirical study, but the central attribution needs the missing control condition.","tokens_in":11525,"tokens_out":3116,"duration_ms":36175,"concrete_test":"Run the missing ablation on Llama 3.2-3B-Instruct: instruction-tune with LoRA using the exact summarization-based prompt (system role, task instruction, no class list) and the same hyperparameters as reported, but feed the original unsummarized passages during both training and inference. Evaluate weighted F1 on the same test set. If the no-summarization condition achieves F1 within a few points of 59.7%, then the improvement is not caused by summarization; if it drops substantially (e.g., more than 10 points), the claim gains support. Repeat with at least three random seeds to assess variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states that 'summarization improves generative RE performance by reducing noise and guiding the model.' The evidence for this is Table 1, where Llama 3.2-3B-Instruct rises from 3.78% weighted F1 in the direct approach to 59.7% in the summarization-based approach. However, the direct approach (Section 3.1) is zero-shot, uses a prompt that includes the full list of 23 relation classes, and is evaluated without any fine-tuning. The summarization-based approach (Section 3.2) adds a LoRA instruction-tuning step, omits the class list from the prompt, and changes the system role and task instruction. The paper presents no ablation that instruction-tunes the same model on the original passages without summaries. Therefore, the measured gain could be due to instruction tuning alone, the removal of the class list, the different prompt, or any combination, rather than to summarization. The claim that summarization is the cause is load-bearing because the paper's central contribution is the summarization component; without an ablation isolating it, the 55-point F1 jump is not attributable to summarization. The paper's own wording in the contributions and Section 5.1 sometimes says 'combining instruction tuning with summarization,' which is more defensible, but the abstract and main discussion overattribute to summarization. Additionally, the summarizer is Llama 3.1, a substantially larger model that may itself perform most of the relational reasoning, leaving the tuned extractor to merely map summary text to labels; this possibility is not tested. The error analysis in Section 5.3 manually inspects only 30 examples and does not address this confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generative relation extraction (RE) pipeline for the low-resource microbiome domain. Two strategies are compared: a zero-shot direct prompting approach and a two-step summarization-based approach in which an LLM first generates a concise summary of the entity-pair context and a second, instruction-tuned model then predicts the relation. Experiments on the MicrobioRel corpus show that the direct zero-shot approach performs poorly (e.g., Llama 3.2-3B-Instruct reaches 3.78% weighted F1), while the summarization-based approach with instruction tuning reaches 59.7% for the same model, still below fine-tuned BERT encoders such as PubMedBERT at 71.3%. The paper also reports summary-quality metrics (cosine similarity and BERTScore) and a small manual error analysis of 30 misclassified examples.","tokens_in":11729,"tokens_out":3984,"duration_ms":50483,"significance":"If the central causal claim were established, this would be a useful contribution to low-resource multi-class generative RE: the two-step pipeline is simple, the target domain is genuinely low-resource, and the comparison against strong BERT baselines is informative. The error analysis is a strength, as it distinguishes summarization-stage errors from extraction-stage errors and acknowledges the role of automatic 'None' labels. The manuscript is honest about its limitations and does not overclaim parity with BERT. However, the headline result is not currently attributable to the summarization step because the compared conditions differ in multiple ways, and all results come from single runs without variance or significance testing. The paper ships no code and does not report hyperparameters for the LoRA tuning, which limits reproducibility. The contribution is therefore promising but not yet established.","major_comments":[{"comment":"The central claim that 'summarization improves generative RE performance' is not isolated by the experimental design. The direct approach (Section 3.1) uses zero-shot prompting and includes the full list of 23 relation classes in the prompt, while the summarization-based approach (Section 3.2) adds a LoRA instruction-tuning step, removes the class list, changes the system role and task instruction, and uses a different input (the generated summary). The 3.78% to 59.7% jump in Table 1 therefore conflates summarization with instruction tuning and prompt changes. No ablation instruction-tunes the same model on the original passages without summaries. The authors should add such an ablation, ideally with matched prompts and matched tuning budgets, to support the abstract's attribution. Without this, the paper's own more cautious phrasing in Section 5.1, 'combining instruction tuning with summarization,' is the only defensible statement.","section":"§3.1–3.2, §5.1, Table 1"},{"comment":"All reported metrics are single runs with no variance, confidence intervals, or significance tests. The difference between 59.7% and 44.6% or between 71.3% and 70.8% may be within noise, particularly given the small test set and the stochastic nature of LLM decoding. The paper should report multiple seeds or decoding runs with means and standard deviations, especially for the generative models. In addition, the weighted F1 metric can be dominated by the majority classes; reporting macro F1 or per-class breakdowns would strengthen the comparison, since the 'None' class and 'Associated_with' are very frequent in MicrobioRel (Table 2 in Appendix A).","section":"§5.1, Table 1; §4.2"},{"comment":"The summarization step itself is performed by Llama 3.1 in a few-shot setting, but the prompt, the demonstration examples, and the decoding parameters are not described, and no details are given for the LoRA configuration (rank, alpha, batch size, learning rate, epochs) beyond 'configurations adapted to each model's architecture.' This makes the experiments difficult to reproduce. Moreover, the error analysis in Section 5.3 shows that the generated summaries sometimes contain explicit relation statements (e.g., the Class 3 example contains 'MPTP-induced neurotoxicity can damage nigral dopaminergic neurons'), so the summarizer may be performing part of the relational reasoning itself. The paper should acknowledge and ideally measure this possibility, because it bears directly on the claimed mechanism of 'reducing noise and guiding the model.'","section":"§4.2, §5.2, §5.3"}],"minor_comments":[{"comment":"There are several typos and grammatical issues: 'low-resources setting' in the abstract, 'as been treated' in Section 2, 'You role' in the prompt in Section 3.2, and 'Parkison’s disease' in Section 5.3. A careful proofreading pass is needed.","section":"Abstract and §2"},{"comment":"The paper alternates between '22 relation types' and '23 relation classes'; clarify that the 23rd class is the automatically introduced 'None' class. Also, the prompt in Section 3.1 lists 'Marker/Mechanism' while Appendix A uses 'Marker-Mechanism'; the label string should be consistent because exact-match evaluation depends on it.","section":"§3.1, §4.1, Appendix A"},{"comment":"Equation (1) is fine as a definition, but the notation 'support_i' and 'total support' should be defined explicitly, and the statement that weighted F1 is derived from per-class F1 scores (not from weighted precision and recall) should be justified or removed, since it is an unusual choice and affects comparability with other work.","section":"§4.3, Equation (1)"},{"comment":"Figure 1 is unclear: it does not specify which textual pairs are plotted (summaries vs. original passages? train, validation, or test split?), and the text mentions a 'moderate to strong positive Pearson correlation' but reports no correlation coefficient. Provide the r value, the number of points, and a description of the pairs.","section":"§5.2, Figure 1"},{"comment":"The observation that 'longer original passages tend to slightly reduce similarity scores' and 'longer summaries are weakly associated with higher similarity' is made without quantitative support. Report the correlation coefficients or regression slopes, or remove the claim.","section":"§5.2"},{"comment":"The Unsloth framework is mentioned only via a URL; it should be cited formally or at least described with a version. The paper also does not state whether code or model checkpoints will be released, which is relevant for the reproducibility claims.","section":"§4.2, References"},{"comment":"The caption says 'other rows correspond to generative models used with or without instruction tuning,' but the 'Summarization-Based' rows all include instruction tuning, so the caption is misleading. Clarify that the direct rows are zero-shot and the summarization rows are instruction-tuned.","section":"Table 1 caption"},{"comment":"The limitations section appropriately notes the absence of comparisons with other generative RE approaches, but given the paper's claim of 'comprehensive comparison' in the contributions, this limitation should be reflected in the contribution statement as well, not only in the limitations section.","section":"§7 Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an 'ongoing work' paper and its main contribution is currently undermined by a confound that a single ablation could resolve. The authors should be encouraged to add the missing ablation and multi-run results. I would not recommend rejection because the domain, the dataset, and the error analysis are of interest, and the paper is honest about its limitations. However, the current version would not meet the bar for a journal publication, as the abstract and discussion attribute the performance gain to summarization without the necessary control condition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a preliminary empirical study, clearly written, about using LLM summarization before instruction-tuned relation extraction on a low-resource microbiome corpus. The headline result — Llama 3.2-3B going from 3.78% to 59.7% F1 when you add summarization — is real in their table, but it is not cleanly attributable to summarization, because the comparison condition also adds instruction tuning and drops the class list from the prompt. The paper itself occasionally says 'combining instruction tuning with summarization,' which is honest; the abstract and discussion overstate the causal role of summarization.\n\nWhat's genuinely useful: the application of summarization-before-RE to a specialized 23-class biomedical relation task is new, and the small error taxonomy (errors from summaries, errors from the RE model, plausible-alternative errors) is a nice analytical contribution, even with only 30 hand-checked examples. They are also honest about BERT models still beating generative ones, and they list plausible limitations.\n\nThe soft spots are real. There is no ablation that instruction-tunes the same model on original passages without summaries, so the 55-point jump could come from instruction tuning alone, from removing the class-list constraint, or from the larger Llama 3.1 summarizer doing the relational reasoning itself. The paper does not test whether the summarizer already encodes the answer. All metrics are single runs with no variance or significance tests, which matters for a small dataset (1,994 instances). Missing code and configs also make the exact setup hard to reproduce. These are not minor nitpicks; they directly bear on whether the central claim holds.\n\nWho this is for: people working on cheap generative RE in low-resource biomedical settings might want the recipe, but they should read it as a hypothesis, not a demonstrated effect. It deserves a serious referee — the method is simple, the paper is short, and a referee can demand the missing ablation and variance reporting. If I were editing, I'd send it out but tell the authors the current attribution needs to be fixed.","headline":"A clearly written empirical recipe that likely helps, but the claimed benefit of summarization is not isolated from instruction tuning, so the headline attribution is under-supported.","tokens_in":12394,"tokens_out":2516,"would_cite":false,"duration_ms":25129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that summarizing a passage around a target entity pair before instruction-tuned classification lifts generative relation extraction on MicrobioRel from 3.78% to 59.7% weighted F1.","keywords":["generative relation extraction","summarization","instruction tuning","microbiome","low-resource NLP","relation classification","large language models","biomedical text mining"],"falsifier":"Run the summarization-based pipeline again with the summarizer removed but with the same instruction-tuned model, the same prompt without a class list, and the same MicrobioRel test set; if weighted F1 stays near 59.7%, summarization is not the source of the gain, and if it drops toward 3.78%, summarization is doing the heavy lifting.","tokens_in":11234,"feed_emoji":"🦠","tokens_out":9292,"duration_ms":89065,"temperature":0.7,"pith_summary":"This paper tries to show that generative large language models can do multi-class relation extraction in a specialized low-resource biomedical domain if the input is first summarized. The authors build a two-step pipeline—summarize the passage around the two entities, then instruction-tune a smaller LLM to classify the relation—and test it on MicrobioRel, a manually annotated gut-microbiome corpus with 22 relation types plus a None class. The headline result is a jump from 3.78% to 59.7% weighted F1 for Llama 3.2-3B-Instruct, attributed to noise reduction and model guidance. A sympathetic reader would care because the result suggests generative extraction can be viable where annotated data is scarce, even though domain-tuned BERT encoders still hold the higher ground.","feed_headline":"Summarization lifts relation extraction from 3.78% to 59.7% F1","feed_subtitle":"A two-step generative pipeline cuts noise in microbiome texts; BERT encoders still lead.","key_machinery":"The central mechanism is a two-stage pipeline: a few-shot summarization pass in which an LLM compresses the passage around a target entity pair into a concise relation-focused summary, followed by instruction tuning of a smaller LLM that learns the relation label set from these summaries and outputs exactly one label. The summary step is designed to strip contextual noise and curb hallucinations before classification; the tuning step transfers the class vocabulary into the model so the prompt no longer needs to enumerate candidate relations.","core_discovery":"The paper's central claim is that inserting an LLM summarization step before instruction-tuned generative relation extraction substantially improves performance on a low-resource biomedical relation extraction corpus. On MicrobioRel, the summarization-based pipeline raises Llama 3.2-3B-Instruct from 3.78% weighted F1 in zero-shot to 59.7%, with fewer hallucinations and better adherence to the predefined label set; the paper attributes this gain to reduced textual noise and focused context. It also reports that BERT-based encoders, led by PubMedBERT at 71.3%, still outperform all generative models on the same test set.","pith_inferences":["An ablation that instruction-tunes the same model on original passages, with the same prompt and learned label set, would isolate how much of the 55.9-point gain comes from summarization rather than from tuning or prompt changes.","The summarizer could be made self-critical by generating several candidate summaries and selecting the one whose predicted relation is most stable across runs, rather than picking by semantic similarity to the original text.","The same recipe is likely transferable to other low-resource biomedical subdomains with multi-class relation schemas, but that transfer needs a second corpus to confirm, since MicrobioRel alone may not represent the difficulty of other relation sets."],"forward_implications":["The summarization-plus-tuning recipe takes small generative models from unusable to competitive, landing within roughly twelve F1 points of the best encoder on MicrobioRel.","Zero-shot prompting with an enumerated class list is the weak mode: every direct model scored below 15% weighted F1 and showed label bias or out-of-set hallucinations.","Error analysis points to the classifier, not the summarizer, as the main failure source, so further gains are more likely from better classification than from better summaries.","Smaller instruction-tuned models adapt better than large ones to a corpus of 1,994 annotated relations, suggesting data scale, not model scale, limits generative relation extraction in low-resource settings."],"supporting_citations":[{"why":"Supplies the entity recognition groundwork and prior corpus work behind MicrobioRel, the dataset used for every experiment.","marker":"El Khettari et al. 2023"},{"why":"Provides the Llama 3 family, including the 3.2-3B-Instruct model that reaches 59.7% F1 in the summarization pipeline.","marker":"Dubey et al. 2024"},{"why":"Provides Mistral 7B-Instruct, one of the generative models compared in zero-shot and fine-tuned settings.","marker":"Jiang et al. 2023"},{"why":"Provides BioMistral 7B, the biomedical-domain generative baseline.","marker":"Labrak et al. 2024"},{"why":"Provides BioBERT, an encoder baseline that outperforms the generative models.","marker":"Lee et al. 2020"},{"why":"Provides PubMedBERT, the best-performing baseline in the comparison at 71.3% F1.","marker":"Gu et al. 2021"},{"why":"Provides SciBERT, another encoder baseline in the comparison.","marker":"Beltagy et al. 2019"},{"why":"Provides BioLinkBERT, the remaining encoder baseline in the comparison.","marker":"Yasunaga et al. 2022"},{"why":"Provides BERTScore, used to assess how well the generated summaries preserve the original passages' semantic content.","marker":"Zhang et al. 2019"}],"fun_headline_variants":["LLM summarization boosts microbiome RE from 3.78% to 59.7%","Summarization lifts generative RE to 59.7% but BERT still tops","Microbiome RE: summarization cuts noise, lifts F1 to 59.7%","Generative RE improves 15x with LLM summarization in microbiome"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes the gain to summarization, but the direct and summarization-based conditions differ in instruction tuning, in whether the relation class list is shown, and in which model weights are used, so the summary effect is not isolated.","fun_headline_variants_meta":{"raw":{"variants":["LLM summarization boosts microbiome RE from 3.78% to 59.7%","Summarization lifts generative RE to 59.7% but BERT still tops","Microbiome RE: summarization cuts noise, lifts F1 to 59.7%","Generative RE improves 15x with LLM summarization in microbiome"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1194,"prompt_tokens":754,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":370,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":370,"tokens_out":440,"duration_ms":4785,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:05:27.104117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the summarization-based pipeline again with the summarizer removed but with the same instruction-tuned model, the same prompt without a class list, and the same MicrobioRel test set; if weighted F1 stays near 59.7%, summarization is not the source of the gain, and if it drops toward 3.78%, summarization is doing the heavy lifting.","supporting_citations":[{"cited_title":"& Chaffron S","cited_arxiv_id":null,"evidence_quote":"Supplies the entity recognition groundwork and prior corpus work behind MicrobioRel, the dataset used for every experiment."},{"cited_title":"Q., Sablayrolles A., Mensch A., Bamford C., Chaplot D","cited_arxiv_id":null,"evidence_quote":"Provides Mistral 7B-Instruct, one of the generative models compared in zero-shot and fine-tuned settings."},{"cited_title":"& Dufour R","cited_arxiv_id":null,"evidence_quote":"Provides BioMistral 7B, the biomedical-domain generative baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BioBERT, an encoder baseline that outperforms the generative models."},{"cited_title":"& Cohan A","cited_arxiv_id":null,"evidence_quote":"Provides SciBERT, another encoder baseline in the comparison."},{"cited_title":"& Liang P","cited_arxiv_id":null,"evidence_quote":"Provides BioLinkBERT, the remaining encoder baseline in the comparison."}],"review_version":1}