{"id":"9c0a123a-d8d1-4678-a65d-f53d669ec5a4","arxiv_id":"2411.15700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAMIE, a retrieval-augmented multi-task instruction-tuned framework, improves LLM information extraction for dietary supplements from clinical records, with RAG recovering accuracy lost in multi-task training.","lead":"Researchers tested a framework called RAMIE that makes one large language model handle four information-extraction jobs on dietary supplement notes from electronic health records. It combines multi-task training with retrieval of similar examples, and the paper reports higher accuracy than single-task fine-tuning on several tasks while training one model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RAG benefit hinges on test-time retrieval not surfacing near-duplicate training sentences; the paper reports no deduplication or similarity analysis, so the improvement may be memorization rather than generalization.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: test-time retrieval from the training set may exploit dataset redundancy, so the reported RAG improvements may not reflect generalization. I agree with that assessment. The paper's central empirical claim, 'RAG significantly boosted overall accuracy,' depends on the retrieved examples being semantically similar but not near-duplicates with known answers. The paper provides no deduplication analysis, no similarity statistics between train and test, and no exclusion of near-duplicate sentences during retrieval. Given that the data come from a single clinical repository with templated phrasing, this is a concrete, testable threat to the conclusion. The paper does have independent support: it reports full result tables, runs 96 experimental configurations, and provides an error analysis; the MTL-only ablation shows a modest drop consistent with prior work. These are genuine contributions. However, the central RAG claim cannot be accepted as stated until the leakage/memorization hypothesis is ruled out. The reader's CONDITIONAL verdict is therefore appropriate; my stress test does not change it. I recommend keeping the verdict CONDITIONAL and requiring the deduplication/similarity analysis as a condition of acceptance.","tokens_in":14580,"tokens_out":2456,"duration_ms":24540,"concrete_test":"Compute the distribution of cosine similarities (using the same embedding model as the retrievers, e.g., Contriever) between each test sentence and its retrieved training example, and also measure the exact-match rate of gold labels between each test instance and its retrieved example. If a substantial fraction of test instances have similarity above 0.9 or identical gold outputs, rerun the RAMIE evaluation with a deduplicated training set from which all near-duplicates (e.g., pairwise similarity > 0.8) of each test sentence have been removed. If the RAG-versus-no-RAG F1 gap on the deduplicated set shrinks by more than a relative 30% (or the reported improvements reverse), the claimed RAG benefit is driven by memorization rather than generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that RAG significantly boosts overall accuracy (abstract, Section 3.3, Fig. 2). This claim rests on the assumption that the retrieved training examples provide helpful analogies rather than near-duplicate answers. Section 2.3 B states that during testing the retriever may select the most relevant example from the entire training set, and that there is no overlap between the input and retrieved examples. However, 'no overlap' only means the exact test sentence is not in the training set; it does not exclude near-duplicate sentences from the same clinical note repository. The datasets are derived from a single CDR with templated clinical language, and no deduplication or sentence-level similarity analysis between train and test is reported. If a test sentence retrieves a training sentence with nearly identical wording and the same gold labels, the model can copy the response, inflating the RAG scores and making the comparison against the no-RAG ablation (which has no such crutch) invalid. The same risk applies during training, where only the exact input sentence is excluded from retrieval, not near-duplicates. Thus the headline 'RAG significantly boosted overall accuracy' may be an artifact of dataset redundancy rather than evidence of improved generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAMIE, a retrieval-augmented multi-task instruction fine-tuning framework for extracting dietary supplement information from clinical text, covering named entity recognition, relation extraction, triple extraction, and usage classification. The authors evaluate eight LLMs under three settings: single-task instruction fine-tuning, the full RAMIE framework (multi-task learning plus retrieval-augmented generation), and multi-task instruction fine-tuning without retrieval. The main claimed findings are that RAG consistently improves accuracy over multi-task learning without RAG, that multi-task learning incurs only a small performance drop while saving storage and training cost, and that RAMIE yields per-task F1 scores up to 87.39 (NER), 93.74 (RE), 79.45 (TE), and 93.45 (UC). The paper includes an ablation study with three retrievers plus a random-example baseline.","tokens_in":14711,"tokens_out":6087,"duration_ms":53417,"significance":"If the central claim is correct, the paper is a useful engineering contribution: it shows that a single instruction-tuned LLM can handle four clinical information extraction tasks with a modest performance trade-off, and that retrieving similar training examples at inference time can help. The internal comparisons are transparent, the RAG-versus-no-RAG ablation is consistently positive across all eight models, and the inclusion of a random-example baseline is a good control. The main caveats are that the evaluation is built on single-run point estimates, the headline results are selected across different retriever configurations, and the RAG benefit may be inflated by near-duplicate training/test sentences from the same clinical data repository. These issues are addressable with additional analysis, so the paper's core idea remains plausible but is not yet fully supported.","major_comments":[{"comment":"The central claim that RAG significantly boosted overall accuracy (abstract, Section 3.3, Fig. 2) depends on the retrieved training examples being helpful analogies rather than near-duplicate answers. The paper says that during testing the retriever may select from the entire training set and that there is no overlap between input and retrieved examples, but 'no overlap' only excludes exact sentence identity. The datasets are drawn from a single CDR with templated clinical language, and the paper reports no deduplication or sentence-level similarity analysis between training and test sentences. If a test sentence retrieves a training sentence with nearly identical wording and the same gold labels, the model can copy the response, inflating the RAG scores and invalidating the comparison with the no-RAG ablation. I request: (a) the distribution of cosine similarities between each test sentence and its retrieved training sentence; (b) an ablation that removes or thresholds near-duplicate retrieved examples; and (c) a report of the RAG gain restricted to test instances whose retrieved neighbor is not near-duplicate. Without this analysis, the headline RAG improvement is not distinguished from memorization.","section":"Section 2.3 B, Tables 3 and 4"},{"comment":"The headline numbers in the abstract are not produced by a single RAMIE configuration. Llama2-13B's NER F1 of 87.39 comes from the MedCPT row, while the same model's RE F1 of 93.74 comes from the BMRetriever row; Llama2-7B's TE F1 of 79.45 is from the BMRetriever row, and MedAlpaca-7B's UC F1 of 93.45 is from the Contriever row. Thus the abstract's phrasing 'with the aid of the RAMIE framework, Llama2-13B achieved...' is misleading because no single model-plus-retriever combination attains all of these scores. The comparison against single-task fine-tuning and against MTL without RAG should be made for a fixed configuration (for example, the retriever selected on the development set), or the paper should explicitly state that the per-task results are oracle selections across retrievers and models. As reported, the performance advantage of RAMIE is overstated relative to any single deployable instantiation of the framework.","section":"Abstract and Table 3"},{"comment":"All results in Tables 2-4 appear to be single-run point estimates, with no confidence intervals, standard deviations, or significance tests. The abstract and Section 3.3 use the word 'significantly' ('RAG significantly boosted overall accuracy'), but the observed differences are often small (for example, 0.94% for UC) and could be within sampling noise. I request that the authors either run multiple seeds and report mean and variance, or soften the language to 'consistently higher in these experiments' with an explicit caveat that the differences were not assessed for statistical significance. This is important because the central claim is a quantitative comparison between conditions.","section":"Section 2.4 and Section 3.3"}],"minor_comments":[{"comment":"The improvement percentages in the abstract appear to be computed as (new - old) / new, whereas the performance-drop percentages in Table 4 are computed as (single - multi) / single. For example, the reported 14.26% TE improvement for Llama2-7B corresponds to (79.45 - 68.12) / 79.45, not the conventional relative improvement (79.45 - 68.12) / 68.12 = 16.6%. Please use a single, clearly stated convention, preferably relative to the baseline.","section":"Abstract and Table 4"},{"comment":"The header of Table 3 lists 'PMC-Llama-7B', but the methods text and Table 2 refer to PMC-Llama-13B. Please correct this inconsistency.","section":"Table 3"},{"comment":"The manuscript does not mention release of code or data. Since the paper claims to be the first benchmark work for LLM-based DS information extraction, releasing the split datasets and evaluation scripts would substantially aid reproducibility and comparison by future work.","section":"General"},{"comment":"There are several typographical errors, including 'Recongition' instead of 'Recognition' in the table captions, 'The motivation for to proposed RAMIE framework' in Section 2.3, and the unfinished question 'The relationship between melatonin and tinnitus is?' in the Table 1 RE example.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the internal ablation is consistently positive, but the evaluation currently does not rule out near-duplicate memorization, and the abstract's best numbers are assembled from different retriever configurations. Both issues are fixable with additional analysis and more careful reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRAMIE is a benchmark paper for a good applied question: can one instruction-tuned LLM handle four dietary supplement IE tasks (NER, RE, TE, UC) at near-single-task accuracy, and does retrieval augmentation close the gap left by multi-task training? The empirical pattern is visible and plausible: MTL alone costs a few F1 points, and adding retrieved training examples brings most of it back. The random-example control is a nice touch—it shows the benefit is not from in-context examples per se but from relevant ones. That is the paper's real contribution, and it is a legitimate one for the clinical NLP audience.\n\nThe execution is where I'd push back. All results are single-run point estimates with no error bars or significance tests, so the reported F1 gaps are hard to weight. The abstract's improvement percentages don't match the tables under any obvious denominator. The text and tables disagree on the best retriever for two of the eight models, and 'PMC-Llama-7B' appears in Table 3 while the methods say 13B. These are fixable but they make the paper look sloppy.\n\nThe bigger concern is the test-time RAG setup. The retriever pulls the most similar training sentence for each test input, and the paper does not report any sentence-level deduplication or similarity analysis between train and test. Given these are small annotated sets drawn from one clinical note repository, near-duplicate templated sentences are likely. If so, the retrieved example can carry the answer, and the RAG-versus-no-RAG comparison becomes a memorization test rather than a generalization test. The authors exclude the exact input sentence during training but that does not address near-duplicates. This needs to be addressed with a similarity/dedup analysis before the headline claim is convincing.\n\nBottom line: the framework is a new combination of known parts, applied to a domain that needs it. For a reviewer, the paper deserves a close look, but as it stands I would not cite the RAG result as evidence of generalization. It is for the dietary supplement IE community; a broader NLP audience will find the MTL/RAG ablation interesting but would want the leakage question resolved.\n\nRecommendation: send to peer review with a request for repeated runs, a dedup analysis, and cleaned-up reporting.\n\nLet me know if you want to chat about it.","headline":"A plausible multi-task/RAG framework for dietary supplement IE, but the RAG benefit may be partly memorization and the reporting needs cleanup.","tokens_in":15335,"tokens_out":3793,"would_cite":false,"duration_ms":31493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single instruction-tuned LLM whose prompts are augmented with retrieved training examples can extract named entities, relations, triples, and usage status from dietary-supplement notes at F1 scores that match or beat single-task models.","keywords":["dietary supplements","information extraction","multi-task learning","retrieval-augmented generation","large language models","clinical records","instruction fine-tuning","named entity recognition"],"falsifier":"Recompute the reported RAG gains after deduplicating or removing training sentences that are near-duplicates of test sentences; if the advantage of retrieved examples over random examples disappears, the claimed RAG boost is an artifact of dataset redundancy rather than retrieval-based generalization.","tokens_in":14280,"feed_emoji":"💊","tokens_out":9431,"duration_ms":77746,"temperature":0.7,"pith_summary":"Dietary supplement information sits in unstructured clinical notes, and extracting it cleanly usually requires separate models for different jobs. This paper proposes RAMIE, a framework that trains one instruction-tuned large language model to do four jobs at once: name supplement and adverse-event entities, classify relations between them, output subject-predicate-object triples, and label whether supplement use started, continued, was discontinued, or is uncertain. The central claim is that retrieval augmentation—showing the model a semantically similar sentence-response pair retrieved from the training set at both training and inference time—raises accuracy, while multi-task learning keeps the whole system to a single cheaper model. In the reported experiments, the best RAMIE setups reach F1 scores of 87.39 on entity recognition, 93.74 on relation extraction, 79.45 on triple extraction, and 93.45 on usage classification, generally matching or beating the same models trained one task at a time.","feed_headline":"One LLM, four tasks: RAG lifts F1 on supplement notes","feed_subtitle":"Retrieval-augmented multi-task model matches single-task baselines on four dietary-supplement extraction tasks.","key_machinery":"The load-bearing mechanism is the retrieval-augmented prompt built from the training set. For each incoming sentence, a retriever compares sentence embeddings by cosine similarity and selects the closest sentence-response pair from the same task's training split; that example is inserted into a task-specific instruction prompt, and the model is fine-tuned with low-rank adaptation on these prompts. Retrieval is restricted during training so the model cannot copy the input sentence's own answer, but at test time the retriever searches the full training set. Three dense retrievers—MedCPT, Contriever, and BMRetriever—supply the similarity search, and instruction fine-tuning supplies the task-switching behavior, so one model can answer all four output formats.","core_discovery":"On the paper's own terms, the discovery is that retrieval-augmented generation is what makes multi-task instruction fine-tuning affordable for dietary-supplement information extraction. With instruction fine-tuning alone, training one model on all four tasks costs an average F1 drop of about 2.72% relative to single-task training, with the largest loss on triple extraction; when the prompt is augmented with a retrieved training example, many of those losses disappear. A random example does not help and can hurt, but each of the three dense retrievers consistently improves scores over the no-retrieval baseline. Across 96 model-retriever-task setups, 46 beat the corresponding single-task fine-tuned model, and the best per-task numbers are 87.39 F1 for NER, 93.74 for RE, 79.45 for TE, and 93.45 for UC. The paper concludes that a single RAMIE-tuned model is comparable to single-task models while being much cheaper to store and train.","pith_inferences":["A stricter test of the RAG claim would split the data by patient or document rather than by sentence; if the retrieval gain vanishes, the retriever is exploiting repeated phrasing instead of improving generalization.","A retriever fine-tuned on dietary-supplement text could push performance further, since all three retrievers used here are general biomedical tools and the paper notes none is supplement-specific.","The consistently larger multi-task drop on triple extraction suggests grouping tasks by output format or giving triple extraction extra capacity could reduce negative transfer beyond what RAG already repairs.","The finding that random examples hurt while retrieved examples help implies that example relevance, not just few-shot formatting, drives the improvement, so retrieval quality thresholds deserve explicit study."],"forward_implications":["A single RAMIE-tuned LLM can replace four separately trained information-extraction models, cutting storage and training cost in a clinical NLP pipeline.","Retrieval-augmented prompts with dense retrievers beat both zero-shot and random-example prompting, so the same recipe is worth testing in other biomedical extraction tasks.","Multi-task learning alone costs a small F1 drop, but pairing it with retrieval augmentation recovers most of the loss, making multi-task deployment practical.","Relation extraction is the most multi-task-friendly task, while triple extraction is the persistent bottleneck.","Because RAMIE is not tied to dietary supplement vocabulary, the framework is claimed to transfer to other domains with annotated training sets."],"supporting_citations":[{"why":"Supplies the four annotated dietary-supplement datasets from clinical notes and the entity and relation definitions the experiments use.","marker":"17"},{"why":"Provides the instruction-prompt template for each task that RAMIE adapts for its prompts.","marker":"19"},{"why":"Benchmarks retrieval-augmented LLMs in biomedical NLP and motivates the RAG-with-retrievers design.","marker":"20"},{"why":"Defines the usage-classification task and dataset that RAMIE includes as one of its four tasks.","marker":"13"},{"why":"Provides the multi-task BERT baseline and the precedent that multi-task learning trades a little performance for efficiency.","marker":"48"},{"why":"Introduces retrieval-augmented generation as the mechanism RAMIE builds on.","marker":"49"},{"why":"Supplies MedCPT, one of the three retrievers used to select similar training examples.","marker":"57"},{"why":"Supplies Contriever, one of the three retrievers used to select similar training examples.","marker":"58"},{"why":"Supplies BMRetriever, one of the three retrievers used to select similar training examples.","marker":"59"}],"fun_headline_variants":["Retrieval-augmented training recovers multi-task F1 in supplement extraction","RAG makes one model match single-task F1 on all four supplement tasks","Retrieval augmentation lifts supplement extraction F1 across four tasks","A retrieved example beats fine-tuning alone for multi-task supplement IE","RAG cancels the multi-task cost in dietary supplement extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The RAG benefit assumes that the retrieved nearest training example at test time is helping the model learn the task, rather than giving away the answer from a near-duplicate sentence; the paper does not report how similar train and test sentences are.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-augmented training recovers multi-task F1 in supplement extraction","RAG makes one model match single-task F1 on all four supplement tasks","Retrieval augmentation lifts supplement extraction F1 across four tasks","A retrieved example beats fine-tuning alone for multi-task supplement IE","RAG cancels the multi-task cost in dietary supplement extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2870,"prompt_tokens":1133,"completion_tokens":1737,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":749,"completion_tokens_details":{"reasoning_tokens":1644}},"tokens_in":749,"tokens_out":1737,"duration_ms":11970,"temperature":1.0,"reasoning_tokens":1644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:00:26.092520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the reported RAG gains after deduplicating or removing training sentences that are near-duplicates of test sentences; if the advantage of retrieved examples over random examples disappears, the claimed RAG boost is an artifact of dataset redundancy rather than retrieval-based generalization.","supporting_citations":[{"cited_title":"& Zhang, R","cited_arxiv_id":null,"evidence_quote":"Supplies the four annotated dietary-supplement datasets from clinical notes and the entity and relation definitions the experiments use."},{"cited_title":"& Zhang, R","cited_arxiv_id":null,"evidence_quote":"Provides the instruction-prompt template for each task that RAMIE adapts for its prompts."},{"cited_title":"& Zhang, R","cited_arxiv_id":null,"evidence_quote":"Defines the usage-classification task and dataset that RAMIE includes as one of its four tasks."},{"cited_title":"& McInnes, B","cited_arxiv_id":null,"evidence_quote":"Provides the multi-task BERT baseline and the precedent that multi-task learning trades a little performance for efficiency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MedCPT, one of the three retrievers used to select similar training examples."}],"review_version":1}