{"id":"ceef924d-ea2a-44f4-9d89-b6af9cc38292","arxiv_id":"2508.06504","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Retrieval-based selection of in-context examples improves few-shot biomedical named entity recognition F1 over random selection, with TF-IDF and SBERT outperforming ColBERT and DPR.","lead":"This paper tests a retrieval-augmented prompting method for few-shot biomedical named entity recognition, where each prompt's examples are chosen by similarity to the input text instead of at random. Across five biomedical datasets and three large language models, dynamic selection improved F1 scores over static prompts, with TF-IDF and SBERT retrieval working best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dynamic prompting gains are measured with a retrieval index built from the full training set, so the headline 'few-shot' claim may not transfer to genuinely limited annotation budgets.","rationale":"The reader's weakest_assumption identifies the same issue, and I agree. The internal comparison between dynamic and static prompting is fair: both use the same optimized static scaffold and differ only in how the k in-context examples are chosen, with random selection averaged over four runs. The reported improvements are internally consistent and several are outside the provided confidence intervals (e.g., GPT-4 BC5CDR 5-shot: TF-IDF 85.88 [84.53, 86.42] vs Base 78.03 [75.51, 80.02]). However, the paper's framing and title promise gains for limited-annotation settings, while the method requires a large annotated corpus to build the retrieval index and to derive dataset-specific high-frequency instances and error-analysis feedback. That gap between the claim and the experimental conditions is the most load-bearing weakness. It does not invalidate the empirical result; it narrows its scope. A reviewer should require either a re-framing of the claims or an ablation with restricted pool sizes before the paper can be accepted as a few-shot NER method. Since the reader already arrived at CONDITIONAL, no verdict change is needed.","tokens_in":26879,"tokens_out":6444,"duration_ms":70562,"concrete_test":"Run the dynamic prompting experiments with the retrieval index restricted to exactly k examples (k=5, 10, 20), matching the shot setting, and compare against random selection from the same k examples. Alternatively, vary the retrieval pool size (k, 2k, 5k, 10k, full training set) and plot the F1 gain of TF-IDF/SBERT over random selection as a function of pool size. If the gain is near zero or negative at pool size=k and increases monotonically with pool size, the 'few-shot' claim is unsupported outside the large-pool setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Dynamic prompting is claimed to improve few-shot biomedical NER, but the experimental setup does not actually operate under limited annotation. In Methods (Dynamic Prompt Engineering), the retrieval engine is 'indexed with the annotated examples from the training set'—the full training set, not a k-shot subset. The static baseline also incorporates high-frequency instances and error analysis computed from the full training set (Static Prompt Engineering and Experimental Setup). Thus 'few-shot' refers only to the number of in-context examples placed in the prompt, not the size of the annotated pool. In a genuinely few-shot deployment, with only 5–20 annotated examples available, the retrieval index would contain exactly those k examples, so TF-IDF/SBERT would merely reorder the same k candidates; the reported 7.3% and 5.6% average F1 improvements over random selection would likely shrink or disappear. The central claim is therefore scoped to the setting where a large annotated corpus already exists, which is precisely the setting the paper's motivation says it aims to avoid. This is a scope limitation on the central claim, not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies prompt engineering for few-shot biomedical NER with LLMs. It designs a static prompt with task descriptions, dataset descriptions, high-frequency entity terms, UMLS knowledge, error-analysis feedback, and k-shot examples, and evaluates GPT-3.5, GPT-4, and LLaMA 3-70B on five biomedical NER datasets. It then compares dynamic prompting, where k in-context examples are retrieved per test instance by TF-IDF, SBERT, ColBERT, or DPR, against random selection. The main reported results are that static prompting improves average F1 by 11-12% over a basic prompt, and dynamic TF-IDF/SBERT retrieval improves average F1 by 7.3% (5-shot) and 5.6% (10-shot) over random examples. The paper concludes that RAG-based dynamic prompting is useful for few-shot biomedical NER.","tokens_in":27104,"tokens_out":4808,"duration_ms":48110,"significance":"If the results held at face value, the paper would provide a practical recipe: retrieve similar examples with TF-IDF/SBERT and improve LLM NER without fine-tuning. Strengths include evaluation across five datasets, three LLMs, four retrieval mechanisms, public code, bootstrap confidence intervals, and a fair baseline (random selection averaged over four runs). The token-level input/output format is a sensible response to LLM tokenization mismatches. However, the headline 'few-shot' claim is scoped to a setting where the retrieval index and static prompt components are built from the full training set, and many reported differences have overlapping confidence intervals; these limits reduce the significance unless addressed.","major_comments":[{"comment":"The central few-shot claim is not evaluated under limited annotation. The retrieval engine is 'indexed with the annotated examples from the training set' (Dynamic Prompt Engineering), and the static base prompt's high-frequency instances are computed from the full training set (Static Prompt Engineering, component 3), while error-analysis feedback is generated from LLM predictions on training instances (component 5). Thus '5-shot/10-shot/20-shot' refers only to the number of in-context examples, not the size of the annotated pool. In a deployment with only 5-20 annotated examples, the retrieval pool would contain exactly those examples, so TF-IDF/SBERT would only reorder the same k candidates and the reported 7.3%/5.6% gains over random selection would not necessarily transfer. Please either reframe the claim as 'dynamic prompting with a pre-existing annotated corpus' or add experiments with the retrieval index and prompt components restricted to k-shot subsets.","section":"Methods: Dynamic Prompt Engineering; Experimental Setup"},{"comment":"The abstract's 'improving average F1-scores by 7.3% and 5.6%' hides that many dataset-specific confidence intervals overlap with the Base condition. For example, GPT-4 5-shot on Med-Mentions, TF-IDF 37.96 [35.90, 39.84] vs Base 37.95 [35.88, 39.90]; and GPT-4 5-shot Reddit_Impacts TF-IDF 28.47 [21.78, 35.47] vs Base 27.60 [19.43, 33.80]. Conversely, some comparisons such as GPT-4 5-shot NCBI TF-IDF 60.08 [56.70, 63.32] vs Base 46.93 [44.85, 51.58] are clearly separated. Please report per-comparison significance or effect sizes, and revise claims of 'significant improvements' accordingly.","section":"Supplementary Table 10; Results: Dynamic Prompting with RAG"},{"comment":"The error-analysis feedback component is derived from the model's own predictions on 'unseen training set instances' and then used in the same model's test prompt. Please clarify whether these instances are part of the training set (i.e., seen during retrieval indexing or high-frequency computation) or a separate development set, and whether the test set was used at any stage. As written, this component constitutes a dataset-specific fitted prompt; its contribution to the static gains (and to the dynamic base) may not generalize to a new annotation budget, especially a small one.","section":"Methods: Static Prompt Engineering, component 5"}],"minor_comments":[{"comment":"The table header says 'eight standardized biomedical datasets' but only five datasets are listed and the text says 'five distinct medical text datasets'; please correct the count.","section":"Table 3"},{"comment":"In Table 2, the shot labels (5-shot, 10-shot, 20-shot) appear as entries in a column shared with model names; moving them to row-group headers would improve readability.","section":"Table 2"},{"comment":"Dataset spellings are inconsistent: REDDIT-IMPACTS, REDDIT -IMPACTS, and Reddit_Impacts are all used; please standardize.","section":"Throughout"},{"comment":"The example output contains 'addict.I-Clinical_Impacts' with a period instead of a hyphen; check the formatting consistency of the token-label separator.","section":"Methods: Static Prompt Engineering"},{"comment":"The 5-shot paragraph lists both TF-IDF (7.28%) and SBERT (7.46%) average gains, while the abstract's '7.3% and 5.6%' refers to 5-shot and 10-shot gains; clarify the mapping to avoid confusion.","section":"Results: Dynamic Prompting with RAG"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for the journal, but the 'few-shot' framing will mislead readers unless revised. The stress-test concern about the retrieval index being built from the full training set is the main reason for major revision. I have no concerns about data or code availability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to the chase: the paper delivers a useful empirical comparison, but the headline claim about few-shot performance is scoped more narrowly than it looks. The dynamic prompting system retrieves examples from the full training set, and the static prompt's high-frequency instances and error analysis also come from that full set. So 'few-shot' means only the number of examples placed in the prompt, not the annotation budget. In a true few-shot deployment with 5–20 annotated examples, the retrieval pool would be nearly empty and the gains would likely shrink. That's a real limitation, but it doesn't invalidate the narrower finding: when you have a decent-sized training pool, selecting similar examples for ICL helps biomedical NER.\n\nWhat's actually new is the systematic comparison of TF-IDF, SBERT, ColBERT, and DPR for dynamic ICL example selection across five datasets and multiple LLMs. The consistent result that TF-IDF and SBERT beat dense retrievers is worth knowing, and it's the kind of practical detail that saves other people from expensive experiments. The static prompt decomposition is also careful—each component's contribution is measured, which is more than most prompt-engineering papers do.\n\nThat said, the statistical support is thinner than the text suggests. Looking at the supplementary CIs, many per-dataset comparisons between retrieval methods and the base have overlapping intervals, especially on noisy datasets like Reddit-Impacts. The paper's average improvements are real, but the claims about specific datasets winning should be softened. Also, there's no comparison against existing NER baselines or standard fine-tuned models, so 'state of the art' is doing a lot of work. The error analysis prompt component is generated from the model's own errors on training data and then used to boost the same model's test performance—minor, but worth a sentence of acknowledgment.\n\nThis deserves a serious referee. It's a solid empirical study with a reusable comparison, but it needs a major revision to reframe the few-shot claim honestly and to present the statistical uncertainty more carefully. I'd send it to review, but I'd expect the authors to rework the framing.","headline":"Useful empirical comparison of retrieval methods for dynamic ICL in biomedical NER, but the few-shot framing overstates the annotation budget.","tokens_in":27616,"tokens_out":2952,"would_cite":false,"duration_ms":30818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that for few-shot biomedical named entity recognition, dynamically selecting in-context examples by similarity to the input text improves F1-score over random static prompts, with TF-IDF and SBERT retrieval giving the…","keywords":["biomedical named entity recognition","few-shot learning","large language models","retrieval-augmented generation","dynamic prompting","in-context learning","TF-IDF","Sentence-BERT"],"falsifier":"Recompute the reported F1 differences using retrieval indexes built only from the same k labeled examples available under each shot setting, with the static prompt components restricted to those same k examples; if the TF-IDF and SBERT gains over random selection fall to near zero or become negative, the few-shot transfer claim would be false.","tokens_in":26695,"feed_emoji":"🩺","tokens_out":7172,"duration_ms":73067,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models do better at biomedical named entity recognition with limited supervision when the few-shot examples placed in the prompt are chosen per input by similarity to the test text rather than fixed and random. It builds a structured static prompt and then a dynamic retrieval-augmented prompt that selects the top similar annotated examples at inference, reporting consistent F1 gains across five biomedical datasets and multiple LLMs. The best retrieval methods improved average F1-scores by about 7.3% in the 5-shot setting and 5.6% in the 10-shot setting, and the structured static prompt alone improved average F1 by 11-12% over a basic prompt. If this holds, cheap similarity-based example selection would be a practical way to improve NER on clinical and social-health texts without additional manual annotation.","feed_headline":"Similar-example prompts lift biomedical NER F1 by up to 7.3%","feed_subtitle":"Retrieving the most relevant in-context examples per input beats fixed random prompts in five datasets with GPT-4 and LLaMA 3.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation applied to in-context example selection. A retrieval index is built from the training sentences; for each test input, the top annotated examples are retrieved by a similarity engine and inserted into the prompt, so the prompt changes instance-by-instance. The paper compares TF-IDF, Sentence-BERT (sentence embeddings), ColBERT (late-interaction token matching), and DPR (dual-encoder dense retrieval), and uses a token-concatenated output format such as 'token-LABEL' to keep generated labels aligned with input tokens.","core_discovery":"The central discovery, stated on the paper's own terms, is that retrieval-augmented dynamic prompting—replacing randomly selected in-context examples with examples retrieved by similarity to the input text—improves few-shot biomedical NER performance. Concretely, TF-IDF and SBERT retrieval produced the best average F1-scores, with about 7.3% and 5.6% average F1 improvements over random-example baselines in 5-shot and 10-shot settings, and GPT-4 consistently outperformed GPT-3.5 and LLaMA 3-70B. The same experiments show that a structured static prompt containing dataset descriptions, high-frequency entity words, UMLS background knowledge, error feedback, and token-level examples is worth roughly 11-12% average F1 over a basic prompt.","pith_inferences":["The reported 'few-shot' gains are measured with a retrieval index built from the full training set; a genuinely few-shot deployment with only five to twenty annotated examples would leave the retrieval pool nearly empty, so the 5-shot/10-shot/20-shot numbers should be read as dynamic prompting with access to a large annotated pool rather than as small-data performance.","If the retrieval index could be built from unlabeled text through pseudo-labeling or self-retrieval, the same mechanism might extend to settings with very little gold annotation; the paper does not test this.","The dataset-dependent winner between TF-IDF and SBERT suggests a cheap selection rule: use lexical TF-IDF on formal entity-dense text and semantic embeddings on informal or paraphrase-heavy text; this could be tested by a per-dataset retriever selector without retraining the LLM.","The token-level output format solves an alignment problem that plagues generative NER, and replicating that format in other generative pipelines may independently improve performance regardless of prompt selection."],"forward_implications":["Similarity-based example selection is a low-cost drop-in change: it requires only a retrieval index over existing annotations and changes each prompt during inference, so it can be layered on top of any fixed prompt template.","In formal, entity-rich biomedical texts such as PubMed abstracts, a simple lexical retriever (TF-IDF) can match or beat dense neural retrieval, whereas on noisy social-media data (the Reddit dataset) sentence-embedding retrieval (SBERT) tends to win.","Static prompt components matter as much as retrieval: adding dataset descriptions, high-frequency entity terms, and error feedback raised average F1 by 11-12%, so dynamic retrieval should be evaluated on top of an already-structured static prompt.","The advantage of dynamic over random selection shrinks as the number of in-context examples grows from 5-shot to 20-shot, because random draws become more diverse and longer prompts may dilute the model's attention.","LLM choice interacts with retrieval: GPT-4 consistently outperforms GPT-3.5 and LLaMA 3-70B, and retrieval helps most on datasets with sparse or noisy entity annotations."],"supporting_citations":[{"why":"Establishes in-context learning as the mechanism by which examples in the prompt teach the model the task.","marker":"[3]"},{"why":"Reports that random in-context examples cause high variance, motivating similarity-based example selection.","marker":"[7]"},{"why":"Introduces retrieval-augmented generation, the paradigm on which the dynamic prompt is built.","marker":"[10]"},{"why":"Provides prior evidence that retrieval-augmented generation improves specialized biomedical text analysis.","marker":"[14]"},{"why":"Defines the TF-IDF term-weighting method used as one of the retrieval engines.","marker":"[20]"},{"why":"Supplies the sentence embeddings used by the SBERT retriever.","marker":"[21]"},{"why":"Describes ColBERT, the late-interaction retriever used for comparison.","marker":"[22]"},{"why":"Describes Dense Passage Retrieval, the dual-encoder retriever used for comparison.","marker":"[23]"}],"fun_headline_variants":["RAG prompts with similar examples lift bio NER F1","Dynamic prompting retrieves best examples for bio NER","Similar-example selection boosts few-shot biomedical NER","Retrieval-augmented prompts raise NER F1 by 7.3%","Adaptive prompts via RAG improve few-shot bio NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments are called few-shot, but the retrieval index and static prompt components are built from the full training set; with only five to twenty labeled examples, the index would be nearly empty and those components could not be derived, so the reported gains may not transfer to genuinely limited annotation.","fun_headline_variants_meta":{"raw":{"variants":["RAG prompts with similar examples lift bio NER F1","Dynamic prompting retrieves best examples for bio NER","Similar-example selection boosts few-shot biomedical NER","Retrieval-augmented prompts raise NER F1 by 7.3%","Adaptive prompts via RAG improve few-shot bio NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1700,"prompt_tokens":947,"completion_tokens":753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":563,"tokens_out":753,"duration_ms":6827,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:09:37.082337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the reported F1 differences using retrieval indexes built only from the same k labeled examples available under each shot setting, with the static prompt components restricted to those same k examples; if the TF-IDF and SBERT gains over random selection fall to near zero or become negative, the few-shot transfer claim would be false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces retrieval-augmented generation, the paradigm on which the dynamic prompt is built."},{"cited_title":"A statistical interpretation of term specificity and its application in retrieval","cited_arxiv_id":null,"evidence_quote":"Defines the TF-IDF term-weighting method used as one of the retrieval engines."},{"cited_title":"& Zaharia, M","cited_arxiv_id":null,"evidence_quote":"Describes ColBERT, the late-interaction retriever used for comparison."},{"cited_title":"& Jia, R","cited_arxiv_id":null,"evidence_quote":"Reports that random in-context examples cause high variance, motivating similarity-based example selection."}],"review_version":1}