{"id":"0ade8be4-9084-4db9-8782-2ac9f30560dc","arxiv_id":"2504.15022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On NER annotation, retrieval-based RAG prompting usually outperforms random in-context learning, but the best LLMs still fall 3 to 20 F1 points below human annotators on hard datasets.","lead":"This paper tested five large language models as automatic labelers for named-entity recognition across four datasets, comparing plain prompting, in-context learning, and a retrieval-based method that selects similar examples automatically. The retrieval method often improved accuracy, and some smaller models matched much larger ones, but the gap to human annotators remains large on complex datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gap-to-human is uninterpretable because the Human baseline protocol and training-set size are undocumented and confounded with the 30% context-sample setup; own tables also contradict the universal 'RAG consistently outperforms ICL' claim.","rationale":"The reader's weakest assumption identifies the undocumented Human baseline as the load-bearing condition, and I agree. My stress-test sharpens this into a concrete confound: the paper's own split definition and sample-space terminology make it likely that the LLM-annotated downstream model is trained on fewer sentences than the Human baseline, so the reported gap is not a pure measure of annotation quality. Even if the Human rows were generated by the same RoBERTa pipeline, the paper must state whether the training set was full Dtrain or the same T used for LLM annotations. In addition, the Section 7 claim that RAG consistently outperforms ICL is contradicted by several cells in Tables 3 and 4. I am not moving the verdict because the reader already assigned CONDITIONAL, and this concern reinforces that assignment rather than requiring a different one: the underlying comparisons are still useful and the reported F1 values may be accurate, but the headline interpretation and the universal RAG claim need revision or qualification before acceptance.","tokens_in":30730,"tokens_out":7138,"duration_ms":64309,"concrete_test":"Request the exact protocol for the Human rows: which Dtrain split (full Dtrain, only T, or T plus X), which seeds, which RoBERTa checkpoint, and which hyperparameters. Then run one decisive experiment: fine-tune RoBERTa with the same five seeds on the human labels for exactly the same T that the LLM annotated, rather than on the full Dtrain, and compare its F1 to the LLM-annotated rows. If the same-size human F1 differs materially from the reported Human values, the headline gap-to-human numbers must be recomputed, settling whether the 'close to human' claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the title claim is that the 'Human' rows in Tables 2-4 measure the same downstream protocol as the LLM-annotated rows. The paper never states how those rows were produced, cites no source, and the only protocol description in Section 4.3 covers LLM annotations, not the human baseline. Section 3.1 splits Dtrain into a context pool X and an annotation target T, while Section 4.2 reports experiments on a 30% sample and Section 5.2 refers to X as the sample space. If the Human row is RoBERTa trained on the full Dtrain while the LLM rows are trained only on T, then the reported gaps of 2.7%, 1.21%, and 3.15% conflate annotation source with training-set size and cannot answer the title question. The conclusion in Section 7 that RAG-based methods consistently outperform ICL across all datasets is also stronger than the paper's own numbers show: for example, Table 3, Qwen2.5-72B on WNUT-17 with 25 context examples, ICL F1 is 51.49 versus RAG-w/OpenAI 49.05 and RAG-w/ST 46.06; Table 4, Qwen2.5-7B with 50 examples, ICL F1 is 48.54 versus RAG-w/OpenAI 44.75. These cells undermine the universal claim, though a more restricted 'best configuration per dataset' claim might still hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the use of LLMs as automatic annotators for named entity recognition (NER), comparing a zero-shot baseline, in-context learning (ICL) with randomly sampled context examples, and a retrieval-augmented generation (RAG) approach in which context examples are retrieved by cosine similarity from a vector store using one of two embedding models. Five LLMs (gpt-4o-mini, Qwen2.5-72B, Llama3.5-70B, Qwen2.5-7B, and Llama3.1-8B) are assessed on CoNLL-2003, WNUT-17, GUM, and SKILLSPAN. Annotation quality is measured by fine-tuning RoBERTa on the LLM-annotated portion of the training set and reporting F1 on the test set, with results averaged over five seeds. The paper reports that RAG-based methods outperform ICL and the baseline, that gpt-4o-mini and Qwen2.5-72B come within a few F1 points of human-level annotation on CoNLL-2003, and that SKILLSPAN remains far below human performance.","tokens_in":31007,"tokens_out":4309,"duration_ms":36096,"significance":"If the results hold, the paper would provide useful practical guidance on when LLM-based NER annotation can substitute for human annotation and whether retrieval-based context selection is the main lever. The study has notable strengths: five LLMs at two parameter scales, three context sizes, two embedding models, four datasets of differing complexity, seed-averaged results with standard deviations, a non-parametric Friedman/Conover analysis, and qualitative examples of model outputs. The evaluation is a genuine measurement on held-out test sets, so circularity is not an issue. However, the headline 'gap to human' claims rest on an undocumented human baseline, the 30% training-sample protocol is underspecified, and the universal claim that RAG consistently outperforms ICL is contradicted by several cells in the paper's own tables. These are load-bearing issues for the central conclusions, though they are potentially fixable in revision.","major_comments":[{"comment":"The Human rows in Tables 2–4 are used as the reference for the title's question and for the gap numbers in §5.1 (2.7%, 1.21%, 3.15%), but the manuscript never states how these Human rows were produced, what data split they used, which model architecture and hyperparameters were used, or what source they come from. Section 4.3 describes only the protocol for fine-tuning RoBERTa on LLM-annotated data; it does not describe a Human row protocol. If the Human rows are RoBERTa trained on the full training set Dtrain while the LLM rows are trained on the 30% subset T, then the reported gaps conflate annotation source with training-set size and cannot answer the paper's title question. Please document the human baseline protocol precisely, or remove/reframe all human-gap claims.","section":"§4.2, §4.3, Tables 2–4"},{"comment":"The paper says experiments are conducted 'on a 30% sample of the training set Dtrain' but does not specify how the 30% sample is drawn, whether this sample is the annotation target T or the context pool X, or how the random split is seeded. Section 3.1 defines a split of Dtrain into disjoint X and T, but §4.2 does not state the sizes of X and T or which of them corresponds to the 30% figure. Appendix D introduces 10% and 20% 'sample space' choices without defining the relationship to X or T. This underspecification matters because the amount of context available for retrieval and the amount of training data used to fine-tune RoBERTa are both confounded with the annotation method. Please give exact definitions of all splits and sizes.","section":"§3.1, §4.2"},{"comment":"The conclusion in Section 7 that 'RAG-based methods consistently outperform both ICL and the baseline across all datasets' is too strong given the paper's own numbers. For example, in Table 3, Qwen2.5-72B on WNUT-17 with 25 context examples has ICL F1 = 51.49 but RAG w/OpenAI F1 = 49.05 and RAG w/ST F1 = 46.06; in Table 4, Qwen2.5-7B on WNUT-17 with 50 context examples has ICL F1 = 48.54 while RAG w/OpenAI has F1 = 44.75. Several other cells show ICL matching or exceeding RAG. The paper should replace the universal claim with a claim restricted to best configurations per dataset, or perform per-dataset, per-context-size statistical comparisons and report which differences are significant.","section":"§7, Tables 3–4"}],"minor_comments":[{"comment":"The text says the best CoNLL-2003 result is 'just 2.7% below human-level annotation', but Table 2 gives Human F1 = 92.12 and the best model F1 = 89.72, an absolute difference of 2.40; if the intended comparison is relative, this should be stated explicitly.","section":"§5.1"},{"comment":"The Friedman test results report a p-value of '7.71−18', which should be written as 7.71e−18 or as an actual p-value, and the report should state the number of datasets N and the number of models K used in the test.","section":"Appendix E"},{"comment":"The sentence 'Dtrain is is split into X and T' contains a duplicated 'is' and should be corrected.","section":"§3.2.1"},{"comment":"The x-axis labels '25 Ex. 50 Ex. 75 Ex.75 Ex.50 Ex.25 Ex.' are confusing and should be replaced with clear per-method grouping labels so the reader can map cells to context sizes and methods.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the main technical risk is the undocumented human baseline, because all headline gap-to-human claims depend on it. The empirical contribution is otherwise a solid engineering comparison; the novelty is modest but the scale of the evaluation is useful. If the authors cannot provide a human baseline produced under the same protocol, the paper should be revised to remove or heavily qualify those claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper is a broad empirical comparison, not a new method. What's genuinely useful is the matrix of measurements—five LLMs, two embeddings, four datasets (including GUM and SKILLSPAN, which are rarely used), three context sizes—with F1/precision/recall and standard deviations. The finding that some 7B models with good retrieval match 70B models is practical and supported by the statistical tests. The RAG recipe itself is not new; it's k-NN retrieval from training data for in-context examples, which Wang et al. (2023) used for GPT-NER. The contribution is the scale of the benchmark and the inclusion of less-common datasets.\n\nWhere it's soft: the Human baseline is undocumented. Tables 2-4 report 'Human' rows with F1 values (92.12, 54.93, 58.26, 54.79) but no source or protocol. Since the title asks how close we are to human performance, this is load-bearing. If those scores come from RoBERTa trained on the full training set while LLM-annotated models train on a 30% subset, the 2.7%, 1.21%, 3.15% gaps conflate annotation source with training data size. This must be fixed before publication.\n\nSecond, the paper claims RAG 'consistently outperforms' ICL, but its own tables show counterexamples. For instance, Qwen2.5-72B on WNUT-17 with 25 examples: ICL 51.49 vs RAG w/OpenAI 49.05; Qwen2.5-7B with 50 examples: ICL 48.54 vs RAG w/OpenAI 44.75. A more restricted claim about 'best configuration per dataset' would hold, but the universal one does not.\n\nThird, minor: the exact split of X and T is under-specified (Section 4.2 says '30% sample' without stating context-pool size), and no code or data is released, so exact numbers can't be checked. Also, the paper cites 'Llama3.5-70B-Instruct,' which doesn't exist publicly as of now; presumably they mean Llama-3.1-70B or similar. That needs correcting.\n\nOverall, the measurements are likely real and the paper gives a useful cost-performance map for NER annotation. The flaws are fixable with documentation and toned-down claims. This deserves peer review, not rejection. I'd send it out, with the expectation of major revisions.","headline":"A useful but overclaimed benchmark of LLM-as-annotator; the undocumented Human baseline and inconsistent RAG-vs-ICL results need fixing before the title claim holds.","tokens_in":31556,"tokens_out":2895,"would_cite":false,"duration_ms":23804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented prompting consistently improves LLM data annotation for named entity recognition, closing most of the gap to human-level F1 on structured datasets.","keywords":["LLM data annotation","Named Entity Recognition","Retrieval-Augmented Generation","In-context learning","token classification","human-level annotation","embedding models","NER evaluation"],"falsifier":"Recompute the human baselines under the paper's own protocol: fine-tune RoBERTa on a 30% human-annotated subset with the same five seeds and report F1. If those numbers are lower than 92.12, 54.93, 58.26, and 54.79, the claimed gaps (2.7%, 1.21%, 3.15%) are underestimates of the true gap to human annotation.","tokens_in":30522,"feed_emoji":"🏷️","tokens_out":6042,"duration_ms":47608,"temperature":0.7,"pith_summary":"The paper asks how close large language models are to human performance when they annotate named entities, and it argues that the gap is much smaller when the context examples are retrieved automatically rather than sampled at random. Across four NER datasets of increasing difficulty, retrieval-augmented generation (RAG) consistently beats both zero-shot and in-context learning, bringing the best models to within about 1 to 3 F1 points of human-level annotation on CoNLL-2003, WNUT-17, and GUM. The exception is SKILLSPAN, a soft-skill extraction dataset where the best LLM still trails human F1 by roughly 20 points. If this holds, retrieval-based context selection is the key lever for making LLM annotation cheap and useful, and the main research frontier moves to hard, ambiguous entity types.","feed_headline":"Retrieval beats hand-picked prompts for LLM annotation","feed_subtitle":"On four NER datasets, auto-retrieved examples put GPT-4o-mini and Qwen within about 1-3% of human F1.","key_machinery":"The central object is retrieval-augmented generation (RAG) applied to prompt construction: each sentence to be annotated is embedded, and the $M$ most similar human-annotated examples are pulled from a vector store to form the in-context context. This replaces the random sample of context examples used by ICL, giving the LLM examples that are semantically close to the input. The other load-bearing piece is structured output formatting, which the paper uses to keep token-label correspondence intact for gpt-4o-mini; it notes that open-weight models still show output inconsistencies.","core_discovery":"The paper's central claim is that for token-level NER annotation, a RAG-based method—embedding each sentence, retrieving the most similar human-annotated examples from a vector store, and placing them in the prompt—consistently outperforms randomly sampled ICL and zero-shot prompting. On CoNLL-2003, gpt-4o-mini with OpenAI embeddings reaches 89.72 F1 with 75 retrieved examples, within 2.7 points of the 92.12 human score; on WNUT-17, Qwen2.5-72B reaches 53.72 versus 54.93 human; on GUM, Qwen2.5-72B reaches 55.11 versus 58.26 human. The paper also reports that with the right retriever, 7B-parameter models can be statistically indistinguishable from 70B models, so model size is not the only driver. On SKILLSPAN, however, the best F1 is 34.06 versus 54.79 human, showing the method does not solve semantically ambiguous entities.","pith_inferences":["A reader should treat the precise gap percentages as provisional: the paper never states how the 'Human' rows in Tables 2-4 were produced, so if those numbers come from full-training-set models, the true gap to a same-protocol human baseline would be larger.","If retrieval quality is the main lever, then a testable extension is to swap the dense retriever for hybrid or re-ranked retrieval and measure annotation F1; the paper uses a 'naive' RAG by its own admission.","The same retrieval principle likely transfers to other token-level labeling tasks such as aspect or event extraction, where similar-example context could reduce annotation cost; this is a conjecture, not a paper claim.","On SKILLSPAN, the failure mode may be the embedding space itself: dense embeddings built for sentence similarity do not represent abstract soft-skill phrases, so a dedicated skill-aware retriever is a natural next test."],"forward_implications":["LLM annotation with retrieval is production-usable for well-structured entity types: CoNLL-2003, WNUT-17, and GUM all land within 1–3 F1 points of the human baseline.","Context selection matters as much as model scale: 7B models paired with strong embeddings can match or approach 70B models, so cost-conscious annotation can use smaller models.","Randomly sampled ICL is a weak default; a vector-store retriever is a cheap, consistent upgrade that needs no additional human effort.","The method does not transfer to soft-skill extraction: SKILLSPAN remains roughly 20 F1 points below human, so progress on ambiguous, multi-token entity types needs another idea."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation concept the paper adapts for automatic context selection.","marker":"(Lewis et al., 2020)"},{"why":"GPT-NER, the prior LLM-based NER annotation approach with k-NN retrieval that this study extends and compares against.","marker":"(Wang et al., 2023)"},{"why":"CoNLL-2003, the well-structured dataset used as an easy benchmark in the evaluation.","marker":"(Tjong Kim Sang and De Meulder, 2003)"},{"why":"WNUT-17, the noisy rare-entity dataset that shows the smallest human-LLM gap under RAG.","marker":"(Derczynski et al., 2017)"},{"why":"GUM, the multi-genre corpus with eleven entity types used to test diversity.","marker":"(Zeldes, 2017)"},{"why":"SKILLSPAN, the hardest dataset with ambiguous soft-skill mentions that exposes current limits.","marker":"(Zhang et al., 2022a)"},{"why":"RoBERTa, the model fine-tuned on LLM-annotated data to measure annotation quality.","marker":"(Liu et al., 2019)"},{"why":"Sentence-BERT embeddings, one of the two retrievers used in the RAG comparison.","marker":"(Reimers and Gurevych, 2019)"}],"fun_headline_variants":["RAG prompts push LLM NER within 3 points of human","Auto-retrieved context beats hand-picked ICL for NER","Retrieval-based prompting narrows LLM-to-human gap","LLM annotation close to human when examples are retrieved","RAG narrows the gap: LLMs annotate NER near humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire 'gap to human' comparison rests on an unstated human baseline: the paper does not say how the Human rows in Tables 2-4 were computed, and if they were not produced by the same RoBERTa-on-30%-subset protocol, the stated gaps are not valid comparisons.","fun_headline_variants_meta":{"raw":{"variants":["RAG prompts push LLM NER within 3 points of human","Auto-retrieved context beats hand-picked ICL for NER","Retrieval-based prompting narrows LLM-to-human gap","LLM annotation close to human when examples are retrieved","RAG narrows the gap: LLMs annotate NER near humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2654,"prompt_tokens":973,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1591}},"tokens_in":589,"tokens_out":1681,"duration_ms":10867,"temperature":1.0,"reasoning_tokens":1591,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:34:41.827003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the human baselines under the paper's own protocol: fine-tune RoBERTa on a 30% human-annotated subset with the same five seeds and report F1. If those numbers are lower than 92.12, 54.93, 58.26, and 54.79, the claimed gaps (2.7%, 1.21%, 3.15%) are underestimates of the true gap to human annotation.","supporting_citations":[],"review_version":1}