{"id":"047e8fa4-e19f-4e96-8ad3-0e1daedf747a","arxiv_id":"2505.03473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-3.5 and Llama-3-70B recall about 59-60% of long-tail historical entities versus ReLiK's 45.7%, but their lower precision leaves F1 scores near 53 versus ReLiK's 56.1.","lead":"This paper tests whether large language models can link rare, historical names in old music magazines to their Wikipedia pages. It finds they retrieve more rare entities than a specialized system, but with more false positives, leaving F1 scores roughly tied.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM recall advantage may be an artifact of matching to ReLiK's QIDs rather than gold QIDs; this invalidates the main comparison as written.","rationale":"The central claim — that Llama3-70b reaches 60.3% recall, about 15 points above ReLiK — stands or falls on how LLM predictions are matched to gold entities. The Section 4 sentence is the weakest link because its literal reading compares LLM predictions to ReLiK's predicted QIDs, a protocol that systematically biases the comparison in favor of LLMs and makes the headline recall gap impossible to interpret. The Reader's weakest_assumption identified exactly this issue, and I agree. Secondary concerns (no error bars, exact title matching, no released code/data) are real but would not independently overturn the claim; the matching protocol is decisive. Since the Reader already marked the paper CONDITIONAL and the proposed re-computation is the condition that would settle the issue, no verdict change is needed.","tokens_in":7719,"tokens_out":7463,"duration_ms":68984,"concrete_test":"Recompute the LLM precision, recall, and F1 using the Wikipedia title of the gold Wikidata QID as the reference instead of ReLiK's QID, and compare Llama3-70b's recall with ReLiK's 45.7%. If the 15-point margin disappears or recall falls below ReLiK's, the central claim is an artifact; if the margin persists, the claim survives the ambiguity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 (Evaluation) reads: \"In contrast, for LLMs we assessed the correct match between the predicted Wikipedia page title and the Wikipedia title retrieved using the baseline's QID.\" The natural referent of \"baseline's QID\" is ReLiK's predicted QID, not the gold QID. Taken literally, an LLM prediction is counted as correct when its title equals the Wikipedia title of ReLiK's output, which measures agreement with ReLiK rather than accuracy on MHERCL. This bias is not neutral: ReLiK's recall is 45.7%, so an LLM cannot score a true positive by recovering a gold entity that ReLiK missed, while it can score a true positive by repeating one of ReLiK's errors. The 60.3%-versus-45.7% recall advantage in Table 2 is therefore unsupported unless \"baseline's QID\" is a typo for the gold QID and the title is taken from the gold Wikidata entry. No code or scoring script is shipped, so the reader cannot resolve this from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates two LLMs (GPT-3.5-turbo-instruct and Llama-3-8B/70B-instruct) on the task of entity linking for long-tail historical entities, using the manually annotated MHERCL v0.1.2 benchmark of sentences from the Polifonia Textual Corpus. The models are prompted in a one-shot JSON-output format to produce Wikipedia page titles for named entity mentions, and their predictions are compared against ReLiK, a state-of-the-art retriever-reader entity linking system. Results reported in Table 2 show that the two largest LLMs achieve higher recall than ReLiK (58.8% and 60.3% vs. 45.7%) but lower precision (48.6% and 47.3% vs. 72.8%), with F1 scores close to or below ReLiK. A threshold analysis based on the number of Wikidata triples per entity is used to examine performance on long-tail entities. The paper concludes that LLMs can serve as high-recall retrievers for long-tail entity linking in historical documents.","tokens_in":7895,"tokens_out":3929,"duration_ms":40128,"significance":"If the results are valid, the paper provides useful evidence for an understudied problem: using LLMs for long-tail entity linking in domain-specific historical texts. The study has several strengths: it uses a manually annotated external benchmark, compares against a strong state-of-the-art baseline (ReLiK), discloses its NIL-exclusion policy, and performs a popularity-threshold analysis that directly targets the long-tail phenomenon. The prompt is given in full, and the study makes falsifiable quantitative claims. However, the central quantitative claim is currently compromised by an ambiguous and potentially circular evaluation protocol, and the absence of error bars or significance tests makes the claimed recall advantage difficult to assess. The paper would be a useful contribution to the community if these issues are resolved.","major_comments":[{"comment":"The evaluation protocol for LLMs is ambiguous in a way that directly affects the central claim. The paper states: 'for LLMs we assessed the correct match between the predicted Wikipedia page title and the Wikipedia title retrieved using the baseline's QID.' Taking this literally, the LLM prediction is compared against the Wikipedia title retrieved from ReLiK's predicted QID, not the gold QID from MHERCL. In that case, the recall values in Table 2 measure agreement with ReLiK's outputs, not correctness, and an LLM could receive credit for repeating one of ReLiK's errors while receiving no credit for recovering a gold entity that ReLiK missed. The 'LLMs exceed ReLiK by about 15%' claim would then be unsupported. Please clarify whether 'baseline's QID' is a typo for the gold QID, and if so, state explicitly that LLM outputs were matched to the Wikipedia titles of the gold QIDs, and describe the QID-to-title mapping procedure (language, redirects, normalization). If the literal reading is intended, the results section must be rewritten because the comparison is not against ground truth.","section":"Section 4, Evaluation"},{"comment":"The main comparative claim—that Llama-3-70B exceeds ReLiK in recall by about 15 percentage points—is presented without any measure of variability or statistical significance. No decoding parameters (temperature, top-p, max tokens, number of runs) are reported, and the models are prompt-based, so outputs can vary across runs. Without confidence intervals or at least multiple runs, the reader cannot tell whether the 60.3% vs. 45.7% gap is stable or within run-to-run noise. Please add this information or explicitly frame the results as single-run exploratory observations and temper the strength of the conclusion accordingly.","section":"Table 2 and Section 5"},{"comment":"The manuscript ships no code, scoring scripts, or detailed parsing instructions. The prompt requests a JSON-style output, but the paper does not state how outputs were parsed, how many outputs were malformed or unparseable, how exact string matching was normalized (case, underscores, parenthetical disambiguation, leading articles), or how the Wikipedia titles were aligned with Wikidata QIDs. These details are necessary for a reader to verify the results and to interpret the statement that 'even a single incorrectly generated character causes the prediction to be considered incorrect.' Please provide the scoring code or an exact algorithmic description of the evaluation procedure.","section":"Section 4 and general reproducibility"}],"minor_comments":[{"comment":"The abstract refers to 'MHERCL v0.1' while Section 4 uses 'MHERCL v0.1.2'; please use a single version identifier consistently throughout.","section":"Abstract and Section 4"},{"comment":"The model name 'Llama' is spelled inconsistently as 'LLama', 'Llama', and 'LLama80b'; please standardize to the official spelling 'Llama' (e.g., Llama-3-70B).","section":"Throughout"},{"comment":"The qualitative example in Section 5 mentions 'LLama80b' where 'Llama-70B' is meant; please correct this typo.","section":"Section 5, Qualitative evaluation"},{"comment":"The text cites specific values for the θ=20 point (ReLiK recall ≈15%, LLM recall below 30%, F1 ≈19%), but the figure is difficult to read without numerical markers; please add explicit data labels or a small table with the values at θ=20 for all models.","section":"Figure 1"},{"comment":"The sentence 'the model needs to identify each entity along with its unique identifier' would be clearer if it immediately states that the identifier used in this work is the Wikipedia page title, rather than leaving that detail to the following subsection.","section":"Section 3"},{"comment":"The paper says 'we leverage ReLiK-base' but Table 2 and the rest of the text refer to 'ReLiK'; please clarify in the table and captions that the results are for the base version.","section":"Section 4, Baseline"}],"recommendation":"major_revision","confidential_remarks":"The most pressing issue is the evaluation-protocol ambiguity in Section 4. If the phrase 'the baseline's QID' is interpreted literally, the paper's main recall comparison is circular and the manuscript would not be publishable. I suspect it is a wording error, but the authors must correct it and provide the title-mapping details before the claims can be trusted. The lack of any variability information is a second concern for a claim of superiority, though it may be acceptable for a preliminary workshop paper once acknowledged. I recommend major revision rather than rejection because the core empirical setup is sound if the protocol is clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline result—Llama-3-70B at 60.3% recall versus ReLiK's 45.7%—is the kind of measurement this community needs, but I cannot trust it from the manuscript as written. Section 4 says the LLM predictions are checked against 'the Wikipedia title retrieved using the baseline's QID.' Taken literally, that means matching against ReLiK's predicted QID, not the gold QID. That would measure agreement with the baseline, not accuracy, and the recall advantage disappears. A sentence in Section 5 says matching is against 'the real one,' so the Section 4 wording is likely a typo. Still, the referee needs the scoring script to be sure.\n\nWhat is genuinely useful: the popularity-threshold analysis using Wikidata triple counts shows the recall gap opens exactly in the tail (θ=20), which supports a hybrid LLM-retriever design. The one-shot JSON prompt is simple, the qualitative OCR examples are informative, and the authors are honest about low precision and QID hallucination.\n\nSoft spots beyond the ambiguity: no code or data, no error bars or significance tests; NIL-labeled entities are excluded from scoring, so over-generation for NIL mentions is not penalized, which may inflate LLM precision. Also, ReLiK's KB mapping goes through KILT (2019 Wikipedia) while the LLMs answer with current titles, so exact-string matching can produce false mismatches in both directions.\n\nThis is a small workshop-grade contribution, incremental beyond LLMAEL, but the specific MHERCL numbers are new. Who gets value: EL researchers in digital humanities, hybrid-system builders, and anyone wanting a clean example of zero-shot prompt behavior on long-tail historical entities. I would not cite the 60.3% number until the QID-matching is clarified and prediction files are released.\n\nRecommendation: send to peer review, conditional on re-running the comparison against gold QIDs and releasing the scoring artifacts. The question is worth a referee's time and the fix is small.","headline":"The headline recall numbers are new and useful, but the 'baseline's QID' matching sentence in Section 4 makes the main comparison unverifiable as written.","tokens_in":8439,"tokens_out":4521,"would_cite":false,"duration_ms":44624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs beat a specialized entity linker on rare-entity recall.","keywords":["entity linking","long-tail entities","large language models","historical documents","Wikidata","Wikipedia page titles","LLM prompting","MHERCL"],"falsifier":"Recompute precision, recall, and F1 for the LLM outputs by matching predicted titles directly against the title of each mention's gold Wikidata entity, with NIL entities excluded, and repeat with a normalised fuzzy match to absorb OCR variants. If Llama-3-70B's recall falls to 45.7% or below under direct gold-title matching, the claimed long-tail recall advantage over ReLiK would be an artifact of the scoring protocol.","tokens_in":1617,"feed_emoji":"📜","tokens_out":1982,"duration_ms":93293,"temperature":0.7,"pith_summary":"The paper argues that off-the-shelf large language models, prompted with a single example, can act as high-recall linkers for rare historical entities. On MHERCL v0.1.2, a manually annotated corpus of noisy OCR music-history sentences, GPT-3.5 reaches 58.8% recall and Llama-3-70B reaches 60.3%, both above ReLiK's 45.7%; precision goes the other way, with the LLMs at 47–49% versus ReLiK's 72.8%. The authors read this as evidence that LLMs can help close the gap between head and long-tail entity linking, proposing them as retrievers or retrieval augmenters rather than replacements. If the paper is right, a generic model with zero task-specific training can recover more obscure entities than a specialized system, though over-generation and exact-title matching keep precision low.","feed_headline":"LLMs beat a specialized entity linker on rare-entity recall","feed_subtitle":"One-shot GPT-3.5 and Llama-3-70B recovered 60.3% of historical entities versus ReLiK's 45.7%.","key_machinery":"The mechanism is the one-shot JSON prompt: the LLM is told to identify key entities in a sentence and output a JSON object mapping each textual mention span to its Wikipedia page title, with one worked example supplied in the prompt. This collapses mention detection and disambiguation into a single autoregressive generation step and removes the need for a candidate index. The comparison rests on exact matching, where a true positive requires the predicted title to match the title obtained from the reference QID, and NIL entities are excluded. On the baseline side, ReLiK is a Retriever-Reader entity linking and relation extraction framework that links mentions to Wikipedia page IDs, which the authors map to Wikidata through KILT.","core_discovery":"The central discovery, stated on the paper's terms, is that a vanilla one-shot prompt converts LLMs into competitive long-tail entity linkers and makes them better retrievers than ReLiK: Llama-3-70B-instruct reaches recall 60.3% and F1 53.0, GPT-3.5-turbo-instruct reaches recall 58.8% and F1 53.2, versus ReLiK's recall 45.7% and F1 56.1. Precision inverts, with 47.3% and 48.6% for the two large LLMs against 72.8% for ReLiK. The prompt asks the model to emit JSON with mention spans as keys and Wikipedia page titles as values, treating entity linking as a sequence-to-sequence translation rather than separate recognition and disambiguation steps. A popularity analysis using Wikidata triple counts shows the same ordering holds for very rare entities at $\\theta = 20$, although all systems remain below 30% recall there, so the paper concludes the long-tail problem is mitigated but not solved.","pith_inferences":["A fuzzy or normalised title match would likely raise measured LLM precision: OCR-era spelling variants like 'Mocre' for 'Moore' are inherently exact-match failures, and the paper's qualitative examples show models sometimes resolve them when context is sufficient.","The same prompting recipe could be tested in other niche domains such as legal, biomedical, or archival text; if the recall advantage transfers, the practical pattern becomes to prompt a general model to propose candidates and then verify with a cheaper knowledge-base lookup.","The paper's evaluation wording leaves room for a stricter re-scoring: comparing LLM titles directly against the gold Wikidata titles, independent of ReLiK's predicted QIDs, would give a cleaner estimate of the claimed recall lead."],"forward_implications":["LLMs can serve as a first-pass entity retriever: their outputs can feed candidates to a high-precision linker, potentially raising recall without sacrificing ReLiK-level precision.","Scale matters: the 70B Llama model recovered 60.3% of entities versus 40.1% for the 8B model, so long-tail recall in this setting tracks model size.","Because LLM errors include both over-generation and exact-title mismatches, adding a validation step that checks generated titles against a knowledge base before counting them is a natural next improvement.","Even for the best models, recall on the rarest entities ($\\theta \\le 20$) stays below 30%, so long-tail entity linking remains unsolved; LLMs narrow the gap but do not close it."],"supporting_citations":[{"why":"Defines long-tail entities and frames the challenge the paper targets.","marker":"[5]"},{"why":"Provides MHERCL v0.1, the manually annotated historical benchmark used for evaluation.","marker":"[7]"},{"why":"Supplies ReLiK, the state-of-the-art baseline whose recall the LLMs are compared against.","marker":"[8]"},{"why":"Describes Llama 3, one of the two model families under test.","marker":"[18]"},{"why":"KILT maps ReLiK's Wikipedia page IDs to Wikidata QIDs for scoring.","marker":"[19]"},{"why":"Provides the Wikidata-triple-count measure of entity popularity used in the threshold analysis.","marker":"[20]"}],"fun_headline_variants":["LLMs out-recall specialized linker on rare historical entities","One-shot LLMs beat ReLiK on long-tail entity recall","Vanilla prompts turn LLMs into strong long-tail entity linkers","LLMs recover more rare entities than a dedicated EL system","GPT and Llama-3 top ReLiK in historical entity recall"],"cache_read_input_tokens":10624,"weakest_assumption_plain":"The recall advantage rests on the scoring protocol: predicted Wikipedia titles must have been checked against the actual Wikipedia titles of the entities marked correct in the benchmark, not against titles supplied by the baseline linker's own guesses.","fun_headline_variants_meta":{"raw":{"variants":["LLMs out-recall specialized linker on rare historical entities","One-shot LLMs beat ReLiK on long-tail entity recall","Vanilla prompts turn LLMs into strong long-tail entity linkers","LLMs recover more rare entities than a dedicated EL system","GPT and Llama-3 top ReLiK in historical entity recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1558,"prompt_tokens":991,"completion_tokens":567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":607,"tokens_out":567,"duration_ms":4935,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:50:41.202709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute precision, recall, and F1 for the LLM outputs by matching predicted titles directly against the title of each mention's gold Wikidata entity, with NIL entities excluded, and repeat with a normalised fuzzy match to absorb OCR variants. If Llama-3-70B's recall falls to 45.7% or below under direct gold-title matching, the claimed long-tail recall advantage over ReLiK would be an artifact of the scoring protocol.","supporting_citations":[{"cited_title":"Ilievski, P","cited_arxiv_id":null,"evidence_quote":"Defines long-tail entities and frames the challenge the paper targets."},{"cited_title":"Graciotti, Knowledge extraction from multilingual and historical texts for advanced question answering, in: C","cited_arxiv_id":null,"evidence_quote":"Provides MHERCL v0.1, the manually annotated historical benchmark used for evaluation."},{"cited_title":"Knowledge Base Completion for Long-Tail Entities","cited_arxiv_id":"2306.17472","evidence_quote":"Provides the Wikidata-triple-count measure of entity popularity used in the threshold analysis."}],"review_version":1}