REVIEW 3 major objections 6 minor 20 references
Evaluation of LLMs on Long-tail Entity Linking in Historical Documents
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read LLMs beat a specialized entity linker on rare-entity recall.
desk verdict The headline recall numbers are new and useful, but the 'baseline's QID' matching sentence in Section 4 makes the main comparison unverifiable as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the one-shot JSON prompt: the LLM is told to identify key entities in a sentence and output a JSON object mapping each textual mention span to its Wikipedia page title, with one worked example supplied in the prompt. This collapses mention detection and disambiguation into a single autoregressive generation step and removes the need for a candidate index. The comparison rests on exact matching, where a true positive requires the predicted title to match the title obtained from the reference QID, and NIL entities are excluded. On the baseline side, ReLiK is a Retriever-Reader entity linking and relation extraction framework that links mentions to Wikipedia page IDs, which the authors map to Wikidata through KILT.
What would settle it
Recompute precision, recall, and F1 for the LLM outputs by matching predicted titles directly against the title of each mention's gold Wikidata entity, with NIL entities excluded, and repeat with a normalised fuzzy match to absorb OCR variants. If Llama-3-70B's recall falls to 45.7% or below under direct gold-title matching, the claimed long-tail recall advantage over ReLiK would be an artifact of the scoring protocol.
Extended reading notes
Core claim
The central discovery, stated on the paper's terms, is that a vanilla one-shot prompt converts LLMs into competitive long-tail entity linkers and makes them better retrievers than ReLiK: Llama-3-70B-instruct reaches recall 60.3% and F1 53.0, GPT-3.5-turbo-instruct reaches recall 58.8% and F1 53.2, versus ReLiK's recall 45.7% and F1 56.1. Precision inverts, with 47.3% and 48.6% for the two large LLMs against 72.8% for ReLiK. The prompt asks the model to emit JSON with mention spans as keys and Wikipedia page titles as values, treating entity linking as a sequence-to-sequence translation rather than separate recognition and disambiguation steps. A popularity analysis using Wikidata triple counts shows the same ordering holds for very rare entities at $\theta = 20$, although all systems remain below 30% recall there, so the paper concludes the long-tail problem is mitigated but not solved.
Load-bearing premise
The recall advantage rests on the scoring protocol: predicted Wikipedia titles must have been checked against the actual Wikipedia titles of the entities marked correct in the benchmark, not against titles supplied by the baseline linker's own guesses.
Editorial extensions
If this is right
- LLMs can serve as a first-pass entity retriever: their outputs can feed candidates to a high-precision linker, potentially raising recall without sacrificing ReLiK-level precision.
- Scale matters: the 70B Llama model recovered 60.3% of entities versus 40.1% for the 8B model, so long-tail recall in this setting tracks model size.
- Because LLM errors include both over-generation and exact-title mismatches, adding a validation step that checks generated titles against a knowledge base before counting them is a natural next improvement.
- Even for the best models, recall on the rarest entities ($\theta \le 20$) stays below 30%, so long-tail entity linking remains unsolved; LLMs narrow the gap but do not close it.
Reading between the lines
- A fuzzy or normalised title match would likely raise measured LLM precision: OCR-era spelling variants like 'Mocre' for 'Moore' are inherently exact-match failures, and the paper's qualitative examples show models sometimes resolve them when context is sufficient.
- The same prompting recipe could be tested in other niche domains such as legal, biomedical, or archival text; if the recall advantage transfers, the practical pattern becomes to prompt a general model to propose candidates and then verify with a cheaper knowledge-base lookup.
- The paper's evaluation wording leaves room for a stricter re-scoring: comparing LLM titles directly against the gold Wikidata titles, independent of ReLiK's predicted QIDs, would give a cleaner estimate of the claimed recall lead.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates two LLMs (GPT-3.5-turbo-instruct and Llama-3-8B/70B-instruct) on the task of entity linking for long-tail historical entities, using the manually annotated MHERCL v0.1.2 benchmark of sentences from the Polifonia Textual Corpus. The models are prompted in a one-shot JSON-output format to produce Wikipedia page titles for named entity mentions, and their predictions are compared against ReLiK, a state-of-the-art retriever-reader entity linking system. Results reported in Table 2 show that the two largest LLMs achieve higher recall than ReLiK (58.8% and 60.3% vs. 45.7%) but lower precision (48.6% and 47.3% vs. 72.8%), with F1 scores close to or below ReLiK. A threshold analysis based on the number of Wikidata triples per entity is used to examine performance on long-tail entities. The paper concludes that LLMs can serve as high-recall retrievers for long-tail entity linking in historical documents.
Significance. If the results are valid, the paper provides useful evidence for an understudied problem: using LLMs for long-tail entity linking in domain-specific historical texts. The study has several strengths: it uses a manually annotated external benchmark, compares against a strong state-of-the-art baseline (ReLiK), discloses its NIL-exclusion policy, and performs a popularity-threshold analysis that directly targets the long-tail phenomenon. The prompt is given in full, and the study makes falsifiable quantitative claims. However, the central quantitative claim is currently compromised by an ambiguous and potentially circular evaluation protocol, and the absence of error bars or significance tests makes the claimed recall advantage difficult to assess. The paper would be a useful contribution to the community if these issues are resolved.
major comments (3)
- [Section 4, Evaluation] The evaluation protocol for LLMs is ambiguous in a way that directly affects the central claim. The paper states: 'for LLMs we assessed the correct match between the predicted Wikipedia page title and the Wikipedia title retrieved using the baseline's QID.' Taking this literally, the LLM prediction is compared against the Wikipedia title retrieved from ReLiK's predicted QID, not the gold QID from MHERCL. In that case, the recall values in Table 2 measure agreement with ReLiK's outputs, not correctness, and an LLM could receive credit for repeating one of ReLiK's errors while receiving no credit for recovering a gold entity that ReLiK missed. The 'LLMs exceed ReLiK by about 15%' claim would then be unsupported. Please clarify whether 'baseline's QID' is a typo for the gold QID, and if so, state explicitly that LLM outputs were matched to the Wikipedia titles of the gold QIDs, and describe the QID-to-title mapping procedure (language, redirects, normalization). If the literal reading is intended, the results section must be rewritten because the comparison is not against ground truth.
- [Table 2 and Section 5] The main comparative claim—that Llama-3-70B exceeds ReLiK in recall by about 15 percentage points—is presented without any measure of variability or statistical significance. No decoding parameters (temperature, top-p, max tokens, number of runs) are reported, and the models are prompt-based, so outputs can vary across runs. Without confidence intervals or at least multiple runs, the reader cannot tell whether the 60.3% vs. 45.7% gap is stable or within run-to-run noise. Please add this information or explicitly frame the results as single-run exploratory observations and temper the strength of the conclusion accordingly.
- [Section 4 and general reproducibility] The manuscript ships no code, scoring scripts, or detailed parsing instructions. The prompt requests a JSON-style output, but the paper does not state how outputs were parsed, how many outputs were malformed or unparseable, how exact string matching was normalized (case, underscores, parenthetical disambiguation, leading articles), or how the Wikipedia titles were aligned with Wikidata QIDs. These details are necessary for a reader to verify the results and to interpret the statement that 'even a single incorrectly generated character causes the prediction to be considered incorrect.' Please provide the scoring code or an exact algorithmic description of the evaluation procedure.
minor comments (6)
- [Abstract and Section 4] The abstract refers to 'MHERCL v0.1' while Section 4 uses 'MHERCL v0.1.2'; please use a single version identifier consistently throughout.
- [Throughout] The model name 'Llama' is spelled inconsistently as 'LLama', 'Llama', and 'LLama80b'; please standardize to the official spelling 'Llama' (e.g., Llama-3-70B).
- [Section 5, Qualitative evaluation] The qualitative example in Section 5 mentions 'LLama80b' where 'Llama-70B' is meant; please correct this typo.
- [Figure 1] The text cites specific values for the θ=20 point (ReLiK recall ≈15%, LLM recall below 30%, F1 ≈19%), but the figure is difficult to read without numerical markers; please add explicit data labels or a small table with the values at θ=20 for all models.
- [Section 3] The sentence 'the model needs to identify each entity along with its unique identifier' would be clearer if it immediately states that the identifier used in this work is the Wikipedia page title, rather than leaving that detail to the following subsection.
- [Section 4, Baseline] The paper says 'we leverage ReLiK-base' but Table 2 and the rest of the text refer to 'ReLiK'; please clarify in the table and captions that the results are for the base version.
Circularity Check
No circularity: the paper is an empirical benchmark comparison with no fitted parameters, no derivation chain, and no load-bearing self-citation.
full rationale
This paper is a direct empirical evaluation. It measures precision, recall, and F1 of GPT-3.5 and Llama-3 variants against the ReLiK baseline on the MHERCL benchmark, using standard confusion-matrix definitions. There are no fitted parameters that are then renamed as predictions, no equation in the paper that defines the claimed result in terms of its own inputs, and no derivation that reduces to an assumed conclusion. The baseline and the LLMs are independent systems evaluated on an external, manually annotated dataset, and the central claim that larger LLMs achieve higher recall than ReLiK is a reported measurement rather than a constructed consequence of the method. The paper does cite prior work, including ReLiK and MHERCL, but none of those citations are self-citations that carry the argument, and the benchmark is externally available. The only notable concern is the ambiguity in Section 4 about whether LLM predictions were matched against the gold-standard Wikipedia title or against the title derived from ReLiK's QID; taken literally, that sentence could undermine the validity of the comparison, but it does not make the analysis circular. A flawed or ambiguous evaluation protocol is a correctness or reproducibility issue, not a case of the result being equivalent to its inputs by construction or by self-reference. Therefore no circularity is present.
Assumptions & free parameters
free parameters (1)
- Popularity threshold theta =
not fully specified; example theta=20 given
assumptions (5)
- domain assumption Wikidata triple count is a valid proxy for entity popularity.
- domain assumption Exact match between predicted and gold Wikipedia page titles correctly measures EL success.
- domain assumption MHERCL ground truth QIDs and their Wikipedia title mapping are accurate.
- domain assumption LLMs have not memorized the MHERCL sentences or the target Wikipedia titles in pretraining.
- domain assumption Excluding NIL-labeled entities does not bias the comparison.
Cite this review
Pith. "Pith review of Evaluation of LLMs on Long-tail Entity Linking in Historical Documents." pith.science (2026). https://pith.science/paper/VQQH2KBQ
@misc{pith2026250503473,
author = {Pith},
title = {Pith review of: Evaluation of LLMs on Long-tail Entity Linking in Historical Documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/VQQH2KBQ}},
note = {Machine review of arXiv:2505.03473}
}
read the original abstract
Entity Linking (EL) plays a crucial role in Natural Language Processing (NLP) applications, enabling the disambiguation of entity mentions by linking them to their corresponding entries in a reference knowledge base (KB). Thanks to their deep contextual understanding capabilities, LLMs offer a new perspective to tackle EL, promising better results than traditional methods. Despite the impressive generalization capabilities of LLMs, linking less popular, long-tail entities remains challenging as these entities are often underrepresented in training data and knowledge bases. Furthermore, the long-tail EL task is an understudied problem, and limited studies address it with LLMs. In the present work, we assess the performance of two popular LLMs, GPT and LLama3, in a long-tail entity linking scenario. Using MHERCL v0.1, a manually annotated benchmark of sentences from domain-specific historical texts, we quantitatively compare the performance of LLMs in identifying and linking entities to their corresponding Wikidata entries against that of ReLiK, a state-of-the-art Entity Linking and Relation Extraction framework. Our preliminary experiments reveal that LLMs perform encouragingly well in long-tail EL, indicating that this technology can be a valuable adjunct in filling the gap between head and long-tail EL.
Figures
Reference graph
Works this paper leans on
-
[1]
D. Banerjee, D. Chaudhuri, M. Dubey, J. Lehmann, Pnel: Pointer network based end-to-end entity linking over knowledge graphs, in: The Semantic Web–ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part I 19, Springer, 2020, pp. 21–38
work page 2020
-
[2]
E. Boros, E. L. Pontes, L. A. Cabrera-Diego, A. Hamdi, J. G. Moreno, N. Sidère, A. Doucet, Robust named entity recognition and linking on historical multilingual documents, in: Conference and Labs of the Evaluation Forum (CLEF 2020), volume 2696, CEUR-WS Working Notes, 2020, pp. 1–17
work page 2020
- [3]
- [4]
-
[5]
F. Ilievski, P. Vossen, S. Schlobach, Systematic study of long tail phenomena in entity linking, in: E. M. Bender, L. Derczynski, P. Isabelle (Eds.), Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, pp. 664–674. URL: https://aclanthology.org/C18-1056
work page 2018
-
[6]
B. Li, G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, S. Zhang, Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness [arxiv: 2304.11633 [cs]], arXiv preprint arXiv:2304.11633 (2023)
arXiv 2023
-
[7]
A. Graciotti, Knowledge extraction from multilingual and historical texts for advanced question answering, in: C. d’Amato, J. Z. Pan (Eds.), Proceedings of the Doctoral Consortium at ISWC 2023 co-located with 22nd International Semantic Web Conference (ISWC 2023), Athens, Greece, November 7, 2023, volume 3678 of CEUR Workshop Proceedings, 2023
work page 2023
-
[8]
R. Orlando, P.-L. Huguet-Cabot, E. Barba, R. Navigli, Relik: Retrieve and link, fast and accurate entity linking and relation extraction on an academic budget, arXiv preprint arXiv:2408.00103 (2024)
arXiv 2024
Show all 20 references
-
[9]
Nguyen, R
K. Nguyen, R. Ichise, Heuristic-based configuration learning for linked data instance matching, in: Semantic Technology: 5th Joint International Conference, JIST 2015, Yichang, China, November 11-13, 2015, Revised Selected Papers 5, Springer, 2016, pp. 56–72
2015
-
[10]
Zheng, M
G. Zheng, M. Liu, B. Liu, Collective entity linking based on dbpedia, in: Knowledge Graph and Semantic Computing. Language, Knowledge, and Intelligence: Second China Conference, CCKS 2017, Chengdu, China, August 26–29, 2017, Revised Selected Papers 2, Springer, 2017, pp. 66–79
2017
-
[11]
P. N. Mendes, M. Jakob, A. García-Silva, C. Bizer, Dbpedia spotlight: shedding light on the web of documents, in: Proceedings of the 7th international conference on semantic systems, 2011, pp. 1–8
2011
-
[12]
De Cao, G
N. De Cao, G. Izacard, S. Riedel, F. Petroni, Autoregressive entity retrieval, arXiv preprint arXiv:2010.00904 (2020)
2020 arXiv
-
[13]
M. P. K. Ravi, K. Singh, I. O. Mulang, S. Shekarpour, J. Hoffart, J. Lehmann, Cholan: A modular approach for neural entity linking on wikipedia and wikidata, arXiv preprint arXiv:2101.09969 (2021)
2021 arXiv
-
[14]
Zhang, W
W. Zhang, W. Hua, K. Stratos, Entqa: Entity linking as question answering, arXiv preprint arXiv:2110.02369 (2021)
2021 arXiv
-
[15]
Y. Ding, Q. Zeng, T. Weninger, Chatel: Entity linking with chatbots, arXiv preprint arXiv:2402.14858 (2024)
2024 arXiv
-
[16]
L. Wu, F. Petroni, M. Josifoski, S. Riedel, L. Zettlemoyer, Scalable zero-shot entity linking with dense entity retrieval, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association ...
2020 doi
-
[17]
A. Xin, Y. Qi, Z. Yao, F. Zhu, K. Zeng, X. Bin, L. Hou, J. Li, Llmael: Large language models are good context augmenters for entity linking, 2024. URL: https://arxiv.org/abs/2407.04020. arXiv:2407.04020
2024
-
[18]
URL: https://arxiv.org/abs/2407.21783
Llama Team, AI @ Meta, The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783
2024 arXiv
-
[19]
Petroni, A
F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, et al., Kilt: a benchmark for knowledge intensive language tasks, arXiv preprint arXiv:2009.02252 (2020)
2020 arXiv
-
[20]
L. Chen, S. Razniewski, G. Weikum, Knowledge base completion for long-tail entities, arXiv preprint arXiv:2306.17472 (2023)
2023 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.