Pith. sign in

REVIEW 3 major objections 6 minor 20 references

Evaluation of LLMs on Long-tail Entity Linking in Historical Documents

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read LLMs beat a specialized entity linker on rare-entity recall.

desk verdict The headline recall numbers are new and useful, but the 'baseline's QID' matching sentence in Section 4 makes the main comparison unverifiable as written. read the letter →

arxiv 2505.03473 v1 pith:VQQH2KBQ submitted 2025-05-06 cs.CL

classification cs.CL
keywords entitylinkinglong-tailentitieslargelanguagemodelshistoricaldocumentsWikidataWikipediapagetitlesLLMpromptingMHERCL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that off-the-shelf large language models, prompted with a single example, can act as high-recall linkers for rare historical entities. On MHERCL v0.1.2, a manually annotated corpus of noisy OCR music-history sentences, GPT-3.5 reaches 58.8% recall and Llama-3-70B reaches 60.3%, both above ReLiK's 45.7%; precision goes the other way, with the LLMs at 47–49% versus ReLiK's 72.8%. The authors read this as evidence that LLMs can help close the gap between head and long-tail entity linking, proposing them as retrievers or retrieval augmenters rather than replacements. If the paper is right, a generic model with zero task-specific training can recover more obscure entities than a specialized system, though over-generation and exact-title matching keep precision low.

What carries the argument

The mechanism is the one-shot JSON prompt: the LLM is told to identify key entities in a sentence and output a JSON object mapping each textual mention span to its Wikipedia page title, with one worked example supplied in the prompt. This collapses mention detection and disambiguation into a single autoregressive generation step and removes the need for a candidate index. The comparison rests on exact matching, where a true positive requires the predicted title to match the title obtained from the reference QID, and NIL entities are excluded. On the baseline side, ReLiK is a Retriever-Reader entity linking and relation extraction framework that links mentions to Wikipedia page IDs, which the authors map to Wikidata through KILT.

What would settle it

Recompute precision, recall, and F1 for the LLM outputs by matching predicted titles directly against the title of each mention's gold Wikidata entity, with NIL entities excluded, and repeat with a normalised fuzzy match to absorb OCR variants. If Llama-3-70B's recall falls to 45.7% or below under direct gold-title matching, the claimed long-tail recall advantage over ReLiK would be an artifact of the scoring protocol.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's terms, is that a vanilla one-shot prompt converts LLMs into competitive long-tail entity linkers and makes them better retrievers than ReLiK: Llama-3-70B-instruct reaches recall 60.3% and F1 53.0, GPT-3.5-turbo-instruct reaches recall 58.8% and F1 53.2, versus ReLiK's recall 45.7% and F1 56.1. Precision inverts, with 47.3% and 48.6% for the two large LLMs against 72.8% for ReLiK. The prompt asks the model to emit JSON with mention spans as keys and Wikipedia page titles as values, treating entity linking as a sequence-to-sequence translation rather than separate recognition and disambiguation steps. A popularity analysis using Wikidata triple counts shows the same ordering holds for very rare entities at $\theta = 20$, although all systems remain below 30% recall there, so the paper concludes the long-tail problem is mitigated but not solved.

Load-bearing premise

The recall advantage rests on the scoring protocol: predicted Wikipedia titles must have been checked against the actual Wikipedia titles of the entities marked correct in the benchmark, not against titles supplied by the baseline linker's own guesses.

Editorial extensions

If this is right

  • LLMs can serve as a first-pass entity retriever: their outputs can feed candidates to a high-precision linker, potentially raising recall without sacrificing ReLiK-level precision.
  • Scale matters: the 70B Llama model recovered 60.3% of entities versus 40.1% for the 8B model, so long-tail recall in this setting tracks model size.
  • Because LLM errors include both over-generation and exact-title mismatches, adding a validation step that checks generated titles against a knowledge base before counting them is a natural next improvement.
  • Even for the best models, recall on the rarest entities ($\theta \le 20$) stays below 30%, so long-tail entity linking remains unsolved; LLMs narrow the gap but do not close it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fuzzy or normalised title match would likely raise measured LLM precision: OCR-era spelling variants like 'Mocre' for 'Moore' are inherently exact-match failures, and the paper's qualitative examples show models sometimes resolve them when context is sufficient.
  • The same prompting recipe could be tested in other niche domains such as legal, biomedical, or archival text; if the recall advantage transfers, the practical pattern becomes to prompt a general model to propose candidates and then verify with a cheaper knowledge-base lookup.
  • The paper's evaluation wording leaves room for a stricter re-scoring: comparing LLM titles directly against the gold Wikidata titles, independent of ReLiK's predicted QIDs, would give a cleaner estimate of the claimed recall lead.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates two LLMs (GPT-3.5-turbo-instruct and Llama-3-8B/70B-instruct) on the task of entity linking for long-tail historical entities, using the manually annotated MHERCL v0.1.2 benchmark of sentences from the Polifonia Textual Corpus. The models are prompted in a one-shot JSON-output format to produce Wikipedia page titles for named entity mentions, and their predictions are compared against ReLiK, a state-of-the-art retriever-reader entity linking system. Results reported in Table 2 show that the two largest LLMs achieve higher recall than ReLiK (58.8% and 60.3% vs. 45.7%) but lower precision (48.6% and 47.3% vs. 72.8%), with F1 scores close to or below ReLiK. A threshold analysis based on the number of Wikidata triples per entity is used to examine performance on long-tail entities. The paper concludes that LLMs can serve as high-recall retrievers for long-tail entity linking in historical documents.

Significance. If the results are valid, the paper provides useful evidence for an understudied problem: using LLMs for long-tail entity linking in domain-specific historical texts. The study has several strengths: it uses a manually annotated external benchmark, compares against a strong state-of-the-art baseline (ReLiK), discloses its NIL-exclusion policy, and performs a popularity-threshold analysis that directly targets the long-tail phenomenon. The prompt is given in full, and the study makes falsifiable quantitative claims. However, the central quantitative claim is currently compromised by an ambiguous and potentially circular evaluation protocol, and the absence of error bars or significance tests makes the claimed recall advantage difficult to assess. The paper would be a useful contribution to the community if these issues are resolved.

major comments (3)
  1. [Section 4, Evaluation] The evaluation protocol for LLMs is ambiguous in a way that directly affects the central claim. The paper states: 'for LLMs we assessed the correct match between the predicted Wikipedia page title and the Wikipedia title retrieved using the baseline's QID.' Taking this literally, the LLM prediction is compared against the Wikipedia title retrieved from ReLiK's predicted QID, not the gold QID from MHERCL. In that case, the recall values in Table 2 measure agreement with ReLiK's outputs, not correctness, and an LLM could receive credit for repeating one of ReLiK's errors while receiving no credit for recovering a gold entity that ReLiK missed. The 'LLMs exceed ReLiK by about 15%' claim would then be unsupported. Please clarify whether 'baseline's QID' is a typo for the gold QID, and if so, state explicitly that LLM outputs were matched to the Wikipedia titles of the gold QIDs, and describe the QID-to-title mapping procedure (language, redirects, normalization). If the literal reading is intended, the results section must be rewritten because the comparison is not against ground truth.
  2. [Table 2 and Section 5] The main comparative claim—that Llama-3-70B exceeds ReLiK in recall by about 15 percentage points—is presented without any measure of variability or statistical significance. No decoding parameters (temperature, top-p, max tokens, number of runs) are reported, and the models are prompt-based, so outputs can vary across runs. Without confidence intervals or at least multiple runs, the reader cannot tell whether the 60.3% vs. 45.7% gap is stable or within run-to-run noise. Please add this information or explicitly frame the results as single-run exploratory observations and temper the strength of the conclusion accordingly.
  3. [Section 4 and general reproducibility] The manuscript ships no code, scoring scripts, or detailed parsing instructions. The prompt requests a JSON-style output, but the paper does not state how outputs were parsed, how many outputs were malformed or unparseable, how exact string matching was normalized (case, underscores, parenthetical disambiguation, leading articles), or how the Wikipedia titles were aligned with Wikidata QIDs. These details are necessary for a reader to verify the results and to interpret the statement that 'even a single incorrectly generated character causes the prediction to be considered incorrect.' Please provide the scoring code or an exact algorithmic description of the evaluation procedure.
minor comments (6)
  1. [Abstract and Section 4] The abstract refers to 'MHERCL v0.1' while Section 4 uses 'MHERCL v0.1.2'; please use a single version identifier consistently throughout.
  2. [Throughout] The model name 'Llama' is spelled inconsistently as 'LLama', 'Llama', and 'LLama80b'; please standardize to the official spelling 'Llama' (e.g., Llama-3-70B).
  3. [Section 5, Qualitative evaluation] The qualitative example in Section 5 mentions 'LLama80b' where 'Llama-70B' is meant; please correct this typo.
  4. [Figure 1] The text cites specific values for the θ=20 point (ReLiK recall ≈15%, LLM recall below 30%, F1 ≈19%), but the figure is difficult to read without numerical markers; please add explicit data labels or a small table with the values at θ=20 for all models.
  5. [Section 3] The sentence 'the model needs to identify each entity along with its unique identifier' would be clearer if it immediately states that the identifier used in this work is the Wikipedia page title, rather than leaving that detail to the following subsection.
  6. [Section 4, Baseline] The paper says 'we leverage ReLiK-base' but Table 2 and the rest of the text refer to 'ReLiK'; please clarify in the table and captions that the results are for the base version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark comparison with no fitted parameters, no derivation chain, and no load-bearing self-citation.

full rationale

This paper is a direct empirical evaluation. It measures precision, recall, and F1 of GPT-3.5 and Llama-3 variants against the ReLiK baseline on the MHERCL benchmark, using standard confusion-matrix definitions. There are no fitted parameters that are then renamed as predictions, no equation in the paper that defines the claimed result in terms of its own inputs, and no derivation that reduces to an assumed conclusion. The baseline and the LLMs are independent systems evaluated on an external, manually annotated dataset, and the central claim that larger LLMs achieve higher recall than ReLiK is a reported measurement rather than a constructed consequence of the method. The paper does cite prior work, including ReLiK and MHERCL, but none of those citations are self-citations that carry the argument, and the benchmark is externally available. The only notable concern is the ambiguity in Section 4 about whether LLM predictions were matched against the gold-standard Wikipedia title or against the title derived from ReLiK's QID; taken literally, that sentence could undermine the validity of the comparison, but it does not make the analysis circular. A flawed or ambiguous evaluation protocol is a correctness or reproducibility issue, not a case of the result being equivalent to its inputs by construction or by self-reference. Therefore no circularity is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

This paper is an empirical benchmark evaluation. It introduces no fitted model parameters. The analysis threshold theta is a hand-chosen binning on Wikidata triple counts. The main assumptions concern the validity of the popularity proxy, exact-match scoring, benchmark annotation quality, exclusion of NIL entities, and absence of pretraining contamination.

free parameters (1)
  • Popularity threshold theta = not fully specified; example theta=20 given
    Hand-chosen cut on Wikidata triple count used in Figure 1 to define popularity bins; not fitted to the central result, but affects the threshold analysis.
assumptions (5)
  • domain assumption Wikidata triple count is a valid proxy for entity popularity.
    Section 5: 'As a measure of popularity, we leveraged the number of Wikidata triples associated with each entity, as also done in [20].'
  • domain assumption Exact match between predicted and gold Wikipedia page titles correctly measures EL success.
    Section 5 states exact matching is used, so a single wrong character counts as wrong; this penalizes LLM outputs and is a strict, not error-tolerant, metric.
  • domain assumption MHERCL ground truth QIDs and their Wikipedia title mapping are accurate.
    The benchmark is manually annotated [7] and the paper does not audit annotation quality.
  • domain assumption LLMs have not memorized the MHERCL sentences or the target Wikipedia titles in pretraining.
    No contamination check is reported; if GPT or Llama saw these texts or titles during training, the long-tail recall numbers would be inflated.
  • domain assumption Excluding NIL-labeled entities does not bias the comparison.
    Section 4 excludes NIL entities because they lack Wikidata/Wikipedia entries; if models differ in how they would treat those mentions, the comparison is incomplete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation of LLMs on Long-tail Entity Linking in Historical Documents." pith.science (2026). https://pith.science/paper/VQQH2KBQ

@misc{pith2026250503473,
  author       = {Pith},
  title        = {Pith review of: Evaluation of LLMs on Long-tail Entity Linking in Historical Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQQH2KBQ}},
  note         = {Machine review of arXiv:2505.03473}
}
read the original abstract

Entity Linking (EL) plays a crucial role in Natural Language Processing (NLP) applications, enabling the disambiguation of entity mentions by linking them to their corresponding entries in a reference knowledge base (KB). Thanks to their deep contextual understanding capabilities, LLMs offer a new perspective to tackle EL, promising better results than traditional methods. Despite the impressive generalization capabilities of LLMs, linking less popular, long-tail entities remains challenging as these entities are often underrepresented in training data and knowledge bases. Furthermore, the long-tail EL task is an understudied problem, and limited studies address it with LLMs. In the present work, we assess the performance of two popular LLMs, GPT and LLama3, in a long-tail entity linking scenario. Using MHERCL v0.1, a manually annotated benchmark of sentences from domain-specific historical texts, we quantitatively compare the performance of LLMs in identifying and linking entities to their corresponding Wikidata entries against that of ReLiK, a state-of-the-art Entity Linking and Relation Extraction framework. Our preliminary experiments reveal that LLMs perform encouragingly well in long-tail EL, indicating that this technology can be a valuable adjunct in filling the gap between head and long-tail EL.

Figures

Figures reproduced from arXiv: 2505.03473 by the authors.

Figure 1
Figure 1. Measurement of the Entity Linking F1 and Recall score Across Different Entity Occurrence Thresholds for all the employed models The plot depicted in Figure 1a, reports the variation of the F1 score in EL at the variation of a threshold 𝜃, which has the role of differentiating real and predicted entities based on their notoriety defined by [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 14 canonical work pages

  1. [1]

    Banerjee, D

    D. Banerjee, D. Chaudhuri, M. Dubey, J. Lehmann, Pnel: Pointer network based end-to-end entity linking over knowledge graphs, in: The Semantic Web–ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part I 19, Springer, 2020, pp. 21–38

  2. [2]

    Boros, E

    E. Boros, E. L. Pontes, L. A. Cabrera-Diego, A. Hamdi, J. G. Moreno, N. Sidère, A. Doucet, Robust named entity recognition and linking on historical multilingual documents, in: Conference and Labs of the Evaluation Forum (CLEF 2020), volume 2696, CEUR-WS Working Notes, 2020, pp. 1–17

  3. [3]

    Sakor, K

    A. Sakor, K. Singh, A. Patel, M.-E. Vidal, Falcon 2.0: An entity and relation linking tool over wikidata, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020, pp. 3141–3148

  4. [4]

    Klang, P

    M. Klang, P. Nugues, Hedwig: A named entity linker, in: Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020, pp. 4501–4508

  5. [5]

    Ilievski, P

    F. Ilievski, P. Vossen, S. Schlobach, Systematic study of long tail phenomena in entity linking, in: E. M. Bender, L. Derczynski, P. Isabelle (Eds.), Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, pp. 664–674. URL: https://aclanthology.org/C18-1056

  6. [6]

    B. Li, G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, S. Zhang, Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness [arxiv: 2304.11633 [cs]], arXiv preprint arXiv:2304.11633 (2023)

  7. [7]

    Graciotti, Knowledge extraction from multilingual and historical texts for advanced question answering, in: C

    A. Graciotti, Knowledge extraction from multilingual and historical texts for advanced question answering, in: C. d’Amato, J. Z. Pan (Eds.), Proceedings of the Doctoral Consortium at ISWC 2023 co-located with 22nd International Semantic Web Conference (ISWC 2023), Athens, Greece, November 7, 2023, volume 3678 of CEUR Workshop Proceedings, 2023

  8. [8]

    Orlando, P.-L

    R. Orlando, P.-L. Huguet-Cabot, E. Barba, R. Navigli, Relik: Retrieve and link, fast and accurate entity linking and relation extraction on an academic budget, arXiv preprint arXiv:2408.00103 (2024)

Show all 20 references
  1. [9]

    Nguyen, R

    K. Nguyen, R. Ichise, Heuristic-based configuration learning for linked data instance matching, in: Semantic Technology: 5th Joint International Conference, JIST 2015, Yichang, China, November 11-13, 2015, Revised Selected Papers 5, Springer, 2016, pp. 56–72

  2. [10]

    Zheng, M

    G. Zheng, M. Liu, B. Liu, Collective entity linking based on dbpedia, in: Knowledge Graph and Semantic Computing. Language, Knowledge, and Intelligence: Second China Conference, CCKS 2017, Chengdu, China, August 26–29, 2017, Revised Selected Papers 2, Springer, 2017, pp. 66–79

  3. [11]

    P. N. Mendes, M. Jakob, A. García-Silva, C. Bizer, Dbpedia spotlight: shedding light on the web of documents, in: Proceedings of the 7th international conference on semantic systems, 2011, pp. 1–8

  4. [12]

    De Cao, G

    N. De Cao, G. Izacard, S. Riedel, F. Petroni, Autoregressive entity retrieval, arXiv preprint arXiv:2010.00904 (2020)

  5. [13]

    M. P. K. Ravi, K. Singh, I. O. Mulang, S. Shekarpour, J. Hoffart, J. Lehmann, Cholan: A modular approach for neural entity linking on wikipedia and wikidata, arXiv preprint arXiv:2101.09969 (2021)

  6. [14]

    Zhang, W

    W. Zhang, W. Hua, K. Stratos, Entqa: Entity linking as question answering, arXiv preprint arXiv:2110.02369 (2021)

  7. [15]

    Y. Ding, Q. Zeng, T. Weninger, Chatel: Entity linking with chatbots, arXiv preprint arXiv:2402.14858 (2024)

  8. [16]

    L. Wu, F. Petroni, M. Josifoski, S. Riedel, L. Zettlemoyer, Scalable zero-shot entity linking with dense entity retrieval, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association ...

  9. [17]

    A. Xin, Y. Qi, Z. Yao, F. Zhu, K. Zeng, X. Bin, L. Hou, J. Li, Llmael: Large language models are good context augmenters for entity linking, 2024. URL: https://arxiv.org/abs/2407.04020. arXiv:2407.04020

  10. [18]

    URL: https://arxiv.org/abs/2407.21783

    Llama Team, AI @ Meta, The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783

  11. [19]

    Petroni, A

    F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, et al., Kilt: a benchmark for knowledge intensive language tasks, arXiv preprint arXiv:2009.02252 (2020)

  12. [20]

    L. Chen, S. Razniewski, G. Weikum, Knowledge base completion for long-tail entities, arXiv preprint arXiv:2306.17472 (2023)

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.