{"id":"c8d1ab77-df15-400f-a0f6-ceb6b987cd9d","arxiv_id":"2502.09168","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MHERCL, a gold-standard dataset of 875 sentences from 19th-century music periodicals, shows that type and time filtering plus NIL-aware heuristics improve entity linking over off-the-shelf models and LLMs.","lead":"A new manually annotated benchmark, MHERCL, brings entity linking to 19th-century music periodicals, where most named entities are rare or absent from knowledge bases. The paper shows that a standard retrieval-based linker with time and type constraints plus NIL prediction can beat much larger models on these historical texts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 7's best F1 is selected by sweeping 1,001 NIL thresholds with no validation split; the claimed edge over LLMs requires a threshold fixed before test.","rationale":"I read the paper as making a practical claim: a constrained retrieval model with a NIL heuristic beats LLMs and specialized EL systems on historical text. The dataset contribution (MHERCL, IAA 0.82) and the breadth of baselines are real strengths, and the paper is unusually transparent about the NIL-error decomposition in Table 14 and the score-magnitude limitation in Section 5.4. My concern is not with the direction of the effect but with the evaluation protocol that produces the headline F1. Sweeping 1,001 thresholds and reporting the best row is a form of test-set selection unless a dev split is used; the paper does not describe one. The deviation-from-mean heuristic is exactly the kind of rule whose operating point must be fixed a priori to support a comparative claim. The gold-NER type filter is a second asymmetry, but it is a component-level design choice; the threshold issue directly threatens the quantitative headline. This is addressable and does not invalidate the resource, so I keep the reader's conditional verdict.","tokens_in":31350,"tokens_out":5758,"duration_ms":55058,"concrete_test":"Hold out a validation split of MHERCL (e.g., 15% of sentences, stratified by NIL ratio) and, using only that split, select τ for the deviation-from-mean NIL heuristic and for the fixed-threshold heuristic. Then freeze τ and evaluate on the held-out test split. Repeat for HIPE-2020 using its official train/dev partitions for threshold selection. Report F1 against the LLM baselines under the same protocol. If the selected-τ F1 is within noise of GPT-4o-mini/LLAMA-3.3, or drops by more than about 0.05 from the Table 7 values, the claim of superiority is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Table 7: C-BLINK† with {ϕd,ϕt} and deviation-from-mean NIL, F1 0.76 on MHERCL and 0.67 on HIPE-2020) is obtained after sweeping every threshold τ in [0,1] with step 0.001 (Section 5.3). No validation split or threshold-selection procedure is described for MHERCL, and for the ML-based NIL classifiers Section 5.3 states only that they are trained on HIPE-2020 while Table 7 also reports HIPE-2020 scores, so the protocol needs clarification. Because the deviation-from-mean heuristic converts a continuous score profile into a binary NIL/link decision, τ is a free parameter that materially changes results: the same C-BLINK† configuration with a fixed threshold reaches only 0.54/0.49, and other heuristics at their best range from 0.47 to 0.74 on MHERCL. The 0.76/0.67 numbers are therefore post-selection maxima over 1,001 thresholds rather than the performance of a model whose NIL policy is fixed before seeing test labels. Section 5.4 itself concedes that the retriever and re-ranker score magnitudes are not a trained proxy for correctness, which makes the threshold sensitivity worse, not better. The central claim that the method 'can outperform' GPT-4o-mini and LLAMA-3.3 on HEL rests on the assumption that the chosen thresholds generalize; the paper provides no evidence for this.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MHERCL, a new gold-standard benchmark for named entity recognition, classification, and linking on historical musical-periodical text, with 875 manually annotated sentences from 76 periodicals (1823–1900), reported Inter-Annotator Agreement of 0.82, and a high share of long-tail and NIL entities. The authors evaluate several off-the-shelf entity linkers and LLMs, finding that they underperform on this benchmark. They then propose two models: ELD, an unsupervised game-theoretic linker, and C-BLINK, an extension of BLINK that filters candidates using time and type constraints from Wikidata and adds NIL prediction through threshold-based or learned heuristics. Experiments on MHERCL and HIPE-2020 report that C-BLINK with the 'deviation from mean' NIL heuristic achieves F1 scores of 0.76 and 0.67, respectively, outperforming GPT-4o-mini and LLAMA-3.3 70B.","tokens_in":31714,"tokens_out":2282,"duration_ms":22384,"significance":"If the results are taken at face value, the paper makes two contributions: a publicly released, carefully annotated benchmark for a neglected domain (historical music periodicals), and evidence that simple KG-based constraints plus score-based NIL prediction can beat much larger closed models on historical entity linking. The dataset construction is a genuine asset: the annotation guidelines are detailed, the IAA of 0.82 is respectable, and the popularity analysis convincingly shows that MHERCL is skewed toward long-tail entities. The code and data are publicly available, which supports reproducibility. However, the headline claim about outperforming LLMs rests on an evaluation protocol whose threshold selection is performed on the test set, so the quantitative edge reported in Table 7 is not yet established as a property of a fixed model.","major_comments":[{"comment":"This is a load-bearing issue because the central claim of outperforming LLMs depends on the NIL decision being made in a way that generalizes beyond the test set; as written, the comparison is not a fair evaluation of a defined method.","section":"Section 5.3 / Table 7"},{"comment":"Additionally, the ML-based NIL classifiers (SVM, logistic regression, decision tree) are stated in Section 5.3 to be trained on HIPE-2020 and evaluated on MHERCL, yet Table 7 also reports HIPE-2020 scores for these classifiers. If those HIPE-2020 numbers are obtained by training and testing on the same set, they are circular and should be removed or replaced with cross-validated or separately trained results.","section":"Section 5.2, Section 5.3"},{"comment":"The paper's NIL heuristics rely on the assumption that, when the correct entity is in the candidate pool, it will have the highest similarity score (Section 4.2). Section 5.4 concedes that the retriever and re-ranker are not explicitly trained to produce calibrated scores, and Table 14 shows that the most frequent C-BLINK error is precisely predicting NIL while the correct QID is among the candidates. This means the score-magnitude assumption is internally acknowledged to be unreliable. To support the claim that score-based NIL heuristics are robust, the authors should provide a distribution of score gaps for correct versus incorrect links, or demonstrate that the chosen heuristic generalizes across documents or datasets rather than only on the test set where the threshold was tuned.","section":"Section 4.2 / Section 5.4"}],"minor_comments":[{"comment":"There is a typo: 'MERCHL' should be 'MHERCL' in the sentence about the context of ELD.","section":"Section 4.1.1"},{"comment":"The text says 'Table 5.4' when referring to results; it should reference Table 7.","section":"Section 5.4"},{"comment":"The phrase 'favour NIL predcitions' contains a typo ('predcitions' should be 'predictions').","section":"Section 5.4"},{"comment":"The model name is written as 'ReFiNeD' in one place and 'ReFinED' elsewhere; please unify the spelling throughout.","section":"Section 2.2"},{"comment":"The reference list contains a duplicate entry for Lacerra et al. (2021); the in-text citation 'Lacerra et al., 2021' appears twice with different reference strings and should be consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is solid and likely useful to the community, but the evaluation protocol for the headline comparison needs substantial rework: the NIL thresholds must be selected on a validation set rather than by test-set grid search, and the oracle use of gold NER types must be acknowledged or replaced with a realistic setting. These issues are fixable without changing the scope of the paper, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe dataset is the real contribution here. MHERCL is a new, manually annotated EL benchmark from historical music periodicals, 875 sentences, about 2,370 mentions, 58 types, 30% NIL, with a careful annotation process (Krippendorff's alpha 0.82) and a nice popularity analysis showing it is long-tail heavy compared to AIDA and even HIPE-2020. The code and data are released. That alone justifies a serious look.\n\nThe models are more mixed. ELD is a reasonable adaptation of existing game-theoretic relaxation labeling to EL, not a fundamentally new idea but a new application. C-BLINK is a straightforward extension of BLINK with type/time filters; the constraints themselves are sensible and clearly described. The paper is honest about the fact that the retriever scores are not trained to support the NIL heuristics, and it reports an error analysis that separates candidate-retrieval failures from threshold failures. That level of transparency earns credit.\n\nThe soft spots are real and one is load-bearing for the headline claim. The best C-BLINK† numbers (0.76 MHERCL, 0.67 HIPE-2020) come from sweeping tau in [0,1] with step 0.001 on the test set. Section 5.3 describes the sweep but no validation split or threshold selection procedure. That means the \"outperforms GPT-4o-mini and LLAMA 3.3\" result is a post-selection maximum, not a test of a model with fixed NIL policy. The paper itself concedes that score magnitude is not a reliable proxy for correctness (Section 5.4), so the threshold sensitivity matters. This is fixable: hold out a validation split, or report performance with a threshold chosen a priori, and show the sensitivity around the chosen tau.\n\nSecond, the type constraint uses gold NER classes (Section 5.2), which LLM baselines never receive. The comparison pit is uneven. It's fine to test a pipeline that assumes gold NEC, but then the claim should be scoped as 'given gold types,' not a fair head-to-head against systems that have to infer types.\n\nMinor issues: no significance tests; the ML-based NIL classifiers are trained on HIPE-2020 but the paper also reports their HIPE-2020 scores, which needs clarification; the paper's own numbers show other heuristics at 0.54–0.74 on MHERCL, so the gap is fragile.\n\nThe central dataset contribution holds up. The model performance claims, as stated, overstate what was tested. I'd send this to peer review—it deserves referee time—but the reviewers should push for a validation-based threshold protocol and a re-scoped comparison. If those are fixed, the paper is useful for the EL and digital humanities communities. I'd cite the dataset; I wouldn't cite the model result without the threshold caveat.","headline":"The dataset is the contribution; the model-vs-LLM claim is post-selected on test-set thresholds and runs with gold type information, so the comparison overstates the win.","tokens_in":32211,"tokens_out":3278,"would_cite":true,"duration_ms":30212,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that a retrieval-based linker constrained by knowledge-graph time and type filters, with a score-based NIL heuristic, outperforms all tested specialized linkers and two large language models on historical…","keywords":["entity linking","historical documents","NIL prediction","knowledge graph constraints","retrieval-based models","game theory","long-tail entities","music heritage"],"falsifier":"Count, on MHERCL, how often the gold QID is present in C-BLINK's top-$k$ candidates but is not the top-scoring plausible candidate; the paper's own error analysis shows many such cases become false NIL predictions. If that share is substantial, the assumption that score magnitude tracks correctness fails. A simpler version: rerun the best C-BLINK configuration with predicted NER types instead of manually annotated types and check whether its F1 advantage over BLINK disappears.","tokens_in":31178,"feed_emoji":"🎼","tokens_out":12316,"duration_ms":112245,"temperature":0.7,"pith_summary":"Linking names in nineteenth-century music periodicals to a knowledge base is hard because most of the entities are long-tail and about a third have no entry at all (NIL). The paper introduces MHERCL, a gold-annotated benchmark of 875 such sentences, and argues that the bottleneck is not model size but candidate discipline. It offers two systems: ELD, an unsupervised game-theoretic linker, and C-BLINK, which filters an existing retrieval linker's candidates by Wikidata type and date. The paper's strongest claim is that C-BLINK without re-ranking, using a deviation-from-mean score heuristic to decide when no candidate is confident enough, reaches F1 0.76 on MHERCL and 0.67 on HIPE-2020, beating all tested specialized systems and the tested large language models. If that is right, a cheap, open, interpretable linker can outperform far larger models on historical text by refusing uncertain links.","feed_headline":"Constrained linker beats GPT-4 and Llama on historical names","feed_subtitle":"Type-and-date filtering plus a score-shape NIL rule reaches F1 0.76 on MHERCL.","key_machinery":"Two mechanisms carry the argument. The first is the plausibility filter $\\Phi$, a set of boolean functions over mention--candidate pairs: $\\phi_d$ requires the Wikidata date (birth, inception, publication, or similar property) to precede the document's date, and $\\phi_t$ requires the Wikidata \"instance of\" types to intersect the mention's NER type through a hand-built type taxonomy. C-BLINK applies these by multiplying the bi-encoder similarity score of any implausible candidate by zero before re-ranking. The second mechanism is the NIL rule acting on the surviving score vector $S = [s_0, \\dots, s_n]$; the winning rule, deviation from mean, predicts NIL when $(s_0 - \\mu(S)) / ((s_0 + \\mu(S))/2) < \\tau$. ELD uses a different engine: replicator dynamics from evolutionary game theory, where mentions, surrounding content words, and candidate entities are players and strategies, and the payoff matrix is built from mBERT and sense embeddings; iterated updates let the sentence settle on a consistent global labeling.","core_discovery":"The central claim is that historical entity linking is best served by making fewer, better-constrained guesses rather than by more expressive models. Concretely, C-BLINK zeroes out the bi-encoder scores of candidate entities whose Wikidata date is not before the document's date ($\\phi_d$) or whose Wikidata type is incompatible with the mention's manually annotated NER class ($\\phi_t$); after that filter, the deviation-from-mean heuristic answers NIL when the top score is not sufficiently separated from the mean of the candidate scores. In the paper's evaluation this configuration reaches F1 0.76 on MHERCL and 0.67 on HIPE-2020, surpassing every specialized entity linker tested and both tested LLMs (GPT-4 o1-mini at 0.60 on MHERCL and 0.68 on HIPE-2020, and LLAMA 3.3 70B at 0.61 and 0.60). The paper likewise claims that ELD, an unsupervised game-theoretic linker where mentions and context words negotiate via replicator dynamics, avoids popularity bias and reaches F1 0.58 on MHERCL and 0.62 on HIPE-2020. The takeaway is that a simpler retrieval-based neural model enhanced with targeted heuristics can outperform larger, more expensive, closed-source models in long-tail, domain-specific scenarios requiring robust NIL handling.","pith_inferences":["An untested pipeline variation that would bound the real-world gain: replace the manually annotated NER classes used by $\\phi_t$ with a predicted NER tagger and measure the F1 drop; the reported 0.76 is an upper bound under perfect type information.","Because the deviation-from-mean rule only looks at the shape of candidate scores, it should transfer to other dense linkers and to retrieval-augmented generation, where the gap between the best and the mean candidate could gate whether the LLM answers at all.","ELD's game-theoretic consensus and C-BLINK's filters are complementary: integrating time and type constraints into ELD's payoff matrix, rather than post-filtering candidates, could raise its 0.58 MHERCL score while preserving its low popularity bias.","The 14% OCR-noise slice of MHERCL provides a ready-made test set for OCR post-correction claims, since the paper's own error analysis shows OCR errors are concentrated in wrongly linked mentions across all models."],"forward_implications":["Retrieval-based linking with knowledge-graph constraints should be a strong default for historical text: it is cheaper than calling a large model, open, and higher-scoring in these experiments.","NIL handling is decisive: because always predicting NIL already earns F1 0.43 on HIPE-2020, models that cannot decline a link are structurally handicapped on historical benchmarks.","Type and date filtering specifically removes anachronistic and wrong-type errors that dense retrieval makes, and it reduces the tendency to choose popular but implausible entities.","By observing that ELD's replicator dynamics reinforce popularity bias even when the underlying representations are less biased, the paper identifies score and representation bias, not just retriever architecture, as a target for future work.","MHERCL, with its long-tail distribution and 30% NIL share, provides a complementary testbed to HIPE-2020 for evaluating long-tail and NIL-sensitive linking."],"supporting_citations":[{"why":"This is BLINK, the retrieval-based linker that C-BLINK extends; its bi-encoder candidate scores are the scores that the time and type filters annihilate.","marker":"Wu et al. (2020)"},{"why":"This work grounds the type-constraint idea: filtering linkable entities by the mention's named-entity class before reranking reduces implausible links.","marker":"Tedeschi et al. (2021)"},{"why":"This is the relaxation-labelling foundation that ELD reformulates as a game; it provides the consistency constraints among labels that drive joint disambiguation.","marker":"Hummel and Zucker (1983)"},{"why":"This gives the replicator dynamic equation that ELD iterates to update strategy probabilities and reach a consistent labeling.","marker":"Taylor and Jonker (1978)"},{"why":"This supplies the mBERT embeddings that ELD uses to represent tokens and entities in a common space for payoff computation.","marker":"Devlin et al. (2019)"},{"why":"This is the HIPE-2020 English test set, one of the two historical benchmarks on which all models are compared.","marker":"Ehrmann et al. (2020b,a)"},{"why":"This defines the HIPE-2022 evaluation setting and the NER, NEC, and EL metrics that the paper adopts when reporting F1 scores.","marker":"Ehrmann et al. (2022)"},{"why":"This is ReFinED, the long-tail-aware baseline that natively includes NIL in its candidates and is the strongest specialized competitor in this comparison.","marker":"Ayoola et al. (2022)"},{"why":"This is mGENRE, the multilingual generative baseline used to represent the retrieval-versus-generation contrast in the experiments.","marker":"De Cao et al. (2022)"}],"fun_headline_variants":["Heuristics trump LLMs for linking old texts","Less is more: rule-based NIL beats GPT-4 on history","C-BLINK: simple filters outsmart large models on old entities","For historical entities, constrained guessing wins","Outsmarting GPT-4 with date and type constraints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after filtering by type and date, the correct entity, when it exists in the knowledge base, will have the highest similarity score; the paper notes that its retriever was not trained to guarantee this, and the type filter is evaluated with manually annotated entity classes rather than classes a real system would predict.","fun_headline_variants_meta":{"raw":{"variants":["Heuristics trump LLMs for linking old texts","Less is more: rule-based NIL beats GPT-4 on history","C-BLINK: simple filters outsmart large models on old entities","For historical entities, constrained guessing wins","Outsmarting GPT-4 with date and type constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1677,"prompt_tokens":997,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":599}},"tokens_in":613,"tokens_out":680,"duration_ms":7767,"temperature":1.0,"reasoning_tokens":599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:24:15.945781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count, on MHERCL, how often the gold QID is present in C-BLINK's top-$k$ candidates but is not the top-scoring plausible candidate; the paper's own error analysis shows many such cases become false NIL predictions. If that share is substantial, the assumption that score magnitude tracks correctness fails. A simpler version: rerun the best C-BLINK configuration with predicted NER types instead of manually annotated types and check whether its F1 advantage over BLINK disappears.","supporting_citations":[{"cited_title":", Conia , S","cited_arxiv_id":null,"evidence_quote":"This work grounds the type-constraint idea: filtering linkable entities by the mention's named-entity class before reranking reduces implausible links."},{"cited_title":", Zucker , S.W","cited_arxiv_id":null,"evidence_quote":"This is the relaxation-labelling foundation that ELD reformulates as a game; it provides the consistency constraints among labels that drive joint disambiguation."},{"cited_title":", Fisher , J","cited_arxiv_id":null,"evidence_quote":"This is ReFinED, the long-tail-aware baseline that natively includes NIL in its candidates and is the strongest specialized competitor in this comparison."}],"review_version":1}