REVIEW 3 major objections 6 minor 22 references
When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read When the same knowledge base is both injected into a translator and used to grade it, the score measures obedience, not quality.
desk verdict A careful measurement of a real evaluation loop in entity-level MT, with one load-bearing provenance assumption that needs checking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the resource-shared evaluation loop: a scoring gold built from the same person-name KB that is injected into the translation prompt, so a string counts as correct merely by surviving into the output. The analysis separates the two pipelines — injection uses an NER model plus KB readings, scoring uses expert person-name tags with a KB/hanja fallback — and splits the gold into an overlapping segment (gold shares the injected resource) and an independent segment (gold does not). The main instrument is a difference-in-differences contrast using the few-shot condition as the reference, which isolates the loop's contribution from the style effect of the prompt. A fixed-denominator contrast separating the set axis (which entities are scored) from the reading axis (what counts as correct) shows the inflation comes entirely from the set axis. The near-invariance of the post-injection ceiling across models is what turns the reported gain into the complement of prior performance.
What would settle it
Trace the construction history of the person-name KB and compare its entries with the expert person-name annotations used as gold: if a substantial share share provenance, the claimed independence is inflated. A complementary check is to run the same difference-in-differences design on a corpus where the KB and the gold are verifiably disjoint and see whether the independent-segment gain stays at or below zero.
Extended reading notes
Core claim
The central discovery is quantitative: of 527 expert-annotated person-name mentions, 31.1% lie outside the injection pipeline, and the remaining 68.9% are scored against a gold that shares the injected KB. In that overlapping segment the injected reading agrees with the human translation 97.8% of the time, while in the independent segment it agrees only 70.1% of the time, so the healthiest-looking part of the answer key is the part the loop is holding up. A difference-in-differences contrast across four models shows the entire measured gain from injection is in the overlapping segment (loop contributions from +0.294 to +0.722), with independent-segment changes at or below zero; the post-injection ceiling is nearly invariant (0.910–0.996) across baselines that differ fivefold. The paper therefore claims that the reported entity-accuracy gain is inflated up to 2.7 times the honest value, and that the loop, not translation quality, is what the metric records.
Load-bearing premise
The load-bearing premise is that the expert gold used for scoring is genuinely independent of the KB used for injection; if the KB was built from the same annotations or the same human translations, the 31.1% independence figure overstates how much of the gold escapes the loop.
Editorial extensions
If this is right
- Entity-accuracy gains reported from KB injection are inflated up to 2.7 times the value measured against an independent gold, so archives should discount headline figures.
- Injection does not generalize: across all four models the independent-segment change is at or below zero, with the strongest model slightly worse.
- Weaker models appear to improve more dramatically because post-injection ceilings cluster in a 0.910–0.996 band while baselines differ fivefold; the reported gain is largely the complement of prior performance.
- A KB-derived gold penalizes more capable models more heavily, so it is unsuitable for cross-model absolute comparison even though it tracks within-condition deltas.
- Splitting the gold into overlapping and independent segments and reporting KB coverage reproduces the paper's central difference-in-differences and lets an archive estimate its own exposure.
Reading between the lines
- Editorial inference: If the person-name KB was assembled from the same expert annotations or the same human translations as the scoring gold, the independence rate and the 70.1% agreement would both be construction artifacts; a provenance audit settles this.
- Editorial inference: The paper's linear projection from 70% KB coverage to lower coverages implies that simply reporting KB coverage alongside an entity metric would let low-resource archives estimate their own distortion before paying for gold.
- Editorial inference: A hardened scoring protocol with over-generation and positional penalties might change absolute levels, but the confinement of the gain to the overlapping segment should persist because the loop operates on which entities are scored, not on how presence is checked.
- Editorial inference: The same design — separate injected from scored resources, fix the denominator, difference by segment — transfers to any low-resource domain that substitutes a dictionary for expert gold, such as clinical or legal corpora.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper measures a 'resource-shared evaluation loop' in entity-level machine translation for the Seungjeongwon Ilgi, in which the same person-name knowledge base is used both to inject names into the prompt and to score the output. Using expert person-name annotations from the National Institute of Korean History as a supposedly independent gold, the authors find that only 164 of 527 expert-annotated mentions (31.1%) lie outside the injection pipeline. In the overlapping segment the injected reading agrees with the human translation 97.8% of the time, versus 70.1% in the independent segment. Across four models, a difference-in-differences analysis shows that the entity-accuracy gain from KB injection is confined to the overlapping segment, with the independent segment at or below zero, and that post-injection ceilings cluster in a narrow 0.910–0.996 band. The main conclusion is that the KB-as-gold metric measures instruction compliance rather than translation quality, and that the reported gain is up to 2.7 times the value obtained under an independent gold.
Significance. If the measurements hold, this is a valuable and timely contribution: it turns a frequently asserted evaluation bias into a quantified, replicable measurement, and it provides a practical protocol for cultural-heritage archives. The paper ships useful transparency assets: exact McNemar paired tests with extreme counts, bootstrap confidence intervals, a replication on an unfiltered sample, released model outputs and evaluation code, and an unusually candid limitations section. It also carefully separates the set axis from the reading axis of circularity, and explicitly notes that the reported gain's complement structure follows from the near-invariant ceiling. The main load-bearing assumption, as the stress-test note correctly identifies, is the provenance independence of the expert gold from the injected KB; the manuscript asserts this independence but does not document it.
major comments (3)
- [§3.1, §4.1, §4.2] The central measurement—31.1% independent mentions, the 97.8% versus 70.1% fidelity split, and the resulting difference-in-differences—rests on the assumption that the NIKH idx_person annotations used as scoring gold are provenance-independent of the person_master KB used for injection, and of the human translations used in §5.5. The manuscript asserts this as a design fact but provides no evidence: it does not state whether person_master was compiled from the same NIKH annotations or from the same human translations, nor whether the human translations were produced independently of the idx_person tags. Because the same institute appears to be the source of both resources, this is not a trivial assumption. If the KB and the gold share provenance, then the 68.9%/31.1% split and the overlap/independent contrast are partly construction artifacts rather than empirical findings. This is externally checkable and should be documented before the 31.1% figure is used as the anchor of the paper.
- [§3.1, §7] The injection gold is a reconstruction, because the original was overwritten by the scoring gold in the original experiment. The segmentation of the 527 expert mentions into overlap and independent segments—and therefore the DiD in §5.2—depends on which entities the NER+KB pipeline actually injected in the original run. If the reconstructed injection set differs from the original, the segment labels used in the analysis may be misassigned, and the reported 31.1% independence figure and the DiD values may not correspond to the experiment as run. The manuscript acknowledges the reconstruction but does not quantify its sensitivity. Please provide a robustness analysis (for example, how much would the DiD change under plausible perturbations of the reconstructed injection set) or argue explicitly why exact set identity is not needed for the qualitative conclusion.
- [§5.1, §10.7] The 'honest' independent-gold value of 0.348 and the '2.7×' inflation figure are computed with presence-based scoring, which the paper itself notes is vulnerable to over-generation (KoBE; Alam et al.). Because injection explicitly lists the target names, a model can satisfy presence-based scoring by copying or appending the injected block, inflating the overlap segment more than the independent segment. This means the headline inflation figure may conflate the resource-sharing loop with the known scoring-protocol artifact. The paper's choice to keep the protocol fixed is defensible for measuring current practice, but the '2.7×' claim should be either repeated under the hardened protocol as a robustness check or explicitly qualified as an upper bound that includes the over-generation artifact.
minor comments (6)
- [Figure 2] Figure 2 is referenced in §3 but does not appear in the manuscript text; the pipeline diagram should be included so that the claimed separation is visually verifiable.
- [Abstract and §4.1] The abstract and §4 use 527 scored mentions, while §5 tables use 515 mentions (or 361 on the common set); the denominator distinction should be stated at the first occurrence in the abstract or in a footnote to Table 1 to avoid confusion.
- [§5.2 Table 2] The repeated value −0.026 for three models is coincidental (−3/117 for each), and while the text explains this, the table itself should carry a footnote or dagger so that readers do not infer a shared cause.
- [§7 Limitations] The bullet 'The style comparison is blind but conducted by the authors' is unclear; please specify what was blinded (e.g., condition labels, model identity) and how blinding was implemented.
- [§3.2, §8] The NER model 'SillokBERT-NER' is used for the injection pipeline but is not described or cited; a reference or a brief description of its training data and output format is needed, especially given the alias/instability discussion for gemma-4-26b in §8.
- [§3.1] The phrase 'This separation is an operational fact before it is a methodological requirement' is misleading: the separation of the gold from the injection pipeline is an assumption about resource provenance, not an operational fact, and this distinction should be stated explicitly.
Circularity Check
No significant circularity: the paper measures a known loop with externally anchored expert annotations and explicitly discloses the only definitional identities.
full rationale
The paper's derivation chain is self-contained and does not reduce to its inputs. The central measurements are anchored in external resources: NIKH expert person-name annotations, human translations, four externally released models, and a fixed presence-based scoring rule inherited from prior work. No parameter is fitted to produce the main numbers, and no prediction is derived from a fitted constant. The two places where the text itself flags definitional structure are explicitly quarantined: the overlap row's 100% KB-registration is stated to hold 'by construction' and is not offered as an empirical finding, and the note in §5.3 concedes that 'Δoverlap = ceiling − pre is an identity by definition' while correctly pointing out that the empirical content lies in the narrow clustering of the ceilings, not in the identity. The difference-in-differences estimator in §5.2 is a standard treatment/control contrast, not a tautology; the overlap gain being large is the phenomenon under study, and the paper uses the negative independent-segment spillover to argue the contrast is not vacuous. The remaining assumption—that the NIKH expert gold is genuinely independent of the person_master KB—is a provenance/validity caveat rather than a circular step, because the paper does not define the KB in terms of the gold or vice versa and offers no self-citation chain in its place. The paper is also candid about its limitations: the injection gold is a reconstruction, the unfiltered replication covers only two models, and the style comparison is preliminary. No enumerated circularity pattern (self-definition, fitted-input-as-prediction, load-bearing self-citation, imported uniqueness, ansatz-by-citation, or renaming) is instantiated with quoteable evidence.
Assumptions & free parameters
free parameters (4)
- BLEU >= 20 construction filter =
20
- Stratified sampling seeds =
42 (original), 20250729 (control)
- Misalignment ratio thresholds =
0.35x and 3x of median source/translation length ratio
- Excluded single-character entity tags =
12 (9 in the independent segment)
assumptions (5)
- domain assumption NIKH expert person-name annotations and the human translations used as reference are correct and independent of the KB and injection pipeline.
- domain assumption Few-shot exemplars contain no substantive person-name information relevant to the evaluation gold.
- domain assumption The reconstructed injection gold is structurally equivalent to the lost original.
- domain assumption Presence-based scoring (gold string appears in output) is adequate for comparing conditions.
- standard math McNemar's exact test and document-level cluster bootstrap are valid for the paired entity data.
Cite this review
Pith. "Pith review of When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation." pith.science (2026). https://pith.science/paper/XF33ALU2
@misc{pith2026260811843,
author = {Pith},
title = {Pith review of: When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/XF33ALU2}},
note = {Machine review of arXiv:2608.11843}
}
read the original abstract
The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name -- a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform -- in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) -- it reflects a property of the model, not of the sample.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction 1.1 One worked example The hanja library renders 沈as chim and 金as geum — the readings for “to sink” and “metal.” As surnames the correct readings are Sim and Kim, and 金is the most common Korean surname. Source ○ 右相沈悅四度呈辭。答曰, 安心調理。 Injected [Persons — use exactly these Korean names] → 沈悅 침열 (Chim-yeol) ← WRONG Human … 우상심열이 Sim-yeol ← correct ...
work page 2021
-
[2]
Of 527 expert annotations, 31.1% lie outside the injection pipeline; for NER Recall the figure is 0%
We quantify how independent the gold actually is. Of 527 expert annotations, 31.1% lie outside the injection pipeline; for NER Recall the figure is 0%. To our knowledge this quantity has never been reported for entity-level MT evaluation
-
[3]
We show what the inflation is a function of. Its magnitude is set by KB coverage and model capability; in particular, the post-injection ceiling is near-invariant across models, which makes the reported gain the complement of prior performance
-
[4]
A Seungjeongwon Ilgi case study: four models, three conditions, a stratified sample of 300 documents, with all model outputs released. Hardening the scoring protocol (over-generation penalty, positional check) is left to §10 as future work. All measurements here use the same presence-based scoring that prior work uses — changing the metric would confound ...
-
[5]
Shared Heritage, Distinct Writing
Related Work 2.1 NLP for Han-character historical records Computational work on Joseon-period Sinitic records accumulated rapidly through the 2020s. HUE (Yoo et al., Findings of NAACL 2022) released a benchmark bundling chronological attribution, topic classification, named entity recognition and summary retrieval over the Annals of the Joseon Dynasty and...
work page 2021
-
[6]
Secondary Findings 6.1 Classical Chinese pretraining does not transfer Model Parameters BLEU(c) baseline ETS Qwen3-8B 8B 14.07 0.396 gemma-3n-E2B 5B (2B effective) 13.97 0.499 Model Parameters BLEU(c) baseline ETS gemma-4-26b 26B 34.41 0.723 Qwen3-8B, which carries a Classical Chinese (文言文) corpus, scores below a 2B-class model without one. Sharing the Ha...
work page 2025
-
[7]
The injection and scoring pipelines, and the segment where they still share a resource
Experimental Design Figure 2. The injection and scoring pipelines, and the segment where they still share a resource. The entity sets are separate but the resource supplying the Korean reading is not, which is why the loop survives on 68.9% of the gold. 3.1 Separating the two pipelines The system and the evaluation must not share resources. Our configurat...
-
[8]
All counts in §4 are stated over the 527 expert tags whose reading could be resolved
How Independent Is the Gold? 4.1 Injection set versus scoring set Count Share of scoring set Scoring set (expert tags) 527 100% ├ Overlap — injected and scored 363 68.9% └ Independent — never injected 164 31.1% Injection-only (NER found, experts did not tag) 22 — Two denominators are used in this paper and they must not be conflated. All counts in §4 are ...
Show all 22 references
-
[9]
of course there is no effect where nothing was injected
How Much Does the Loop Inflate? 5.1 A three-level decline under changing gold provenance The same outputs, scored against golds of differing provenance (Qwen3-8B, NERinject): Gold provenance Loop Value NER Recall (injection set = scoring set) complete 0.944 ETS overall (expert...
-
[10]
append the term at the end
Future Work (1) Extending gold coverage. Our distortion estimate comes from a setting with 70% KB coverage. The projection that lower-coverage domains are more exposed is a linear extrapolation and should be measured directly in a low-coverage archive. (2) Broader entity types...
2021
-
[11]
Limitations • Prior-art coverage comprises ACL Anthology, arXiv, Springer, the public CCL/CNKI indices, and the KCI / DBpia / KISS web indices. Korean work on classical-Chinese MT, historical entity recognition and automatic translation evaluation was identified, but no Korean...
-
[12]
Reproducibility as a Result, Not an Excuse During this study the served behaviour of the alias gemma-4-26b-a4b-it changed. Under identical code, parameters and prompts, mean output length fell from 244 to 34 characters and 185 of 300 outputs became empty: the model emits <thou...
-
[13]
But a zero budget also means no money for an expert gold standard, so a knowledge base is substituted — and that is exactly where the loop appears
Implications for Cultural-Heritage Practice 9.1 Why this arises only in low-resource archives The chain runs: no fine-tuning means no GPU is needed; if the method works on free models, an archive can adopt it at zero budget. But a zero budget also means no money for an expert ...
2023
-
[14]
Here we injected from NER output and scored against expert annotations; once separated, an independence rate can be reported
Build the evaluation key from a resource disjoint from the injected one. Here we injected from NER output and scored against expert annotations; once separated, an independence rate can be reported
-
[15]
Where separation is impossible, report KB coverage and the distortion projected from it, so readers can discount the reported figure accordingly
-
[16]
Separate the segment whose gold overlaps the system’s resources from the segment that does not; this alone reproduces the difference-in-differences of §5.2
Do not report an entity metric as a single number. Separate the segment whose gold overlaps the system’s resources from the segment that does not; this alone reproduces the difference-in-differences of §5.2
-
[17]
If the gain approximates the complement of prior performance, it is likely an artefact of the measurement rather than an effect of the method
When a weak model shows a large gain, check for a ceiling effect first. If the gain approximates the complement of prior performance, it is likely an artefact of the measurement rather than an effect of the method
-
[18]
As §5.5 shows, such a gold penalises the more competent model more heavily
Do not compare absolute performance across models with a KB-derived gold. As §5.5 shows, such a gold penalises the more competent model more heavily. 9.4 Relation to adjacent work in the field Applying large language models to cultural-heritage material and dissecting the resu...
-
[21]
development of reliable translation evaluation methods
References Korean-language sources Kim, Woo-jeong [Kim W.] (2021). Status and Challenges of Machine Translation of Classical Chinese ( 古典文言文기계번역의현황과과제 ). Chinese Literature 109, 21–51. — names “development of reliable translation evaluation methods” among the required tasks. B...
2021 arXiv
-
[2008]
circularity — leaking the evaluation signal into the system
showed that using other systems’ outputs as pseudo-references favours systems of the same family. More recently Panickssery et al. (NeurIPS 2024) quantified self-preference: LLM judges recognise and favour their own generations. The benchmark-contamination literature covers th...
2024
-
[2024]
Our contribution to that line is methodological: rather than adding a performance figure, we ask under what conditions such figures can be trusted
and domain-specific entity recognition for archaeology (Brandsen et al., 2022), both published in this journal. Our contribution to that line is methodological: rather than adding a performance figure, we ask under what conditions such figures can be trusted
2022
-
[2557]
circularity — leaking the evaluation signal into the system
— introduces M-ETA (Manual Entity Translation Accuracy). Dietz, L. et al. (2025). LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations. arXiv:2504.19076. — names “circularity — leaking the evaluation signal into the system” as the first evaluation trope. Diet...
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.