REVIEW 3 major objections 7 minor 12 references
CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that feeding a GPT-4o translator gold Wikidata entity labels and descriptions, then having the model critique and revise its own output, raises entity-aware machine translation scores from the low 50s to the low 90s…
desk verdict Solid SemEval system description with a big oracle-driven RAG gain and a small, under-identified refinement gain; the +Refine claim needs a control before I would trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the RAG prompt plus the self-refinement loop. Given a source text $x$, guidance prompt $p_{\mathrm{gen}}$, and entity information $e$ (target-language label, English label, description) fetched from Wikidata, the initial translation is $y_0 = M(p_{\mathrm{gen}} \| e \| x)$. A feedback prompt $p_{\mathrm{fb}}$ makes the same model score the translation on two equally weighted criteria, entity correctness and overall quality, on a 10-point scale aligned with the task metric; a refinement prompt $p_{\mathrm{rf}}$ then rewrites the translation using the full history $y_0, fb_0, \dots, y_t, fb_t$. The design lets the model correct both wrong entity names (e.g., choosing the established Korean title '붉은 군대는 가장 강력하다' over a literal rendering) and semantic errors in the sentence.
What would settle it
Run the same pipeline on test sentences whose gold Wikidata IDs are replaced by IDs from an off-the-shelf entity linker, or by randomly corrupted IDs, and measure the M-ETA and COMET harmonic mean. If the score collapses toward the GPT-4o baseline of 33–63, the gains come from the oracle lookup rather than from the RAG-and-refine mechanism being tested.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that retrieval-augmented generation with gold entity information from Wikidata substantially improves entity-aware machine translation, and that self-refinement consistently adds further improvement. The pipeline first retrieves entity labels and descriptions using the Wikidata REST API, inserts them into a few-shot prompt, and generates an initial translation $y_0 = M(p_{\mathrm{gen}} \| e \| x)$. A feedback step scores the translation on entity-label accuracy and overall quality, and a refinement step produces $y_{t+1} = M(p_{\mathrm{rf}} \| e \| x \| y_0 \| fb_0 \| \dots \| y_t \| fb_t)$. The process stops at a perfect score or after two iterations. Across Arabic, German, Spanish, French, Italian, Japanese, Korean, Thai, Turkish, and Chinese, the harmonic mean of M-ETA and COMET rises from a GPT-4o baseline of 33–63 to 85–94 after RAG, and then by 0.19–1.66 points after refinement.
Load-bearing premise
The system assumes every test sentence comes with a correct gold Wikidata ID and that the Wikidata REST API returns an accurate target-language label and description; the model performs no entity linking or retrieval of its own, so the reported gains depend entirely on that oracle lookup.
Editorial extensions
If this is right
- Adding RAG with gold Wikidata entity info improves the harmonic mean of M-ETA and COMET by large margins on all ten language pairs tested.
- Self-refinement with the two-criteria scoring rubric improves scores on all ten language pairs, with gains between 0.19 and 1.66 percentage points.
- The model's feedback loop can correct both entity label errors (literal translation) and general translation errors (misreading the source intent).
- The two-criteria feedback design aligns the model's self-evaluation with the official metric, the harmonic mean of M-ETA and COMET.
- Entity label similarity between English and the target language (measured by edit distance) has little influence on translation accuracy, so other factors drive errors.
Reading between the lines
- Without the gold Wikidata IDs, the pipeline would need an entity linking step; the reported margin over the baseline would likely shrink because retrieval errors would propagate into the prompt. This is a testable extension flagged by the authors as future work.
- The small gains from self-refinement suggest the model already uses much of the injected entity information in the initial RAG pass; the feedback loop mainly catches residual errors.
- The two-criteria scoring scheme could transfer to other knowledge-grounded generation tasks where a named resource (a label, a definition, a canonical form) must be respected in the output.
- Because the few-shot feedback examples for nine languages were machine-translated from Korean templates, the feedback quality may vary by language; checking per-language refinement gains (0.19–1.66) against human feedback quality would clarify this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes CHILL, the Samsung Research submission to SemEval-2025 Task 2 on entity-aware machine translation (EA-MT). The system takes the gold Wikidata ID attached to each English test sentence, fetches the corresponding target-language label and description from the Wikidata REST API, and inserts this entity information into a GPT-4o translation prompt; it then runs a self-refinement loop in which the same model scores its own translation on entity correctness and overall quality (5 points each, total 10) and iteratively revises the translation, with a maximum of two iterations. On the official test set, the harmonic mean of M-ETA and COMET rises from 33-62 for a plain GPT-4o baseline to 85-94 with RAG, and by an additional 0.19-1.66 percentage points with refinement. The paper reports per-language results for all ten target languages, per-metric COMET and M-ETA scores in Appendix B, two qualitative case studies, full prompt templates, and a correlation analysis between English-foreign label edit distance and entity accuracy.
Significance. If the results are taken at face value, the paper shows that prompt-injected gold entity labels and descriptions drive large, uniform gains in entity-aware translation across ten languages, and that a second pass with self-feedback yields modest further gains on the official harmonic-mean metric. The manuscript's strengths are its transparency and simplicity: Table 5 reports COMET and M-ETA separately so the headline harmonic means can be independently verified; the initial-translation and feedback prompts are given verbatim in Listings 1-2; the system uses a single off-the-shelf model with no fine-tuning; and Section 7 openly acknowledges the reliance on gold Wikidata IDs. The scientific scope is, however, narrower than the abstract suggests: the 'RAG' step is an oracle lookup keyed to the test set's gold IDs rather than a retrieval system, and the distinctive self-refinement claim is under-identified (major comments 1-2). The paper is a solid, reproducible system description whose conclusions about the refinement mechanism require further controlled evidence before they can be credited beyond the specific task setup.
major comments (3)
- [§4.2, Eqs. (2)-(3); Tables 2 and 3; §7] The attribution of the +Refine gains to the self-evaluation mechanism is under-identified, and this is load-bearing for component (ii) of the central claim in §7. Both the feedback prompt (Eq. 2) and the refinement prompt (Eq. 3) contain the entity information e, which includes the gold target-language label, and the case study in Table 3 shows that the model's own feedback text spells out the correct label verbatim ('붉은군대는가장강력하다'). The same case study shows the model had that label in its initial prompt and did not use it, so the gain is not simply re-exposure to previously withheld information; nevertheless, the current design cannot separate the claimed self-scoring mechanism from (a) the explicit gold-labeled correction that the feedback step inserts into the context, or (b) the benefit of an additional decoding pass. No ablation removes the feedback text, removes e from the refine prompt, or re-translates without feedback. Table 2 further reports a single run per configuration, with no variance or significance testing, so 'consistently enhances translation quality across all language pairs' rests on differences as small as 0.19 percentage points that could be sampling noise. I recommend control conditions on the validation set (refine with feedback but without e; re-translation without feedback; repeated decoding) to identify the active ingredient.
- [Appendix B (Table 5); §§6.1 and 7] The paper's own per-metric results partially contradict the claim that refinement improves entity handling. From +RAG to +Refine, M-ETA decreases in 6 of 10 languages (AR 92.17 to 91.86, DE 85.46 to 85.23, ES 90.61 to 89.88, FR 90.61 to 89.95, IT 92.64 to 92.43, ZH 78.06 to 77.77), while COMET increases in all 10 languages; the harmonic mean in Table 2 consequently rises even where the entity-specific metric falls. Since §4.2 motivates the feedback mechanism with an equally weighted 5-point entity-correctness criterion, the mechanism's advertised benefit is absent from the entity metric for most language pairs. The §7 statement that 'self-refinement mechanism consistently enhances translation quality across all language pairs' is defensible only for COMET and the harmonic mean; it should be qualified, and the authors should discuss why the entity score degrades in the majority of language pairs despite the entity-focused feedback criterion.
- [§4.1 and §7] The RAG component is an oracle lookup: the test set provides the gold Wikidata IDs, and the retrieved entity information includes the gold target-language label, so Eq. (1) places the answer-key label directly in the prompt. The headline gain in Table 2 (harmonic mean from roughly 33-62 to 85-94) is therefore largely a property of the task setup rather than evidence for retrieval-augmented translation as a general method; the M-ETA values of 78-92 after RAG show the model does not always copy the provided label, so the gain is not fully automatic, but the experiment never tests entity retrieval or linking. Section 7 does acknowledge the gold-entity reliance in its future-work sentence, but the abstract and §7 conclusion still present RAG as a generally effective component. I suggest reframing the contribution as an in-task upper bound for entity-aware prompting with perfect per-sentence entity information, and stating explicitly in the abstract that no retrieval is performed.
minor comments (7)
- [§2.1] There is a subject-verb agreement error in 'These retrieved information has been successfully applied'; the manuscript needs a careful copy-edit pass.
- [Title] The title's phrasing 'Make Your LLM to Get Them Right' is ungrammatical; consider 'Make Your LLM Get Them Right'.
- [§4.2] Please report the average number of feedback-refinement cycles actually executed and the fraction of sentences that terminated on a perfect score; this would substantiate the linear-cost discussion in the same section.
- [§5.2] Please state whether the GPT-4o-generated few-shot examples for the nine non-Korean languages were manually checked, since the quality of the feedback exemplars directly affects the refinement behavior being evaluated.
- [§6.3, Table 4] Since M-ETA is treated as a binary outcome in this analysis, Spearman's rho and the point-biserial r convey nearly the same association; reporting confidence intervals or significance levels would make the 'little impact' conclusion quantitative.
- [Appendix A] The iteration prompt template is given only with placeholders such as '(entity information)' and '(language)'; a fully instantiated example, matching the level of detail in Listings 1-2, would make the refinement step fully reproducible.
- [Appendix B, Table 5] The column header 'GPT-4o +RAG +Refine' with C/M subcolumns is ambiguous; expanding 'C' and 'M' to 'COMET' and 'M-ETA' would improve readability.
Circularity Check
The +RAG entity-accuracy gain is by construction an oracle label lookup, but the self-refinement claim is not circular; overall low circularity.
-
self definitional
[Section 4.1, Eq. (1); Section 6.1]
"This significant enhancement is attributed to our utilization of oracle Wikidata IDs from the dataset, from which we extract precise entity labels and descriptions."
Equation (1) defines the initial translation as y0 = M(pgen||e||x), where e is the entity information containing the gold target-language label and description. The task metric M-ETA scores whether the entity translation exactly matches that same target-language label. Thus the +RAG entity-accuracy improvement is obtained by inserting the answer-key label into the prompt and copying it out; the entity 'prediction' is an input by construction. The paper discloses this oracle dependence, and the COMET component and the +Refine gains are not forced by this construction, so the overall circularity is partial and mild.
full rationale
The paper is a SemEval system description rather than a formal derivation, so most of its claims are empirical. The one by-construction element is the +RAG entity-accuracy gain: the gold Wikidata ID provides the exact target-language label that M-ETA checks, so the entity portion of the improvement reduces to reading the supplied label. This is disclosed as 'oracle Wikidata IDs' and is a task-setup property, not hidden circular reasoning. The distinctive +Refine claim is not circular: both +RAG and +Refine configurations contain the same entity information e (Eqs. 1-3), so the reported 0.19-1.66 percentage-point refinements are not simply the answer key reappearing; they could reflect a second exposure to e or extra decoding passes, but that is a control/attribution weakness rather than a definitional reduction. The only self-citation (Lee and Shin 2024) appears in related work and is not load-bearing. No uniqueness theorem or ansatz is smuggled in via self-citation. Overall score 2 reflects one disclosed oracle-input confound with no load-bearing self-citation chain.
Assumptions & free parameters
free parameters (2)
- max_iterations =
2
- feedback_criteria_weights =
5/5
assumptions (3)
- domain assumption The Wikidata REST API returns the correct target-language label and description for every gold Wikidata ID used.
- domain assumption COMET and M-ETA are valid automatic proxies for translation quality and entity accuracy.
- domain assumption GPT-4o's outputs are stable enough across API calls that a single evaluation run is representative.
Cite this review
Pith. "Pith review of CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right." pith.science (2026). https://pith.science/paper/SZJWMED7
@misc{pith2026250613070,
author = {Pith},
title = {Pith review of: CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZJWMED7}},
note = {Machine review of arXiv:2506.13070}
}
read the original abstract
In this paper, we describe our approach for the SemEval 2025 Task 2 on Entity-Aware Machine Translation (EA-MT). Our system aims to improve the accuracy of translating named entities by combining two key approaches: Retrieval Augmented Generation (RAG) and iterative self-refinement techniques using Large Language Models (LLMs). A distinctive feature of our system is its self-evaluation mechanism, where the LLM assesses its own translations based on two key criteria: the accuracy of entity translations and overall translation quality. We demonstrate how these methods work together and effectively improve entity handling while maintaining high-quality translations.
Reference graph
Works this paper leans on
-
[5]
arXiv preprint arXiv:2407.01626
Sparkle: En- hancing sparql generation with direct kg integration in decoding. arXiv preprint arXiv:2407.01626. Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih
-
[6]
Efficient one-pass end-to-end entity linking for questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6433–6441, Online. Association for Computational Linguistics. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao
work page 2020
-
[8]
COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Gaetano Rossiello, Nandana Mihindukulasooriya, Ibrahim Abdelaziz, Mihaela Bornea, Alfio Gliozzo, Tahira Naseem, and Pavan Kapanipathi
work page 2020
-
[9]
In The Semantic Web – ISWC 2021, pages 321–337, Cham
Gen- erative relation linking for question answering over knowledge bases. In The Semantic Web – ISWC 2021, pages 321–337, Cham. Springer International Publishing. Kugatsu Sadamitsu, Itsumi Saito, Taichi Katayama, Hisako Asano, and Yoshihiro Matsuo
work page 2021
-
[10]
Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6397–6407, Online. Association for Computa- tional Linguistics. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Has- san Awadalla
work page 2020
-
[11]
Extract and attend: Improving entity translation in neural machine translation. In Findings of the As- sociation for Computational Linguistics: ACL 2023 , pages 1697–1710, Toronto, Canada. Association for Computational Linguistics. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Gra- ham Neubig, and Satoshi Nakamura
work page 2023
-
[2018]
Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long Papers), pages 1325–1335, New Orleans, Louisiana. Association for Computa- tional Linguistics. A Iteration Prompt Template...
work page 2018
-
[2020]
Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Tom Kocmi and Christian Federmann
work page 2020
Show all 12 references
-
[2021]
In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7,
Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7,
2021
-
[2023]
In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 12009–12024, Singapore
Leveraging GPT- 4 for automatic translation post-editing. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 12009–12024, Singapore. Association for Computational Linguistics. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie
2023
-
[2024]
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16343–16360, Miami, Florida, USA
Towards cross-cultural machine translation with retrieval- augmented generation from multilingual knowledge graphs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16343–16360, Miami, Florida, USA. Associa- tion for Computation...
2024
-
[2025]
In Proceedings of the 19th In- ternational Workshop on Semantic Evaluation (Se- mEval2025)
Semeval-2025 task 2: Entity-aware machine translation. In Proceedings of the 19th In- ternational Workshop on Semantic Evaluation (Se- mEval2025). Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.