Pith. sign in

REVIEW 3 major objections 7 minor 12 references

CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper claims that feeding a GPT-4o translator gold Wikidata entity labels and descriptions, then having the model critique and revise its own output, raises entity-aware machine translation scores from the low 50s to the low 90s…

desk verdict Solid SemEval system description with a big oracle-driven RAG gain and a small, under-identified refinement gain; the +Refine claim needs a control before I would trust it. read the letter →

arxiv 2506.13070 v1 pith:SZJWMED7 submitted 2025-06-16 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords entity-awaremachinetranslationretrieval-augmentedgenerationself-refinementWikidatalargelanguagemodelstranscreationM-ETACOMET
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses entity-aware machine translation, where named entities must be rendered by their established target-language names rather than literal translations. It argues that giving GPT-4o the gold Wikidata labels and descriptions for the entities in each sentence, via retrieval-augmented generation, lifts the harmonic mean of entity accuracy (M-ETA) and overall quality (COMET) from roughly 33–63 to 85–94 across ten languages. Adding a self-refinement loop, in which the same model scores its own translation (entity correctness 5, translation quality 5) and revises accordingly, yields consistent but smaller gains of 0.19–1.66 percentage points. The paper's core message is that entity information must be actively injected and explicitly checked; the model will not reliably use it otherwise.

What carries the argument

The load-bearing machinery is the RAG prompt plus the self-refinement loop. Given a source text $x$, guidance prompt $p_{\mathrm{gen}}$, and entity information $e$ (target-language label, English label, description) fetched from Wikidata, the initial translation is $y_0 = M(p_{\mathrm{gen}} \| e \| x)$. A feedback prompt $p_{\mathrm{fb}}$ makes the same model score the translation on two equally weighted criteria, entity correctness and overall quality, on a 10-point scale aligned with the task metric; a refinement prompt $p_{\mathrm{rf}}$ then rewrites the translation using the full history $y_0, fb_0, \dots, y_t, fb_t$. The design lets the model correct both wrong entity names (e.g., choosing the established Korean title '붉은 군대는 가장 강력하다' over a literal rendering) and semantic errors in the sentence.

What would settle it

Run the same pipeline on test sentences whose gold Wikidata IDs are replaced by IDs from an off-the-shelf entity linker, or by randomly corrupted IDs, and measure the M-ETA and COMET harmonic mean. If the score collapses toward the GPT-4o baseline of 33–63, the gains come from the oracle lookup rather than from the RAG-and-refine mechanism being tested.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that retrieval-augmented generation with gold entity information from Wikidata substantially improves entity-aware machine translation, and that self-refinement consistently adds further improvement. The pipeline first retrieves entity labels and descriptions using the Wikidata REST API, inserts them into a few-shot prompt, and generates an initial translation $y_0 = M(p_{\mathrm{gen}} \| e \| x)$. A feedback step scores the translation on entity-label accuracy and overall quality, and a refinement step produces $y_{t+1} = M(p_{\mathrm{rf}} \| e \| x \| y_0 \| fb_0 \| \dots \| y_t \| fb_t)$. The process stops at a perfect score or after two iterations. Across Arabic, German, Spanish, French, Italian, Japanese, Korean, Thai, Turkish, and Chinese, the harmonic mean of M-ETA and COMET rises from a GPT-4o baseline of 33–63 to 85–94 after RAG, and then by 0.19–1.66 points after refinement.

Load-bearing premise

The system assumes every test sentence comes with a correct gold Wikidata ID and that the Wikidata REST API returns an accurate target-language label and description; the model performs no entity linking or retrieval of its own, so the reported gains depend entirely on that oracle lookup.

Editorial extensions

If this is right

  • Adding RAG with gold Wikidata entity info improves the harmonic mean of M-ETA and COMET by large margins on all ten language pairs tested.
  • Self-refinement with the two-criteria scoring rubric improves scores on all ten language pairs, with gains between 0.19 and 1.66 percentage points.
  • The model's feedback loop can correct both entity label errors (literal translation) and general translation errors (misreading the source intent).
  • The two-criteria feedback design aligns the model's self-evaluation with the official metric, the harmonic mean of M-ETA and COMET.
  • Entity label similarity between English and the target language (measured by edit distance) has little influence on translation accuracy, so other factors drive errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Without the gold Wikidata IDs, the pipeline would need an entity linking step; the reported margin over the baseline would likely shrink because retrieval errors would propagate into the prompt. This is a testable extension flagged by the authors as future work.
  • The small gains from self-refinement suggest the model already uses much of the injected entity information in the initial RAG pass; the feedback loop mainly catches residual errors.
  • The two-criteria scoring scheme could transfer to other knowledge-grounded generation tasks where a named resource (a label, a definition, a canonical form) must be respected in the output.
  • Because the few-shot feedback examples for nine languages were machine-translated from Korean templates, the feedback quality may vary by language; checking per-language refinement gains (0.19–1.66) against human feedback quality would clarify this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper describes CHILL, the Samsung Research submission to SemEval-2025 Task 2 on entity-aware machine translation (EA-MT). The system takes the gold Wikidata ID attached to each English test sentence, fetches the corresponding target-language label and description from the Wikidata REST API, and inserts this entity information into a GPT-4o translation prompt; it then runs a self-refinement loop in which the same model scores its own translation on entity correctness and overall quality (5 points each, total 10) and iteratively revises the translation, with a maximum of two iterations. On the official test set, the harmonic mean of M-ETA and COMET rises from 33-62 for a plain GPT-4o baseline to 85-94 with RAG, and by an additional 0.19-1.66 percentage points with refinement. The paper reports per-language results for all ten target languages, per-metric COMET and M-ETA scores in Appendix B, two qualitative case studies, full prompt templates, and a correlation analysis between English-foreign label edit distance and entity accuracy.

Significance. If the results are taken at face value, the paper shows that prompt-injected gold entity labels and descriptions drive large, uniform gains in entity-aware translation across ten languages, and that a second pass with self-feedback yields modest further gains on the official harmonic-mean metric. The manuscript's strengths are its transparency and simplicity: Table 5 reports COMET and M-ETA separately so the headline harmonic means can be independently verified; the initial-translation and feedback prompts are given verbatim in Listings 1-2; the system uses a single off-the-shelf model with no fine-tuning; and Section 7 openly acknowledges the reliance on gold Wikidata IDs. The scientific scope is, however, narrower than the abstract suggests: the 'RAG' step is an oracle lookup keyed to the test set's gold IDs rather than a retrieval system, and the distinctive self-refinement claim is under-identified (major comments 1-2). The paper is a solid, reproducible system description whose conclusions about the refinement mechanism require further controlled evidence before they can be credited beyond the specific task setup.

major comments (3)
  1. [§4.2, Eqs. (2)-(3); Tables 2 and 3; §7] The attribution of the +Refine gains to the self-evaluation mechanism is under-identified, and this is load-bearing for component (ii) of the central claim in §7. Both the feedback prompt (Eq. 2) and the refinement prompt (Eq. 3) contain the entity information e, which includes the gold target-language label, and the case study in Table 3 shows that the model's own feedback text spells out the correct label verbatim ('붉은군대는가장강력하다'). The same case study shows the model had that label in its initial prompt and did not use it, so the gain is not simply re-exposure to previously withheld information; nevertheless, the current design cannot separate the claimed self-scoring mechanism from (a) the explicit gold-labeled correction that the feedback step inserts into the context, or (b) the benefit of an additional decoding pass. No ablation removes the feedback text, removes e from the refine prompt, or re-translates without feedback. Table 2 further reports a single run per configuration, with no variance or significance testing, so 'consistently enhances translation quality across all language pairs' rests on differences as small as 0.19 percentage points that could be sampling noise. I recommend control conditions on the validation set (refine with feedback but without e; re-translation without feedback; repeated decoding) to identify the active ingredient.
  2. [Appendix B (Table 5); §§6.1 and 7] The paper's own per-metric results partially contradict the claim that refinement improves entity handling. From +RAG to +Refine, M-ETA decreases in 6 of 10 languages (AR 92.17 to 91.86, DE 85.46 to 85.23, ES 90.61 to 89.88, FR 90.61 to 89.95, IT 92.64 to 92.43, ZH 78.06 to 77.77), while COMET increases in all 10 languages; the harmonic mean in Table 2 consequently rises even where the entity-specific metric falls. Since §4.2 motivates the feedback mechanism with an equally weighted 5-point entity-correctness criterion, the mechanism's advertised benefit is absent from the entity metric for most language pairs. The §7 statement that 'self-refinement mechanism consistently enhances translation quality across all language pairs' is defensible only for COMET and the harmonic mean; it should be qualified, and the authors should discuss why the entity score degrades in the majority of language pairs despite the entity-focused feedback criterion.
  3. [§4.1 and §7] The RAG component is an oracle lookup: the test set provides the gold Wikidata IDs, and the retrieved entity information includes the gold target-language label, so Eq. (1) places the answer-key label directly in the prompt. The headline gain in Table 2 (harmonic mean from roughly 33-62 to 85-94) is therefore largely a property of the task setup rather than evidence for retrieval-augmented translation as a general method; the M-ETA values of 78-92 after RAG show the model does not always copy the provided label, so the gain is not fully automatic, but the experiment never tests entity retrieval or linking. Section 7 does acknowledge the gold-entity reliance in its future-work sentence, but the abstract and §7 conclusion still present RAG as a generally effective component. I suggest reframing the contribution as an in-task upper bound for entity-aware prompting with perfect per-sentence entity information, and stating explicitly in the abstract that no retrieval is performed.
minor comments (7)
  1. [§2.1] There is a subject-verb agreement error in 'These retrieved information has been successfully applied'; the manuscript needs a careful copy-edit pass.
  2. [Title] The title's phrasing 'Make Your LLM to Get Them Right' is ungrammatical; consider 'Make Your LLM Get Them Right'.
  3. [§4.2] Please report the average number of feedback-refinement cycles actually executed and the fraction of sentences that terminated on a perfect score; this would substantiate the linear-cost discussion in the same section.
  4. [§5.2] Please state whether the GPT-4o-generated few-shot examples for the nine non-Korean languages were manually checked, since the quality of the feedback exemplars directly affects the refinement behavior being evaluated.
  5. [§6.3, Table 4] Since M-ETA is treated as a binary outcome in this analysis, Spearman's rho and the point-biserial r convey nearly the same association; reporting confidence intervals or significance levels would make the 'little impact' conclusion quantitative.
  6. [Appendix A] The iteration prompt template is given only with placeholders such as '(entity information)' and '(language)'; a fully instantiated example, matching the level of detail in Listings 1-2, would make the refinement step fully reproducible.
  7. [Appendix B, Table 5] The column header 'GPT-4o +RAG +Refine' with C/M subcolumns is ambiguous; expanding 'C' and 'M' to 'COMET' and 'M-ETA' would improve readability.

Circularity Check

1 steps flagged · score 2.0 of 10

The +RAG entity-accuracy gain is by construction an oracle label lookup, but the self-refinement claim is not circular; overall low circularity.

  1. self definitional [Section 4.1, Eq. (1); Section 6.1]
    "This significant enhancement is attributed to our utilization of oracle Wikidata IDs from the dataset, from which we extract precise entity labels and descriptions."

    Equation (1) defines the initial translation as y0 = M(pgen||e||x), where e is the entity information containing the gold target-language label and description. The task metric M-ETA scores whether the entity translation exactly matches that same target-language label. Thus the +RAG entity-accuracy improvement is obtained by inserting the answer-key label into the prompt and copying it out; the entity 'prediction' is an input by construction. The paper discloses this oracle dependence, and the COMET component and the +Refine gains are not forced by this construction, so the overall circularity is partial and mild.

full rationale

The paper is a SemEval system description rather than a formal derivation, so most of its claims are empirical. The one by-construction element is the +RAG entity-accuracy gain: the gold Wikidata ID provides the exact target-language label that M-ETA checks, so the entity portion of the improvement reduces to reading the supplied label. This is disclosed as 'oracle Wikidata IDs' and is a task-setup property, not hidden circular reasoning. The distinctive +Refine claim is not circular: both +RAG and +Refine configurations contain the same entity information e (Eqs. 1-3), so the reported 0.19-1.66 percentage-point refinements are not simply the answer key reappearing; they could reflect a second exposure to e or extra decoding passes, but that is a control/attribution weakness rather than a definitional reduction. The only self-citation (Lee and Shin 2024) appears in related work and is not load-bearing. No uniqueness theorem or ansatz is smuggled in via self-citation. Overall score 2 reflects one disclosed oracle-input confound with no load-bearing self-citation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The system introduces no free parameters in the sense of fitted values, aside from the hand-chosen iteration count and equal feedback weights. It relies on three domain assumptions about the Wikidata API, the automatic metrics, and GPT-4o stability. No invented entities.

free parameters (2)
  • max_iterations = 2
    Set to 2 due to budget constraints (Section 4.2); the +Refine results in Table 2 depend on this stopping rule.
  • feedback_criteria_weights = 5/5
    Entity correctness and overall translation are weighted equally (5 points each) to align with the task metric's harmonic mean (Section 4.2); this choice is not derived from data.
assumptions (3)
  • domain assumption The Wikidata REST API returns the correct target-language label and description for every gold Wikidata ID used.
    Section 4.1 retrieves entity information using the Wikidata REST API; the large M-ETA gains in Table 2 depend on these labels being present and correct.
  • domain assumption COMET and M-ETA are valid automatic proxies for translation quality and entity accuracy.
    Section 5.3 defines the evaluation as the harmonic mean of COMET and M-ETA; all reported improvements are measured with respect to these metrics.
  • domain assumption GPT-4o's outputs are stable enough across API calls that a single evaluation run is representative.
    Section 5.1 uses GPT-4o without fine-tuning and Section 6.1 reports one set of numbers per configuration, with no repeated runs or confidence intervals; the consistency of the small +Refine gains is therefore assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right." pith.science (2026). https://pith.science/paper/SZJWMED7

@misc{pith2026250613070,
  author       = {Pith},
  title        = {Pith review of: CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZJWMED7}},
  note         = {Machine review of arXiv:2506.13070}
}
read the original abstract

In this paper, we describe our approach for the SemEval 2025 Task 2 on Entity-Aware Machine Translation (EA-MT). Our system aims to improve the accuracy of translating named entities by combining two key approaches: Retrieval Augmented Generation (RAG) and iterative self-refinement techniques using Large Language Models (LLMs). A distinctive feature of our system is its self-evaluation mechanism, where the LLM assesses its own translations based on two key criteria: the accuracy of entity translations and overall translation quality. We demonstrate how these methods work together and effectively improve entity handling while maintaining high-quality translations.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [5]

    arXiv preprint arXiv:2407.01626

    Sparkle: En- hancing sparql generation with direct kg integration in decoding. arXiv preprint arXiv:2407.01626. Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih

  2. [6]

    In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6433–6441, Online

    Efficient one-pass end-to-end entity linking for questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6433–6441, Online. Association for Computational Linguistics. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao

  3. [8]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online

    COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Gaetano Rossiello, Nandana Mihindukulasooriya, Ibrahim Abdelaziz, Mihaela Bornea, Alfio Gliozzo, Tahira Naseem, and Pavan Kapanipathi

  4. [9]

    In The Semantic Web – ISWC 2021, pages 321–337, Cham

    Gen- erative relation linking for question answering over knowledge bases. In The Semantic Web – ISWC 2021, pages 321–337, Cham. Springer International Publishing. Kugatsu Sadamitsu, Itsumi Saito, Taichi Katayama, Hisako Asano, and Yoshihiro Matsuo

  5. [10]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6397–6407, Online

    Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6397–6407, Online. Association for Computa- tional Linguistics. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Has- san Awadalla

  6. [11]

    In Findings of the As- sociation for Computational Linguistics: ACL 2023 , pages 1697–1710, Toronto, Canada

    Extract and attend: Improving entity translation in neural machine translation. In Findings of the As- sociation for Computational Linguistics: ACL 2023 , pages 1697–1710, Toronto, Canada. Association for Computational Linguistics. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Gra- ham Neubig, and Satoshi Nakamura

  7. [2018]

    Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long Papers), pages 1325–1335, New Orleans, Louisiana. Association for Computa- tional Linguistics. A Iteration Prompt Template...

  8. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online

    Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Tom Kocmi and Christian Federmann

Show all 12 references
  1. [2021]

    In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7,

    Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7,

  2. [2023]

    In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 12009–12024, Singapore

    Leveraging GPT- 4 for automatic translation post-editing. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 12009–12024, Singapore. Association for Computational Linguistics. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie

  3. [2024]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16343–16360, Miami, Florida, USA

    Towards cross-cultural machine translation with retrieval- augmented generation from multilingual knowledge graphs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16343–16360, Miami, Florida, USA. Associa- tion for Computation...

  4. [2025]

    In Proceedings of the 19th In- ternational Workshop on Semantic Evaluation (Se- mEval2025)

    Semeval-2025 task 2: Entity-aware machine translation. In Proceedings of the 19th In- ternational Workshop on Semantic Evaluation (Se- mEval2025). Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.