{"id":"e232fa02-b9b5-47b0-b398-daae5bd7907f","arxiv_id":"2411.08348","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-step prompting method using keyword extraction, dictionary retrieval, and iterative self-checking yields modest and inconsistent BLEU gains for LLM translation.","lead":"This paper describes a prompt method that helps large language models translate rare words more faithfully by picking key words, looking up their translations in a dictionary, and asking the model to check and refine its output. It is tested on 10 language pairs and recent WMT data, with mostly small gains and a few losses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim unsupported: no comparison to the dictionary-prompting baselines cited in the paper, and own tables contain unquantified degradations.","rationale":"The paper's strongest claim is an empirical one, so the most load-bearing requirement is that the proposed multi-step prompting chain outperforms the relevant prior methods. The manuscript itself points to Ghazvininejad et al. (2023) and Lu et al. (2023) as dictionary-prompting predecessors, yet the experimental setup in Section 4 compares only against a plain zero/few-shot prompt. This omission is not cosmetic: the proposed components (LLM keyword scoring, MUSE retrieval, iterative self-check) are layered on top of a dictionary-constrained prompt, and without those cited systems in the tables the observed BLEU deltas could come entirely from the known practice of appending translated terms. The negative results in Table 2 reinforce the concern because they demonstrate that injecting constraints can hurt, and the paper's explanation in Section 5.1 is post-hoc and not tested. I partially agree with the reader's weakest assumption about dictionary retrieval and priority scores; those are genuine risks, but they are secondary to the missing baseline comparison. Even a perfect retrieval oracle would not validate the claim unless the method also beats prior dictionary prompting with uncertainty quantification. Consequently, the appropriate verdict is the same rejection reached by the reader, although the decisive condition is the absent controlled comparison rather than the dictionary-retrieval assumption.","tokens_in":11170,"tokens_out":6867,"duration_ms":67760,"concrete_test":"Re-run all Table 2 language pairs and the WMT22/23/24 EN-DE sets with three systems: (1) the proposed method, (2) Ghazvininejad et al. (2023) dictionary-based phrase-level prompting, and (3) Lu et al. (2023) Chain-of-Dictionary prompting, using the same MUSE dictionary, same Llama-3.1-8B and Qwen2-7B models, and same evaluation scripts. Report BLEU and chrF++ with paired bootstrap 95% confidence intervals over the sentences. If the proposed method does not beat both dictionary-prompting baselines by a margin exceeding the interval on the majority of language pairs, the central effectiveness claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is empirical: the proposed constraint-aware prompting 'significantly' outperforms baselines and achieves state-of-the-art results. The load-bearing condition is that the method beats the dictionary-prompting approaches it builds on. Section 2.1 cites Ghazvininejad et al. (2023) and Lu et al. (2023), both dictionary-based prompting methods, but Section 4 defines the baseline as only a plain 'Translate the following sentence' prompt with randomly selected few-shot examples. None of the tables compare against those cited systems or against a simple prompt that just appends dictionary translations. Consequently, even if keyword selection and bge-m3 retrieval are perfect, the experiments cannot distinguish the proposed contribution from simply adding dictionary entries to the prompt. Table 2 also contains negative pairs (Llama hr-en BLEU 33.42 to 29.91, ms-en 37.38 to 36.20, no-en 39.47 to 39.04; Qwen nl-en 28.44 to 26.77, sk-en 30.99 to 29.14), and no variance or paired significance test is reported anywhere, including Table 3. The reader's dictionary-retrieval concern is real but secondary: a perfect oracle dictionary would still leave the missing-baseline problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, multi-step prompting method for LLM-based machine translation. It first asks the LLM to assign priority scores to source words, selects the top-k keywords, retrieves their translations from a bilingual dictionary using bge-m3 embeddings, injects these translations as lexical constraints into the prompt, and then applies iterative self-checking and a final self-selection step between the initial and refined translations. Experiments on FLORES-200 low-resource language pairs and WMT22-24 EN-DE test sets, using Meta-Llama-3.1-8B-Instruct and Qwen2-7B-Instruct, report BLEU and chrF++ scores relative to a simple prompt baseline.","tokens_in":11468,"tokens_out":4763,"duration_ms":47609,"significance":"If the claimed improvements were robust, the method would be a useful training-free recipe for reducing rare-word errors and hallucinations in 7B-scale LLM translation. The paper uses external sacreBLEU metrics, evaluates on contamination-aware WMT test sets, has no fitted parameters, and candidly lists dictionary dependence as a limitation. However, the current evidence does not support the abstract's claim of 'significant improvements': the baseline is too weak, several language pairs degrade under the proposed method, and no significance testing is provided. The core idea is plausible but remains unvalidated as presented.","major_comments":[{"comment":"The baseline is only a plain 'Translate the following sentence' prompt with randomly selected few-shot examples. Section 2.1 cites Ghazvininejad et al. (2023) and Lu et al. (2023) as dictionary-based prompting methods, but the experiments never compare against these systems or against a simple prompt that appends the same dictionary translations. Consequently, the experiments cannot isolate the proposed keyword-selection and self-checking contribution from the effect of merely adding dictionary entries to the prompt, which is the load-bearing claim of the paper.","section":"Section 4, Baseline; Tables 2 and 3"},{"comment":"Several language pairs show degraded scores under the proposed method, including Llama hr-en BLEU 33.42 to 29.91 and chrF++ 60.72 to 55.38, Llama ms-en BLEU 37.38 to 36.20, Llama no-en BLEU 39.47 to 39.04, Qwen nl-en BLEU 28.44 to 26.77, and Qwen sk-en BLEU 30.99 to 29.14. The abstract states that 'significant improvements over baselines' were demonstrated, and Section 5.1 only vaguely mentions a 'slight decline' without quantifying these drops. The authors need to report paired significance tests, confidence intervals, or at least a detailed error analysis explaining these negative cases.","section":"Table 2"},{"comment":"No variance, confidence interval, or paired significance test is reported anywhere. Many differences are less than 1 BLEU point (e.g., ca-en 41.48 to 41.95, da-en 43.74 to 44.24), and without statistical testing the claimed improvements cannot be distinguished from noise. This is especially important given the observed degradations in other language pairs; an aggregate or per-pair significance analysis is needed to support the central empirical claim.","section":"Tables 2 and 3; Section 5"},{"comment":"The retrieval step assumes that the nearest-neighbor match in the MUSE bilingual dictionary gives the correct translation for the keyword in its actual sentential context. The paper calls these 'ground-truth bilingual dictionaries' in Section 4, but the Conneau et al. (2017) MUSE dictionaries are induced, not ground-truth, and they do not resolve polysemy. If a retrieved dictionary translation is contextually wrong, injecting it as a hard constraint can harm output; the degradations in Table 2 are consistent with this failure mode. The paper should evaluate retrieval accuracy in context and report how often a retrieved constraint is actually used correctly, rather than assuming dictionary entries are always reliable.","section":"Section 3.2, Eq. (4); Section 4, Dictionary"},{"comment":"The final selection step uses the same LLM to choose between the initial translation Y1 and the refined translation Yfinal based on the lexical constraints and fluency. Since the LLM generated both candidates, this self-selection may be biased, and the paper does not validate that the model's chosen output correlates with the external BLEU/chrF++ metrics used in Tables 2 and 3. Reporting an oracle comparison (which of Y1 and Yfinal scores better externally) or the agreement rate between the self-selection and the external metric would clarify whether Eq. (8) actually helps or hurts the final scores.","section":"Section 3.3, Eq. (8)"}],"minor_comments":[{"comment":"The notation is inconsistent: the text defines the source sentence as X = {x1,...,xn} and then refers to si and wi, while Eq. (1) uses xi and Eq. (2) uses wi with wi ∈ X. Please unify the notation.","section":"Section 3.1, Eqs. (1)-(2)"},{"comment":"The model name appears as 'Mete-Llama-3.1-8B-Instruct' in the text but 'Meta-Llama-3.1-8B-Instruct' in Table 2; please correct the typo.","section":"Section 4, Models"},{"comment":"The sentence 'These dictionaries command.3 were meticulously crafted using Meta's internal translation tool' is garbled and incomplete; it should be rewritten.","section":"Section 4, Dictionary"},{"comment":"There is a typo 'free from he potential issues'; also Table 3 reports only BLEU without chrF++ and does not state the translation direction, so the results are not directly comparable to Table 2.","section":"Section 5.2"},{"comment":"The value of k is described as 'an adaptive threshold based on the length of the sentence,' but no formula or procedure is given. Without this detail, the method is not reproducible. The same applies to the number of self-checking iterations used in the experiments.","section":"Section 3.1, k selection"},{"comment":"Zhong and Chiang (2020) is marked as retracted but is cited as support for contamination-free WMT evaluation; please clarify whether the retracted version is still the appropriate citation, or replace it with the relevant non-retracted literature.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an early arXiv preprint with substantial presentation issues. The decisive problem is experimental: the missing dictionary-prompting baselines and the absence of significance testing make the central claim unverifiable. The negative results in Table 2 are not fatal by themselves, but they are not analyzed, and the abstract overstates the findings. I would like to see a revision with the missing baselines, significance tests, and a corrected abstract before this can be considered for publication in a serious venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper combines three known ideas—LLM-based keyword importance scoring, dictionary retrieval via RAG, and iterative self-checking—into one prompt chain for low-resource LLM translation. The combination is reasonable and clearly described, and the ablations for keyword selection and self-checking are sensible. Evaluating on WMT22-24 to avoid contamination is also good practice. That's real value.\n\nThe problem is the empirical claim. The baseline is only a plain 'translate this sentence' prompt, zero- or few-shot. The paper cites dictionary-prompting predecessors (Ghazvininejad et al. 2023; Lu et al. 2023) but never compares against them, nor against a simple prompt that just appends dictionary entries. So you cannot tell whether keyword selection helps at all beyond just putting dictionary translations in the prompt. That's a load-bearing gap, and the stress-test note is right.\n\nSecond, Table 2 shows several pairs where the method degrades on both metrics (Llama hr-en, ms-en, no-en; Qwen nl-en, sk-en). The abstract says 'significant improvements,' the text calls these 'slight declines,' but there is no variance or significance testing anywhere. Some drops are not slight—hr-en BLEU from 33.42 to 29.91. These need a real explanation, not hand-waving about Romance language training data.\n\nThird, there are formal slips: Eq. 7 defines the final output as only the constraint words, which is wrong. k is said to be adaptive but never specified. The number of self-checking iterations is not reported. Minor, but they make the paper harder to trust.\n\nWho is this for? Researchers working on prompt-based MT with bilingual dictionaries. They will find a useful recipe to beat, but not a reliable SOTA claim.\n\nMy recommendation: it deserves peer review, not a desk reject. A referee should demand the missing dictionary-prompting baselines and per-pair significance testing before acceptance. I would not cite it in its current form.","headline":"Reasonable incremental prompting recipe for low-resource MT, but the empirical claims outrun the experiments: missing dictionary-prompting baselines and untested degradations.","tokens_in":11991,"tokens_out":3307,"would_cite":false,"duration_ms":30660,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A constraint-aware prompt chain--LLM-chosen keywords, dictionary lookups, and iterative self-checking--improves low-resource machine translation for 7B-parameter LLMs without fine-tuning.","keywords":["machine translation","large language models","low-resource languages","retrieval-augmented generation","prompt engineering","bilingual dictionary","self-checking","hallucination mitigation"],"falsifier":"Construct a test set of sentences built around polysemous words for which the dictionary's nearest-neighbor translation is contextually wrong; if the full constraint-and-check chain on those sentences does not lower BLEU or human faithfulness scores relative to unconstrained translation, the dictionary-retrieval step is not doing the causal work the paper claims. A second check is to replace the self-checking loop with best-of-n sampling over unconstrained translations of equal inference budget; if that matches the constrained chain's gains, iterative self-checking is not the source of the improvement.","tokens_in":11004,"feed_emoji":"🌐","tokens_out":14224,"duration_ms":121529,"temperature":0.7,"pith_summary":"Large language models translate familiar text well but stumble on rare or specialized words, particularly in low-resource languages. This paper tries to fix that with a training-free prompt chain: the model first scores every source word for translation importance, retrieves dictionary translations for the top-scoring keywords, includes those translations in the translation prompt as constraints, and then repeatedly asks the model to check and refine its own output against them. On ten FLORES-200 language pairs and contamination-free WMT22-24 data, the authors report BLEU and chrF++ gains with Llama-3.1-8B-Instruct and Qwen2-7B-Instruct relative to zero- and few-shot baselines, with the clearest gains for Qwen, whose hallucinated Chinese output in non-Chinese targets is largely suppressed. The payoff, if the results generalize, is a recipe for more faithful LLM translation that needs no fine-tuning and no parallel data.","feed_headline":"Prompt chain cuts rare-word errors in LLM translation","feed_subtitle":"Dictionary lookups plus self-checking lift BLEU for low-resource pairs with Llama and Qwen.","key_machinery":"The carrying object is the constraint-aware prompt chain, a four-stage pipeline that runs entirely at inference time. Stage one computes an LLM priority score $p_i = f_{LLM}(x_i|X)$ for every source word and keeps the $k$ highest-scoring words as keyword set $W$. Stage two embeds each keyword, finds its nearest-neighbor translation $t_i$ in the paper's bilingual dictionary, and packages the word-translation pairs as constraint set $C$. Stage three produces an initial constrained translation $Y^1 = f_{LLM}(X, C)$. Stage four is the iterative self-checking loop $Y^{i+1} = f_{LLM}(X, C, Y^i)$, which revises until the constraints are satisfied, and a final call lets the model choose the better of the initial and refined outputs. The dictionary supplies lexical facts the model may not know; the self-checking loop is what the paper credits with suppressing hallucinations and enforcing the constraints.","core_discovery":"On its own terms, the paper establishes that constraining an LLM with dictionary translations of the words it judges most important, and then having the LLM revise its work against those constraints, produces measurably more faithful translations than unconstrained prompting. The central empirical claim is that this holds across multilingual 7B-parameter models (Llama-3.1-8B-Instruct and Qwen2-7B-Instruct), across ten low-resource FLORES-200 language pairs in both directions, and on recent WMT22-24 EN-DE test sets chosen to reduce data-contamination effects. The ablation evidence supports the two load-bearing choices: LLM-guided keyword selection outperforms fixed, random, or no constraints, and iterative self-checking improves on a single constrained pass. The paper's own results also contain exceptions--some Llama pairs such as hr-en, ms-en, and no-en lose points--and the stated cause is dictionary retrieval producing contextually wrong translations on polysemous words.","pith_inferences":["An extension the paper leaves implicit is a confidence filter on retrieved dictionary translations: the negative Llama results on hr-en, ms-en, and no-en suggest that letting the model flag or veto a contextually wrong dictionary entry could recover those losses.","The paper does not compare the self-checking loop against equally priced best-of-n sampling, so the specific contribution of iterative refinement versus general test-time compute is not isolated; that comparison would sharpen the recipe.","Because the dictionaries used cover only the languages they were built for, transferring the method to genuinely under-resourced languages will require inducing lexicons, which the paper acknowledges as a limitation.","A natural combination with few-shot prompting is untested: constrained revision could be applied after in-context exemplars, potentially stacking two independent faithfulness gains."],"forward_implications":["Any language pair covered by a decent bilingual dictionary can apply this recipe to a stock 7B instruct model, improving faithfulness without fine-tuning or parallel data.","For models with a dominant-language bias, the constraint-and-check loop can suppress hallucinated content in the wrong target language, as it does for Qwen2's Chinese outputs.","Because the gains persist on WMT22-24 sets that the paper treats as contamination-free, the improvement is unlikely to be a memorization artifact.","The ablation's message--that keyword selection quality drives the gain--points future work toward better importance scoring and richer dictionaries rather than heavier training."],"supporting_citations":[{"why":"Supplies the ground-truth bilingual dictionaries from which keyword translations are retrieved.","marker":"Conneau et al. (2017)"},{"why":"Establishes the dictionary-based prompting line the paper extends with keyword selection and self-checking.","marker":"Ghazvininejad et al. (2023)"},{"why":"Introduces retrieval-augmented generation, the mechanism adopted for dictionary lookup.","marker":"Lewis et al. (2020a)"},{"why":"Provides the FLORES-200 benchmark and the low-resource language pairs used in the main experiments.","marker":"NLLB Team et al. (2022)"},{"why":"Documents benchmark contamination in LLM evaluation, motivating the contamination-free WMT22-24 experiments.","marker":"Zhou et al. (2023)"},{"why":"Defines the Llama-3.1-8B-Instruct base model used for keyword extraction and translation.","marker":"Dubey et al. (2024)"},{"why":"Defines the Qwen2-7B-Instruct base model whose hallucinations the method targets.","marker":"Yang et al. (2024)"},{"why":"Provides the embedding model used to map keywords to dictionary translations.","marker":"Chen et al. (2024)"}],"fun_headline_variants":["Dictionary constraints and self-checks improve LLM translation","LLM translation gets a boost from dictionary-guided iterative prompting","Self-checking LLM prompts refine rare-word translations","Iterative self-checking boosts LLM translation for rare words","Dictionary-guided LLM prompting cuts rare-word translation errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each retrieved dictionary translation is the right rendering of the keyword in its actual sentence, and that the LLM's priority scores identify the words whose translation truly matters; if either fails, the injected constraint can hurt the output, as the paper's own hr-en, ms-en, and no-en results for Llama show.","fun_headline_variants_meta":{"raw":{"variants":["Dictionary constraints and self-checks improve LLM translation","LLM translation gets a boost from dictionary-guided iterative prompting","Self-checking LLM prompts refine rare-word translations","Iterative self-checking boosts LLM translation for rare words","Dictionary-guided LLM prompting cuts rare-word translation errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00102,"raw_usage":{"total_tokens":4284,"prompt_tokens":905,"completion_tokens":3379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":3300}},"tokens_in":521,"tokens_out":3379,"duration_ms":25442,"temperature":1.0,"reasoning_tokens":3300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:40:24.031246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of sentences built around polysemous words for which the dictionary's nearest-neighbor translation is contextually wrong; if the full constraint-and-check chain on those sentences does not lower BLEU or human faithfulness scores relative to unconstrained translation, the dictionary-retrieval step is not doing the causal work the paper claims. A second check is to replace the self-checking loop with best-of-n sampling over unconstrained translations of equal inference budget; if that matches the constrained chain's gains, iterative self-checking is not the source of the improvement.","supporting_citations":[],"review_version":1}