{"id":"24aad65d-275d-4624-a782-90754ea635bb","arxiv_id":"2505.21172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Word-alignment rewards for RL-trained translation raise terminology accuracy on RTT from 54.42 to 56.42 TA without hurting general translation quality.","lead":"This paper trains a translation model with reinforcement learning and a new kind of reward based on word alignment, which helps it translate technical terms correctly. It improves terminology accuracy on a German test set while keeping overall translation quality about the same.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evidence underpins the reported 2-point terminology accuracy gain; without seed variance or significance testing, the central claim of significant improvement is not established.","rationale":"I read the paper in good faith: the method is coherent, the reward design is clearly specified, and the ablations are directionally consistent. However, the central empirical claim rests on a single training run per configuration, and the key TA difference is only 2.0 points on a 500-sentence test set. RL training of this kind is noisy, so without repeated seeds and significance testing the reported gain cannot be distinguished from chance. The reader identified SimAlign/NER alignment quality as the weakest assumption; that is a valid mechanism-level concern, but it is secondary because the empirical result could in principle hold even with noisy alignments. The statistical fragility is more load-bearing because it directly threatens whether the claimed improvement is real at all. The existing CONDITIONAL verdict is therefore appropriate; my concern reinforces it rather than moving it. No ad hominem is intended, and I credit the authors for releasing plans for code and data, but those artifacts are not yet available for independent verification.","tokens_in":11789,"tokens_out":7659,"duration_ms":91061,"concrete_test":"Train RL-Rcomet and TAT-R1 at least five times from the same Qwen2.5-7B-Instruct checkpoint with different GRPO seeds, keeping all hyperparameters fixed, and report mean ± 95% bootstrap CI for RTT TA, BLEU, COMETKiwi, XCOMET, and the composite average. Apply a two-sided paired permutation test over the 500 RTT sentences for TAT-R1 vs RL-Rcomet TA. If the 56.42 vs 54.42 gap is not replicated with non-overlapping CIs or p < 0.05, the central claim of significant terminology improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 reports one run: on RTT EN->DE, TAT-R1 reaches TA 56.42 vs 54.42 for RL-Rcomet, and the composite average rises from 59.84 to 62.14. No confidence intervals, standard deviations, or significance tests are reported, and the ablation increments in Figure 4 are all within a few points. GRPO uses temperature 1.0 with 16 sampled rollouts per group, so training is stochastic; the difference between TAT-R1 and RL-Rcomet could easily fall within run-to-run noise on a 500-sentence test set. This is load-bearing because every conclusion in the paper, including the claimed OOD EN->DE generalization and the value of each alignment reward, rests on this single comparison. Even if SimAlign and NER identify key terms perfectly, the empirical claim that word-alignment rewards 'significantly improve' terminology accuracy is statistically unsupported until replicated. The reader's alignment-quality concern is a mechanism-level worry; the more fundamental issue is that the observed effect itself has no measured robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAT-R1, a terminology-aware translation model trained with GRPO using three word-alignment-based rewards (answer-align-word, answer-align-order, think-align-word) combined with a COMET reward and a format reward. The method first extracts word alignments with SimAlign, restricts the key alignments to nouns via NER, and then uses these alignments to compute rewards that encourage accurate translation of key terms. Experiments on WMT23 ZH->EN, WMT24 EN->ZH, and the RTT EN->DE terminology test set report improvements in BLEU, COMETKiwi, XCOMET, and terminology accuracy (TA), with ablations showing incremental gains from each alignment reward. The abstract claims that TAT-R1 significantly improves terminology translation accuracy while maintaining comparable general translation quality.","tokens_in":11967,"tokens_out":3900,"duration_ms":47086,"significance":"If the reported results hold, the contribution is useful and timely: it provides a simple, rule-based way to inject terminology awareness into RL training for translation without requiring terminology detection at inference, and the ablation sequence cleanly isolates the effect of each reward component. The terminology metric TA is computed from test-set annotations rather than from the reward functions, so the main claim is externally grounded and not circular. The method is also transparent and reproducible in principle, with code, data, and models promised for release. However, the significance is currently limited by the lack of statistical rigor: all conclusions rest on single-run point estimates, and the alignment-quality assumption underlying the rewards is not validated. These issues are load-bearing for the central claim and require attention before the results can be accepted as established.","major_comments":[{"comment":"All reported results are single-run point estimates. GRPO with sampling temperature 1.0 and 16 rollouts per group is stochastic, and the RTT test set contains only 500 sentences; the reported 2.00-point TA improvement (56.42 vs 54.42) and the smaller ablation increments in Figure 4 could easily fall within run-to-run noise. The abstract's \"significantly improves\" requires multiple seeds (at least three), standard deviations, and a significance test (e.g., paired bootstrap or approximate randomization) on the key metrics. Without this, the central claim that word-alignment rewards improve terminology accuracy is not statistically established.","section":"§3.2.1, Tables 1-3 and Figure 4"},{"comment":"The rewards are computed from SimAlign alignments filtered by NER to retain only nouns, but the paper provides no validation of either component on the training or test data. Noisy alignments, or an aggressive noun filter that misses key terminology, would reward the wrong tokens and could produce apparent RTT gains through reward hacking rather than genuine terminology learning. Please report alignment quality (e.g., alignment error rate on a sample), the coverage of RTT source terms by the noun filter, and ideally an analysis of whether TA improvements are concentrated in correctly aligned, noun-filtered terms.","section":"§2.1, Eqs. (8)-(11)"},{"comment":"The three reward weights alpha=1, beta=0.1, and gamma=0.1 are chosen without sensitivity analysis, and the ablation increments in Figure 4 are small enough that the conclusions may hinge on this particular weighting. A small grid search or one-at-a-time sensitivity study is needed to show that the method is not brittle to these hyperparameters and that the reported ordering of configurations is stable.","section":"§2.1, Eq. (11) and §3.1"}],"minor_comments":[{"comment":"The SFT model scores 0.00 on BLEU, COMETKiwi, and XCOMET and 0.08 on TA because it apparently translates English-to-German input into Chinese, but the table does not explain this; consider a footnote or exclude SFT from the RTT comparison since it is not a valid EN->DE system.","section":"Table 2, SFT row"},{"comment":"The average metric combines raw BLEU, COMETKiwi, XCOMET, and TA scores, which have different scales and distributions; this makes the composite average difficult to interpret. Please report normalized scores or justify the raw average, and temper the claim that the average rises from 59.84 to 62.14.","section":"Tables 1 and 2, \"Avg.\""},{"comment":"There are several typos: \"Experimetal Setups\" in §3.1, \"Algorithom\" in §2.2, and \"DeeeSeek-R1\" in §4.1; please proofread.","section":"§3.1 and §2.2"},{"comment":"The KL divergence approximation is written as an equality with the exact KL, but it is an unbiased estimator of the forward KL used in GRPO; please clarify this to avoid notation confusion.","section":"§2.2, Eq. (13)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is promising and the core idea is worth publishing after revision. The main blocker is the single-run evidence for the central comparison; the statistical robustness issue should be addressed with multiple seeds and significance testing. Also, note that the COMET-22 training reward shares model family with the COMETKiwi/XCOMET evaluation metrics, which can inflate semantic-score gains; the externally grounded TA metric mitigates this, but the authors should discuss the potential overlap explicitly. The alignment-quality validation is also important because the reward design depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is simple: the reward design is new and sensible, but the empirical claim that it works rests entirely on single-run numbers. On a 500-sentence test set the TA gain over RL-Rcomet is about two points. GRPO with temperature 1.0 and 16 rollouts is noisy; without seeds, confidence intervals, or significance tests that gap is not yet a result.\n\nWhat is actually new: the three alignment rewards (Raaw, Raao, Rtaw) defined via SimAlign on nouns selected by NER, combined with COMET and format rewards in GRPO. Prior RL-for-MT papers used COMET, BLEU, or model scoring; term-level alignment rewards are new. The method doesn't need terminology detection at inference, which is a genuine practical advantage. The ablation story is coherent: adding each reward nudges average metrics up, and the comparison with BLEU reward—where BLEU improves but COMET drops—supports their claim that n-gram overlap is a bad reward for translation semantics. The OOD robustness of RL over SFT on EN->DE is also worth reporting.\n\nThe soft spots are the load-bearing ones. First, the central comparison is single-run. The difference between TAT-R1 and RL-Rcomet in TA (56.42 vs 54.42) and average (62.14 vs 59.84) could easily be run-to-run noise on 500 sentences. This is not a minor flaw; it underdetermines every conclusion about the value of alignment rewards. Second, the alignment reward mechanism itself is unvalidated. SimAlign and the NER-based noun filter are off-the-shelf and known to be noisy; there is no alignment quality check on training or test data. If the alignments reinforce wrong tokens, reward-game artifacts are real. Third, code, data, and models are only promised, so nothing is independently reproducible yet. The missing comparison to earlier terminology methods (constrained decoding, RAG-based term integration) is also a gap, though not a fatal one.\n\nThe honest limitations paragraph in the paper is a point in its favor: the authors note the reasoning is shallow and that other metrics weren't explored. That's the tone of a serious group. But the paper currently overstates certainty: 'significantly improves' is not supported without statistics.\n\nThis is a competent, well-written paper with a plausible reward idea. It deserves a serious referee: an editor should send it out, but reviewers should ask for multi-run experiments with variance, significance testing, alignment-quality validation, and the missing baselines. I'd bring it to a reading group as a case study in reward design, but I would not cite it as is.","headline":"New reward design, but single-run results make the two-point terminology gain unproven.","tokens_in":12524,"tokens_out":2607,"would_cite":false,"duration_ms":28232,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Word-alignment rewards sharpen RL translation on terminology, lifting the terminology accuracy on the RTT test from 54.42 to 56.42 while holding general quality.","keywords":["terminology-aware translation","reinforcement learning","word alignment","GRPO","machine translation","reward design","reasoning LLMs","terminology accuracy"],"falsifier":"Compute the alignment rewards on the RTT training samples and compare them against a human-annotated set of correct terminology translations; if the reward frequently scores a mistranslated term as correct, or a correct term as wrong, then the RTT gain likely comes from reward hacking rather than genuine terminology learning. A simpler check: replace SimAlign alignments with random noun alignments and see whether terminology accuracy still rises.","tokens_in":11583,"feed_emoji":"🎯","tokens_out":7006,"duration_ms":66340,"temperature":0.7,"pith_summary":"This paper argues that reinforcement learning for machine translation can be made terminology-aware by rewarding the model for aligning key source nouns to their translations, rather than relying on terminology databases or detection at inference. It proposes TAT-R1, a 7B instruction-tuned model trained with GRPO on a hybrid reward that combines a format reward, a COMET quality reward, and three word-alignment-based rewards. On the English-to-German RTT terminology test, TAT-R1 raises terminology accuracy from 54.42 to 56.42 while the average of BLEU, COMETKiwi, XCOMET, and TA rises from 59.84 to 62.14. The paper also reports that RL generalizes to an out-of-distribution language pair where supervised fine-tuning collapses, and that adding a BLEU reward improves BLEU but hurts semantic metrics.","feed_headline":"Word-alignment rewards sharpen RL translation on terminology","feed_subtitle":"TAT-R1 lifts RTT terminology accuracy from 54.42 to 56.42 while preserving general translation quality.","key_machinery":"The carrying mechanism is the word-alignment reward family computed from SimAlign alignments restricted to source nouns. For a source-reference pair and a source-prediction pair, the aligner produces $A_{\\text{ref}}$ and $A_{\\text{pre}}$; NER keeps only noun-token alignments, yielding key alignments. $R_{\\text{aaw}}$ is the overlap of key aligned word pairs between prediction and reference, normalized by source plus prediction length to discourage verbose outputs; $R_{\\text{aao}}$ is the fraction of reference order pairs preserved in the prediction; $R_{\\text{taw}}$ counts how many key aligned word pairs appear in the <think> section. These rewards are combined as $R_{\\text{all}} = R_{\\text{comet}} + \\alpha R_{\\text{aaw}} + \\beta R_{\\text{aao}} + \\gamma R_{\\text{taw}}$ when format is valid, and zero otherwise, grounding the RL signal in localized terminology correctness.","core_discovery":"The central claim is that word-alignment information extracted from parallel data can serve as a dense, rule-based reward signal that teaches a translation model to render key terms faithfully. Given a source sentence, its reference, and the model's prediction, the paper runs an unsupervised aligner (SimAlign) to obtain token alignments, prunes them to nouns via NER, and computes three rewards: answer-align-word (overlap of aligned noun pairs between prediction and reference), answer-align-order (preservation of their relative order), and think-align-word (whether the model's reasoning text mentions the aligned pairs). Trained with GRPO, the resulting TAT-R1 improves terminology accuracy on the RTT benchmark while keeping general translation quality on WMT ZH-to-EN and EN-to-ZH at or above the COMET-reward-only baseline. The authors report that these gains transfer to an out-of-distribution EN-to-DE setting where SFT fails, and that adding the reward components one at a time yields steady improvement.","pith_inferences":["Beyond the paper, the central mechanism predicts that alignment quality is the bottleneck: if SimAlign misaligns a key term, the reward reinforces a wrong association, so evaluating alignment accuracy on the training set would clarify how much of the RTT gain is genuine terminology learning.","The paper restricts key tokens to NER-detected nouns; a direct extension would test whether the same reward still works when key terminology is verbal, multi-word, or non-named-entity, which the current design leaves invisible.","The out-of-distribution EN-to-DE result suggests RL with alignment rewards might adapt terminology skills to new language pairs with little data, but the paper does not test larger terminology benchmarks, so that generalization remains a hypothesis.","A natural next experiment, which the paper lists as future work, is to pair the alignment rewards with a learned reference-based metric such as MetricX; that would test whether the gains come from the term-level signal or from COMET's semantic scoring."],"forward_implications":["Terminology translation improves without a terminology database or inference-time term detection, because the alignment rewards are computed only during training.","The gains transfer out of distribution: RL-trained TAT-R1 improves EN-to-DE RTT scores even though training is ZH-to-EN and EN-to-ZH, while SFT on the same data collapses the EN-to-DE output to near-zero.","Each alignment reward component is additive: adding answer-align-word, then answer-align-order, then think-align-word successively raises average scores on WMT and RTT.","A BLEU reward trade-off is confirmed: it inflates BLEU but lowers COMET and fluency, whereas the alignment rewards improve lexical and semantic quality together.","The think-align-word reward changes the reasoning trace from boilerplate statements to explicit consideration of key terms, and this correlates with higher terminology accuracy."],"supporting_citations":[{"why":"Supplies SimAlign, the unsupervised word aligner that generates A_ref and A_pre for the alignment rewards.","marker":"Sabet et al. (2020)"},{"why":"Provides the R1-T1 paradigm and the rounding convention for the COMET reward.","marker":"He et al. (2025)"},{"why":"MT-R1-Zero gives the training-data recipe and the RL-with-COMET baseline that TAT-R1 compares against.","marker":"Feng et al. (2025)"},{"why":"Defines GRPO, the group-relative policy optimization algorithm that trains TAT-R1.","marker":"Shao et al. (2024)"},{"why":"Supplies the DeepSeek-R1-style think/answer template and the RL reasoning paradigm being adapted.","marker":"DeepSeek-AI et al. (2025)"},{"why":"Defines COMET, the semantic quality reward and evaluation metric used throughout.","marker":"Rei et al. (2020)"},{"why":"Defines COMETKiwi, the reference-free evaluation metric used for general-quality assessment.","marker":"Rei et al. (2022)"},{"why":"Defines XCOMET-XL, the reference-based metric used for general-quality evaluation.","marker":"Guerreiro et al. (2024)"},{"why":"Provides the RTT English-German terminology test set and the TA metric that measures the main improvement.","marker":"Zhang et al. (2023)"}],"fun_headline_variants":["Terminology-aware RL translation via word-alignment rewards","Word alignment guides RL to translate key terms correctly","RL translation improves key terms with alignment-based rewards","TAT-R1 applies alignment rewards for better terminology output","Word-aligned rewards enhance RL translation of terminology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that SimAlign's alignments, after NER keeps only nouns, reliably identify the key terms that must be rewarded; noisy or non-noun key terminology would make the reward reinforce the wrong tokens.","fun_headline_variants_meta":{"raw":{"variants":["Terminology-aware RL translation via word-alignment rewards","Word alignment guides RL to translate key terms correctly","RL translation improves key terms with alignment-based rewards","TAT-R1 applies alignment rewards for better terminology output","Word-aligned rewards enhance RL translation of terminology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1912,"prompt_tokens":944,"completion_tokens":968,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":894}},"tokens_in":560,"tokens_out":968,"duration_ms":10418,"temperature":1.0,"reasoning_tokens":894,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:32:50.984955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the alignment rewards on the RTT training samples and compare them against a human-annotated set of correct terminology translations; if the reward frequently scores a mistranslated term as correct, or a correct term as wrong, then the RTT gain likely comes from reward hacking rather than genuine terminology learning. A simpler check: replace SimAlign alignments with random noun alignments and see whether terminology accuracy still rises.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RTT English-German terminology test set and the TA metric that measures the main improvement."}],"review_version":1}