{"id":"4db8610b-b4a6-42cb-b3a3-cba925155f44","arxiv_id":"2506.05070","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RIVAL iteratively re-trains a reward model adversarially against the current translator and adds a BLEU-predicting head, improving in-domain WMT and subtitle translation over SFT baselines.","lead":"RIVAL trains a translator and its reward model in an alternating adversarial loop so the reward model keeps adapting to the policy's shifting outputs. The method improves in-domain translation over SFT baselines, but part of the reported gain relies on GPT-4o serving as both the training target and the judge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD-preservation claim (Contribution 2) is contradicted by the paper's own Table 2: RIVAL-Iter2-Qual+Quant lowers OOD COMETkiwi versus the base model by 2.79 (EN->DE) and 1.95 (DE->EN), so the claim as written does not hold.","rationale":"The reader's weakest_assumption focused on the GPT-4o circularity in the subtitle evaluation. That concern is real, but it affects only the subtitle portion of the evidence and depends on an assumption about GPT-4o's alignment with human preferences. The OOD contradiction I identify is stronger because it uses only the paper's own Table 2 and the literal wording of Contribution 2: the reported numbers refute the 'without compromising OOD performance' claim regardless of whether GPT-4o is a faithful judge. The reader did note in the rationale that 'the OOD claim is contradicted by the paper's own tables,' so there is partial agreement, but the reader did not elevate this to the primary load-bearing concern. I also considered the algorithmic issue in Equation 8, where the quantitative loss is written as E[|ys_BLEU - yw_BLEU|], which has no dependence on the learned reward model and therefore cannot train the quantitative head as written. However, Algorithm 1 clarifies the intended supervision (BLEU of generated translations against strong reference), so a corrected equation is plausible; the OOD contradiction is not similarly fixable by a typo. The in-domain WMT improvements (BLEU 39.39 vs. 38.77; COMETkiwi 72.60 vs. 71.39) remain the strongest evidence for the method's core mechanism, so the appropriate outcome is not rejection but a conditional revision: the OOD claim must be weakened or redefined, and significance testing should be reported. Since the reader already issued CONDITIONAL, my read does not change that verdict; it strengthens the reasons behind it.","tokens_in":19362,"tokens_out":4536,"duration_ms":54075,"concrete_test":"Re-analyze the three repeated runs behind Table 2: compute the per-run COMETkiwi difference between Qwen2.5-7B-Chat and RIVAL-Iter2-Qual+Quant on OOD EN->DE and DE->EN, and test significance with a paired bootstrap or paired t-test over sentence-level scores. Also check whether the run-to-run standard deviations reported as 'averaged by 3 times' overlap zero. If the OOD drops are significant—or even consistently negative—the contribution should be revised from 'without compromising' to 'degrades less than SFT.' A useful additional table entry would report OOD deltas relative to both the base model and SFT, making the reference point explicit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Contribution 2 states that RIVAL improves in-domain translation quality 'without compromising the out-of-distribution performance of the model.' This requires OOD performance not to degrade relative to the base model. The paper's own data show the opposite. In Table 2, the best in-domain variant, RIVAL-Iter2-Qual+Quant, has OOD WMT EN->DE COMETkiwi 68.74 versus Qwen2.5-7B-Chat's 71.53 (drop of 2.79), and DE->EN COMETkiwi 72.23 versus 74.18 (drop of 1.95). OOD BLEU also falls on both directions (25.25 vs. 27.23 and 35.25 vs. 36.85). The paper's narrative compares only against SFT, arguing that RIVAL shows 'significantly less degradation,' which is a different claim. If 'without compromising' is meant relative to SFT, the contribution text is mis-stated; if it is meant relative to the base model, it is empirically false. This is not an external judgment about GPT-4o or a typo in an equation; it is an internal contradiction between the headline claim and the reported numbers, and it is load-bearing because Contribution 2 is the paper's advertised advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RIVAL, an iterative adversarial RL framework for LLM-based machine translation. The reward model and the translation LLM are alternately updated in a min-max game: the RM learns to rank strong translations above the current policy's outputs (qualitative preference reward), while an additional head predicts BLEU-like quantitative scores to stabilize training. Experiments on a new colloquial Chinese-English subtitle dataset and on WMT ZH-EN (with EN-DE and Medical ZH-DE as OOD tasks) report that RIVAL-Iter2-Qual+Quant outperforms SFT on BLEU and COMETkiwi in-domain while degrading less than SFT on OOD tasks. The paper also releases the processed subtitle dataset. The central in-domain WMT result is plausible if the reported training objective is corrected and the OOD claim is restated precisely.","tokens_in":19646,"tokens_out":5770,"duration_ms":67598,"significance":"If the corrected training objective reproduces the reported numbers, RIVAL is a useful contribution: it offers a concrete alternative to SFT for LLM-based MT, with an adversarial reward-model update that addresses distributional shift, and it provides an open subtitle-translation resource for a challenging colloquial domain. The paper's strongest evidence is the in-domain WMT comparison in Table 2, where RIVAL-Iter2-Qual+Quant reaches BLEU 39.39/33.42 and COMETkiwi 72.60/73.61 versus SFT's 38.77/32.22 and 71.39/71.88 on EN-ZH/ZH-EN. The MAE-vs-MSE analysis (Table 3) is a concrete, useful ablation. However, the paper does not ship code or machine-checked proofs, and the central claims rest on an equation that is printed in a way that cannot train the quantitative head and on a contribution statement that is contradicted by the paper's own OOD numbers. These issues are correctable but load-bearing.","major_comments":[{"comment":"Equation (8) defines the quantitative term as E_DRM[|ys_BLEU - yw_BLEU|]. This expression does not involve the reward model parameters rϕ, so its gradient with respect to the RM is zero and the quantitative head cannot be trained by this loss as printed. Algorithm 1, line 7, stores (x, BLEU(ys_i, ys_i), BLEU(ys_i, yw_i)), which suggests the intended loss is an MAE between predicted scores and these BLEU targets, e.g., |rϕ(x,ys)-BLEU(ys,ys)| + |rϕ(x,yw)-BLEU(ys,yw)|. Please correct the equation and confirm that the Qual+Quant results in Tables 2 were obtained with a loss that actually updates the quantitative head; if they were, the printed formula is a typo, but it is load-bearing and must be fixed.","section":"§4.2, Eq. (8) and Algorithm 1"},{"comment":"The claim that RIVAL improves in-domain translation quality 'without compromising the out-of-distribution performance of the model' is not supported by Table 2. Relative to the base Qwen2.5-7B-Chat, RIVAL-Iter2-Qual+Quant decreases OOD WMT EN-DE COMETkiwi from 71.53 to 68.74 and BLEU from 27.23 to 25.25, and DE-EN COMETkiwi from 74.18 to 72.23 and BLEU from 36.85 to 35.25. The paper's own discussion later narrows the claim to 'significantly less degradation' than SFT, which is a different statement. Please revise Contribution 2 and the relevant text to state the comparison explicitly, and if the degradation relative to the base model is considered acceptable, justify that standard with evidence.","section":"§1, Contribution 2 and Table 2 (OOD rows)"},{"comment":"The subtitle evaluation and the construction of training data are both anchored to GPT-4o: GPT-4o produces the 'strong translations' used as RM targets, and GPT-4o is used as the primary judge. The only human check (Appendix C, Figure 5) reports three annotators but provides no numeric agreement, correlation, inter-annotator reliability, or error bars. This makes it difficult to rule out that the subtitle gains reflect the RM and the judge favoring GPT-4o-like outputs rather than human-perceived quality. Please report quantitative human-model agreement (e.g., Pearson/Spearman correlation, quadratic-weighted kappa, per-dimension agreement) and, if possible, add a reference-based metric or human evaluation on a subset to break the circularity.","section":"§3.1, §5.1, Appendix C"},{"comment":"The formal objectives in Equations (4) and (7) impose the constraint KL[πθ∥πref] < η, but Appendix D states that the KL penalty coefficient β is set to 0 and the constraint is 'effectively removed.' If no KL regularization is used, the distributional-shift story motivating RIVAL is tested without the safeguard present in the formal method. Please clarify whether the constraint is active, and if β=0 is intentional, justify this choice and report the actual divergence between πθ and πref over training iterations.","section":"§4, Eqs. (4) and (7), Appendix D"}],"minor_comments":[{"comment":"The captions say all results are averaged over 3 runs, but no standard deviations, confidence intervals, or significance tests are reported. Some differences are small (e.g., +0.62 BLEU for EN-ZH in Table 2), so variance information is needed to assess whether the gains are reliable.","section":"Tables 1 and 2"},{"comment":"The figure alone does not support the statement that GPT-4o assessments show 'a high degree of consistency' with human judgments. Please add numeric correlation coefficients, per-dimension agreement, inter-annotator agreement, and the number of annotated items.","section":"Appendix C, Figure 5"},{"comment":"The claim that 'more iterations lead to better performance' is not consistently supported by Table 1: RIVAL-Iter2-Qual has lower GPT-4o average (3.53) than RIVAL-Iter1-Qual (3.68), and COMETkiwi only improves from 66.27 to 66.49. Please restrict the claim to the Qual+Quant setting or qualify it with the observed variance.","section":"Section 5.2 and Table 1"},{"comment":"There are small presentation errors: 'accross' appears in Related Work, and the text refers to the Verl framework while citing Sheng et al. (2024), whose system is named HybridFlow. These should be harmonized.","section":"Related Work and Appendix D"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection because the in-domain WMT results are credible and the main problems are internal inconsistencies (the printed Eq. (8) and the overstated Contribution 2) that can be corrected in a revised manuscript. The GPT-4o circularity is a real risk for the subtitle claims, but it can be mitigated with quantitative human agreement and external metrics; I would not treat it as grounds for rejection on its own."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RIVAL is a real attempt at a real failure mode: offline reward models diverge from the online policy during RLHF, and the paper demonstrates this concretely in subtitle MT with a clean pilot experiment. The proposed fix—iterative adversarial RM updating with an auxiliary BLEU-regression head on the same model—is a plausible, simple recipe. What is genuinely new is the specific combination, and the WMT results are the strongest evidence: Iter2-Qual+Quant beats SFT on both BLEU and COMETkiwi for EN->ZH and ZH->EN, with external references. The subtitle dataset and processing pipeline are a useful contribution on their own.\n\nThe paper is not ready to accept. The most serious issue is Contribution 2. The claim that RIVAL improves in-domain quality without compromising OOD performance is contradicted by Table 2. Iter2-Qual+Quant drops OOD COMETkiwi by about 2.8 points on EN->DE and 2.0 on DE->EN relative to the base Qwen2.5-7B-Chat. What the data actually show is that RIVAL degrades less than SFT, which is a different and defensible claim. The contribution text needs to be redone.\n\nSecond, Equation 8 as printed is wrong. The MAE term E[|ys_BLEU - yw_BLEU|] has no dependence on rϕ, so it cannot train the quantitative head. Likely a typo—presumably the predicted BLEU regression is inside the absolute value—but as written it is a mathematical error.\n\nThe subtitle evaluation has the circularity the reader flagged: GPT-4o produces the strong translations, the RM is trained to prefer them, and GPT-4o is the judge. Appendix C's human agreement is a reasonable check but reports no numeric agreement. The WMT results rescue the paper because those use independent references.\n\nMinor: no variance or significance reported despite three runs, and no comparison to iterative DPO or self-rewarding baselines.\n\nThese are addressable. The core approach is sound and the central empirical claim on WMT holds. This deserves a serious referee. If you are the editor, send it out; if you are the referee, ask for a corrected equation, a re-framed OOD claim, and variance numbers.","headline":"Solid recipe for iterative adversarial reward modeling in MT, but the OOD-preservation claim is overstated and Equation 8 has a typo; deserves peer review.","tokens_in":20246,"tokens_out":3716,"would_cite":true,"duration_ms":40094,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes iterative RM-LLM adversarial training as a viable alternative to SFT for LLM translation, with gains in WMT BLEU and smaller out-of-distribution degradation.","keywords":["machine translation","RLHF","reward hacking","distributional shift","adversarial training","iterative optimization","subtitle translation","GRPO"],"falsifier":"Run a blind side-by-side human evaluation on held-out subtitle and WMT samples, comparing RIVAL-Iter2-Qual+Quant, SFT, and GPT-4o; if human raters do not prefer RIVAL over SFT in the same direction GPT-4o's scores do, the reward proxy is not faithfully tracking human translation quality.","tokens_in":19085,"feed_emoji":"🌐","tokens_out":8778,"duration_ms":97308,"temperature":0.7,"pith_summary":"Reinforcement learning from feedback is meant to sharpen translation quality, but the paper reports that ordinary offline RLHF fails on colloquial subtitle translation: the fixed reward model drifts out of sync with the translator as it trains, and the model starts adding content that is not in the source text to please the reward model. RIVAL instead runs reward-model training and translator training as an iterative min-max game, with the reward model repeatedly retrained to widen the gap between strong and weak translations and the translator trained to close that gap, guided by both qualitative preference and a BLEU-based quantitative reward predicted by the same reward model. On WMT Chinese-English test sets, two RIVAL iterations with both rewards reach BLEU 39.39/33.42 and COMETkiwi 72.60/73.61, above the SFT baseline of 38.77/32.22 and 71.39/71.88, while out-of-distribution English-German quality degrades far less than under SFT. The paper's central claim is that iteratively updated adversarial reward modeling is a viable alternative to supervised fine-tuning for LLM-based machine translation.","feed_headline":"Reward-loop RL beats supervised fine-tuning for LLM translation","feed_subtitle":"An iterative RM-versus-LLM game raises WMT BLEU and avoids the out-of-domain collapse seen with SFT.","key_machinery":"The load-bearing object is the RIVAL min-max objective $$\\min_{r_\\phi}\\max_{\\pi_\\$\\theta$} \\mathbb{E}[r_\\$\\varphi$(x,\\pi_\\$\\theta$(y|x))] - \\mathbb{E}_{y\\sim P_{\\text{strong}}}[r_\\$\\varphi$(x,y)]$$ subject to a KL constraint, which frames the RM as a discriminator that keeps chasing the current translator while the LLM acts as a generator that keeps closing the quality gap. Two concrete mechanisms carry the result: the RM step retrains $r_\\phi$ with a pair-wise rank loss (the qualitative preference reward) on strong-versus-current-weak translation pairs, and the LLM step uses GRPO to maximize the RM reward; the quantitative preference reward is a second output head on the same RM that regresses BLEU scores through a noise-robust MAE loss, injecting lexical fidelity without making the reward a brittle reference-dependent metric.","core_discovery":"The paper claims that the failure of vanilla RLHF in translation is caused by distributional shift: as the LLM improves during RL training, its output distribution moves away from the weak translations the offline reward model was trained on, so reward scores stop tracking true quality and reward hacking appears. The proposed fix is RIVAL, which rewrites the RLHF pipeline as a min-max game between the reward model and the LLM: the RM is trained to maximize the score gap between outputs of a strong translator, defined as gold references plus GPT-4o outputs, and the current LLM, while the LLM is trained to minimize that gap, with the RM rebuilt each iteration from current LLM outputs plus replayed earlier outputs. To keep the qualitative preference signal from drifting during open-ended exploration, the RM also predicts a quantitative preference reward that approximates BLEU, trained on noisy data with a multi-head loss $L_{\\text{qualitative}} + \\alpha L_{\\text{quantitative}}$. With this setup, two iterations on WMT EN-ZH reach BLEU 39.39 and 33.42 versus 38.77 and 32.22 for SFT, and the out-of-distribution EN-DE results degrade far less than SFT, which the paper reads as evidence that RIVAL learns more general translation strategies rather than memorized patterns.","pith_inferences":["Beyond the paper's claims, the framework implies that reward hacking in translation is primarily a distribution-shift problem rather than a pure signal-design problem; if so, periodically re-fitting the RM is more important than the exact preference loss, a hypothesis testable by ablating RIVAL against offline RM refreshment without adversarial framing.","Because the subtitle references are GPT-4o-generated and GPT-4o also scores the outputs, the reported quality gains may partly reflect imitation of that model's style; a human side-by-side evaluation on a fresh subtitle sample would separate real quality gain from proxy alignment.","A direct extension would be to replace the BLEU head with a learned reference-free quality estimator as the quantitative reward; if the multi-head stabilization effect persists, the framework's success does not depend on BLEU specifically."],"forward_implications":["If RIVAL is right, iterative adversarial RM updating is a working alternative to SFT for LLM translation, and the gains are not bought by sacrificing BLEU: quantitative and qualitative rewards together raise both lexical and semantic metrics.","The method preserves out-of-distribution language performance far better than SFT: on WMT EN-DE and DE-EN the RIVAL models sit close to the original Qwen2.5-7B-Chat, whereas SFT drops sharply.","Because the reward model is rebuilt each round from current LLM outputs, the framework should keep providing reliable supervision as the model improves, avoiding the reward-hacking failure mode documented in the pilot experiment.","The paper's own results suggest further iterations may push the model past strong translators; it explicitly lists surpassing the strong translator as a plausible continuation."],"supporting_citations":[{"why":"Supplies the generative-adversarial min-max formulation that RIVAL applies to RM-LLM training.","marker":"(Goodfellow et al., 2014)"},{"why":"Provides the pairwise preference model behind the qualitative preference reward.","marker":"(Bradley and Terry, 1952)"},{"why":"Supplies the GRPO algorithm used to optimize the LLM against the RM reward.","marker":"(Shao et al., 2024)"},{"why":"Defines BLEU, the quantitative preference target the RM is trained to approximate.","marker":"(Papineni et al., 2002)"},{"why":"Defines COMETkiwi, the reference-free semantic metric used to evaluate translation quality.","marker":"(Rei et al., 2022)"},{"why":"Justifies adding noise when training the BLEU-regression head as a form of regularization.","marker":"(Bishop, 1995)"}],"fun_headline_variants":["Adversarial RL beats SFT for LLM translation","Min-max game between RM and LLM boosts BLEU","Iterative RM-LLM competition fixes translation drift","RIVAL: adversarial training improves LLM translation","Fighting reward model drift with adversarial RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline assumes that GPT-4o generates genuinely strong translations and that its scores faithfully track human quality on the subtitle task, since GPT-4o both writes the strong translations the reward model learns from and evaluates the final outputs.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial RL beats SFT for LLM translation","Min-max game between RM and LLM boosts BLEU","Iterative RM-LLM competition fixes translation drift","RIVAL: adversarial training improves LLM translation","Fighting reward model drift with adversarial RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1469,"prompt_tokens":1016,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":632,"tokens_out":453,"duration_ms":5261,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:26:00.980923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a blind side-by-side human evaluation on held-out subtitle and WMT samples, comparing RIVAL-Iter2-Qual+Quant, SFT, and GPT-4o; if human raters do not prefer RIVAL over SFT in the same direction GPT-4o's scores do, the reward proxy is not faithfully tracking human translation quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the generative-adversarial min-max formulation that RIVAL applies to RM-LLM training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies adding noise when training the BLEU-regression head as a form of regularization."}],"review_version":1}