Pith. sign in

REVIEW 3 major objections 4 minor 14 references

Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reference-free RL post-training improves multilingual translation across 46 languages.

desk verdict Solid empirical study of reference-free RL post-training for multilingual MT with released models, but the headline quality gains are measured on the same QE metrics used as the reward, so the claims outrun the evidence. read the letter →

arxiv 2608.10812 v2 pith:7DQUPHZL submitted 2026-08-11 cs.CL cs.AI

Chris Han , Pengzhi Gao , Pei Fu , Jian Luan This is my paper · ORCID
classification cs.CLcs.AI
keywords multilingualmachinetranslationreference-freequalityestimationreinforcementlearningGRPOcheckpointinterpolationon-policydistillationlanguageidentificationMiLMMT-46
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an open multilingual translation model can be improved after supervised fine-tuning without any reference translations, using only source sentences and automatic quality scores. It claims the answer is yes: group-relative policy optimization with a reward that averages two reference-free quality estimation models and is gated by language identification improves translation quality over the SFT starting point across 46 languages at 1B, 4B, and 12B scales. The released v1.0 checkpoints, formed by linearly interpolating the SFT and RL parameters at equal weight, keep most of the neural-quality gains while recovering most of the lexical-overlap loss. The 12B variant leads the reference-free scores among the evaluated open and proprietary systems.

What carries the argument

The load-bearing mechanism is the language-gated reference-free reward: $R(x,y,\ell)=\frac{1}{2}(\mathrm{XCOMET}(x,y)+\mathrm{COMETKiwi}(x,y))$ if OpenLID-v3 identifies the candidate's language as the intended target $\ell$, and $0$ otherwise. This gives GRPO a training signal on source-candidate pairs without any reference translation. The second mechanism is SFT-RL linear interpolation $\theta_\alpha = \alpha\theta_{\mathrm{SFT}} + (1-\alpha)\theta_{\mathrm{RL}}$, which costs no extra training and lets the final checkpoint trade lexical-overlap fidelity (spBLEU) against neural quality metrics. The paper also studies on-policy distillation as an alternative transfer mechanism, obtaining students that nearly match but do not beat the interpolated RL checkpoints.

What would settle it

Run a blinded human adequacy and faithfulness evaluation on a stratified sample of the 46 languages comparing MiLMMT-46-v1.0 with MiLMMT-46-v0.1; if human raters find no improvement or prefer the SFT outputs, the central claim that reference-free post-training improves translation quality is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that reference-free post-training can push the quality frontier of open multilingual translation models beyond what supervised fine-tuning alone achieves, and that the resulting gains survive checkpoint interpolation. Starting from MiLMMT-46-v0.1, GRPO optimizes a reward $R(x,y,\ell)=\frac{1}{2}(s_X+s_K)$ when the predicted output language matches the target, and zero otherwise; averaging XCOMET and COMETKiwi and gating by OpenLID is enough to produce consistent improvements in reference-free and reference-based neural metrics. The final MiLMMT-46-v1.0 models are obtained by $\theta_\alpha = \alpha\theta_{\mathrm{SFT}} + (1-\alpha)\theta_{\mathrm{RL}}$ with $\alpha=0.5$, which the paper shows is a favorable operating point on a spBLEU versus reference-based-XCOMET trade-off curve. On WMT24++ and FLORES+, the 12B model attains the best reference-free XCOMET and COMETKiwi scores in the comparison set, including proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5, and the 1B model beats TranslateGemma-4B across all reported metrics. A parallel investigation of on-policy distillation shows it reaches but does not extend this frontier.

Load-bearing premise

Everything rests on trusting that the averaged automatic quality scores, gated by predicted language, reward genuinely better translations rather than merely fluent-looking ones; if they do not, the reported RL gains would be an artifact of the reward metric.

Editorial extensions

If this is right

  • Reference-free post-training is not confined to small models: at 1B, 4B, and 12B, gains over the SFT baseline appear on both WMT24++ and FLORES+ directions.
  • The interpolation coefficient $\alpha$ provides a free, controllable trade-off between spBLEU and reference-based XCOMET, with $\alpha=0.5$ selected as a balanced default operating point.
  • Because the pipeline uses only source-side text plus the two quality-estimation scorers, it can be applied where parallel data is scarce but monolingual source text is abundant.
  • MiLMMT-46-12B-v1.0's leading reference-free scores imply that fully open models can be competitive with large proprietary translation systems on these metrics.
  • On-policy distillation transfers most of the post-training gains to smaller students, offering a cheaper deployment path, though it does not advance the quality frontier beyond RL with checkpoint interpolation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the gains on learned quality metrics correspond to what human users prefer; a natural testable extension is a blinded human evaluation of v1.0 against v0.1 outputs.
  • The interpolation coefficient can be treated as a deployment knob: systems that need to honor a lexical-overlap contract can raise $\alpha$, while systems optimizing neural fluency can lower it.
  • The same reward-and-interpolation recipe should transfer to other conditional generation tasks with a trustworthy reference-free evaluator, such as summarization or code generation.
  • Because the RL data is built by discarding references from existing SFT data, the method's scaling is bounded by the availability of source-side text; languages with very little source text may see uneven gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies reference-free post-training for multilingual machine translation. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, the authors apply GRPO with a reward that averages XCOMET and COMETKiwi reference-free QE scores and gates the result by OpenLID-v3 language identification. They then linearly interpolate the SFT and RL checkpoints to produce MiLMMT-46-v1.0 at 1B, 4B, and 12B scales, and they report consistent gains on XCOMET and COMETKiwi over the SFT models across 46 languages on WMT24++ and FLORES+. The paper also compares against open and proprietary baselines, claims leading reference-free scores, and analyzes on-policy distillation as an alternative way to transfer the post-trained teacher's gains to smaller students.

Significance. If the reported gains reflect human-perceived translation quality, this is a valuable result: it shows that source-only data can be used for reference-free RL post-training of open multilingual models, and the released models and code would support follow-up work. The three-scale study, the 46-language coverage, and the checkpoint-interpolation analysis are useful contributions. However, the main evidence for quality improvement comes from the same XCOMET and COMETKiwi metrics that constitute the reward, so a large part of the measured gain is expected from optimizing the training objective. The paper provides no human evaluation of the final checkpoints to establish that the optimized metrics correspond to true translation quality. The reference-based XCOMET improvement is partial support because it is also a learned neural metric correlated with the QE models, while spBLEU actually declines by 1.21 points on average. The conclusion's statement that future work will explore 'more robust and human-aligned reward models' implicitly acknowledges this gap.

major comments (3)
  1. [Sections 2 and 3; Tables 1-11] The reward in Section 2 is R(x,y,ell) = 0.5*(XCOMET + COMETKiwi) gated by OpenLID, and the evaluation in Section 3 and Tables 1-11 uses exactly these two reference-free metrics as the primary yardstick. Since GRPO directly maximizes the average of these scores, the reported improvements of 2.75/2.44 points on WMT24++ and 1.41/1.17 points on FLORES+ are partly a consequence of optimizing the training objective, not independent evidence of human-perceived quality. The reference-based XCOMET gains provide partial support, but that metric is also a learned neural metric whose training signal is correlated with the reference-free QE models. The paper's own conclusion states that 'in future work, we will explore more robust and human-aligned reward models,' which concedes that the current reward-evaluation loop is not human-validated. The headline claim that the models 'consistently improve translation quality over their SFT counterparts' therefore needs human evaluation of the final checkpoints, or at least a strong non-optimized held-out metric, before it can be accepted as stated.
  2. [Section 2, RL data filtering; Section 4.2, interpolation] The RL data filtering thresholds (0.30 < group mean < 0.95 and sigma >= 0.05) and the interpolation coefficient alpha = 0.5 are free parameters chosen by the authors, and the retained set is only 31,572 of 263,982 instances, roughly 12% of the source data. The paper provides no sensitivity analysis over the filtering thresholds, and alpha = 0.5 is justified only through the same metric family that is being optimized. Because data filtering, reward design, checkpoint selection, and evaluation all use the same XCOMET/COMETKiwi scores, the possibility of selection bias inflating the reported gains is not addressed. A sensitivity analysis over the filtering thresholds and a clearly stated criterion for choosing alpha would make the claimed robustness of the recipe more convincing.
  3. [Section 4.1 and Figure 2] The trade-off claim in Section 4.1 and Figure 2 rests on the same metric family: reference-free post-training raises XCOMET and COMETKiwi while lowering spBLEU by an average of 1.21 points, and the paper dismisses the spBLEU decrease as 'less informative' because spBLEU measures lexical overlap. This is defensible only if the learned metrics are known to track human judgment for these models. Without a human evaluation or an external metric that was not used in training or model selection, the statement that alpha = 0.5 provides a 'favorable balance' is an assertion rather than a demonstrated property. The per-direction tables also give no confidence intervals or significance tests, so the consistency claims across 46 languages are not statistically quantified.
minor comments (4)
  1. [Abstract and Section 4.1] The text repeatedly says the models 'improve translation quality,' but the evidence is specific to XCOMET and COMETKiwi scores. Recommend rewording to 'improve reference-free quality-estimation scores' or adding the human evaluation that would justify the stronger phrasing.
  2. [Section 2, training objective] In the GRPO objective, the importance ratio ri,t is used before it is defined; consider defining ri,t before displaying the objective, and clarify the token indexing (the displayed formula mixes yi,t and an earlier notation "xi,t" that is not explained).
  3. [Figure 1 caption] The caption reads 'evaluated every50training steps'; there is a missing space between 'every' and '50'.
  4. [Appendix E] The text refers to 'the k1 estimator for PG-OPD' but never defines or cites k1; add a definition or reference.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluator circularity: GRPO maximizes the same XCOMET/COMETKiwi scores that are then reported as translation-quality gains.

  1. fitted input called prediction [Section 2 Reward design; Section 3 Evaluation; Section 4.1 Effects of reference-free post-training]
    "Let sX and sK denote the scores assigned to (x,y) by XCOMET2 ... and COMETKiwi3 ... We define the reward as R(x,y,ℓ) = ( 1/2(sX + sK), ℓhat(y)=ℓ, 0, otherwise). ... Evaluation. We use XCOMET and COMETKiwi as reference-free evaluators ... Averaged over the three model scales, XCOMET and COMETKiwi scores on WMT24++ improve by 2.75 and 2.44 points, respectively."

    The reward optimized by GRPO is exactly R(x,y,ℓ) = 0.5(s_X + s_K) for language-matched outputs, where s_X and s_K are XCOMET and COMETKiwi scores. The paper's headline evidence of quality gain is the increase in these same two scores on WMT24++ and FLORES+. Because RL directly maximizes this average, those improvements are expected from the training objective itself and are not independent confirmation of translation quality. The reference-based XCOMET result is not in the reward and provides some outside signal, and spBLEU actually declines, so the circularity is partial rather than total. No human evaluation of the final checkpoints is provided to validate that the optimized QE metrics track human judgment.

full rationale

The central derivation is not a formal theorem; it is an RL post-training experiment. The load-bearing circular step is evaluator identity: the reward used for GRPO is the arithmetic mean of XCOMET and COMETKiwi, and the primary evaluation of the final models reports the same two reference-free metrics as evidence of improved translation quality. Improvements on these metrics are therefore partly forced by the optimization objective. The paper does include an outside metric, reference-based XCOMET, which improves, and spBLEU, which decreases; external baselines and proprietary systems are also compared on the same metrics. These elements give the empirical claims some independent content, but the reference-based XCOMET is itself a learned neural metric correlated with the QE rewards, and no human evaluation is supplied. The self-citations to the authors' prior MiLMMT-46-v0.1 and GemmaX2 work are normal antecedents, not load-bearing uniqueness arguments. Overall, the central claim of quality improvement reduces in part to reporting an increase in the very scores being optimized, so a substantial but not total circularity score of 6 is warranted.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two hand-chosen operating points (alpha and data filtering thresholds) and on the unverified assumption that the QE reward models faithfully capture human quality. No new physical or architectural entities are introduced.

free parameters (2)
  • SFT-RL interpolation coefficient alpha = 0.5 (all scales)
    Fixed at 0.5 after exploring alpha values from 0.1 to 0.9; this is an operating point chosen to balance spBLEU and reference-based XCOMET, not derived from theory.
  • RL data filtering thresholds = 0.30 < mu < 0.95; sigma >= 0.05
    Instances are discarded unless the rollout group mean and standard deviation satisfy these thresholds, reducing 263982 to 31572 instances; the bounds are hand chosen to retain reward variation.
assumptions (3)
  • domain assumption XCOMET and COMETKiwi are valid proxies for human translation quality.
    The reward and the primary reference-free evaluation both rely on these QE models; the paper cites high correlation with humans (Freitag et al., 2023) but provides no human evaluation of the final models.
  • domain assumption OpenLID-v3 correctly identifies the output language.
    The language gate gives zero reward to any output classified as the wrong language, so misclassification would discard valid translations or reward incorrect ones.
  • domain assumption MiLMMT-46-v0.1 provides a strong SFT starting point.
    The entire study builds on this base model; if the base were weak or skewed, the post-training results would not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation." pith.science (2026). https://pith.science/paper/7DQUPHZL

@misc{pith2026260810812,
  author       = {Pith},
  title        = {Pith review of: Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DQUPHZL}},
  note         = {Machine review of arXiv:2608.10812}
}
read the original abstract

We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation (OPD) and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.

Figures

Figures reproduced from arXiv: 2608.10812 by the authors.

Figure 1
Figure 1. Training and validation rewards during GRPO [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Trade-off between spBLEU and reference-based XCOMET under SFT–RL checkpoint interpolation [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 linked inside Pith

  1. [3]

    Multilingual machine translation with open large language models at practical scale: An empirical study. InProceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages 5420–5443, Albuquerque, New Mexico. Asso- ciation for Computatio...

    2025

  2. [4]

    InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 12257–12284, Vienna, Austria

    WMT24++: Ex- panding the language coverage of WMT24 to 55 languages & dialects. InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 12257–12284, Vienna, Austria. Association for Com- putational Linguistics. Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jindˇrich Helcl, Stephan Oepen, Egil Rønningstad, and Yves Scherrer

    2025

  3. [6]

    Preprint, arXiv:2607.02770

    Gemma 4 technical report. Preprint, arXiv:2607.02770. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Mar- tins

    arXiv

  4. [7]

    In Findings of the Association for Computational Lin- guistics: ACL 2026, pages 33699–33723, San Diego, California, United States

    Mending the holes: Mitigating reward hacking in re- inforcement learning for multilingual translation. In Findings of the Association for Computational Lin- guistics: ACL 2026, pages 33699–33723, San Diego, California, United States. Association for Computa- tional Linguistics. Kevin Lu and Thinking Machines Lab

    2026

  5. [8]

    Ricardo Rei, Nuno M

    Qwen3 technical report.Preprint, arXiv:2505.09388. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André F. T. Martins

    arXiv

  6. [10]

    Preprint, arXiv:2506.17080

    Tower+: Bridging generality and translation specialization in multilingual llms. Preprint, arXiv:2506.17080. Yuzhe Shang, Pengzhi Gao, Wei Liu, Jian Luan, and Jinsong Su

    arXiv

  7. [11]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y

    Scaling model and data for multi- lingual machine translation with open large language models.Preprint, arXiv:2602.11961. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo

    arXiv

  8. [12]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.Preprint, arXiv:2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu

    arXiv

Show all 14 references
  1. [14]

    Mao Zheng, Zheng Li, Tao Chen, Mingyang Song, and Di Wang

    Hy-mt2: A family of fast, efficient and powerful multilingual translation models in the wild.Preprint, arXiv:2605.22064. Mao Zheng, Zheng Li, Tao Chen, Mingyang Song, and Di Wang. 2025a. Hy-mt1.5 technical report. Preprint, arXiv:2512.24092. Mao Zheng, Zheng Li, Bingxin Qu, Mi...

    2026 arXiv

  2. [2022]

    Mao Zheng, Zheng Li, Tao Chen, Bo Lv, Mingrui Sun, Mingyang Song, Jinlong Song, Hong Huang, Decheng Wu, Hai Wang, Yifan Song, Yanfeng Chen, and Guanwei Zhang

    No language left behind: Scal- ing human-centered machine translation.Preprint, arXiv:2207.04672. Mao Zheng, Zheng Li, Tao Chen, Bo Lv, Mingrui Sun, Mingyang Song, Jinlong Song, Hong Huang, Decheng Wu, Hai Wang, Yifan Song, Yanfeng Chen, and Guanwei Zhang

    arXiv

  3. [2023]

    InProceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore

    Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. InProceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore. Association for Compu- tational Linguistics. Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alv...

    2023

  4. [2024]

    InInternational Conference on Learning Representations, volume 2024, pages 21246–21263

    On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263. Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jing- wen Chen, ...

    2024

  5. [2025]

    Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, and Bin Wang

    Seed-x: Building strong multilingual translation llm with 7b parameters.Preprint, arXiv:2507.13618. Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, and Bin Wang

    arXiv

  6. [2026]

    Translategemma technical report.Preprint, arXiv:2601.09012. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Elefthe- rios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stew- art, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.