Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

MBR decoding—choosing by expected agreement with sampled annotations—beats MAP for error-span detection in machine translation evaluation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:52 UTC pith:ZP2NUE2Z

load-bearing objection Useful, honest application of MBR to generative ESD, but the headline span-level gain is measured with the same function used as the optimization objective; the paper says so itself, and the official F1 numbers only partly support it. the 3 major comments →

arxiv 2512.07540 v4 pith:ZP2NUE2Z submitted 2025-12-08 cs.CL cs.AIcs.LG

Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation

Boxuan Lyu , Haiyue Song , Hidetaka Kamigaito , Chenchen Ding , Hideki Tanaka , Masao Utiyama , Kotaro Funakoshi , Manabu Okumura This is my paper
classification cs.CL cs.AIcs.LG
keywords minimum Bayes risk decodingerror span detectionmachine translation evaluationgenerative evaluation modelsspan-level utility functionsdecoding by consensusLLM structured outputsdistillation of decoding decisions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the standard decoding choice for generative machine-translation evaluators—picking the output the model deems most probable—is the wrong objective for identifying error spans, because a model can assign higher likelihood to a wrong annotation than to the human one. It replaces this with Minimum Bayes Risk decoding, which samples many candidate annotations and selects the one with the highest average similarity to the rest of the sample. To measure that similarity the paper introduces SOFTF1, a continuous span-level score that avoids the hard-zero penalty the standard metric imposes on any non-empty annotation compared against an empty one. With sufficiently many candidates, the SOFTF1-driven decoder improves span-level accuracy across all tested models and language pairs, and matches or exceeds probability decoding at sentence and system level. A final distillation step packs the decoding decision into model weights, so a fast greedy search reproduces the reranking advantage.

Core claim

The paper's central claim is that Maximum A Posteriori decoding of generative ESD models rests on a false premise—model probabilities do not reliably track similarity to human error annotations—and that Minimum Bayes Risk decoding corrects it. The decision rule becomes: sample a diverse set of candidate annotations, define a utility function over pairs of annotations, and output the candidate with the highest average utility against the sampled set, treating the sample as a stand-in for the unknown human annotation. The paper identifies a flaw in the standard span-level utility: any non-empty candidate compared with an empty support annotation receives the minimum score, even if it contains

What carries the argument

The central object is SOFTF1, a soft span-level similarity function. An annotation is converted into a dense severity vector over character positions (major and minor penalties), and the similarity between two annotations is the harmonic mean of soft precision and soft recall defined via ℓ1 distance normalized by translation length; unlike the standard character-level F1, it never collapses to zero when one annotation is empty. SOFTF1 plays two roles: it is the utility function that MBR decoding maximizes, and it is the primary span-level evaluation metric. The second mechanism is MBR decoding itself: each candidate's score is its average utility against a set of model-generated support hypo

Load-bearing premise

The span-level advantage rests on SOFTF1 being a faithful proxy for human error-span judgment; since SOFTF1 is both the optimization target and the evaluation metric, a gain measured in SOFTF1 does not automatically translate to a gain measured by human perception.

What would settle it

Take the released MBR-SOFTF1 and MAP outputs and score both against human error-span annotations using a neutral span metric that was not used as the utility—for instance, the original character-level F1 computed only over sentences where the human annotation is non-empty. If MAP equals or exceeds MBR-SOFTF1 on that neutral metric, the reported span-level improvement is an artifact of optimizing the evaluation metric itself.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Span-level error localization becomes a reranking problem solvable without retraining: any generative evaluator can be improved by sampling and averaging under SOFTF1, at the cost of more inference compute.
  • The same decoding change does not hurt—and sometimes helps—system-level ranking and sentence-level pairwise decisions, so the improvement does not come at the expense of the evaluator's other functions.
  • The distillation result means the reranking advantage can be transferred into model weights, eliminating the latency bottleneck while preserving the gain; this suggests decoding-time objectives can serve as training-time objectives.
  • The SOFTF1 utility also stands alone as an evaluation metric that handles error-free translations more sensibly than the standard F1, which is relevant to any ESD benchmark.
  • Oracle experiments indicate the remaining headroom is in the quality of the sampled support set, not in the decision rule, pointing to diversity of candidate generation as the lever for further gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If SOFTF1 tracks human judgment, the same 'choose the consensus candidate' recipe transfers to other span-annotation tasks produced by LLMs—e.g., harmful-content attribution, grammatical-error correction, or factuality localization—wherever outputs are sets of labeled spans.
  • Because the evaluation metric and the objective are the same, the reported span-level magnitude is suspect until confirmed by a neutral human-aligned metric; a clean test would compare MBR and MAP outputs against human annotations using a different scoring function.
  • The plateau between N=256 and N=1024 suggests that simple sampling with fixed temperature hits a diversity ceiling; raising temperature, varying prompts, or clustering candidates before reranking may push the scaling curve further.
  • Distillation working here hints that the model's own preference signal—which candidate looks most representative—is a valuable training signal that could be mined without any external human labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes Minimum Bayes Risk (MBR) decoding for generative Error Span Detection (ESD) in reference-free MT evaluation. Instead of MAP decoding, the method generates N candidate hypotheses and reranks them by expected similarity to model-generated support hypotheses. The authors introduce sentence-level (SCORESIM) and span-level (SOFTF1) utility functions, and experimentally compare MBR variants against MAP on WMT24 Metrics Shared Task data with four LLMs and three language pairs. They report that MBR-SOFTF1 significantly improves span-level SOFTF1 scores and generally matches or outperforms MAP at system and sentence levels, while MBR distillation via DPO lets a greedy-decoded model match full MBR performance. The main caveat, acknowledged in §6.2, is that the span-level evaluation uses SOFTF1, the same function used as the MBR utility, producing metric bias.

Significance. If the span-level claim were established, this would be a meaningful contribution: it is the first application of reranking to generative ESD, it offers a concrete utility-function design that addresses a known defect in the official F1 metric, and the distillation result has practical value for removing MBR latency. The experimental breadth is good: four models, three language pairs, multiple N values, and statistical significance tests. The paper also ships code and model releases. However, the central span-level claim is currently not established because the primary metric is the same function being optimized, and the alternative F1 results are inconsistent across models. The system- and sentence-level results appear robust enough to support a narrower claim, but the headline 'significantly improves span-level performance' needs a neutral metric or a reframing to be valid.

major comments (3)
  1. [§6.2, Eq. (5), §5.2.3] The headline span-level claim relies on SOFTF1 as the evaluation metric, but SOFTF1 is also the MBR utility function. The paper acknowledges this in §6.2 ('the evaluation metric intrinsically favors this method'), but the consequence is that the central conclusion is circular: MBR-SOFTF1 is selected specifically to maximize average SOFTF1 against support hypotheses, and then evaluated on SOFTF1 against the human annotation. This does not establish that MBR decoding improves span-level error detection in any metric-neutral sense. The official F1 column in Tables 2–3 provides the only available neutral check, and it is inconsistent: e.g., for Llama-3.3-70B at N=256, MBR-SOFTF1 F1 is 0.513 vs. MAP 0.531, and at N=16 it is 0.510 vs. 0.537. Only 2–3 of 4 models show consistent F1 gains. The span-level claim must either be supported by a genuinely neutral span metric (e.g., the official F1, or
  2. [Appendix B, §5.2.3] SOFTF1's empty-annotation behavior is the core motivation for replacing F1, but the metric has not been validated as a measure of human-perceived span quality. The soft penalty is governed by ad hoc choices: beta=1, gamma=0.5, and the L in the denominators. Additionally, Appendix B states that for empty translations (L=0) the denominators are increased by 1 'in practice,' but no analysis is given for how this fudge affects ranking or scores. Since SOFTF1 is both the utility and the evaluation metric, these arbitrary components directly influence the reported improvement. A sensitivity analysis or validation against the official F1/human agreement is needed before SOFTF1 can be used to support the paper's central claim.
  3. [§7.3, Table 6] The span-distribution analysis reveals a likely mechanism for the SOFTF1 gain: MBR-SOFTF1 reduces the number of major spans from 18.9K (MAP) to 13.1K, closer to the human count of 9.0K. This sparsity is rewarded by SOFTF1 when support hypotheses are empty, because a non-empty candidate receives a positive soft score instead of the official F1's zero. The paper does not test whether this sparsity actually corresponds to better error detection or simply to a lenient bias. The F1 column in Table 2 (Llama N=256) suggests the latter in at least one configuration. The analysis should be connected to the metric-bias concern, and the claim that MBR-SOFTF1 'mitigates' severity-proportion bias should be separated from the unsupported claim of improved span-level accuracy.
minor comments (3)
  1. [Figure 1 caption] Typo: 'Llama-3.3-70B-Inston' should be 'Llama-3.3-70B-Instruct'.
  2. [§6.2 / Tables 2–3] The F1 columns are reported but never discussed in the results narrative. Since F1 is the official WMT span metric and is not the MBR objective, the authors should explicitly address the discrepancy, especially the Llama N=256 case where MBR-SOFTF1 is below MAP on F1.
  3. [§5.2.1 / §6.1] The severity weights w_maj=-5, w_min=-1, alpha=-25, beta=1, gamma=0.5 are taken from prior work, but no ablation or sensitivity analysis is provided. A short discussion of how these choices affect the utility ranking would strengthen the paper.

Circularity Check

1 steps flagged

Primary span-level claim is measured with the same SOFTF1 function that MBR-SOFTF1 is optimized to maximize; the paper acknowledges this metric bias.

specific steps
  1. self definitional [§5.2.3, Eq. (5); §6.1 Evaluation; §6.2 Results]
    "For the span level, we used SOFTF1(·, Ey) as the metric and reported the average scores over test instances. ... Since MBR-SOFTF1 was directly optimized by SOFTF1(·,·) as the utility function, the evaluation metric intrinsically favors this method, a phenomenon known as metric bias (Kovacs et al., 2024). Lacking a neutral metric, we leave further investigation of this bias to future work."

    Eq. (5) sets scoreMBR_E = (1/|S|) Σ_s SOFTF1(E, E_s), so MBR-SOFTF1 selects the candidate with the highest average SOFTF1 against support hypotheses. The paper then evaluates span-level performance with SOFTF1(E, Ey), the same function against human annotations, and bases its headline improvement on that score. Thus the selection objective and the measured outcome coincide: a SOFTF1 maximizer is graded with SOFTF1, making part of the reported advantage built into the objective rather than discovered. The paper explicitly concedes the metric bias in the quoted passage. The independently reported F1 numbers are inconsistent with the claim (e.g., Llama-3.3-70B, N=256: MBR-SOFTF1 0.513 vs MAP 0.531), and the practical +1 denominator fudge for empty translations further couples the utility to t

full rationale

The central span-level claim is partly circular: MBR-SOFTF1 is decoded by maximizing SOFTF1 against sampled support hypotheses, and then evaluated by SOFTF1 against the human annotation. The paper itself flags this in §6.2, saying the evaluation metric intrinsically favors the method and that a neutral metric is lacking. This is not merely a self-citation or a stylistic issue; the same function appears on both sides of the derivation. The alternative F1 metric does not consistently support the span-level conclusion, and the empty-annotation design of SOFTF1 (Appendix A.2) creates a concrete mechanism by which sparser MBR output (Table 6) can score higher on SOFTF1 without improving F1. No load-bearing self-citation was found: citations to Lyu et al. (2025) and Kamigaito et al. (2025) are contextual background on MBR and diversity, not used to force the paper's conclusion. The system-level and sentence-level results rest on SPA and Acc*eq, which are independent of the proposed utility, and the distillation result is internally coherent as a faithful replication of MBR decisions. Therefore the paper is not wholly circular, but its primary span-level improvement is substantially tied to the choice of SOFTF1 as both objective and metric, warranting a score of 6.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central empirical claim rests on standard MBR assumptions (support set approximates human distribution), on the proposed SOFTF1 being both a good utility and a good metric, and on prior MQM severity weights. SOFTF1's dual role is the largest circularity burden; the oracle gap in Table 4 additionally indicates that the support-hypothesis approximation itself is a major uncontrolled factor.

free parameters (5)
  • Soft severity weights beta (major), gamma (minor) = beta=1, gamma=0.5
    Used in SOFTF1 utility/evaluation; adopted from WMT MQM practice (§6.1), not fitted here, but the metric's behavior and rankings depend on this choice.
  • MQM scoring weights w_maj, w_min, alpha = -5, -1, -25
    Used by SCORESIM and sentence-level scores; from prior WMT MQM recommendations (§6.1).
  • Denominator offset for empty translations = +1 added to L in SOFTF1 denominators
    Introduced in Appendix B to avoid division by zero for L=0 translations; changes SOFTF1 for the degenerate case and is not part of the formal definition.
  • Candidate/support set size N = 256 (headline)
    Results are reported for 16/64/256/1024; the main conclusion is drawn at N=256, chosen after seeing performance across N.
  • Sampling temperature / top-k = T=2.0, k=10
    High-temperature sampling chosen to increase candidate diversity (§6.1); affects utility estimation and final scores.
axioms (5)
  • domain assumption The set of model-generated support hypotheses with uniform weights approximates the expectation over human ESD annotations (Eq. 5).
    Section 5.1 Eq. 5 replaces the unknown human distribution with an average over sampled hypotheses; §7.1's oracle experiment shows this approximation is a major limiting factor, so the central comparison depends on it.
  • ad hoc to paper SOFTF1 is a valid span-level evaluation metric whose soft penalty for empty annotations reflects true annotation quality.
    Introduced in §5.2.3 and used as evaluation metric (§6.1); its validity is assumed from its construction, and the paper notes there is no neutral metric to confirm it.
  • domain assumption Generative ESD prompts (GEMBA-MQM style) on the four LLMs produce parseable, meaningful annotations whose probabilities are usable for sampling and MAP baselines.
    Required for all hypothesis generation; the paper uses guided generation to enforce JSON format (Appendix B) and verifies training-data cutoffs, but the calibration of the LLM probabilities is taken as given.
  • domain assumption Adopted MQM severity weights (w_maj=-5, w_min=-1, alpha=-25, beta=1, gamma=0.5) are the correct cost structure for error severity.
    Section 6.1 takes these from prior WMT practice; final MBR decisions and scores inherit this choice.
  • domain assumption The WMT24 human MQM annotations are a reliable ground truth for error spans, and the reference-free setting has no leakage.
    Used for all evaluation and for the oracle experiment (§7.1); authors state they verified model cutoffs to mitigate leakage (§6.1).
invented entities (1)
  • SOFTF1 (soft F1 metric/utility) no independent evidence
    purpose: Continuous span-level utility for MBR decoding and span-level evaluation; avoids collapse to zero when one annotation is empty.
    No external meta-evaluation against human-preference judgments is provided; its fitness as a metric is asserted from its construction and the known F1 defect. The paper itself calls for future work on metric bias.

pith-pipeline@v1.3.0-alltime-deepseek · 15756 in / 13487 out tokens · 119302 ms · 2026-08-03T17:52:56.829080+00:00 · methodology

0 comments
read the original abstract

Error Span Detection (ESD) extends automatic machine translation (MT) evaluation by localizing translation errors and labeling their severity. Current generative ESD methods typically use Maximum a Posteriori (MAP) decoding, assuming that the model-estimated probabilities are perfectly correlated with similarity to the human annotation, but we often observe higher likelihood assigned to an incorrect annotation than to the human one. We instead apply Minimum Bayes Risk (MBR) decoding to generative ESD. We use a sentence- or span-level similarity function for MBR decoding, which selects candidate hypotheses based on their approximate similarity to the human annotation. Experimental results on the WMT24 Metrics Shared Task show that MBR decoding significantly improves span-level performance and generally matches or outperforms MAP at the system and sentence levels. To reduce the computational cost of MBR decoding, we further distill its decisions into a model decoded via greedy search, removing the inference-time latency bottleneck.

Figures

Figures reproduced from arXiv: 2512.07540 by Boxuan Lyu, Chenchen Ding, Haiyue Song, Hideki Tanaka, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Masao Utiyama.

Figure 1
Figure 1. Figure 1: Performance of decoding methods with Llama-3.3-70B-Inst on the WMT24 Metrics Shared Task. MBR indicates our MBR-SOFTF1. et al., 2024), hiding diagnostic information such as error location and severity, which are crucial for model analysis and post-editing. Error Span Detection (ESD) extends MT evaluation by ask￾ing metrics not only to score a translation but also to identify error spans and their severitie… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of our MBR decoding for gen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics

    cs.CL 2026-04 unverdicted novelty 7.0

    XQ-MEval provides the first parallel-quality benchmark across languages, revealing that common translation metrics assign inconsistent scores to equally good translations and offering a normalization fix.

Reference graph

Works this paper leans on

58 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905

    Pith/arXiv arXiv 2024

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  5. [5]

    Pavan Baswani, Ananya Mukherjee, and Manish Shrivastava. 2023. https://doi.org/10.18653/v1/2023.eval4nlp-1.13 LTRC \_ IIITH ' s 2023 submission for prompting large language models as explainable metrics task . In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 156--163, Bali, Indonesia. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.eval4nlp-1.13 2023

  6. [6]

    Guerreiro, Diptesh Kanojia, Jos \'e G

    Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, Jos \'e G. C. de Souza, Beatriz Silva, T \^a nia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and Andr \'e Martins. 2023. https://doi.org/10.18653/v1/2023.wmt-1.52 Findings of the WMT 2023 shared task on quality estimation . In Proceedings of the Eighth Conference on...

    doi:10.18653/v1/2023.wmt-1.52 2023

  7. [7]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. 2023. https://arxiv.org/abs/2302.06675 Symbolic discovery of optimization algorithms

    Pith/arXiv arXiv 2023

  8. [8]

    Julius Cheng and Andreas Vlachos. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.767 Faster minimum B ayes risk decoding with confidence-based pruning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12473--12480, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.767 2023

  9. [9]

    Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe, Hideki Tanaka, and Masao Utiyama. 2024. https://doi.org/10.18653/v1/2024.findings-acl.654 Centroid-based efficient minimum B ayes risk decoding . In Findings of the Association for Computational Linguistics ACL 2024, pages 11009--11018, Bangkok, Thailand and virtual meeting. Association fo...

    work page doi:10.18653/v1/2024.findings-acl.654 2024

  10. [10]

    Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. https://doi.org/10.1162/tacl_a_00417 A statistical analysis of summarization evaluation metrics using resampling methods . Transactions of the Association for Computational Linguistics, 9:1132--1146

    work page doi:10.1162/tacl_a_00417 2021

  11. [11]

    Daniel Deutsch, George Foster, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.798 Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914--12929, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.798 2023

  12. [12]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1082 2018

  13. [13]

    Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu. 2025. https://doi.org/10.18653/v1/2025.acl-long.351 M - MAD : Multidimensional multi-agent debate for advanced machine translation evaluation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    doi:10.18653/v1/2025.acl-long.351 2025

  14. [14]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation . In Proceedings of the Eighth Confer...

    doi:10.18653/v1/2023.wmt-1.100 2023

  15. [15]

    Mara Finkelstein and Markus Freitag. 2024. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods. In The Twelfth International Conference on Learning Representations

    2024

  16. [16]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021 a . https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Computational Linguistics, 9:1460--1474

    doi:10.1162/tacl_a_00437 2021

  17. [17]

    Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022 a . https://doi.org/10.1162/tacl_a_00491 High quality rather than high model probability: Minimum B ayes risk decoding with neural metrics . Transactions of the Association for Computational Linguistics, 10:811--825

    doi:10.1162/tacl_a_00491 2022

  18. [18]

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. https://doi.org/10.18653/v1/2024.wmt-1.2 Are LLM s breaking MT metrics? results of the WMT 24 metrics shared task . In Proc...

    doi:10.18653/v1/2024.wmt-1.2 2024

  19. [19]

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innoc...

    doi:10.18653/v1/2023.wmt-1.51 2023

  20. [20]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022 b . https://aclanthology.org/2022.wmt-1.2/ Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics are better and more robust . In Proceedings of the Seventh Conference on Mac...

    2022

  21. [21]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ond r ej Bojar. 2021 b . https://aclanthology.org/2021.wmt-1.73/ Results of the WMT 21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain . In Proceedings of the Sixth Conference on Machine Translation, page...

    2021

  22. [22]

    Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 2(14):115--135

    2000

  23. [23]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  24. [24]

    Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F

    Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F. T. Martins. 2024. https://doi.org/10.1162/tacl_a_00683 xcomet: Transparent machine translation evaluation through fine-grained error detection . Transactions of the Association for Computational Linguistics, 12:979--995

    doi:10.1162/tacl_a_00683 2024

  25. [25]

    David Heineman, Yao Dou, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1255 Improving minimum B ayes risk decoding with multi-prompt . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22525--22545, Miami, Florida, USA. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.emnlp-main.1255 2024

  26. [26]

    Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.211 Lost in the source language: How large language models evaluate the quality of machine translation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 3546--3562, Bangkok, Thailand. Association ...

    work page doi:10.18653/v1/2024.findings-acl.211 2024

  27. [27]

    Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. https://doi.org/10.18653/v1/2024.wmt-1.35 M etric X -24: The G oogle submission to the WMT 2024 metrics shared task . In Proceedings of the Ninth Conference on Machine Translation, pages 492--504, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.wmt-1.35 2024

  28. [28]

    Hidetaka Kamigaito, Hiroyuki Deguchi, Yusuke Sakai, Katsuhiko Hayashi, and Taro Watanabe. 2025. https://doi.org/10.18653/v1/2025.acl-long.1410 Diversity explains inference scaling laws: Through a case study of minimum B ayes risk decoding . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    work page doi:10.18653/v1/2025.acl-long.1410 2025

  29. [29]

    Tom Kocmi and Christian Federmann. 2023. https://doi.org/10.18653/v1/2023.wmt-1.64 GEMBA - MQM : Detecting translation quality error spans with GPT -4 . In Proceedings of the Eighth Conference on Machine Translation, pages 768--775, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.wmt-1.64 2023

  30. [30]

    Philipp Koehn. 2004. https://aclanthology.org/W04-3250/ Statistical significance tests for machine translation evaluation . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388--395, Barcelona, Spain. Association for Computational Linguistics

    2004

  31. [31]

    Geza Kovacs, Daniel Deutsch, and Markus Freitag. 2024. Mitigating metric bias in minimum bayes risk decoding. In Proceedings of the Ninth Conference on Machine Translation, pages 1063--1094

    2024

  32. [32]

    Shankar Kumar and Bill Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169--176

    2004

  33. [33]

    Shankar Kumar and William Byrne. 2002. https://doi.org/10.3115/1118693.1118712 Minimum B ayes-risk word alignments of bilingual texts . In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing ( EMNLP 2002) , pages 140--147. Association for Computational Linguistics

    arXiv 2002

  34. [34]

    Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.207 BRIO : Bringing order to abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890--2903, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.207 2022

  35. [35]

    Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. https://aclanthology.org/2025.coling-main.374/ MQM - APE : Toward high-quality error annotation predictors with automatic post-editing in LLM translation evaluators . In Proceedings of the 31st International Conference on Computational Linguistics, pages 5570--5587, Abu Dhabi, UAE....

    2025

  36. [36]

    Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, and Manabu Okumura. 2025. https://doi.org/10.18653/v1/2025.acl-long.149 Unveiling the power of source: Source-based minimum B ayes risk decoding for neural machine translation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2976--2...

    work page doi:10.18653/v1/2025.acl-long.149 2025

  37. [37]

    Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ond r ej Bojar. 2020. https://aclanthology.org/2020.wmt-1.77/ Results of the WMT 20 metrics shared task . In Proceedings of the Fifth Conference on Machine Translation, pages 688--725, Online. Association for Computational Linguistics

    2020

  38. [38]

    Mathias M \"u ller and Rico Sennrich. 2021. https://doi.org/10.18653/v1/2021.acl-long.22 Understanding the properties of minimum B ayes risk decoding in neural machine translation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1...

    doi:10.18653/v1/2021.acl-long.22 2021

  39. [39]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics

    arXiv 2002

  40. [40]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/w15-3049 2015

  41. [41]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . In Thirty-seventh Conference on Neural Information Processing Systems

    Pith/arXiv arXiv 2023

  42. [42]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. http://arxiv.org/abs/1910.02054 Zero: Memory optimizations toward training trillion parameter models

    Pith/arXiv arXiv 2020

  43. [43]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52/ COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578--585...

    2022

  44. [44]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.213 2020

  45. [45]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.704 2020

  46. [46]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. http://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Pith/arXiv arXiv 2024

  47. [47]

    Andreas Stolcke, Yochai Konig, and Mitch Weintraub. 1997. https://api.semanticscholar.org/CorpusID:6874209 Explicit word error minimization in n-best list rescoring . In EUROSPEECH

    1997

  48. [48]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    Pith/arXiv arXiv 2024

  49. [49]

    Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. https://doi.org/10.18653/v1/2024.wmt-1.118 Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy . In Proceedings of the Ninth Conference on Machine Translation, pages 1222--1234, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.wmt-1.118 2024

  50. [50]

    Roy Tromble, Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. https://aclanthology.org/D08-1065 Lattice M inimum B ayes- R isk decoding for statistical machine translation . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 620--629, Honolulu, Hawaii. Association for Computational Linguistics

    2008

  51. [51]

    Jannis Vamvas and Rico Sennrich. 2024. https://doi.org/10.18653/v1/2024.acl-short.71 Linear-time minimum B ayes risk decoding with reference aggregation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 790--801, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.acl-short.71 2024

  52. [52]

    Jun Wang, Eleftheria Briakou, Hamid Dadkhahi, Rishabh Agarwal, Colin Cherry, and Trevor Cohn. 2025. Don't throw away data: Improving sequence knowledge distillation with minimum bayes risk decoding. In Scaling Self-Improving Foundation Models without Human Supervision

    2025

  53. [53]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  54. [54]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. 2023. http://arxiv.org/abs/2307.09702 Efficient guided generation for large language models

    Pith/arXiv arXiv 2023

  55. [55]

    Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Khoshfetrat Pakazad, and Graham Neubig. 2025. https://openreview.net/forum?id=7xCSK9BLPy Better instruction-following through minimum bayes risk . In The Thirteenth International Conference on Learning Representations

    2025

  56. [56]

    Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.365 INSTRUCTSCORE : Towards explainable text generation evaluation with automatic feedback . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5967--5994, Singapore...

    doi:10.18653/v1/2023.emnlp-main.365 2023

  57. [57]

    Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. 2024. https://doi.org/10.18653/v1/2024.naacl-short.34 Direct preference optimization for neural machine translation with minimum B ayes risk decoding . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    doi:10.18653/v1/2024.naacl-short.34 2024

  58. [58]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr BERTScore : Evaluating text generation with bert . In International Conference on Learning Representations

    2020