Pith. sign in

REVIEW 2 major objections 6 minor 55 references

A Bayesian Optimization Approach to Machine Translation Reranking

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Framing machine-translation reranking as Bayesian optimization finds the best candidate with a fraction of the scoring calls, matching a 180-candidate random baseline with only 70 evaluations.

desk verdict A solid, carefully measured first BayesOpt treatment of MT reranking whose efficiency claim holds for the tested setup, with real soft spots around the untested smoothness assumption, an unreported bandwidth, and a non-random test split. read the letter →

arxiv 2411.09694 v2 pith:MENT43KI submitted 2024-11-14 cs.CL

Julius Cheng , Maike Züfle , Vilém Zouhar , Andreas Vlachos This is my paper · ORCID
classification cs.CL
keywords machinetranslationrerankingBayesianoptimizationGaussianprocessqualityestimationmulti-fidelityexpectedimprovementCometKiwiefficientdecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine-translation systems often improve output quality by reranking a list of candidate translations with an external quality scorer, but the best scorers are so expensive that reranking nearly doubles the cost of translation. This paper tries to make reranking cheap by treating the candidate list as an unknown score surface and using Bayesian optimization with a Gaussian process to decide which candidates to score next. The method chooses candidates by balancing exploration of uncertain translations against exploitation of promising ones, and on the paper's test set it reaches the same CometKiwi score with 70 scoring calls that a random-subset baseline needs 180 calls to match. The paper also introduces a multi-fidelity version that first scores candidates with a cheaper, noisier proxy model, and shows this helps when the proxy is well correlated with the main scorer.

What carries the argument

The load-bearing object is the kernel that measures similarity between candidate translations, $K_{MT}(y_i,y_j) = K_{RBF}(\mathrm{emb}(y_i), \mathrm{emb}(y_j))$, where $\mathrm{emb}$ is the mean-pooled, unit-normalized output of the generator's final decoder layer and $K_{RBF}$ is the radial-basis kernel with a single bandwidth $w$ tuned on the validation set. This kernel encodes the prior assumption that similar translations receive similar scores, and it makes the Gaussian-process posterior computable from only the few observed scores. The acquisition function is closed-form expected improvement, which scores each unscored candidate by its expected gain over the current best observation, balancing exploration of uncertain regions against exploitation of promising ones. For the multi-fidelity variant, the kernel becomes $K_{mult} = K_{MT} \cdot K_{score}(s_k, s_l)$, where $K_{score}$ is the empirical covariance between the main scorer and the proxy scorer measured on a validation set; this product kernel lets proxy observations shape the posterior over the main score.

What would settle it

Apply the same algorithm to the same candidate lists and embeddings but replace the CometKiwi scores with a deliberately non-smooth scoring function—for example, scores assigned by exact string match against a reference or scores drawn randomly per instance—and run BayesOpt+GP with the same kernel; if the quality-cost curve no longer beats random subset selection, the smoothness assumption is confirmed as the load-bearing premise. Alternatively, use a different generator whose decoder embeddings do not place similar translations near each other and measure whether the advantage diminishes.

Watch

Extended reading notes

Core claim

The paper's central claim is that machine-translation reranking can be viewed as a black-box optimization problem: given a list of candidate translations and a scoring function, a Gaussian-process model of the score surface—using an RBF kernel over mean-pooled final-decoder embeddings of the candidates—lets an expected-improvement acquisition function find near-best candidates while scoring only a fraction of the list. On the test set, with 200 sampled candidates per source sentence (about 178 after deduplication), BayesOpt+GP achieves an average CometKiwi score of 0.8210 with 70 scoring calls, versus 0.8149 for 70 random candidates and 0.8216 when roughly the full list is scored; the maximum attainable score is 0.8216. The method also outperforms search-based baselines (hill-climbing and log-probability selection) on the quality-cost tradeoff curve. The multi-fidelity extension multiplies the candidate kernel by an empirical covariance kernel between scorers, so observations from a cheap, distilled proxy scorer inform the posterior over the main scorer; with a well-correlated proxy, the method matches or exceeds the full-list score with about 70–90 CometKiwi calls and further reduces runtime.

Load-bearing premise

The method's efficiency relies on the assumption that similar translations, as judged by the embedding vectors, usually receive similar quality scores, so the Gaussian process's uncertainty estimates faithfully reflect where good candidates are; if that smoothness fails for a different scorer or generator, the query selection becomes no better than random.

Editorial extensions

If this is right

  • If the smoothness assumption holds, translation systems can use far larger candidate lists for the same computational budget, because the number of expensive scorer calls no longer scales with list size.
  • The multi-fidelity result implies that a cheap, well-distilled proxy scorer can replace a large fraction of the main scorer's calls; in the paper's runtime profile this cuts the total reranking time from 984.7 seconds to 873.6 seconds per 350 instances at matched output quality.
  • Because the method treats the scorer as a black box, it should carry over to any quality estimator whose scores vary smoothly in the chosen embedding space, including quality-estimation models other than CometKiwi.
  • The quality-cost curves show diminishing returns after roughly 70–90 scoring calls, so the method gives practitioners a principled way to choose the scoring budget without scoring the full list.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is applying the same search to human evaluation or an LLM used as a judge, where smoothness of the score surface in embedding space is less established; the method's assumptions would need to be re-validated for such scorers.
  • The RBF bandwidth $w$ is fixed after tuning on the validation set; making it instance-specific or learned online might further improve the efficiency when candidate lists come from different language pairs or generators.
  • The product-kernel formulation extends to more than two scorers, so a cascade of increasingly expensive scoring models could be exploited, although the paper only demonstrates the two-scorer case.
  • The candidate embeddings come for free from the generator; using a different embedding source (for example, a cross-lingual sentence encoder) may change the smoothness and therefore the performance, which a careful reader could test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper frames machine translation list reranking as a Bayesian optimization problem. Given a candidate list generated by a neural MT system, a Gaussian process (GP) with an RBF kernel defined over mean-pooled final-decoder embeddings models the dependence between candidates and their CometKiwi scores. Expected improvement is used to select which candidates to score next under a fixed budget, and the highest-scoring observed candidate is returned. A multi-fidelity extension, BayesOpt+GP+P, incorporates cheaper proxy scores through a product kernel. Experiments on WMT23 with an NLLB generator and CometKiwi-22 show that BayesOpt+GP reaches the score that random selection achieves with 180 evaluations using only about 70 evaluations, and that the multi-fidelity variant can further improve early candidate quality when the proxy is well correlated. The paper includes several baselines, statistical significance tests, runtime profiling, and a code repository.

Significance. If the reported results hold, the paper makes a useful practical contribution to reducing the computational cost of MT reranking, which is increasingly dominated by large quality-estimation models. The experimental design is careful: it compares against multiple baselines, reports statistical significance in the appendix, profiles the full pipeline, and makes the code available. The empirical claim is evaluated on an external benchmark (CometKiwi on WMT23), so the core result is not circular with respect to the method's objective. The main risk is that the central efficiency gain rests on an untested smoothness assumption about the scoring function in the embedding space induced by the candidate generator, which limits the generality of the claims as currently stated.

major comments (2)
  1. [Section 3.1, Eq. (1); Section 2.1] The central efficiency claim depends on the assumption that CometKiwi scores vary smoothly in the mean-pooled NLLB decoder embedding space, since the kernel KMT(y_i,y_j) = KRBF(emb(y_i),emb(y_j)) uses the RBF of Eq. (1). The paper provides no direct evidence for this smoothness: there is no semivariogram or correlation analysis between embedding distances and absolute score differences, no comparison with a kernel defined on an unrelated representation, and no ablation of the embedding source. Because the kernel is tied to the candidate generator's internal states, the method is not an 'arbitrary black-box' optimizer as claimed in Section 2.1; it is coupled to the generator. The reported gains on CometKiwi/NLLB are suggestive, but transferability to other scoring functions or generators is not established. I recommend adding a direct smoothness diagnostic (e.g., correlation of pairwise kernel distances with score differences on the validation set) or explicitly limiting the method's scope to settings where such a diagnostic is favorable.
  2. [Section 4 (experimental setup)] The single RBF bandwidth w is grid-searched on the entire validation set and fixed for all experiments, but its value is never reported. The text states that results are 'not statistically significantly different within a range of settings' without providing that range or any supporting numbers. Since the acquisition function's exploration-exploitation tradeoff is controlled entirely by w, omitting the actual value and the stability range prevents readers from assessing how finely the smoothness assumption must be tuned. Please report the tuned w and the range over which the quality-cost curves remain statistically indistinguishable.
minor comments (6)
  1. [Abstract] The abstract's claim that BayesOpt+GP 'achieves the same CometKiwi score using only 70 scoring evaluations compared a baseline system using 180' is slightly imprecise: from Table 3, BayesOpt+GP at 70 runs scores 0.8210 while UniqRandom at 180 scores 0.8211, a difference of 0.0001. This is well within noise, but the wording should be qualified (e.g., 'statistically indistinguishable') to avoid an overly strong reading.
  2. [Appendix A, Table 4] The legend for the significance tables is confusing: it says '↑ means that the column method is significantly better than the row method,' but the example given in the caption ('third row and first column, ← means that HillClimbing is significantly better than UniqRandom') does not obviously match the printed layout. Please clarify the table orientation or provide a clearer legend.
  3. [Section 4.4, Table 2] The 'Comet Loading' value for BayesOpt+GP+P (11.27s) differs from the other two columns (8.43s) even though the same CometKiwi model is loaded. If the extra time includes loading the proxy model or other overhead, please state this explicitly in the table caption or the text.
  4. [Section 1] The sentence 'GPs make very few assumptions about the distribution and base their predictions are mostly on observed points' is grammatically garbled, and the claim that GPs make 'very few assumptions' is misleading because the kernel choice itself is a strong prior. Please rephrase.
  5. [Section 3.1] The statement that the kernel is 'fast to compute' is supported by the runtime measurements, but the O(|C|^2) similarity computation and O(|C|^3) matrix inversion costs are only mentioned in the Limitations section. For reproducibility and to set expectations, these costs should also be stated in the main method section.
  6. [Section 4.2] The baselines are introduced as 'Logprob{Avg,Sum}' but one sentence refers to them as 'LogprobFirst{Avg,Sum}'. Please make the naming consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the efficiency claim is an empirical comparison against an external CometKiwi benchmark, and no load-bearing step reduces to its inputs by construction.

full rationale

The paper's central claim is that BayesOpt+GP reaches a given CometKiwi score with fewer scoring calls than random subset scoring. This is an empirical result evaluated on WMT23 test sentences with CometKiwi as an external target; the GP is not fitted to the test outcome, and the final answer is always the arg max over actually observed scores (Algorithm 1, line 14), so no unobserved-score prediction is presented as the result. The RBF bandwidth w is tuned on the validation set and fixed for test evaluation, which is standard hyperparameter selection rather than a fitted input renamed as a prediction. The multi-fidelity proxy models are distilled from CometKiwi, and the paper explicitly measures and discloses their correlation; the Kscore term in Equation 6 is estimated from validation-set covariances, not from the test results it is used to explain. The only self-citations (e.g., Cheng and Vlachos 2023) are related-work references for efficient MBR and are not load-bearing for the reranking derivation. The smoothness assumption behind the RBF kernel is a modeling premise that could fail, but it is not circular: it is not defined in terms of the claimed efficiency gain, and the paper's comparisons would be falsifiable if the assumption were wrong. No uniqueness theorem, ansatz-smuggling citation, or definitional equivalence is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The load-bearing elements are a tuned RBF bandwidth (not reported), hand-chosen initialization sizes (alpha, beta), sampling parameters (epsilon, k), an empirical covariance estimate between scorers, and the domain assumption that translation quality varies smoothly in the decoder-embedding space. The paper introduces no new theoretical entities; the proxy models are trained artifacts.

free parameters (6)
  • RBF bandwidth w = Not reported in text
    Tuned by grid search on the validation set (Section 4); fixed for all experiments. The single value affects the smoothness assumption of the GP and is load-bearing for the acquisition function.
  • Initial main scores alpha = 10
    Chosen by hand as the random initialization size for BayesOpt (Section 4).
  • Initial proxy scores beta = 50 or 200
    Chosen per experiment; when beta=200 all proxy scores are used, when beta=50 a subset is used (Section 4.3).
  • Batch size k = 1 for main experiments; 10 for runtime
    Number of candidates scored per acquisition step; large k speeds up but slightly reduces quality (Section 4.5).
  • Kscore empirical covariance = Estimated from validation set
    The covariance between main scorer and proxy scorer is estimated on validation data and used in the multi-fidelity kernel (Section 3.2).
  • Epsilon for sampling = 0.02
    Truncation sampling parameter for candidate list generation (Section 3.4).
assumptions (6)
  • domain assumption The target scoring function s is adequately modeled by a Gaussian process with an RBF kernel over candidate embeddings.
    The entire acquisition strategy (Equations 2, 3, 5) assumes this; no empirical check of GP calibration is provided.
  • domain assumption Candidates that are close in mean-pooled final-decoder embedding space receive similar CometKiwi scores.
    This smoothness assumption justifies KMT in Section 3.1; if it fails, expected improvement is not informative for selecting top candidates.
  • domain assumption Per-instance normalization of scores to mean 0 and variance 1 preserves the argmax and is a stable transformation across acquisitions.
    Algorithm 1 step 6 renormalizes the observed set at every iteration; this changes the modeled function and may distort GP variance estimates.
  • standard math The EI closed form (Equation 5) applies because observations are treated as noiseless, while the GP posterior includes a noise term sigma squared.
    Section 3.1 says 'assuming no observation noise' for EI, but Equations 2 and 3 include sigma squared; the paper does not specify how sigma is set.
  • domain assumption A CometKiwi score difference of 0.006 is meaningful and likely to be human-detectable.
    Used in the abstract and Section 1 to argue that the efficiency gain does not sacrifice quality; relies on Kocmi et al. 2024b.
  • domain assumption The epsilon-sampling candidate lists (epsilon=0.02) contain the high-quality candidates needed for reranking.
    Section 3.4 adopts this setting from prior MBR work; the conclusion depends on the candidate pool being sufficiently diverse.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Bayesian Optimization Approach to Machine Translation Reranking." pith.science (2026). https://pith.science/paper/MENT43KI

@misc{pith2026241109694,
  author       = {Pith},
  title        = {Pith review of: A Bayesian Optimization Approach to Machine Translation Reranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MENT43KI}},
  note         = {Machine review of arXiv:2411.09694}
}
read the original abstract

Reranking a list of candidates from a machine translation system with an external scoring model and returning the highest-scoring candidate remains a simple and effective method for improving the overall output quality. Translation scoring models continue to grow in size, with the best models being comparable to generation models. Thus, reranking can add substantial computational cost to the translation pipeline. In this work, we pose reranking as a Bayesian optimization (BayesOpt) problem. By strategically selecting candidates to score based on a balance of exploration and exploitation, we show that it is possible to find top-scoring candidates when scoring only a fraction of the candidate list. For instance, our method achieves the same CometKiwi score using only 70 scoring evaluations compared a baseline system using 180. We present a multi-fidelity setting for BayesOpt, where the candidates are first scored with a cheaper but noisier proxy scoring model, which further improves the cost-performance tradeoff when using smaller but well-trained distilled proxy scorers.

Figures

Figures reproduced from arXiv: 2411.09694 by the authors.

Figure 1
Figure 1. A machine translation system generates candi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: Performance of reranking methods measured as the average CometKiwi score of the selected [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Average CometKiwi score of the selected top candidate (y-axis) for BayesOpt+GP+P with Distilled-S [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Average CometKiwi score of the selected candidate (y-axis) for BayesOpt+GP+P with different choices of proxy score. observation point until min(n, |C|) candidates are scored. UniqRandom simulates the effect of iteratively sampling candidates until n unique candidates a…
Figure 5
Figure 5. Figure 5: Difference between BayesOpt+GP with batch [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 25 canonical work pages

  1. [1]

    Daniel Beck and Trevor Cohn. 2017. https://aclanthology.org/I17-2012 Learning kernels over strings using G aussian processes . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 67--73. Asian Federation of Natural Language Processing

    work page 2017

  2. [2]

    Daniel Beck, Trevor Cohn, and Lucia Specia. 2014. https://doi.org/10.3115/v1/D14-1190 Joint emotion analysis via multi-task G aussian processes . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , 1798--1803. Association for Computational Linguistics

    work page doi:10.3115/v1/d14-1190 2014

  3. [3]

    Daniel Beck, Kashif Shah, Trevor Cohn, and Lucia Specia. 2013. https://aclanthology.org/W13-2241 SHEF - L ite: When less is more for translation quality estimation . In Proceedings of the Eighth Workshop on Statistical Machine Translation, 337--342. Association for Computational Linguistics

    work page 2013

  4. [4]

    James Bergstra, R\' e mi Bardenet, Yoshua Bengio, and Bal\' a zs K\' e gl. 2011. https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf Algorithms for hyper-parameter optimization . In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc

    2011

  5. [5]

    Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2021. https://doi.org/10.18653/v1/2021.acl-long.349 Energy-based reranking: Improving neural machine translation using energy-based models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In...

    work page doi:10.18653/v1/2021.acl-long.349 2021

  6. [6]

    Eugene Charniak and Mark Johnson. 2005. https://doi.org/10.3115/1219840.1219862 Coarse-to-fine n-best parsing and M ax E nt discriminative reranking . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics ( ACL ' 05) , 173--180. Association for Computational Linguistics

    arXiv 2005

  7. [7]

    Julius Cheng and Andreas Vlachos. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.767 Faster minimum B ayes risk decoding with confidence-based pruning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 12473--12480. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.767 2023

  8. [8]

    Michael Collins and Terry Koo. 2005. https://doi.org/10.1162/0891201053630273 Discriminative reranking for natural language parsing . Computational Linguistics, 31(1):25--70

    doi:10.1162/0891201053630273 2005

Show all 55 references
  1. [9]

    Conn, Katya Scheinberg, and Luis N

    Andrew R. Conn, Katya Scheinberg, and Luis N. Vicente. 2009. https://doi.org/10.1137/1.9780898718768 Introduction to Derivative-Free Optimization . Society for Industrial and Applied Mathematics

    2009 doi

  2. [10]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . Co...

    2019 arXiv

  3. [11]

    Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe, Hideki Tanaka, and Masao Utiyama. 2024. https://doi.org/10.18653/v1/2024.findings-acl.654 Centroid-based efficient minimum B ayes risk decoding . In Findings of the Association for Computational Linguistics ACL...

    2024 doi

  4. [12]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. https://doi.org/10.18653/v1/D18-1045 Understanding back-translation at scale . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 489--500. Association for Computational Li...

    2018 doi

  5. [13]

    Bryan Eikema and Wilker Aziz. 2020. https://doi.org/10.18653/v1/2020.coling-main.398 Is MAP decoding all you need? the inadequacy of the mode in neural machine translation . In Proceedings of the 28th International Conference on Computational Linguistics, 4506--4520. Internati...

    2020 doi

  6. [14]

    Bryan Eikema and Wilker Aziz. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.754 Sampling-based approximations to minimum B ayes risk decoding for neural machine translation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 10978--...

    2022 doi

  7. [15]

    Patrick Fernandes, Ant \'o nio Farinhas, Ricardo Rei, Jos \'e G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. https://doi.org/10.18653/v1/2022.naacl-main.100 Quality-aware decoding for neural machine translation . In Proceedings of the 2022 Conference of t...

    2022 doi

  8. [16]

    Mara Finkelstein, Subhajit Naskar, Mehdi Mirzazadeh, Apurva Shah, and Markus Freitag. 2024. http://arxiv.org/abs/2309.10966 MBR and QE finetuning: Training-time distillation of the best and most expensive decoding methods

    2024 arXiv

  9. [17]

    Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.617 Epsilon sampling rocks: Investigating sampling strategies for minimum B ayes risk decoding for machine translation . In Findings of the Association for Computa...

    2023 doi

  10. [18]

    Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022. https://doi.org/10.1162/tacl\_a\_00491 High quality rather than high model probability: Minimum B ayes risk decoding with neural metrics . Transactions of the Association for Computational Linguistics, 10:811--825

    2022 doi

  11. [19]

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023 b . https://doi.org/10.18653/v1/2023.wmt-1.51 Results ...

    2023 doi

  12. [20]

    Philipp Hennig and Christian J. Schuler. 2011. https://api.semanticscholar.org/CorpusID:166832 Entropy search for information-efficient global optimization . ArXiv, abs/1112.1217

    2011 arXiv

  13. [21]

    John Hewitt, Christopher Manning, and Percy Liang. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.249 Truncation sampling as language model desmoothing . In Findings of the Association for Computational Linguistics: EMNLP 2022, 3414--3427. Association for Computational ...

    2022 doi

  14. [22]

    Donald R. Jones. 2001. https://api.semanticscholar.org/CorpusID:8723392 A taxonomy of global optimization methods based on response surfaces . Journal of Global Optimization, 21:345--383

    2001

  15. [23]

    Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.wmt-1.63 M etric X -23: The G oogle submission to the WMT 2023 metrics shared task . In Proceedings of the Eighth Conference on Machin...

    2023 doi

  16. [24]

    Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabas Poczos. 2016. https://proceedings.neurips.cc/paper_files/paper/2016/file/605ff764c617d3cd28dbbdd72be8f9a2-Paper.pdf Gaussian process bandit optimisation with multi-fidelity evaluations . In A...

    2016

  17. [25]

    Cenk Kaynak and Ethem Alpaydin. 2000. MultiStage Cascading of Multiple Classifiers: One Man’s Noise is Another Man’s Data . In Proceedings of the 17th International Conference on Machine Learning , 455--462. Morgan Kaufmann

    2000

  18. [26]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, ...

    2024

  19. [27]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Nov \'a k, Ma...

    2022

  20. [28]

    Tom Kocmi, Vil \'e m Zouhar, Christian Federmann, and Matt Post. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.110 Navigating the metrics maze: Reconciling score magnitudes and accuracies . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024 doi

  21. [29]

    Andreas Krause, Ajit Singh, and Carlos Guestrin. 2008. https://www.jmlr.org/papers/volume9/krause08a/krause08a.pdf Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies . J. Mach. Learn. Res., 9:235–284

    2008

  22. [30]

    Ann Lee, Michael Auli, and Marc ' Aurelio Ranzato. 2021. https://doi.org/10.18653/v1/2021.acl-long.563 Discriminative reranking for neural machine translation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa...

    2021 doi

  23. [31]

    Jonas Mockus. 1974. https://link.springer.com/content/pdf/10.1007/978-3-662-38527-2_55.pdf On bayesian methods for seeking the extremum . In Proceedings of the IFIP Technical Conference, 400--404

    1974 doi

  24. [32]

    Noack, Harinarayan Krishnan, Mark D

    Marcus M. Noack, Harinarayan Krishnan, Mark D. Risser, and Kristofer G. Reyes. 2023. https://doi.org/10.1038/s41598-023-30062-8 Exact gaussian processes for massive datasets via non-stationary sparsity-discovering kernels . Scientific Reports, 13(1)

    2023 doi

  25. [33]

    Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. https://doi.org/10.18653/v1/W18-6301 Scaling neural machine translation . In Proceedings of the Third Conference on Machine Translation: Research Papers, 1--9. Association for Computational Linguistics

    2018 doi

  26. [34]

    Slav Petrov. 2011. https://dl.acm.org/doi/abs/10.5555/2103611 Coarse-to-Fine Natural Language Processing (Theory and Applications of Natural Language Processing) . Springer Publishing Company, Incorporated

    2011 doi

  27. [35]

    Rasmussen and C.K.I

    C.E. Rasmussen and C.K.I. Williams. 2005. https://books.google.co.uk/books?id=GhoSngEACAAJ Gaussian Processes for Machine Learning . Adaptive Computation and Machine Learning series. MIT Press

    2005

  28. [36]

    de Souza, Pedro G

    Ricardo Rei, Ana C Farinha, Jos \'e G.C. de Souza, Pedro G. Ramos, Andr \'e F.T. Martins, Luisa Coheur, and Alon Lavie. 2022 a . https://aclanthology.org/2022.eamt-1.9 Searching for COMETINHO : The little metric that could . In Proceedings of the 23rd Annual Conference of the ...

    2022

  29. [37]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2685--2702. Associ...

    2020 doi

  30. [38]

    Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G

    Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr \'e F. T. Martins. 2022 b . https://aclanthology.org/2022.wmt-1.60 C omet K iwi: IST -u...

    2022

  31. [39]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7881--7892. Association for C...

    2020 doi

  32. [40]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1009 Improving neural machine translation models with monolingual data . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2016 doi

  33. [41]

    Adams, and Nando de Freitas

    Bobak Shahriari, Kevin Swersky, Ziyun Wang, Ryan P. Adams, and Nando de Freitas. 2016. https://api.semanticscholar.org/CorpusID:14843594 Taking the human out of the loop: A review of bayesian optimization . Proceedings of the IEEE, 104:148--175

    2016

  34. [42]

    Prasann Singhal, Jiacheng Xu, Xi Ye, and Greg Durrett. 2023. https://doi.org/10.18653/v1/2023.acl-long.517 EEL : Efficiently encoding lattices for reranking . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 92...

    2023 doi

  35. [43]

    Kakade, and Matthias W

    Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. 2009. https://api.semanticscholar.org/CorpusID:59031327 Information-theoretic regret bounds for gaussian process optimization in the bandit setting . IEEE Transactions on Information Theory, 58:3250--3265

    2009

  36. [44]

    Felix Stahlberg. 2020. https://www.jair.org/index.php/jair/article/view/12007 Neural machine translation: A review . Journal of Artificial Intelligence Research, 69:343--418

    2020

  37. [45]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  38. [46]

    Firas Trabelsi, David Vilar, Mara Finkelstein, and Markus Freitag. 2024. http://arxiv.org/abs/2406.02832 Efficient minimum bayes risk decoding using low-rank matrix completion algorithms

    2024 arXiv

  39. [47]

    Jannis Vamvas and Rico Sennrich. 2024. https://doi.org/10.18653/v1/2024.acl-short.71 Linear-time minimum B ayes risk decoding with reference aggregation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 790--...

    2024 doi

  40. [48]

    Giorgos Vernikos and Andrei Popescu-Belis. 2024. https://aclanthology.org/2024.acl-long.653 Don ' t rank, combine! combining machine translation hypotheses using quality estimation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  41. [50]

    Jun Wang, Eleftheria Briakou, Hamid Dadkhahi, Rishabh Agarwal, Colin Cherry, and Trevor Cohn. 2024 b . https://api.semanticscholar.org/CorpusID:271213690 Don't throw away data: Better sequence knowledge distillation . ArXiv, abs/2407.10456

    2024 arXiv

  42. [51]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. http://arxiv.org/abs/2002.10957 MiniLM : Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    2020 arXiv

  43. [52]

    Frazier, and Andrew Gordon Wilson

    Jian Wu, Saul Toscano-Palmerin, Peter I. Frazier, and Andrew Gordon Wilson. 2020 . https://proceedings.mlr.press/v115/wu20a.html Practical multi-fidelity bayesian optimization for hyperparameter tuning . In Proceedings of The 35th Uncertainty in Artificial Intelligence Confere...

    2020

  44. [53]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith St...

    2016 arXiv

  45. [54]

    Kyra Yee, Yann Dauphin, and Michael Auli. 2019. https://doi.org/10.18653/v1/D19-1571 Simple and effective noisy channel modeling for neural machine translation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International...

    2019 doi

  46. [55]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  47. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.