{"id":"a704801e-0ac2-4cfc-9b2f-cc196f66019a","arxiv_id":"2411.09694","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Bayesian optimization with Gaussian processes finds top-scoring machine translation candidates while scoring only a fraction of the candidate list, and multi-fidelity proxies improve the cost-performance tradeoff.","lead":"This paper uses Bayesian optimization to choose which machine translation candidates to score with an expensive quality model, finding near-best translations while scoring fewer than half the candidates. The method could make high-quality reranking cheaper for production translation systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency gain rests on an untested smoothness assumption for CometKiwi in NLLB embedding space; a direct semivariogram or embedding-ablation check is needed.","rationale":"The reader's weakest assumption correctly identifies the smoothness of CometKiwi in the NLLB embedding space as the load-bearing premise. The paper provides indirect evidence via HillClimbing, but no direct diagnostic of the GP's modeling assumption. This concern is significant because it determines whether the method generalizes beyond the specific generator/scorer pair tested, but it does not invalidate the empirical results as reported. The code is released and the experimental protocol is clear enough that the proposed test can be run by a third party. Because the concern is addressable and the central empirical claim is supported by the data, the reader's CONDITIONAL verdict remains appropriate; my read does not change it.","tokens_in":16242,"tokens_out":8123,"duration_ms":80678,"concrete_test":"On a held-out subset of the test instances, compute the empirical semivariogram: for random candidate pairs, plot |s(y_i) - s(y_j)| against ||emb(y_i) - emb(y_j)|| after unit-normalization, and check for a monotonically increasing trend consistent with the RBF kernel. Additionally, run the full BayesOpt+GP pipeline with the same settings but with embeddings replaced by random unit vectors; if the 70-vs-180 efficiency gap disappears or reverses, the specific embedding geometry is load-bearing rather than the generic exploration-exploitation mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — BayesOpt+GP attains the same CometKiwi score with ~70 evaluations as random scoring of ~180 — depends on the RBF kernel KMT(y_i,y_j) in Section 3.1 being an informative prior over the score function s in the mean-pooled NLLB decoder embedding space. If s is not smooth in this space, the GP posterior uncertainty is miscalibrated and EI degenerates toward random selection, so the reported gains would not transfer to other scorers or generators. The paper does not directly validate this smoothness; it only shows that HillClimbing, a smoothness-exploiting heuristic, is the strongest baseline, which is suggestive but not conclusive. A further subtlety is that emb is taken from the candidate generator, so the kernel is coupled to the generation model, while the paper frames the method as applicable to 'arbitrary black-box scoring functions' (Section 2.1). The single bandwidth w tuned on the validation set is not reported, making it impossible to assess how finely tuned the smoothness assumption must be.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames machine translation list reranking as a Bayesian optimization problem. Given a candidate list generated by a neural MT system, a Gaussian process (GP) with an RBF kernel defined over mean-pooled final-decoder embeddings models the dependence between candidates and their CometKiwi scores. Expected improvement is used to select which candidates to score next under a fixed budget, and the highest-scoring observed candidate is returned. A multi-fidelity extension, BayesOpt+GP+P, incorporates cheaper proxy scores through a product kernel. Experiments on WMT23 with an NLLB generator and CometKiwi-22 show that BayesOpt+GP reaches the score that random selection achieves with 180 evaluations using only about 70 evaluations, and that the multi-fidelity variant can further improve early candidate quality when the proxy is well correlated. The paper includes several baselines, statistical significance tests, runtime profiling, and a code repository.","tokens_in":16428,"tokens_out":6375,"duration_ms":59094,"significance":"If the reported results hold, the paper makes a useful practical contribution to reducing the computational cost of MT reranking, which is increasingly dominated by large quality-estimation models. The experimental design is careful: it compares against multiple baselines, reports statistical significance in the appendix, profiles the full pipeline, and makes the code available. The empirical claim is evaluated on an external benchmark (CometKiwi on WMT23), so the core result is not circular with respect to the method's objective. The main risk is that the central efficiency gain rests on an untested smoothness assumption about the scoring function in the embedding space induced by the candidate generator, which limits the generality of the claims as currently stated.","major_comments":[{"comment":"The central efficiency claim depends on the assumption that CometKiwi scores vary smoothly in the mean-pooled NLLB decoder embedding space, since the kernel KMT(y_i,y_j) = KRBF(emb(y_i),emb(y_j)) uses the RBF of Eq. (1). The paper provides no direct evidence for this smoothness: there is no semivariogram or correlation analysis between embedding distances and absolute score differences, no comparison with a kernel defined on an unrelated representation, and no ablation of the embedding source. Because the kernel is tied to the candidate generator's internal states, the method is not an 'arbitrary black-box' optimizer as claimed in Section 2.1; it is coupled to the generator. The reported gains on CometKiwi/NLLB are suggestive, but transferability to other scoring functions or generators is not established. I recommend adding a direct smoothness diagnostic (e.g., correlation of pairwise kernel distances with score differences on the validation set) or explicitly limiting the method's scope to settings where such a diagnostic is favorable.","section":"Section 3.1, Eq. (1); Section 2.1"},{"comment":"The single RBF bandwidth w is grid-searched on the entire validation set and fixed for all experiments, but its value is never reported. The text states that results are 'not statistically significantly different within a range of settings' without providing that range or any supporting numbers. Since the acquisition function's exploration-exploitation tradeoff is controlled entirely by w, omitting the actual value and the stability range prevents readers from assessing how finely the smoothness assumption must be tuned. Please report the tuned w and the range over which the quality-cost curves remain statistically indistinguishable.","section":"Section 4 (experimental setup)"}],"minor_comments":[{"comment":"The abstract's claim that BayesOpt+GP 'achieves the same CometKiwi score using only 70 scoring evaluations compared a baseline system using 180' is slightly imprecise: from Table 3, BayesOpt+GP at 70 runs scores 0.8210 while UniqRandom at 180 scores 0.8211, a difference of 0.0001. This is well within noise, but the wording should be qualified (e.g., 'statistically indistinguishable') to avoid an overly strong reading.","section":"Abstract"},{"comment":"The legend for the significance tables is confusing: it says '↑ means that the column method is significantly better than the row method,' but the example given in the caption ('third row and first column, ← means that HillClimbing is significantly better than UniqRandom') does not obviously match the printed layout. Please clarify the table orientation or provide a clearer legend.","section":"Appendix A, Table 4"},{"comment":"The 'Comet Loading' value for BayesOpt+GP+P (11.27s) differs from the other two columns (8.43s) even though the same CometKiwi model is loaded. If the extra time includes loading the proxy model or other overhead, please state this explicitly in the table caption or the text.","section":"Section 4.4, Table 2"},{"comment":"The sentence 'GPs make very few assumptions about the distribution and base their predictions are mostly on observed points' is grammatically garbled, and the claim that GPs make 'very few assumptions' is misleading because the kernel choice itself is a strong prior. Please rephrase.","section":"Section 1"},{"comment":"The statement that the kernel is 'fast to compute' is supported by the runtime measurements, but the O(|C|^2) similarity computation and O(|C|^3) matrix inversion costs are only mentioned in the Limitations section. For reproducibility and to set expectations, these costs should also be stated in the main method section.","section":"Section 3.1"},{"comment":"The baselines are introduced as 'Logprob{Avg,Sum}' but one sentence refers to them as 'LogprobFirst{Avg,Sum}'. Please make the naming consistent.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with careful experimentation and a clear potential for practical impact. The main concern is the untested smoothness assumption that underlies the efficiency result; this is not a circularity issue but a transferability issue. I would encourage the editors to request the smoothness diagnostic and the bandwidth value, as these are load-bearing for the generality claim. The paper fits the scope of a top NLP venue, and with the requested revisions the contribution would be significantly stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful empirical paper. It is the first to formulate MT candidate reranking as Bayesian optimization, with a GP over the generator's mean-pooled decoder embeddings, plus a multi-fidelity kernel that folds in cheaper proxy scores. The experiments are careful: quality-cost curves, statistical tests in Appendix A, runtime profiling, and a code repository. The central efficiency claim — same CometKiwi score with roughly 70 scoring calls instead of 180 for random selection — is supported by the numbers in Table 3 across seven language pairs. I believe that result holds for the setup they test.\n\nWhat is genuinely new is the application and the embedding-space kernel, not the optimization machinery. Each component is standard, but the packaging is a real contribution, and the multi-fidelity extension with distilled proxies is a nice practical addition.\n\nSoft spots, in rough order of importance. First, the smoothness assumption on which the whole gain rests — CometKiwi scores are smooth in NLLB mean-pooled decoder embedding space — is never directly validated. The strong HillClimbing baseline is suggestive, but a semivariogram or a simple experiment with a deliberately non-smooth scoring function would have made the mechanism much more convincing. This matters for transfer more than for the reported result: their numbers show it works on their setup, but the 'arbitrary black-box scorer' framing overreaches because the kernel needs an embedding space from the generator. Second, the RBF bandwidth w is tuned on the validation set and never reported; the authors claim the results are stable over a range, but the actual value is needed to assess how fragile the smoothness assumption is. Third, the test set is the first 1000 and 500 sentences per language pair from WMT23, not a random sample; that is a moderate representativeness concern. Fourth, no data or checkpoints are released, only code, so exact reproduction of the proxy distillation would be difficult.\n\nNone of this undermines the core empirical claim for the tested conditions. The related work is covered fairly, including the prior pruning and MBR-approximation literature. I would send this to a serious referee; it deserves review rather than desk rejection.","headline":"A solid, carefully measured first BayesOpt treatment of MT reranking whose efficiency claim holds for the tested setup, with real soft spots around the untested smoothness assumption, an unreported bandwidth, and a non-random test split.","tokens_in":16967,"tokens_out":1904,"would_cite":true,"duration_ms":19157,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Framing machine-translation reranking as Bayesian optimization finds the best candidate with a fraction of the scoring calls, matching a 180-candidate random baseline with only 70 evaluations.","keywords":["machine translation reranking","Bayesian optimization","Gaussian process","quality estimation","multi-fidelity optimization","expected improvement","CometKiwi","efficient decoding"],"falsifier":"Apply the same algorithm to the same candidate lists and embeddings but replace the CometKiwi scores with a deliberately non-smooth scoring function—for example, scores assigned by exact string match against a reference or scores drawn randomly per instance—and run BayesOpt+GP with the same kernel; if the quality-cost curve no longer beats random subset selection, the smoothness assumption is confirmed as the load-bearing premise. Alternatively, use a different generator whose decoder embeddings do not place similar translations near each other and measure whether the advantage diminishes.","tokens_in":16031,"feed_emoji":"🎯","tokens_out":11037,"duration_ms":87730,"temperature":0.7,"pith_summary":"Machine-translation systems often improve output quality by reranking a list of candidate translations with an external quality scorer, but the best scorers are so expensive that reranking nearly doubles the cost of translation. This paper tries to make reranking cheap by treating the candidate list as an unknown score surface and using Bayesian optimization with a Gaussian process to decide which candidates to score next. The method chooses candidates by balancing exploration of uncertain translations against exploitation of promising ones, and on the paper's test set it reaches the same CometKiwi score with 70 scoring calls that a random-subset baseline needs 180 calls to match. The paper also introduces a multi-fidelity version that first scores candidates with a cheaper, noisier proxy model, and shows this helps when the proxy is well correlated with the main scorer.","feed_headline":"Rerank with 70 scores, match a 180-score baseline","feed_subtitle":"A Gaussian process search over candidate translations cuts the number of expensive quality-model calls by more than half.","key_machinery":"The load-bearing object is the kernel that measures similarity between candidate translations, $K_{MT}(y_i,y_j) = K_{RBF}(\\mathrm{emb}(y_i), \\mathrm{emb}(y_j))$, where $\\mathrm{emb}$ is the mean-pooled, unit-normalized output of the generator's final decoder layer and $K_{RBF}$ is the radial-basis kernel with a single bandwidth $w$ tuned on the validation set. This kernel encodes the prior assumption that similar translations receive similar scores, and it makes the Gaussian-process posterior computable from only the few observed scores. The acquisition function is closed-form expected improvement, which scores each unscored candidate by its expected gain over the current best observation, balancing exploration of uncertain regions against exploitation of promising ones. For the multi-fidelity variant, the kernel becomes $K_{mult} = K_{MT} \\cdot K_{score}(s_k, s_l)$, where $K_{score}$ is the empirical covariance between the main scorer and the proxy scorer measured on a validation set; this product kernel lets proxy observations shape the posterior over the main score.","core_discovery":"The paper's central claim is that machine-translation reranking can be viewed as a black-box optimization problem: given a list of candidate translations and a scoring function, a Gaussian-process model of the score surface—using an RBF kernel over mean-pooled final-decoder embeddings of the candidates—lets an expected-improvement acquisition function find near-best candidates while scoring only a fraction of the list. On the test set, with 200 sampled candidates per source sentence (about 178 after deduplication), BayesOpt+GP achieves an average CometKiwi score of 0.8210 with 70 scoring calls, versus 0.8149 for 70 random candidates and 0.8216 when roughly the full list is scored; the maximum attainable score is 0.8216. The method also outperforms search-based baselines (hill-climbing and log-probability selection) on the quality-cost tradeoff curve. The multi-fidelity extension multiplies the candidate kernel by an empirical covariance kernel between scorers, so observations from a cheap, distilled proxy scorer inform the posterior over the main scorer; with a well-correlated proxy, the method matches or exceeds the full-list score with about 70–90 CometKiwi calls and further reduces runtime.","pith_inferences":["A natural extension the paper does not test is applying the same search to human evaluation or an LLM used as a judge, where smoothness of the score surface in embedding space is less established; the method's assumptions would need to be re-validated for such scorers.","The RBF bandwidth $w$ is fixed after tuning on the validation set; making it instance-specific or learned online might further improve the efficiency when candidate lists come from different language pairs or generators.","The product-kernel formulation extends to more than two scorers, so a cascade of increasingly expensive scoring models could be exploited, although the paper only demonstrates the two-scorer case.","The candidate embeddings come for free from the generator; using a different embedding source (for example, a cross-lingual sentence encoder) may change the smoothness and therefore the performance, which a careful reader could test."],"forward_implications":["If the smoothness assumption holds, translation systems can use far larger candidate lists for the same computational budget, because the number of expensive scorer calls no longer scales with list size.","The multi-fidelity result implies that a cheap, well-distilled proxy scorer can replace a large fraction of the main scorer's calls; in the paper's runtime profile this cuts the total reranking time from 984.7 seconds to 873.6 seconds per 350 instances at matched output quality.","Because the method treats the scorer as a black box, it should carry over to any quality estimator whose scores vary smoothly in the chosen embedding space, including quality-estimation models other than CometKiwi.","The quality-cost curves show diminishing returns after roughly 70–90 scoring calls, so the method gives practitioners a principled way to choose the scoring budget without scoring the full list."],"supporting_citations":[{"why":"Supplies the Bayesian optimization framework and the standard acquisition-function setup.","marker":"Shahriari et al., 2016"},{"why":"Provides the Gaussian-process posterior equations and proves that a product of kernels is a valid kernel, which the multi-fidelity kernel relies on.","marker":"Rasmussen and Williams, 2005"},{"why":"Gives the closed-form expected-improvement acquisition function used to pick the next candidate.","marker":"Jones, 2001"},{"why":"CometKiwi is the main scoring model being optimized in all reranking experiments.","marker":"Rei et al., 2022b"},{"why":"Motivates reranking with quality-estimation models and supplies the two-stage pruning baseline related to the multi-fidelity setting.","marker":"Fernandes et al., 2022"},{"why":"Provides the sampling-based approximate MBR view and the use of a faster scorer for two-stage reranking.","marker":"Eikema and Aziz, 2022"},{"why":"Introduces multi-fidelity Bayesian optimization, which the paper's product-kernel extension adapts to list reranking.","marker":"Kandasamy et al., 2016"},{"why":"Describes training smaller distilled COMET models that serve as proxy scorers in the multi-fidelity experiments.","marker":"Rei et al., 2022a"},{"why":"Defines the epsilon-sampling method used to generate the 200-candidate lists for all reranking experiments.","marker":"Hewitt et al., 2022"}],"fun_headline_variants":["BayesOpt cuts rerank scoring to 70 calls","Machine translation rerank: 70 scores beat 180","Match full-list rerank with 70 scorer calls","Multi-fidelity rerank: cheaper proxy, same quality","GP-guided reranking halves quality-model calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's efficiency relies on the assumption that similar translations, as judged by the embedding vectors, usually receive similar quality scores, so the Gaussian process's uncertainty estimates faithfully reflect where good candidates are; if that smoothness fails for a different scorer or generator, the query selection becomes no better than random.","fun_headline_variants_meta":{"raw":{"variants":["BayesOpt cuts rerank scoring to 70 calls","Machine translation rerank: 70 scores beat 180","Match full-list rerank with 70 scorer calls","Multi-fidelity rerank: cheaper proxy, same quality","GP-guided reranking halves quality-model calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1239,"prompt_tokens":966,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":582,"tokens_out":273,"duration_ms":3095,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:22:50.058575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same algorithm to the same candidate lists and embeddings but replace the CometKiwi scores with a deliberately non-smooth scoring function—for example, scores assigned by exact string match against a reference or scores drawn randomly per instance—and run BayesOpt+GP with the same kernel; if the quality-cost curve no longer beats random subset selection, the smoothness assumption is confirmed as the load-bearing premise. Alternatively, use a different generator whose decoder embeddings do not place similar translations near each other and measure whether the advantage diminishes.","supporting_citations":[{"cited_title":"Adams, and Nando de Freitas","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian optimization framework and the standard acquisition-function setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces multi-fidelity Bayesian optimization, which the paper's product-kernel extension adapts to list reranking."}],"review_version":1}