{"id":"27b5648c-fe9f-4513-8ecb-f6a3c9f586f5","arxiv_id":"2506.20544","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hedged sampling, checklist-based one-pass selection (CHOPS), and cross-lingual MBR (X-MBR) improve multilingual LLM output quality when scaling from one to five samples.","lead":"This paper tests whether generating several LLM answers and picking the best one, a common trick to spend more compute at inference time, can be made to work across languages. It proposes three new sampling and selection techniques and reports consistent quality gains on multilingual benchmarks for 8B and 111B models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains rest on untested cross-lingual LLM-judge alignment; a judge-bias artifact could invalidate the multilingual win-rate claims.","rationale":"The paper's central contribution is empirical: it claims that adapted sampling and selection methods produce quality gains across languages. The evidence for that claim is entirely mediated by LLM judges — GPT-4o for evaluation and Command A for selection. The reader's weakest_assumption identifies exactly this reliance, and I agree it is the most load-bearing assumption. The paper acknowledges the general risk in Limitations but does not test whether judge preferences are language-dependent; in fact, X-MBR's design of using English/Chinese evidence increases the chance that selection optimizes for judge-specific cross-lingual artifacts. This is not an internal inconsistency, because the paper's metric is defined as GPT-4o win rate, but it is a correctness risk for the broader claim about multilingual quality and democratization. A human-evaluation check would settle the concern. Since the reader already marked the verdict CONDITIONAL with moderate confidence, and this concern supports rather than changes that judgment, the appropriate recommendation is UNCHANGED.","tokens_in":26905,"tokens_out":5866,"duration_ms":73660,"concrete_test":"Run a blinded human preference study on a stratified sample of 50 m-ArenaHard-v2.0 prompts per language (350 total), pitting the CHOPS- and X-MBR-selected outputs against the greedy baseline used in Table 4, and compare the human win-rate deltas per language against the GPT-4o-computed deltas. If human deltas do not reproduce the reported gains (e.g., shrink below noise or reverse in Japanese/Russian), the headline results are judge artifacts; if human deltas track the GPT-4o deltas, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All headline deltas (e.g., +17.3, +9.0, +6.8) are measured by GPT-4o win rates on m-ArenaHard, while the proposed CHOPS and X-MBR select using Command A as judge. The paper's only guard is the Limitations note 'Reliance on judge alignment', which says the methods 'do not directly address this issue' and offers no cross-lingual check. This matters specifically for X-MBR: it scores target-language candidates against English/Chinese pseudo-references (eq. 8), so selection can favor outputs that best match the judge's stylistic or length preferences in those evidence languages rather than genuine quality. Command A was chosen partly for M-RewardBench alignment, which measures preference accuracy, not human agreement on open-ended win rates. Without knowing whether GPT-4o and Command A agree with human preferences per language, the central claim that the methods 'democratize' multilingual quality is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies parallel inference-time scaling (sampling N=5 outputs and selecting one) for multilingual, multi-task generation. The authors evaluate Aya-Expanse-8B, Qwen3-8B, and Command-A (111B) across seven languages on m-ArenaHard open-ended tasks, MGSM math reasoning, and WMT24++ translation. They introduce hedged sampling (mixing greedy with stochastic samples), CHOPS (one-pass checklist-based judge selection), and X-MBR (MBR with cross-lingual evidence in English/Chinese), reporting consistent win-rate, accuracy, and XComet gains over greedy decoding, with headline improvements such as +17.3 win-rate points for Aya on m-ArenaHard and +9.0 for Command-A. The paper argues that English-centric inference-time scaling strategies do not transfer across languages and tasks.","tokens_in":27121,"tokens_out":7548,"duration_ms":77733,"significance":"The contribution is practically relevant and empirically broad: the authors compare against strong reward-model BoN baselines, use separate dev, devtest, and test splits, and report results on open-ended generation, math, and MT across seven languages. If the judge-based gains are real, the proposed methods are low-cost and useful for production multilingual systems. However, the central quantitative claims rest entirely on LLM-judge win rates, and the paper explicitly acknowledges in its Limitations that judge alignment is not addressed; the absence of any cross-lingual validation of the judges is a significant gap that must be closed before the headline deltas can be taken at face value.","major_comments":[{"comment":"The manuscript's central win-rate claims are evaluated with GPT-4o (m-ArenaHard) while CHOPS and X-MBR select using Command A. The Limitations section ('Reliance on judge alignment') concedes that the methods 'do not directly address' the issue, and no per-language agreement analysis between either judge and human preferences is provided. This matters because X-MBR (Eq. 8) scores target-language candidates against English/Chinese pseudo-references, so selection can favor outputs that match the judge's stylistic or length preferences in those evidence languages rather than genuine quality. M-RewardBench (Table 5) measures preference accuracy on a different task, not agreement on open-ended win rates. Please add a cross-lingual judge-bias check: either per-language human agreement for GPT-4o and Command A on a sample of m-ArenaHard, or a judge-substitution experiment (e.g., re-running the final evaluation with a different judge model) to demonstrate that the reported deltas are not artifacts.","section":"Limitations; §3.3.2; Eq. (8)"},{"comment":"All quantitative results, including the headline '+17.3', '+9.0', and '+6.8', are point estimates without confidence intervals or significance tests. In Table 3, many deltas are small (e.g., +0.20 XComet for Aya WMT), and Table 8 shows negative per-language X-MBR deltas on WMT; a paired bootstrap over the 250/498 prompts could establish which gains are stable. Without such intervals, the claim of 'consistent gains across languages' is not supported at the reported level of precision. Please add uncertainty estimates for the main tables.","section":"§3.3.2; Tables 3 and 4"},{"comment":"The text in §3.3.2 states 'We find improvements over the greedy baseline in all tasks, languages, and methods,' but Appendix F Table 8 reports negative X-MBR deltas for most WMT language pairs (e.g., Aya: Chinese -1.05, Japanese -3.62, Russian -2.73; Qwen: Chinese -0.23, Japanese -0.60, Russian -1.26). Even the averaged test result in Table 3 is only +0.20 for Aya WMT, and X-MBR underperforms RM-BoN and CHOPS on translation in both tables. The blanket claim and the recipe's recommendation of X-MBR for 'language disparities' therefore need to be qualified, for example restricted to open-ended generation.","section":"§3.3.2 vs Appendix F, Table 8"}],"minor_comments":[{"comment":"Please clarify in the abstract that the +6.8 average against Gemini is the X-MBR result specifically; CHOPS gives +5.0 and BoN +6.4 in Table 4, so the current wording ('our combined sampling and selection methods') overstates the consistency across methods.","section":"Abstract; §3.3.2, Table 4"},{"comment":"Section 1 contains a typo ('and and how well'), and §3.1 defines X-MBR without stating explicitly that the M=3 evidence samples are an additional inference budget beyond the N=5 candidates; please state this in the method description.","section":"§1; §3.1"},{"comment":"The x-axis labels ('Single Temperature Hedged Single Hedged Random Random Temperature') are hard to parse; consider a cleaner legend that distinguishes single vs multi-temperature and hedged vs non-hedged.","section":"Figure 4a"},{"comment":"M-RewardBench is a preference-accuracy benchmark; the text should clarify that it does not directly measure agreement with human open-ended quality judgments, which is precisely the gap noted in the Limitations.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The authors are from Cohere Labs and the judge model Command A is a same-lab model; while the evaluation judge is GPT-4o, the selection judge is a Cohere model. I would ask the editor to require explicit disclosure of any internal benchmarking that motivated the choice of Command A, and to require the cross-lingual judge-alignment analysis before publication. The manuscript fits the journal's scope, but the current evidence is conditional on judge validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a real empirical study, not a position piece. The authors test sampling and selection strategies across three tasks (open-ended generation, math, translation) and seven languages, with held-out dev/test splits and a sensible N=5 budget. The new pieces are hedged sampling (adding a greedy decode to the stochastic pool), CHOPS (one-pass checklist-based selection), and X-MBR (cross-lingual evidence for MBR). They are simple, practical, and—as far as I can tell—genuinely new in combination. The central finding, that sampling and selection need to be adapted to languages beyond English, is plausible and supported by the data.\n\nThe paper does several things well. The dev/test discipline is better than most: they tuned on dev, tested once on test. They compare against a strong RM-based BoN baseline and against Gemini, so the wins are not cheap. They report non-judge metrics (MGSM accuracy, WMT XComet) alongside win rates, and those also show gains. The Limitations section is honest: it flags judge alignment and the high-resource-language restriction. The CHOPS efficiency argument (O(1) vs O(N^2)) is real and practically useful.\n\nNow the soft spots. The headline m-ArenaHard numbers all come from GPT-4o win rates, and selection is done with Command A—same lab as the authors. The stress-test note is right that cross-lingual judge agreement with humans is never checked. That is a genuine gap, and for X-MBR specifically the English/Chinese evidence set could in principle steer selection toward judge stylistic preferences rather than target-language quality. The authors admit they do not address this. I would not call it fatal: the non-judge metrics show the same broad direction, and the consistency across methods and tasks argues against a pure artifact. But the per-language win-rate deltas without error bars or significance tests are hard to interpret—some numbers (e.g., X-MBR on WMT) are negative, and a few gains are within noise. The paper would be stronger with confidence intervals and a per-language judge-agreement analysis.\n\nWho is this for? Anyone doing test-time scaling for multilingual production systems, or working on multilingual evaluation. It deserves a serious referee; the core claims are substantive and the authors are transparent about limits. I would ask for error bars, a judge-bias sensitivity check, and code/data release. I would cite it if I worked in this area.","headline":"A genuinely multi-task, seven-language study of sampling and selection for inference-time scaling, with practical methods and honest limitations, but the headline win-rate results would be stronger with per-language judge agreement checks and error bars.","tokens_in":27640,"tokens_out":2589,"would_cite":true,"duration_ms":32046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that generating five candidate outputs and selecting among them with multilingual-aware sampling and judging outperforms single-sample decoding across languages, adding up to 17.3 win-rate points for an 8B model.","keywords":["inference-time compute","multilingual LLMs","hedged sampling","CHOPS","cross-lingual MBR","Best-of-N","test-time scaling","open-ended generation"],"falsifier":"Run the same hedged sampling with CHOPS and X-MBR on m-ArenaHard in the seven test languages, then have human annotators rate the selected outputs and compare win rates per language against GPT-4o's ratings. If the methods' advantage shrinks or reverses for some languages when ground truth is human preference rather than the automated judge, the central claim would be shown to be judge-dependent.","tokens_in":26732,"feed_emoji":"🌍","tokens_out":5801,"duration_ms":60589,"temperature":0.7,"pith_summary":"This paper argues that spending a small amount of extra inference-time compute—generating five candidate outputs instead of one—can reliably improve multilingual LLM quality, provided both how samples are drawn and how the best one is chosen are adapted to language diversity. English-centric recipes fail in non-English languages because high-temperature sampling raises the risk of poor outputs more sharply outside English; the authors' 'hedged sampling' counters this by always including a greedy decoding in the pool. For selection, they propose two methods: CHOPS, which asks an LLM judge to write a task-specific checklist and pick the best answer in one pass, and X-MBR, which extends minimum Bayes risk selection with evidence samples drawn in a stronger language. Together these methods add up to +17.3 win-rate points for Aya Expanse 8B and +9.0 for Command-A (111B) on multilingual open-ended benchmarks, against single-sample decoding, and +6.8 average points against Gemini 2.0 Flash.","feed_headline":"Five samples, chosen well, lift multilingual LLM win rates 17 points","feed_subtitle":"A language-aware recipe of hedged sampling and judge-based selection beats single decoding across seven languages.","key_machinery":"The machinery is a two-stage pipeline for parallel inference-time scaling. Sampling creates a pool of $N=5$ candidates. The paper's hedged sampling mixes stochastic samples at temperature $\\tau \\in [0.7, 0.9]$ with one deterministic greedy ($\\tau=0$) output, because non-English languages show higher 'risk'—the worst sample's quality drops faster at high temperature—so the greedy sample acts as a safety net. Selection then picks one candidate. The paper compares Best-of-N with a reward model (scoring each sample independently), Minimum Bayes Risk (MBR) with pairwise LLM-judge comparisons, and two new methods: CHOPS, which asks an LLM judge to first write a prompt-specific evaluation checklist and then select the best candidate in a single pass ($O(1)$ judge calls); and X-MBR, which extends the MBR evidence set with $M=3$ extra samples written in a dominant language (Chinese for English prompts, English otherwise), leveraging cross-lingual generation to sharpen selection.","core_discovery":"The paper's central claim is that parallel inference-time scaling—sample multiple outputs, then select one—generalizes to multilingual, multi-task settings only if both stages are language-aware. Existing selection methods that work in English (likelihood, similarity-based MBR) fail or underperform cross-lingually, while Best-of-N with a specialized reward model is a strong but not optimal baseline. The paper introduces hedged sampling (mixing high-temperature stochastic samples with the deterministic greedy output), CHOPS (one-pass checklist-based judge selection), and X-MBR (cross-lingual evidence for MBR). On test sets spanning open-ended generation (m-ArenaHard-v2.0), math reasoning (MGSM) and machine translation (WMT24++), across seven languages and two 8B models, the best combinations beat single-sample greedy decoding on every task and language, and also improve the outputs of the much larger Command-A model when it judges its own samples.","pith_inferences":["The cross-lingual evidence idea suggests a testable extension: for languages with no strong auxiliary language, evidence drawn from several languages or from machine-translated prompts may give a similar boost.","The checklist mechanism in CHOPS may be doing the real calibration work: asking the judge to operationalize quality criteria before reading candidates could reduce the length or style biases that plague LLM judges; this could be tested by comparing CHOPS against judges primed with fixed rubrics.","Because the gains appear in open-ended generation and translation, synthetic-data pipelines that produce multilingual training data could adopt the same selection step to filter higher-quality samples at negligible extra cost.","The paper only tests high-resource languages; if the pattern holds, the risk-hedging motivation is even stronger for lower-resource languages, where sample variance is larger."],"forward_implications":["Five well-chosen samples can replace a much larger and more expensive model for many multilingual generation tasks.","Existing English-tuned reward models are not the ceiling: generalist multilingual LLM judges can outperform a specialized RM for Best-of-N in open-ended multilingual generation.","The same recipe transfers to a 111B model selecting its own outputs, so the gains do not depend on a size gap between generator and judge.","The recipe is cheap: CHOPS costs $O(1)$ judge calls, and X-MBR requires only $N+M=8$ total samples and $O(N(N+M))$ pairwise comparisons.","Scaling curves are steep at $N=5$; beyond $N=10$ gains become less predictable, so small budgets are a sensible operating point."],"supporting_citations":[{"why":"Supplies the greedy-decoding baseline recommendation and the task-dependent view of deterministic vs stochastic sampling that the paper benchmarks against.","marker":"[Song et al., 2025]"},{"why":"Formalizes Best-of-N and its scaling limits, motivating the BoN baseline and the observation that gains flatten beyond N=10.","marker":"[Huang et al., 2025]"},{"why":"Introduces sampling-based MBR decoding for neural machine translation, the selection paradigm X-MBR extends.","marker":"[Eikema & Aziz, 2022]"},{"why":"Establishes MBR with separate hypothesis and evidence sets, used to justify the cross-lingual evidence extension.","marker":"[Bertsch et al., 2023]"},{"why":"Introduced Aya Expanse and the m-ArenaHard benchmarks, supplying the models and the open-ended evaluation setup.","marker":"[Dang et al., 2024b]"},{"why":"Command A is the 111B multilingual judge used for CHOPS and X-MBR, and the model used in the self-improvement experiment.","marker":"[Cohere et al., 2025]"},{"why":"Qwen3-8B is the second 8B generator used to test whether the methods transfer across model families.","marker":"[Yang et al., 2025a]"},{"why":"Establishes the limits of resampling with imperfect verifiers, framing the paper's reliance on judge alignment.","marker":"[Stroebl et al., 2024]"}],"fun_headline_variants":["Language-aware sampling lifts multilingual LLM wins by 9","Scaling inference compute: multilingual LLMs need language-aware sampling","Hedged sampling and cross-lingual selection boost LLM win rates","Language-aware inference scaling lifts multilingual LLM wins","Multilingual LLMs: sampling strategy beats single decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the judges used for selection (Command A) and for evaluation (GPT-4o) agreeing with human preferences in every language; the paper acknowledges this reliance but does not measure whether judge preferences are biased toward certain languages, styles, or output lengths.","fun_headline_variants_meta":{"raw":{"variants":["Language-aware sampling lifts multilingual LLM wins by 9","Scaling inference compute: multilingual LLMs need language-aware sampling","Hedged sampling and cross-lingual selection boost LLM win rates","Language-aware inference scaling lifts multilingual LLM wins","Multilingual LLMs: sampling strategy beats single decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":4041,"prompt_tokens":1028,"completion_tokens":3013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2930}},"tokens_in":644,"tokens_out":3013,"duration_ms":24448,"temperature":1.0,"reasoning_tokens":2930,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:46:07.565317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same hedged sampling with CHOPS and X-MBR on m-ArenaHard in the seven test languages, then have human annotators rate the selected outputs and compare win rates per language against GPT-4o's ratings. If the methods' advantage shrinks or reverses for some languages when ground truth is human preference rather than the automated judge, the central claim would be shown to be judge-dependent.","supporting_citations":[{"cited_title":"The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism","cited_arxiv_id":null,"evidence_quote":"Supplies the greedy-decoding baseline recommendation and the task-dependent view of deterministic vs stochastic sampling that the paper benchmarks against."},{"cited_title":"\"\" Listing 1: Prompt Used for Checklisted One Pass Selection (CHOPS) 30","cited_arxiv_id":null,"evidence_quote":"Establishes the limits of resampling with imperfect verifiers, framing the paper's reliance on judge alignment."}],"review_version":1}