{"id":"aa40ac60-10a9-4200-b4e8-05ef0ef2b663","arxiv_id":"1908.06288","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Selecting a representative output via range voting over beam search candidates improves BLEU, length, and diversity in image captioning and machine translation.","lead":"This paper proposes a decoding method that picks the most representative sentence from a beam, using range voting and similarity scores, instead of the single most probable sentence. It reports better BLEU scores and human ratings in image captioning and machine translation, especially with large beams.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-eval support for the similarity proxy is fragile: in the captioning study the voted output is 'much better' 27 times but 'much worse' 40 times, opposite the headline direction; the paper's sign test collapses this reversal.","rationale":"The paper has genuine merits: it provides a clean generalization of MBR decoding, evaluates several training-free decoding objectives on two tasks, and is unusually transparent in reporting the weak 'much better' counts. My concern is not that the method is useless; it is that the strongest qualitative evidence for representativeness—the human ratings—is fragile in exactly the category that matters. The reader's weakest assumption concerned whether the chosen similarity measures correctly capture representativeness; the human-evaluation reversal is direct evidence that this proxy can fail in the strong-preference tail. A secondary concern is that the best similarity metric appears to be selected on the same evaluation set, which may make the BLEU gains optimistically chosen; however, the human-eval issue is more tightly connected to the paper's central qualitative claim. A conditional verdict remains appropriate: the empirical claims should be re-analysed with ordinal tests and, ideally, with a held-out metric-selection split before the 'rated higher' statement is treated as established.","tokens_in":14720,"tokens_out":9272,"duration_ms":94085,"concrete_test":"Re-analyse the stored pairwise ratings for both human evaluations using an ordinal paired test (e.g., Wilcoxon signed-rank test, or a pre-specified weighted score with 'much better' = +2, 'slightly better' = +1, tie = 0, etc.) and report the 'much better' vs 'much worse' counts as primary evidence. If the ordinal effect is not positive and significant in both tasks, or if the captioning much-better/much-worse reversal persists, the human-evaluation claim in the abstract and conclusion should be qualified or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own human evaluations are the direct test of whether similarity-based range voting captures 'informative' or 'representative' quality, and that test is weaker than the conclusion states. In Section 4.1.4 (image captioning, 500 images, beam search k=10 vs precision2 k=100), the voted caption is rated 'better' 106 times vs 'worse' 73 times, p=0.0165 by a two-tailed sign test with ties discarded. But for the strongest category, the voted caption is 'much better' 27 times and 'much worse' 40 times (p=0.142), so the raw strong-preference direction is reversed. The translation study in Section 4.2.4 shows the same pattern in miniature: overall 69 vs 44 (p=0.0235) but 'much better' 36 vs 28 (not significant). Because Eq. (2) and (3) deliberately reward candidates that contain all of a voter's n-grams plus extra material, the method can add content not well supported by the model distribution; the human data suggest these failures concentrate in the 'much worse' tail. Collapsing a five-point scale into 'better/worse' hides this, so the headline 'rated higher' is less robust than the paper's conclusion implies.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a decoding strategy based on range voting: given a set of candidate sequences and voter sequences from beam search, it scores each candidate as the probability-weighted average similarity to all voters. Several similarity measures are tested (n-gram overlap, bag precision, averaged hidden-state cosine). Experiments on MSCOCO image captioning and WMT14 En-De translation report higher BLEU, longer and more diverse outputs, and a human evaluation that the authors claim shows the method is preferred. The method is positioned as an alternative to MBR decoding.","tokens_in":15019,"tokens_out":9632,"duration_ms":89200,"significance":"The proposed method is simple, model-agnostic, and addresses the known issue of beam-search degradation at large beam sizes. The empirical gains in BLEU at large k (e.g., 28.23 vs 26.31 BLEU-4 on captioning at k=100; 24.62 vs 22.59 on translation at k=100) are substantial. The paper also provides a useful discussion of vote splitting and its connection to MBR. However, the human-evaluation evidence is weaker than claimed, and the theoretical claim about not needing a model-distribution approximation is inaccurate. The contribution is a solid empirical study of a simple re-ranking objective, but the paper needs revisions to validate the headline claims.","major_comments":[{"comment":"The human evaluations compare different beam sizes between conditions: in captioning, standard beam search at k=10 is compared with precision2 at k=100, and in translation, length normalisation at k=4 is compared with precision2 at k=10. This confounds the effect of range voting with the effect of beam size and output length. Additionally, the 'much better' counts in captioning show the voted caption is judged much worse 40 times versus much better 27 times (p=0.142), so the strong-preference direction is reversed. The conclusion that the method is 'rated higher' is therefore overstated. Please run matched-beam-size comparisons and report the strong-preference statistics in the conclusion.","section":"§4.1.4, §4.2.4, Conclusion"},{"comment":"The claim that range voting 'does not require the assumption that we can approximate the true distribution by the model's distribution' is incorrect. Eq. (1) weights each voter by P(v), which is exactly the model distribution. Score(c) = sum_v P(v) sim(v,c) is the expected utility under the model posterior, i.e., MBR with a different utility function. The difference from classic MBR is in the choice and asymmetry of sim, not in the absence of the approximation.","section":"§3.3"},{"comment":"The best-performing similarity metric is selected on the test set (lstm states for captioning, overlap2 for translation, precision2 for the human evaluations). For example, in §4.2.5 'we fix the similarity metric to overlap2, which was the best performing metric for large k≥4 (§4.2.1)'. This test-set peeking can inflate the reported gains. Please report metric selection on a validation set or provide a multiple-comparison analysis, and acknowledge this limitation.","section":"§4.1.1, §4.2.1, §4.2.5"},{"comment":"The BLEU tables report point estimates without confidence intervals or significance tests, except for a single paired bootstrap test in §4.1.1. Several conclusions rely on small differences (e.g., captioning at k=10). Please provide bootstrap confidence intervals or p-values for the key comparisons to establish that the improvements are not within noise.","section":"Tables 1 and 4"}],"minor_comments":[{"comment":"The measure precisionn(v,c) divides by |bag_n(v)|, so it is a recall score relative to v, not a precision score. This naming is confusing, particularly in the comparison with BLEU in §3.3. Consider renaming it to 'recalln' or clarifying the definition.","section":"Eq. (2)"},{"comment":"The title contains 'Range V oting' with a space, and §2.2 contains 'doesnt' (missing apostrophe). Please proofread.","section":"Title and §2.2"},{"comment":"The description of the rating scale reads 'one caption much better, slightly better, or no difference'; it should clarify that each caption can be rated on the five-point scale (e.g., 'one caption much better, one slightly better, or no difference').","section":"§4.1.4"},{"comment":"The copy-filtering heuristic is applied only to the range-voting rows and to the 'beam search (no copy)' baseline, while the other baselines (length normalisation, diverse decoding, diverse beam search) are not filtered. This asymmetry should be stated explicitly or the baselines should be re-run with filtering.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but the human evaluation is confounded and the MBR independence claim is inaccurate. I believe these can be fixed within the scope of a revision. The core decoding method and the BLEU experiments are valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the MBR idea re-read through range voting, with the useful twist that similarity is asymmetric so informative candidates get more votes. If you work on NLG decoding, it's worth a read. The paper does not oversell much; the authors explicitly acknowledge the equivalence to MBR when sim=BLEU, and the voting framing gives a clean motivation for using other similarity measures. The experiments cover two standard tasks, and the length/diversity analysis is more systematic than most decoding papers.\n\nWhat's actually new: the specific asymmetric n-gram measures (precision_n, overlap_n) and the demonstration that they beat BLEU-based MBR on BLEU while producing longer, more diverse outputs. The \"more voters helps\" experiment (Sec 4.2.5) is a nice sanity check that the benefit comes from the voter distribution, not just candidate quality. The derivation is a definition, not circular, and there are no self-citation issues.\n\nSoft spots, in order of importance. First, the best similarity metric is selected on the test set (overlap2 for MT is chosen from Table 4 on newstest2014). That is peeking, mild but real. Second, most BLEU tables lack error bars; only one paired bootstrap is reported. Third, the human eval: 500 items per task, and the 'much better' direction is reversed in captioning (27 vs 40, p=0.142) and not significant in translation (36 vs 28). The overall better/worse sign test is significant, but collapsing a five-point scale hides the fact that the method's failures concentrate in the worst tail. To be fair, the paper discloses this reversal and gives a sensible explanation (when the model is wrong, adding extra content makes things worse), so the stress-test note that the paper \"collapses\" it is itself unfair. Still, the abstract's \"rated higher\" is less robust than the strong-preference data support. Also minor: the captioning human eval compares precision2 k=100 against beam search k=10, so it does not isolate the similarity measure from the beam-size difference.\n\nWho it's for: people working on decoding for MT and captioning, especially MBR variants. It deserves a serious referee. A revision should add error bars, fix the test-set metric selection (e.g., select on a held-out dev split), and report the full ordered human ratings rather than only the collapsed sign test. I'd send it to review.","headline":"A training-free decoding reranker that extends MBR with asymmetric n-gram similarities; the core experiments are solid, but the human-eval claim leans on a sign test that hides a reversed 'much worse' tail.","tokens_in":15487,"tokens_out":2688,"would_cite":true,"duration_ms":26898,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that for language generation the right objective is not the most likely sequence but the most representative one — the winner of a range-voting election in which every beam hypothesis votes for every other, weighted by…","keywords":["range voting","beam search","natural language generation","representativeness","similarity measure","minimum Bayes risk decoding","image captioning","machine translation"],"falsifier":"On the two tasks, decode with range voting using a similarity measure that has nothing to do with content—for example, how many function words two sentences share, or random scores drawn once per pair. If BLEU and human preference still match or beat standard beam search as beam size grows, the reported gains are produced by probability-weighted voting itself, not by similarity-based representativeness. A second check: sample many sequences from the model and test whether the range-voting winner is closer to the probability-weighted average of hidden states than the beam-search argmax; if it is not, the \"representative\" label does not hold.","tokens_in":14545,"feed_emoji":"🗳️","tokens_out":8409,"duration_ms":75132,"temperature":0.7,"pith_summary":"Beam search for language generation traditionally returns the single most probable sequence, but this paper argues that the mode of a discrete distribution over sentences is often the wrong answer: probability mass split across many similar long sentences can make a short, boring sentence the winner. The proposed fix is to make the beam hold an election. Each candidate sentence receives votes from every other sentence in the beam, weighted by the voter's model probability and by a similarity score between voter and candidate, and the sentence with the highest total is the \"most representative\" output. On image captioning and English-German translation, this range-voting re-ranking beats plain beam search, length normalisation, and diverse decoding on BLEU, produces longer and more diverse outputs, and wins head-to-head human evaluations. The method is model-agnostic and needs no retraining.","feed_headline":"Range voting beats max-likelihood beam search for text generation","feed_subtitle":"Beam-search hypotheses vote for one another, yielding longer, more diverse, higher-rated captions and translations.","key_machinery":"The central object is a range-voting election over a beam. Each sequence $v$ in the voter set $V$ casts a vote of strength $\\mathrm{sim}(v,c)$ for each candidate sequence $c$, weighted by its model probability $P(v)$, and the winner maximizes $\\mathrm{score}(c)=\\sum_{v\\in V} P(v)\\,\\mathrm{sim}(v,c)$. Range voting is independent of clones, so splitting probability mass across near-identical sequences does not dilute their collective strength. The paper pairs this with asymmetric n-gram similarities (precision and overlap) that let longer, more informative candidates collect votes from shorter voters, plus a cosine similarity between averaged decoder hidden states.","core_discovery":"The central discovery is that representativeness, not likelihood, should drive decoding when a language model puts comparable probability on a cluster of similar outputs. The paper formalizes representativeness as the winner of a range-voting election over beam hypotheses: $\\mathrm{score}(c)=\\sum_{v\\in V} P(v)\\,\\mathrm{sim}(v,c)$. This recovers Minimum Bayes Risk decoding when the similarity measure is BLEU, but with an independent motivation and with freedom to choose asymmetric similarity measures that reward informative candidates. Empirically, the method removes the usual BLEU drop as beam size grows in image captioning, shrinks it in translation, and improves human preference ratings on both tasks.","pith_inferences":["If the similarity measure is the true engine of the gains, replacing n-gram overlap with a calibrated semantic similarity should widen the gap over beam search; if gains persist under a semantically meaningless similarity, the mechanism is probability-weighted averaging rather than representativeness.","The candidate/voter separation suggests an inference-time recipe the paper does not fully exploit: keep candidates small and high-precision while drawing a large, diverse voter set by sampling, a direction Figure 3 indicates should keep improving BLEU.","The human-evaluation tails show representative decoding can also amplify model errors, so a natural extension is to gate voting with a faithfulness or quality filter to avoid rewarding confident wrong information."],"forward_implications":["On image captioning, BLEU no longer falls as beam size grows from 10 to 100, so the method removes one source of sensitivity to the beam-size hyperparameter.","Because the asymmetric n-gram similarities reward candidates that contain the voter plus more information, outputs become longer and lexically more diverse, moving caption and translation length closer to that of references.","Range voting is a superset of MBR decoding: swapping in any similarity measure, not just the evaluation metric, lets practitioners bias decoding toward informativeness or faithfulness without retraining the model.","Decoupling voters from candidates pays off: with candidates fixed at beam size 4, increasing the voter set to 1000 via sampling, stochastic beam search, or filtered beam search raises BLEU, implying search quality and selection quality can be improved separately."],"supporting_citations":[{"why":"This work introduces Minimum Bayes Risk decoding, the framework range voting generalizes and against which the paper positions its own similarity choices.","marker":"Kumar and Byrne (2004)"},{"why":"This work defines independence of clones, the voting-theoretic property used to motivate range voting against vote splitting.","marker":"Tideman (1987)"},{"why":"This work documents that beam search covers little probability mass and identifies the copy problem; its filtering heuristic is an MT baseline the method builds on.","marker":"Ott et al. (2018)"},{"why":"This work documents degrading translation performance for larger beam sizes, the failure mode the method targets.","marker":"Koehn and Knowles (2017)"},{"why":"This work shows the most likely output can be empty or bad, motivating generation of a representative output instead.","marker":"Stahlberg and Byrne (2019)"},{"why":"This work names the 'boring output problem' of high-probability uninformative responses that motivates the paper.","marker":"Cao and Clark (2017)"},{"why":"This work provides the Show and Tell image captioning architecture used in the captioning experiments.","marker":"Vinyals et al. (2015)"},{"why":"This work provides the Transformer architecture trained for the WMT English-German translation experiments.","marker":"Vaswani et al. (2017)"},{"why":"This work defines BLEU, the automatic metric used for evaluation and for the MBR/range-voting comparison.","marker":"Papineni et al. (2002)"},{"why":"This work inspires the hidden-state cosine similarity measure tested in the experiments.","marker":"Mueller and Thyagarajan (2016)"}],"fun_headline_variants":["Range voting outshines likelihood in beam search decoding","Voting beam hypotheses: more diverse, human-approved text","Representativeness wins: range voting for text generation","Beam search with a vote: longer, richer captions and translations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the similarity measure actually captures which sentences are alike in the way that matters; if the measure is a poor proxy for representativeness, the \"representative\" winner is only representative under that flawed metric.","fun_headline_variants_meta":{"raw":{"variants":["Range voting outshines likelihood in beam search decoding","Voting beam hypotheses: more diverse, human-approved text","Representativeness wins: range voting for text generation","Beam search with a vote: longer, richer captions and translations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1850,"prompt_tokens":816,"completion_tokens":1034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":966}},"tokens_in":432,"tokens_out":1034,"duration_ms":10672,"temperature":1.0,"reasoning_tokens":966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:49:29.128931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the two tasks, decode with range voting using a similarity measure that has nothing to do with content—for example, how many function words two sentences share, or random scores drawn once per pair. If BLEU and human preference still match or beat standard beam search as beam size grows, the reported gains are produced by probability-weighted voting itself, not by similarity-based representativeness. A second check: sample many sequences from the model and test whether the range-voting winner is closer to the probability-weighted average of hidden states than the beam-search argmax; if it is not, the \"representative\" label does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work introduces Minimum Bayes Risk decoding, the framework range voting generalizes and against which the paper positions its own similarity choices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work defines independence of clones, the voting-theoretic property used to motivate range voting against vote splitting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work documents that beam search covers little probability mass and identifies the copy problem; its filtering heuristic is an MT baseline the method builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work documents degrading translation performance for larger beam sizes, the failure mode the method targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work shows the most likely output can be empty or bad, motivating generation of a representative output instead."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work names the 'boring output problem' of high-probability uninformative responses that motivates the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work provides the Show and Tell image captioning architecture used in the captioning experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work provides the Transformer architecture trained for the WMT English-German translation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work defines BLEU, the automatic metric used for evaluation and for the MBR/range-voting comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work inspires the hidden-state cosine similarity measure tested in the experiments."}],"review_version":1}