{"id":"7bfe4cb4-bca5-47cb-8f71-26db437ddecf","arxiv_id":"2507.17797","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Asking a reasoning LLM to compare and select the best among N math solutions at once beats pointwise scoring and majority voting on competition math benchmarks.","lead":"This paper introduces GenSelect, a method that gives a reasoning language model all N candidate math solutions at once and asks it to pick the best one. The authors test it on competition math problems and find it beats pointwise scoring and majority voting while using fewer comparison rounds than pairwise tournaments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unreported summary-faithfulness assumption is load-bearing: GenSelect and GenRM both judge Qwen2.5-32B-Instruct summaries of QwQ traces, and if those summaries distort mathematical content, the headline comparison may not transfer to the actual candidate solutions.","rationale":"The reader's weakest-assumption selection matches my own: the paper's entire experimental comparison is built on solution summaries generated by Qwen2.5-32B-Instruct, yet the faithfulness of these summaries is supported only by an unreported 'initial experiments' claim (Section 3). If summaries are unfaithful, both GenRM and GenSelect are evaluated on modified content. The internal comparison between methods remains coherent because both use the same summaries, but the central contribution—that an LLM can directly select the best among N candidate solutions—is stated about the original candidates, not about summaries of them. The Appendix A.1 prompt is particularly concerning: it presumes the input solution is eventually correct and asks for a 'clean version of the final correct solution,' which could cause the summarizer to fabricate or sanitize steps for incorrect traces. This is not a question of consensus; it is a correctness risk that is directly testable. The other issues the reader notes (missing error bars, unreleased benchmark data, no pairwise baseline, LLM-judged HMMT answers) are real but secondary: they weaken confidence in the magnitude of the gains, whereas summary faithfulness threatens the validity of the object being selected. The reader's CONDITIONAL verdict is appropriate: the concern is addressable by reporting the promised initial experiments and by running a full-trace ablation. I find no reason to move the verdict to reject or accept without that evidence.","tokens_in":9825,"tokens_out":14388,"duration_ms":160208,"concrete_test":"On a random sample of 50 problems (e.g., the AIME24 and AIME25 splits), run the full GenSelect@8 and GenRM pipelines twice: once with the Qwen2.5-32B-Instruct summaries (current setup) and once with the complete original QwQ reasoning traces (no summarization), keeping all other settings identical. Also manually audit 100 summary-trace pairs for preservation of the boxed final answer and key mathematical steps. If GenSelect's advantage over GenRM (72.1% @1 / 73.4% @8 vs. 69.1%) shrinks by more than 2 points on the sampled problems, or the audit shows more than 5% of summaries changing the final answer or a crucial derivation step, the summary-faithfulness assumption is violated and the headline claim as stated—selecting among the actual N candidate solutions—is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the Qwen2.5-32B-Instruct summaries (Section 3; Appendix A.1) faithfully preserve the mathematical content of the original QwQ reasoning traces. All methods—pointwise GenRM and GenSelect—score these summaries, so the relative 73.4% vs. 69.1% comparison is internally coherent. But the paper's central claim is that GenSelect selects the best among the N candidate solutions, and the only support for faithfulness is an unreported 'initial experiments' claim (Section 3). The Appendix A.1 prompt instructs the summarizer to write a 'clean version of the final correct solution' while presupposing that 'eventually that solution gets to the right approach and solves the problem.' For incorrect or partially corrected traces, this could systematically produce clean, plausible-looking summaries that drop or alter key steps, or even the boxed final answer, making the selection task about artifacts rather than the actual candidates. If summaries change what is compared, the headline advantage of GenSelect may not transfer to deployments that use the original reasoning traces. The paper does not report any faithfulness audit, error analysis, or ablation on full traces, despite the claim in Section 3 that complete traces gave 'no significant benefit' over summaries.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenSelect, a best-of-N selection method in which a reasoning LLM is prompted to compare N candidate solutions simultaneously and return the index of the best one; for candidate pools larger than the context window, the method uses an N-ary knockout tournament. The experiments on a self-curated Comp-Math-24-25 benchmark (256 AIME/HMMT problems) show that QwQ-32B and DeepSeek-R1-0528 with GenSelect outperform majority voting and a pointwise GenRM baseline, and that GenSelect is stable across tournament widths and compute-allocation choices.","tokens_in":10080,"tokens_out":5530,"duration_ms":61555,"significance":"If the result holds, GenSelect is practically significant: it offers a training-free, prompt-only method for improving test-time scaling in math reasoning, and its N-ary tournament has lower latency than pairwise knockout tournaments. The paper makes concrete falsifiable claims, reports prompts in the appendix, and includes a useful compute-allocation analysis for two open-weight reasoning models. However, several load-bearing pieces of evidence are missing: summary faithfulness is not audited, no pairwise baseline is included, error bars or significance tests are absent, and the HMMT evaluation relies on an LLM judge from the authors' own prior work. The central claim is therefore defensible but not yet established at the required level.","major_comments":[{"comment":"The solution summaries are load-bearing for every method, but their faithfulness is not established. The summary prompt tells Qwen2.5-32B-Instruct to write \"a clean version of the final correct solution\" and presupposes that \"eventually that solution gets to the right approach and solves the problem.\" For incorrect or abandoned reasoning traces, this instruction may systematically produce clean, plausible-looking summaries that omit the actual errors or even change the boxed final answer. Since both GenRM and GenSelect score these summaries rather than the raw QwQ traces, a systematic distortion would mean the comparison is between artifacts rather than candidate solutions. The paper's statement in Section 3 that \"initial experiments revealed no significant benefit from using complete reasoning traces\" is not supported by any reported data. I ask for a faithfulness audit (e.g., checking whether the summary preserves the original final answer and key steps), an error analysis of summaries, or an ablation running GenSelect and GenRM on full traces.","section":"§3, Appendix A.1"},{"comment":"The empirical comparison omits a pairwise baseline, even though the paper's motivation is framed around the limitations of pairwise GenRMs. Table 3 includes majority voting, a discriminative RM, and a pointwise QwQ GenRM, but no pairwise GenRM knockout tournament such as PairJudge-RM or a pairwise prompt converted to a tournament. Without this baseline, the headline claim that GenSelect \"outperforms existing scoring approaches\" is only established against pointwise scoring, not against the pairwise paradigm that the paper explicitly discusses as the main alternative. A pairwise baseline using the same QwQ model and the same tournament structure is needed to isolate the benefit of N-ary comparison.","section":"§4.3, Table 3"},{"comment":"No error bars, confidence intervals, or significance tests are reported. The headline difference in Table 3 is 73.4% vs. 69.1% on 256 problems; under an independent-proportion approximation, the standard error of each proportion is roughly 2.8 points and the standard error of the difference is roughly 4.0 points, so the observed 4.3-point advantage is on the order of one standard error. Since the methods are evaluated on the same candidate pools, a paired test would be more powerful, but no paired analysis is reported. Please report variance across problem subsets, confidence intervals, and a paired significance test for the main comparisons in Tables 3 and 5.","section":"§4.3, Tables 3 and 5"},{"comment":"The HMMT evaluation relies on the \"LLM-judge setup of (Toshniwal et al., 2025)\", a work that shares authors with this paper. The GenSelect gains are largest on the HMMT-24-25 split (e.g., QwQ-32B goes from 56.1% maj@64 to 70.4% with Self GenSelect@32, compared with 86.7% to 90.0% on AIME24), so the accuracy and neutrality of the judge are load-bearing. No validation of this judge on the HMMT subset is reported. I ask for a validation against human labels on a sample, or a sensitivity analysis using a second, independent judge, or a clear statement of how the judge is scored and why it is not a source of bias.","section":"§4.3, Table 5 footnote"}],"minor_comments":[{"comment":"The term \"Self-GenSelect\" is used without a definition; please state explicitly that the same model generates the candidate solutions and performs the GenSelect comparisons.","section":"§4.3, Table 5"},{"comment":"Table 5 reports top-p 0.95, but Table 3 and the verification sampling in Section 3.1 do not state the top-p used; please report the exact sampling parameters for all experiments.","section":"§4.2, Tables 3 and 5"},{"comment":"The GenRM baseline says the model is sampled multiple times, but it is not specified how multiple verifications are aggregated (e.g., majority vote over Yes/No judgments, average token probability, or another method); please clarify, since this affects the fairness of the comparison with GenSelect@8.","section":"§3.1, Figure 1"},{"comment":"For N=8 with 64 candidates, the tournament uses 8 parallel GenSelect calls in the first round and one call in the second round, so the text \"two rounds of scoring eight solutions each\" is ambiguous; please state the total number of GenSelect calls per problem.","section":"§3.2, Table 5"},{"comment":"The benchmark is self-curated and not released; for reproducibility, please provide the exact problem list or release the benchmark, and state the filtering criteria for excluding proof-based or partial-credit questions.","section":"§4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's HMMT evaluation uses an LLM judge from the authors' own prior work (Toshniwal et al., 2025), and the benchmark itself is self-curated from AIME/HMMT. This is not inherently disqualifying, but it creates a perception of benchmark ownership that the authors should address with a neutral judge or external validation. The absence of a pairwise baseline is the most surprising gap given that the paper's motivation is explicitly about the inefficiency of pairwise methods; editors may want to weigh this when deciding whether the empirical claim is sufficiently supported for the target venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful bit: GenSelect asks a reasoning LLM to compare N candidate solutions at once, and on competition math it beats pointwise GenRM and majority voting. On their 256-problem Comp-Math-24-25, QwQ GenSelect@8 gets 73.4% vs 69.1% for GenRM and 68.4% for majority. The N-ary knockout tournament is a natural extension of PairJudge-RM, and the stability across N (72–73% for N=2..16) is a genuinely useful finding: you can make N large, cut compute, and not lose accuracy. The compute-allocation comparison (spending budget on generation vs verification) is also a nice touch.\n\nWhat is new here is the empirical result, not the concept. The paper is a clear, well-written evaluation of a simple prompt, and I believe the headline comparison is internally coherent. Both GenSelect and the GenRM baseline score the same Qwen2.5-32B-Instruct summaries of raw QwQ traces, so the relative advantage is not an artifact of the representation.\n\nThe soft spots are real but not fatal. The biggest is the summary-faithfulness assumption. The summary prompt tells the summarizer that 'eventually that solution gets to the right approach and solves the problem.' For incorrect traces, that is a biasing instruction; it may produce clean, plausible summaries that drop exactly the errors a verifier should catch. The paper claims complete traces gave 'no significant benefit' but reports no experiment for that claim. If summaries distort mathematical content, the method may not transfer to deployments that score raw traces. Second, the paper motivates pairwise comparison as inefficient but never runs a pairwise GenRM or PairJudge-RM baseline, so 'outperforming existing scoring approaches' is too broad. Third, HMMT answers are graded by the authors' own LLM-judge from Toshniwal et al. (2025), the benchmark is not released, and there are no error bars. None of these is a load-bearing flaw in the central comparison, but together they make the paper weaker than its headline suggests.\n\nWho is this for? Researchers working on test-time scaling and best-of-N for math reasoning. It is a useful empirical datapoint and a practical trick, not a paradigm shift. I would send it to peer review. A revision that adds a pairwise baseline, a summary-faithfulness audit, and error bars (or a released benchmark) would make it a solid conference paper. As is, it is a credible workshop contribution.","headline":"GenSelect is a simple and useful empirical result—N-ary comparison with reasoning models beats pointwise scoring—but the unvalidated solution-summarization step and missing pairwise baseline keep it from being a clean win.","tokens_in":10628,"tokens_out":3178,"would_cite":true,"duration_ms":31482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reasoning LLM can pick the best of N candidate math solutions in one shot, beating pointwise scoring.","keywords":["best-of-N selection","generative reward models","reasoning models","test-time scaling","N-ary comparison","knockout tournament","mathematical reasoning","majority voting"],"falsifier":"On a held-out set of QwQ solutions with full traces, run GenSelect twice per problem: once on full traces and once on summarized versions, and compare the selected answer to the oracle best answer. If the two pipelines disagree on more than a small fraction of problems, or if full-trace GenSelect is meaningfully more accurate than summary GenSelect, the summary-faithfulness premise fails and the baselines were compared on different content.","tokens_in":9599,"feed_emoji":"🧮","tokens_out":8631,"duration_ms":82367,"temperature":0.7,"pith_summary":"The paper proposes GenSelect, a test-time method that asks a reasoning language model to compare all $N$ candidate solutions to a math problem in one pass and return the index of the best one. Its central claim is that this direct N-ary comparison is both more accurate and more computationally efficient than the two existing paradigms: pointwise scoring of each solution and pairwise comparison tournaments. On the 256-problem Comp-Math-24-25 benchmark, QwQ-32B with GenSelect reaches 73.4% accuracy versus 69.1% for a pointwise generative-reward-model baseline and 68.4% for majority voting at the same 64-sample budget, and DeepSeek-R1-0528 with self-GenSelect raises its majority-voting score from 84.0% to 87.1%. If the claim holds, the best-of-N selection problem can be solved with a simple prompt and no trained reward model, and larger sampling budgets can be exploited by widening the comparison rather than running more pairwise battles.","feed_headline":"Comparing N math answers at once beats pointwise scoring","feed_subtitle":"Zero-shot GenSelect lifts QwQ to 73.4% on competition math, above 69.1% GenRM and 68.4% majority voting.","key_machinery":"The load-bearing object is the GenSelect prompt, which converts best-of-N selection into a single N-ary comparison: the model must screen all candidate solutions, identify conflicting answers, eliminate unsound ones, and emit exactly one index as its judgment. Candidate solutions are first compressed from full reasoning traces into clean summaries by Qwen2.5-32B-Instruct, because the authors found in initial experiments that summaries work as well as full traces. For $N$ larger than the context window, the method uses an N-ary knockout tournament adapted from pairwise knockout design: with 64 candidates, a 16-way tournament needs two rounds and five total comparisons, whereas binary comparison needs six rounds and 63 comparisons.","core_discovery":"The authors establish that a long-reasoning LLM, given the problem and $N$ candidate solutions together, can reliably act as an N-ary judge: it groups solutions by final answer, explains contradictions, eliminates clearly wrong ones, evaluates the rest, and ends with a judgment that names the best index. They report that current open reasoning models do this zero-shot, and that the solution selected this way beats pointwise GenRM and majority voting when the same QwQ model both generates and selects. They further show that selection accuracy is stable as $N$ varies from 2 to 16, which justifies using large N-ary knockout tournaments to cut the number of sequential rounds and total comparisons. The paper's headline numbers are QwQ GenSelect@8 at 73.4% on Comp-Math-24-25, versus 69.1% for QwQ GenRM and 68.4% for majority voting, with an oracle Pass@64 ceiling of 85.2%.","pith_inferences":["An unstated consequence is that the comparison width $N$ itself becomes a tunable test-time knob: one could search for the optimal $N$ for a given model and budget, rather than just using the largest $N$ the context window allows.","A testable extension beyond math: the same 'screen, eliminate, justify' prompt could be used for code repair or multi-document question answering, where candidates are alternate outputs and correctness is less cleanly verifiable.","Because summaries are generated by a smaller instruct model, the faithfulness of those summaries is a bottleneck; if a better summarizer or full traces improve GenSelect results, the reported gains would shift accordingly.","The GenSelect formulation invites an RL-trained verifier that distills N-ary comparison into a smaller model, which the authors mention only as future work but which would turn the method from a prompting trick into a scalable component."],"forward_implications":["On competition math, GenSelect with no training beats majority voting, a 72B discriminative outcome reward model, and a pointwise GenRM built from the same QwQ model.","GenSelect accuracy is stable across $N = 2, 4, 8, 16$, so widening the comparison width is a safe way to cut latency and compute in knockout tournaments.","When inference compute is split between generation and selection, GenSelect is the better use of a fixed budget for QwQ at every tested $N$, and for DeepSeek-R1-0528 at $N \\ge 8$.","A model selecting among its own generations (Self-GenSelect) raises QwQ from 68.4% majority voting to 73.0%, and DeepSeek-R1-0528 from 84.0% to 87.1% on the full Comp-Math-24-25 set.","The method applies immediately at test time to any current long-reasoning model without specialized reward-model training."],"supporting_citations":[{"why":"Supplies the majority-voting baseline that GenSelect must beat.","marker":"(Wang et al., 2023)"},{"why":"Introduces outcome-based reward models, the discriminative-verification lineage GenSelect compares against.","marker":"(Cobbe et al., 2021)"},{"why":"Defines pointwise generative reward models and their token-level yes/no scoring, which form the main GenRM baseline.","marker":"(Zhang et al., 2025a)"},{"why":"Introduces pairwise generative reward models, the comparative-scoring approach that GenSelect generalizes to N-ary comparison.","marker":"(Mahan et al., 2024)"},{"why":"Provides the pairwise knockout-tournament structure that GenSelect adapts for N-ary comparisons.","marker":"(Liu et al., 2025a)"},{"why":"Releases Qwen2.5-Math-RM-72B, the discriminative reward model used as a baseline.","marker":"(Yang et al., 2024)"},{"why":"Gives empirical evidence that LLMs verify better by comparing responses than by judging them in isolation.","marker":"(Zhao et al., 2025)"},{"why":"One of the two open reasoning models evaluated for GenSelect.","marker":"(DeepSeek-AI, 2025)"}],"fun_headline_variants":["N-wise selection beats pointwise scoring for math reasoning","Zero-shot GenSelect picks best math answer from N candidates","Comparing all N answers at once outperforms scoring each","Long reasoning LLM judges N solutions in one step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The compressed solution summaries preserve every step that determines correctness, so judging a summary is equivalent to judging the full reasoning trace; the paper says only that initial experiments showed no benefit from full traces, and gives no faithfulness measurement.","fun_headline_variants_meta":{"raw":{"variants":["N-wise selection beats pointwise scoring for math reasoning","Zero-shot GenSelect picks best math answer from N candidates","Comparing all N answers at once outperforms scoring each","Long reasoning LLM judges N solutions in one step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":1969,"prompt_tokens":854,"completion_tokens":1115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1051}},"tokens_in":470,"tokens_out":1115,"duration_ms":9484,"temperature":1.0,"reasoning_tokens":1051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:45:06.213536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of QwQ solutions with full traces, run GenSelect twice per problem: once on full traces and once on summarized versions, and compare the selected answer to the oracle best answer. If the two pipelines disagree on more than a small fraction of problems, or if full-trace GenSelect is meaningfully more accurate than summary GenSelect, the summary-faithfulness premise fails and the baselines were compared on different content.","supporting_citations":[],"review_version":1}