{"id":"7eb4500f-ff4b-4a94-9622-4b2933dbd76b","arxiv_id":"2502.06703","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Small LLMs with compute-optimal test-time scaling can outperform much larger models on math benchmarks, but the reported strategy is selected on the same test sets used for evaluation.","lead":"This paper claims that small language models, given extra inference-time compute and a process reward model, can match or beat much larger models like GPT-4o and o1 on math benchmarks. The claim is significant only if the strategy generalizes, but the paper's 'compute-optimal' strategy appears to pick the best configuration after seeing the test scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 'compute-optimal TTS' results in Table 3 are selected as the best of a large method/PRM/budget grid on the same MATH-500/AIME24 test sets, and no validation split or selection rule is given; the small-vs-large crossings are therefore not established as a deployable strategy.","rationale":"The reader's REJECT verdict is warranted. The central claim depends on the compute-optimal TTS strategy being a method that can be chosen for unseen problems; the paper instead selects the best configuration on the very benchmarks used for evaluation. Equations (2)-(3) formalize an oracle argmax, but the experimental protocol never defines a validation split or a selection rule, and Section 3.2 uses Pass@1 accuracy on the same test problems to define difficulty. Many reported margins are small and within plausible selection noise, especially on AIME24 with only 30 problems. A held-out validation test would settle whether the crossings survive. The paper's descriptive findings about PRM generalization, length bias, and voting sensitivity are useful and may be independently valuable, but they do not rescue the headline claim. The concern is about experimental design and generalization, not about the honesty of the authors; the evidence as reported simply does not support a deployable compute-optimal strategy. Verdict remains REJECT.","tokens_in":29458,"tokens_out":6562,"duration_ms":60367,"concrete_test":"Hold out a random half of MATH-500 (250 problems) as validation and the other half as test; on the validation half, for each policy model and difficulty bin select the configuration (method, PRM, budget, scoring/voting) with highest accuracy, then apply that fixed configuration to the held-out half. Repeat over 10 random splits; for AIME24, use all 30 problems for selection and report bootstrap CIs on the resulting 30-point differences. If the fixed-configuration gaps (1B vs 405B, 7B vs o1, 0.5B vs GPT-4o) do not persist consistently, Table 3's crossings are selection artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 sweeps multiple policy models, seven PRMs, three TTS methods, budgets {4,16,64,256} (plus N=512 for Llama-3.2-1B), and several scoring/voting schemes, then Table 3 reports the best configuration per small model as 'compute-optimal TTS.' Equations (2)-(3) define the strategy as an argmax involving the ground-truth response y*(x), but the paper never specifies a validation split or a rule for selecting the configuration on unseen problems. Difficulty bins in Section 3.2 are defined by Pass@1 accuracy on the same test problems, so the 'optimal' allocation itself uses oracle labels. On AIME24 (30 problems), crossings like Qwen2.5-0.5B vs GPT-4o (10.0 vs 9.3) and Llama-3.2-3B vs 405B (30.0 vs 23.3) are consistent with selection over dozens of configurations; expected max over noise can produce such margins even when no real advantage exists. Thus the central claim—that a small model with a real compute-optimal TTS strategy beats frontier models—is not supported by the reported design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large empirical study of external test-time scaling (TTS) methods—Best-of-N, beam search, and Diverse Verifier Tree Search—across a wide grid of policy models (0.5B to 72B), seven process reward models, several scoring/voting schemes, and compute budgets. It proposes a \"reward-aware compute-optimal TTS\" framework, analyzes how the best TTS method depends on policy model, verifier, and problem difficulty, and reports that compute-optimal TTS lets small models outperform much larger models: e.g., Llama-3.2-1B over Llama-3.1-405B on MATH-500, Qwen2.5-0.5B over GPT-4o, and DeepSeek-R1-Distill-7B over o1 and DeepSeek-R1 on both benchmarks. The paper also claims large FLOPS savings.","tokens_in":29737,"tokens_out":5714,"duration_ms":51772,"significance":"If the headline claim were supported—that a deployable, compute-optimal TTS strategy allows small models to beat frontier models on these benchmarks—the result would be practically important for efficient inference. The descriptive component of the paper is also valuable: the sweep over policy models, PRMs, and budgets is broad, and the qualitative taxonomy of PRM failure modes in Appendix C (over-criticism, error neglect, localization bias, scoring bias) is a useful contribution. However, the central comparison is not currently established because the strategy is selected on the same test benchmarks used to report the gains, and the reported crossings are often within one or two problems on a 30-problem set. The paper needs an out-of-sample selection protocol and uncertainty quantification before the small-vs-large claims can be taken as evidence.","major_comments":[{"comment":"The \"compute-optimal TTS strategy\" is selected on the test benchmarks themselves. Equations (2)-(3) define the strategy as an argmax involving the ground-truth answer y*(x), and Section 4 sweeps multiple policy models, seven PRMs, three TTS methods, budgets {4,16,64,256} (plus N=512 in one case), and multiple scoring/voting schemes; Table 3 then reports the best configuration per small model as the \"compute-optimal\" result. No validation split or selection rule is specified that would allow the strategy to be chosen without access to test labels. The reported MATH-500 and AIME24 numbers are therefore selected maxima over a large configuration grid, not out-of-sample predictions. The headline claims that a 1B model exceeds a 405B model, that a 0.5B model beats GPT-4o, and that a 7B model beats o1 require a validation-based selection protocol; as written, they are not supported by the experimental design.","section":"§4 and §5.1, especially Table 3"},{"comment":"The difficulty bins used to allocate compute are defined by Pass@1 accuracy on the same test problems that are later used to evaluate the strategy. The paper states that problems are labeled easy/medium/hard using absolute Pass@1 thresholds, but Pass@1 is estimated on MATH-500 and AIME24 with oracle labels. This means the \"optimal\" allocation per problem uses information that a deployable strategy would not have at inference time. The authors need either a validation-based difficulty classifier or a measurable proxy for difficulty, and the selection of thresholds must be decoupled from the test set.","section":"§3.2 and §4.3"},{"comment":"Several claimed crossings are within sampling noise, and no error bars or significance tests are reported. AIME24 has 30 problems: the Qwen2.5-0.5B vs GPT-4o comparison (10.0 vs 9.3) is a gap of roughly one question, and DeepSeek-R1-Distill-7B vs o1 (83.3 vs 79.2) is a gap of roughly one or two questions; on MATH-500, the 7B-vs-o1 gap (95.2 vs 94.8) is two questions. Since Table 3 reports the best of many configurations, the expected maximum over noise can produce these margins even without a real advantage. The authors should report per-configuration results, bootstrap confidence intervals, and either a multiple-testing adjustment or a pre-registered selection rule.","section":"Table 3 and AIME24 sample size"},{"comment":"The FLOPS efficiency claim does not appear to account for the compute of the process reward model. For example, the reported inference FLOPS for Llama-3.2-3B with compute-optimal TTS is 3.07×10^17, but if the strategy uses a 72B PRM to score many steps (as suggested by the sweep in Section 4), the verifier compute should be included in a fair comparison with a large-model CoT baseline. The claim of higher inference efficiency needs to specify exactly which components (policy, verifier, sampling, scoring) are included in the FLOPS estimate.","section":"§5.1, Table 4"}],"minor_comments":[{"comment":"There is a typo: \"pervious experiments\" should be \"previous experiments.\"","section":"§4.4"},{"comment":"Using Qwen2.5-32B-Instruct to extract answers for Llama-3.2-1B because the model does not emit \\boxed{...} introduces a non-identical evaluation pipeline for that row; the potential effect on the reported 66.2 and 72.2 numbers should be discussed or the extraction method should be applied uniformly.","section":"§5.1, footnote 5"},{"comment":"The statement \"we improve the result by 487.0% (23×→ 135×)\" is unclear: the comparison base and the definition of the percentage improvement over prior work should be stated precisely.","section":"§5.1, text near Table 3"},{"comment":"Several figures (e.g., Figures 10-11) are dense and difficult to read at print resolution; providing numeric tables in the appendix or a separate supplementary file would improve verifiability of the per-configuration results that underlie Table 3.","section":"Tables and figures"}],"recommendation":"major_revision","confidential_remarks":"The gating issue is the lack of a validation split: the headline small-vs-large claim is currently an in-sample selected maximum. If the authors cannot supply out-of-sample results, the paper should not be accepted; a major revision with a proper selection protocol and confidence intervals is the appropriate path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. This paper is a broad empirical sweep of test-time scaling (TTS) methods across policy models, PRMs, and difficulty levels. The headline——1B beats 405B, 7B beats o1——is a post hoc best-of-grid result on the test benchmarks, not a validated deployable strategy. But the sweep itself is useful, and the descriptive findings are worth having.\n\nWhat's actually new: the scale of the comparison. They run seven PRMs, multiple policy models from 0.5B to 72B, and three TTS methods over compute budgets {4,16,64,256} (plus 512 for one model) on MATH-500 and AIME24. That lets them document that the optimal TTS method depends on the policy model and PRM, that PRMs generalize poorly across policy families, and that PRMs have measurable biases (length, step scoring, voting sensitivity). The 'reward-aware' framing in Eq. (3) is a modest but sensible generalization: the verifier should be treated as part of the strategy, not a fixed oracle. These are legitimate contributions for people working on inference-time computation.\n\nThe soft spots are in the central claim. Eq. (3) defines the compute-optimal strategy as an argmax that uses the ground-truth answer y*(x), and Section 5 implements it by picking the best configuration on the same MATH-500 and AIME24 test sets. The difficulty bins in Section 3.2 are also derived from Pass@1 accuracy on those same problems. So Table 3 reports the maximum of a large grid, not the performance of a strategy selected on a validation set. On AIME24 (30 problems), the margins are tiny——Qwen2.5-0.5B vs GPT-4o is 10.0 vs 9.3, Llama-3.2-3B vs 405B is 30.0 vs 23.3——and selection over noise can easily produce such gaps. No error bars or repeated-run variance are given. If the strategy has to generalize to new problems, the crossing claims are not supported as stated.\n\nThe descriptive analysis doesn't depend on the headline. The patterns about PRM generalization, difficulty-dependent method choice, and PRM biases stand on their own. The paper deserves serious peer review, but the authors should be required to either add a validation split and show the selected strategy generalizes, or reframe the result as an oracle-selected upper bound. A careful referee should not let the 1B>405B crossing stand without that.","headline":"Useful empirical sweep of TTS methods, but the headline small-beats-large crossings are post hoc best-of-grid results on the test set, not a deployable strategy.","tokens_in":30305,"tokens_out":3462,"would_cite":false,"duration_ms":29307,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By allocating test-time computation optimally, a 1B LLM can beat a 405B LLM on MATH-500, and a 7B LLM can beat o1 and DeepSeek-R1 on both MATH-500 and AIME24.","keywords":["test-time scaling","compute-optimal inference","process reward models","mathematical reasoning","small language models","inference efficiency","best-of-n sampling","beam search"],"falsifier":"Take a held-out set of competition math problems, fix a small policy (e.g., Llama-3.2-3B-Instruct) and a compute budget, and choose the strategy (method, PRM, budget) using only a separate validation split. If the best-on-validation configuration fails to beat the paired large model (e.g., Llama-3.1-405B-Instruct or GPT-4o with chain-of-thought) on the held-out test set, then the claimed size-crossing results do not generalize beyond the benchmark they were selected on.","tokens_in":29262,"feed_emoji":"🧮","tokens_out":9027,"duration_ms":65609,"temperature":0.7,"pith_summary":"Test-time scaling (TTS) spends extra computation at inference to improve a language model's answers, and this paper asks how to spend it optimally and how far that can push small models. The authors argue that the best strategy—which search method, which process reward model, and how much compute to give each problem—depends strongly on the policy model and the problem's difficulty. Under their 'compute-optimal' recipe, they report that a 1B model exceeds a 405B model on MATH-500, a 0.5B model beats GPT-4o, and a 7B model outperforms o1 and DeepSeek-R1, while using far less inference compute. The practical stakes: if correct, small models plus smart search and process-reward scoring could replace frontier models on math reasoning at a small fraction of the cost.","feed_headline":"A 1B model can beat a 405B model on math","feed_subtitle":"Tuned test-time compute and process-reward search let tiny LLMs top frontier ones on MATH-500 and AIME24.","key_machinery":"The central object is the compute-optimal TTS strategy, a rule that selects the scaling method, compute budget, and PRM for a given policy model and problem. The paper's version is reward-aware (Eq. 3), meaning the choice depends on the reward function, and it is evaluated empirically across three TTS methods: Best-of-N (sample N answers and score them), beam search (keep the top-N/M steps per depth according to the PRM), and Diverse Verifier Tree Search (run independent beam searches in subtrees). The other load-bearing piece is the difficulty criterion: instead of Pass@1 quantiles, the authors bin problems by absolute Pass@1 accuracy into easy (50–100%), medium (10–50%), and hard (0–10%) levels, arguing that quantiles are misleading when different policy models have very different baseline abilities.","core_discovery":"The paper's central claim is that compute-optimal test-time scaling is reward-aware: the best way to scale inference compute cannot be chosen from the policy model and budget alone, but must also account for the process reward model (PRM) that scores reasoning steps. Concretely, the authors define the reward-aware compute-optimal strategy in Eq. (3) as the argmax over strategies of the expected probability of the correct answer given the policy, the compute budget, the problem, and the reward function. Running that selection across Llama and Qwen policy models from 0.5B to 72B, seven PRMs, and three scaling methods (Best-of-N, beam search, and Diverse Verifier Tree Search) on MATH-500 and AIME24, they find that small models with the best configuration outperform much larger models: Llama-3.2-3B-Instruct beats Llama-3.1-405B-Instruct on both benchmarks, Llama-3.2-1B-Instruct beats the 405B model on MATH-500 at budget N=512, Qwen2.5-0.5B-Instruct beats GPT-4o, and DeepSeek-R1-Distill-Qwen-7B beats o1 and DeepSeek-R1. They also report that the optimal method shifts with model size—search-based methods win for small policies while Best-of-N wins for larger ones—and with difficulty, with BoN better on easy problems and beam search better on hard ones.","pith_inferences":["A direct extension the paper leaves implicit is that a practical deployment needs a selection rule that chooses the strategy before seeing the test answers; the reported crossings are the best configuration on the test benchmarks, so a validation-based selection rule is the natural next experiment.","The length-bias findings suggest that comparing TTS methods by raw accuracy without fixing token budgets can conflate search quality with PRM preferences; a token-normalized comparison would sharpen the efficiency claims.","The weak-to-strong result (a 7B PRM supervising a 72B policy) points toward building automated supervisors that can pick methods and budgets per problem, possibly trained on a development split, rather than assuming one global strategy.","The same recipe may transfer to other multi-step tasks like coding, but only if step-level reward models there are strong enough; the paper's own AIME24 results show TTS gains are smaller on harder problems, so the transfer is not automatic."],"forward_implications":["If the compute-optimal strategy is followed, small policy models (0.5B–7B) can beat frontier models on MATH-500 and AIME24 while consuming less inference FLOPS, with total FLOPS reductions of 100x–1000x.","The optimal TTS method is not universal: search-based methods (beam search/DVTS) help small models most, while Best-of-N is best for large models, so practitioners must match the method to the policy.","PRM choice matters as much as the search method: PRMs trained on different policies generalize poorly, and PRM quality (measured by process supervision ability) is positively correlated with TTS performance.","TTS with external search outperforms several long-CoT training recipes (rStar-Math, Eurus-2, SimpleRL, Satori) on both benchmarks, but is less effective than distilling from a strong reasoning model like DeepSeek-R1.","TTS gains shrink as policy models get stronger; for small models compute-optimal TTS improves accuracy over CoT by up to 154.6% and can be 256x more efficient than majority voting."],"supporting_citations":[{"why":"Defines the original compute-optimal test-time scaling framework and the Pass@1 quantile difficulty division that this paper replaces with absolute thresholds and reward-awareness.","marker":"Snell et al. (2024)"},{"why":"Introduces Diverse Verifier Tree Search, the search method the paper finds best for small policies on hard problems, and the prompt format used for Llama models.","marker":"Beeching et al. (2024)"},{"why":"Provides MATH-500 and the step-level verification data and methodology behind process reward models.","marker":"Lightman et al. (2024)"},{"why":"Supplies the AIME24 problems used as the harder benchmark.","marker":"AI-MO (2024)"},{"why":"Releases the Qwen2.5-Math PRMs and reports the process-supervision ability scores that the paper correlates with TTS performance.","marker":"Zhang et al. (2025)"},{"why":"Releases the RLHFlow PRMs and the response-length statistics of their training data, which motivate the length-bias analysis.","marker":"Xiong et al. (2024)"},{"why":"Defines the scoring and voting methods (PRM-Min, PRM-Last, PRM-Avg, and the three voting rules) used to aggregate search results.","marker":"Wang et al. (2024a)"},{"why":"Provides DeepSeek-R1 and its distilled 1.5B and 7B models, the strongest baselines and the small policies used for the headline crossings.","marker":"DeepSeek-AI et al. (2025)"},{"why":"Provides the MATH dataset whose difficulty labels the paper tests and rejects in favor of absolute Pass@1 thresholds.","marker":"Hendrycks et al. (2021)"}],"fun_headline_variants":["Reward-aware test-time scaling lets 1B beat 405B","Tiny LLMs outsmart giants with compute-optimal TTS","1B LLM beats 405B on MATH-500 with right scaling","Process reward search lets 0.5B beat GPT-4o","Compute-optimal TTS: small models surpass giants"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best configuration on the test benchmarks can be chosen without using the test answers; the paper reports the best method, PRM, and budget found on MATH-500 and AIME24 themselves, and does not give a selection rule that would work on unseen problems.","fun_headline_variants_meta":{"raw":{"variants":["Reward-aware test-time scaling lets 1B beat 405B","Tiny LLMs outsmart giants with compute-optimal TTS","1B LLM beats 405B on MATH-500 with right scaling","Process reward search lets 0.5B beat GPT-4o","Compute-optimal TTS: small models surpass giants"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1873,"prompt_tokens":1174,"completion_tokens":699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":790,"completion_tokens_details":{"reasoning_tokens":607}},"tokens_in":790,"tokens_out":699,"duration_ms":5631,"temperature":1.0,"reasoning_tokens":607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:35:52.444934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of competition math problems, fix a small policy (e.g., Llama-3.2-3B-Instruct) and a compute budget, and choose the strategy (method, PRM, budget) using only a separate validation split. If the best-on-validation configuration fails to beat the paired large model (e.g., Llama-3.1-405B-Instruct or GPT-4o with chain-of-thought) on the held-out test set, then the claimed size-crossing results do not generalize beyond the benchmark they were selected on.","supporting_citations":[{"cited_title":"Aime 2024, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the AIME24 problems used as the harder benchmark."},{"cited_title":"Measuring mathematical problem solving with the MATH dataset","cited_arxiv_id":null,"evidence_quote":"Provides the MATH dataset whose difficulty labels the paper tests and rejects in favor of absolute Pass@1 thresholds."}],"review_version":1}