{"id":"3f0c3e30-258c-445f-9dca-70d511820441","arxiv_id":"2505.19634","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Test-time scaling should be tuned for wall-clock latency rather than token count; branches plus speculative decoding win under time budgets.","lead":"This paper shows that when time is the budget, running many LLM attempts in parallel beats letting one attempt think for a long time, and that combining parallel branches with speculative decoding gives the best accuracy under a strict time limit. The authors report a 32B model reaching 82.3% on MATH-500 in under a minute, but the winning settings were selected on the benchmark itself.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline numbers are selected on the test set: Algorithm 1 and Table 2 choose (B, gamma) by accuracy on MATH-500 and then report that same accuracy, so the 82.3% and 72.4% figures overstate deployable performance.","rationale":"The paper's memory-bound analysis is supported by the roofline model in Figure 3 and by direct latency measurements, and the Limitations section honestly scopes the hardware regime. Those parts are credible. However, the empirical central claim is not secured by the current protocol: the optimal configuration is found by maximizing accuracy on the same test set that is later used to report that accuracy. This is a form of test-set selection, and it applies to all headline numbers. The reader's verdict is conditional, and our concern makes the condition more specific: either add a validation-based configuration selection procedure, or explicitly label Algorithm 1 and Tables 1-2 as offline analysis of achievable upper bounds rather than a deployable method. We therefore recommend the same conditional verdict, with the condition tightened around the evaluation protocol.","tokens_in":15261,"tokens_out":5516,"duration_ms":61074,"concrete_test":"Hold out 100 of the 500 MATH-500 problems as a configuration-selection split. Run Algorithm 1 on that 100-problem split under the 1-minute latency budget to select (B, gamma), then evaluate the chosen configuration on the remaining 400 held-out problems. Repeat across five random splits. If the mean held-out gain over the baseline is substantially below the reported 7.3-point gain, or is not statistically significant, the central 82.3% claim is a selection artifact and should be reframed as an offline upper bound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is the evaluation protocol for the headline results. Algorithm 1 selects (B, gamma) by calling tts_task, an accuracy evaluation under a latency budget, on the same benchmark used to report final accuracy. Table 2 validates greedy search against grid search on MATH-500, and Appendix A.2 lists per-dataset optimal configurations (e.g., B=16, gamma=5 for s1.1-32B on MATH-500) that were chosen using ground-truth answers on the test set. Thus the reported 82.3% and 72.4% accuracies are test-set-selected maxima, not the expected accuracy of a method that chooses its concurrency configuration without seeing the test answers. The comparison between greedy and grid search in Section 5.3 only shows that greedy recovers the test-set-optimal configuration; it does not show that the configuration transfers to unseen problems. Because the central claim is that latency-optimal TTS enables a given accuracy within a time budget, this selection bias directly inflates every headline number. The memory-bound limitation is real but the paper scopes it explicitly and provides a roofline model; the test-set selection issue is unaddressed and affects all reported gains regardless of hardware regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that test-time scaling (TTS) should be optimized under a wall-clock latency budget rather than a token-count budget, because LLM decoding is memory-bound at small batch sizes and therefore parallel branches and speculative decoding can increase throughput at little additional latency. The authors propose branch-wise parallelism and sequence-wise parallelism (speculative decoding) as two concurrency mechanisms, formulate latency-optimal TTS as a configuration search over branch count B and draft length γ, and present a greedy search algorithm to find (B, γ). Experiments on MATH-500, AIME24, AIME25, and GPQA-Diamond across several 3B--32B models show that parallel scaling can outperform sequential scaling within a fixed time budget while using more tokens, and that a jointly optimized configuration reaches 82.3% on MATH-500 for s1.1-32B within one minute and 72.4% for s1.1-3B within ten seconds. The paper also reports that greedy search reproduces grid-search configurations with fewer steps and includes ablations on aggregation strategies.","tokens_in":15512,"tokens_out":2429,"duration_ms":26531,"significance":"If the reported accuracy numbers were obtained by a method that selects its concurrency configuration without seeing the test answers, the paper would make a useful and falsifiable systems-level point: on memory-bound, small-batch deployments, latency, not token count, is the appropriate optimization target for test-time scaling. The roofline argument in Section 3.1 and Figure 3 is principled, and the fixed-configuration comparisons in Figures 1--3 and Table 1 are informative. The speculative-decoding-plus-majority-voting combination as a unified concurrency resource-allocation problem is a reasonable framing, and the paper is unusually explicit in its Limitations section that the memory-bound premise does not apply to large-scale server workloads. However, the headline 'latency-optimal' accuracies are not yet supported as deployable, expected-performance claims because the configuration search is evaluated on the same test benchmark used for the final numbers (Algorithm 1 and Table 2, with per-dataset configurations in Appendix A.2). This selection bias directly inflates the headline numbers and must be fixed before the central quantitative claims can be accepted.","major_comments":[{"comment":"The headline results 82.3% on MATH-500 and 72.4% within 10 seconds are chosen as the maximum of the search objective on the same benchmark on which they are reported. Algorithm 1 calls tts_task, an accuracy evaluation on the test set, and Table 2 reports the accuracy of the selected (B, γ) on that same set. Appendix A.2 lists per-dataset optimal configurations (e.g., B=16, γ=5 for s1.1-32B on MATH-500) selected using ground-truth answers. These are test-set-selected maxima, not the expected accuracy of a deployment that must choose its configuration without seeing the answers. This is load-bearing because the abstract and Section 5.1 present these numbers as achieved accuracies within a time budget. The authors should either report accuracy on a held-out set after selecting the configuration on a disjoint validation set, or explicitly re-label the reported numbers as upper bounds / selected-configuration ceilings and provide an unbiased estimate.","section":"§3.3, Algorithm 1; §5.3, Table 2; Appendix A.2"},{"comment":"Equation (1) defines the latency-optimal strategy as the argmax over θ of the expected indicator that the sampled answer equals the ground-truth y*(x). This makes the ground-truth answer part of the optimization objective, which is not a practical procedure. The greedy search in Algorithm 1 is a practical proxy, but its objective is still test-set accuracy. The paper should distinguish between (i) an oracle upper bound that knows ground-truth answers and (ii) a configurable policy that must be selected without them; the current presentation conflates the two, and the reported 82.3% and 72.4% figures belong to the former category as currently measured.","section":"§3.3, Eq. (1)"},{"comment":"The claim that greedy search is 'effective' is supported only by showing that greedy recovers the same (B*, γ*) as grid search on the test set. This does not establish that the selected configuration transfers to unseen problems, nor does it provide an estimate of the accuracy penalty from using a validation-set selection. In addition, Table 2 reports no standard deviations or repeated runs, unlike Table 1, so the equality of greedy and grid accuracies may be within noise. The authors should add a validation/test split experiment and report confidence intervals for the selected configurations.","section":"§5.3, Table 2"}],"minor_comments":[{"comment":"The header 'Bnh-wise' is a typo for 'Branch-wise'; please correct it.","section":"Table 1"},{"comment":"The sentence 'Conversely, for LLMs hat exhibit improvements from branches' contains a typo: 'hat' should be 'that'.","section":"§4.5"},{"comment":"The sentence 'The results of sequence-wise parallelism under different branch counts are shown in Figure 13..' has a double period; please fix the punctuation.","section":"Appendix A.2"},{"comment":"The caption label 'Latenct-Optimal scaling' is a typo for 'Latency-Optimal scaling'.","section":"Figures 9 and 10"},{"comment":"The claim that Qwen2.5-32B-Instruct has 64GB of weights should specify the precision assumption (presumably FP16/BF16); otherwise the number is confusing because 32B parameters at 2 bytes each is 64GB.","section":"Appendix A.1"},{"comment":"The sentence 'latency-optimal TTS can achieve 6% accuracy improvement on average' is not directly derivable from Table 1, which reports improvements of 7.3, 2.5, 6.8, and 7.3 percentage points; please clarify whether '6%' refers to a different aggregation or is a typo.","section":"§5.2"},{"comment":"The scaling curves are plotted without error bars or repeated-run variability, while Table 1 reports standard deviations; adding uncertainty information to the curves would make the comparisons more robust.","section":"Figures 7--10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's qualitative message is sound and worth publishing after the evaluation protocol is fixed. The central concern is not the memory-bound premise — the authors explicitly scope it in the Limitations section — but the fact that the headline accuracy numbers are selected on the same test set used for reporting. I would ask the authors to add a validation/test split or otherwise provide an unbiased estimate of accuracy under a fixed latency budget, and to re-state the headline claims accordingly. If they can do that, the paper would be a solid contribution to the latency-aware inference literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the observation that for memory-bound LLM inference, parallel branches are nearly free, so the optimal test-time scaling under a latency budget is not the same as under a token budget. That insight is well-supported by the roofline analysis and the fixed-config comparisons in Figures 1–3. The paper does a thorough job across several models and datasets, and it is upfront about the memory-bound scope: the Limitations section and Appendix A.4 both acknowledge that on large server workloads the token-count view returns.\n\nWhat is genuinely new is the unified concurrency framing, treating branch count and speculative-draft length as two competing uses of the same memory-bandwidth resource, plus a simple greedy search to set them. The greedy search matching grid search in Table 2 is a nice efficiency result.\n\nThe soft spot is the evaluation protocol for the headline numbers. Algorithm 1 and the appendix list configurations chosen by accuracy on the test set itself. The claimed 82.3% on MATH-500 within 1 minute is the accuracy of the configuration that the search selected on that same benchmark. That is test-set selection, not a prediction. Table 1's baseline comparison is not controlled for search effort, so the 7.3% and 1.7x improvements are optimistic. The stress-test note is right: this inflates every headline number. It is fixable: they should either use a validation split for configuration selection, or clearly present Algorithm 1 as an offline analysis tool that maps out the Pareto frontier rather than as a deployable method. As written, the claimed \"latency-optimal TTS enables...\" is misleading.\n\nThat said, the core qualitative finding holds. The fixed-config experiments do show parallel scaling reaching the same accuracy faster than sequential under a latency budget in memory-bound settings. And the authors explicitly scope the claim to small/medium hardware, so the workload dependence is not hidden. The circularity is the main issue, and it is moderate rather than fatal: the paper's contribution is the framing and the analysis, not the specific configuration-selection procedure.\n\nI'd send this to peer review, but it needs a revision. The authors should add a validation-based selection or reposition the search as a way to generate hypotheses about configurations. I'd also want the search cost included in the comparison. For a reader in the LLM-inference subfield, the latency-aware perspective is worth citing, especially given the growing interest in device-side deployment.\n\nRecommendation: engage with it, but require the test-set-selection issue to be addressed.","headline":"Latency-aware framing is real and useful, but the headline accuracy numbers are selected on the test set, so they predict the past rather than the future.","tokens_in":16030,"tokens_out":2990,"would_cite":true,"duration_ms":29602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that on memory-bound hardware, test-time scaling should be budgeted by wall-clock latency rather than generated tokens, and that tuning parallel branches and speculative decoding finds configurations that are both faster…","keywords":["test-time scaling","latency-aware inference","speculative decoding","branch-wise parallelism","memory-bound decoding","majority voting","greedy search","MATH-500"],"falsifier":"Run the same latency-aware configurations on a compute-heavy accelerator or under a large server batch and measure per-token latency as branch count grows from 1 to 64; if latency rises proportionally with branches even at small batch size — or if a roofline measurement shows the decode workload already sits above the machine's ridge point — the premise that extra branches are nearly free fails and the latency-optimal results become hardware-specific artifacts.","tokens_in":15049,"feed_emoji":"⏱️","tokens_out":7509,"duration_ms":57285,"temperature":0.7,"pith_summary":"This paper tries to establish that test-time scaling — spending extra inference compute to improve an LLM's answers — should be measured and optimized against wall-clock latency, not generated-token count, in latency-sensitive deployments. Prior work treats token efficiency (accuracy gained per token) as the objective, but the paper shows that a compute-optimal strategy can be slower in real time than a parallel strategy that wastes more tokens yet uses otherwise idle memory bandwidth. The authors propose optimizing two kinds of generation concurrency — multiple parallel solution branches and speculative decoding with a chosen draft length — and give a greedy search that finds the best configuration cheaply. If the claim is right, a 32B model can reach 82.3% accuracy on the MATH-500 math benchmark within one minute, and a 3B model 72.4% within ten seconds.","feed_headline":"Latency, not tokens, sets the test-time scaling budget","feed_subtitle":"Tuning parallel branches and speculative draft length lifts a 32B model to 82.3% on MATH-500 in one minute.","key_machinery":"The machinery is generation concurrency, treated as two resource-competing dimensions. Branch-wise parallelism runs B independent solution branches and aggregates them by majority voting, adding accuracy almost without adding latency while decoding is memory-bound. Sequence-wise parallelism applies speculative decoding with draft length gamma, letting several tokens be verified in one forward pass and shifting the accuracy-latency curve left. A greedy search algorithm starts at B = 1, gamma = 0 and each step expands whichever dimension improves accuracy more, stopping when neither helps; this finds the latency-optimal configuration with far fewer evaluations than grid search.","core_discovery":"The central claim is that compute-optimal test-time scaling is not latency-optimal in memory-bound regimes: the strategy that maximizes accuracy per token can require more wall-clock time than a parallel strategy that generates more tokens but exploits otherwise idle memory bandwidth. Demonstrating this on s1.1-32B with MATH-500, the paper finds sequential scaling needs about 9x fewer tokens than parallel majority voting but 1.6x more time to reach the same accuracy. The authors then define latency-optimal TTS as the joint choice of parallel branches B and speculative draft length gamma, and show that a greedy search recovers the same configuration as exhaustive grid search with 8–10 evaluation steps instead of 56. The headline quantitative results are 82.3% accuracy for a 32B model within one minute and 72.4% for a 3B model within ten seconds on MATH-500.","pith_inferences":["If the memory-bound premise holds broadly, then token-efficiency rankings reported on one hardware class may invert on another; benchmark suites should report latency-accuracy curves for the target deployment setting, not only tokens-per-accuracy.","The same concurrency-allocation view could be applied to other latency-sensitive LLM services such as interactive agents or edge assistants, where the greedy search could run at calibration time on the actual device.","A natural extension the authors do not develop is a hybrid controller that starts in latency-optimal parallel mode and switches to token-count budgeting as request concurrency grows and the workload becomes compute-bound.","Because confidence-based aggregation improved accuracy without changing latency, combining confidence scores with the latency-optimal search could push the same curves slightly higher."],"forward_implications":["On memory-bound hardware, a latency budget can change which TTS strategy is best: parallel majority voting beat sequential scaling by 1.6x in time to reach the same accuracy for s1.1-32B on MATH-500.","Tuning both branch count B and speculative draft length gamma together outperforms either alone: on s1.1-32B the latency-optimal configuration improved accuracy by up to 7.3 points and reduced latency by up to 1.7x versus the baseline.","The optimal configuration is model-dependent: reasoning models such as QwQ-32B get more from speculative decoding, while models not trained for long reasoning chains get more from extra branches.","Greedy search finds the same (B, gamma) as grid search using 8–10 evaluation steps instead of 56, so latency-optimal configurations can be identified cheaply.","When the workload becomes compute-bound, the paper's own analysis says the token-count budget is the right metric again, so latency-optimal TTS is specifically a small- and medium-scale deployment result."],"supporting_citations":[{"why":"Supplies the s1.1 models and budget forcing used for sequential scaling and the baseline scaling curves.","marker":"Muennighoff et al., 2025"},{"why":"Foundational speculative decoding method that enables sequence-wise parallelism.","marker":"Leviathan et al., 2023"},{"why":"Foundational speculative sampling method that enables sequence-wise parallelism.","marker":"Chen et al., 2023"},{"why":"Establishes the compute-optimal test-time scaling paradigm the paper argues is incomplete under latency budgets.","marker":"Snell et al., 2024"},{"why":"Provides the MATH-500 benchmark used for the main latency-accuracy evaluations.","marker":"Hendrycks et al., 2021"},{"why":"Provides self-consistency and majority voting, the aggregation mechanism for branch-wise parallelism.","marker":"Wang et al., 2022"},{"why":"Supplies the confidence-based aggregation baselines and the parallel-scaling perspective the paper extends.","marker":"Liu et al., 2025"},{"why":"Provides the Eagle3 draft model used for the LLaMA-3.1-8B speculative decoding setup.","marker":"Li et al., 2025"}],"fun_headline_variants":["Parallel branches beat brute compute for fast LLM answers","Speedy test-time scaling: concurrency is the missing lever","82% on MATH-500 in a minute: latency-aware TTS","Rethink TTS: wall-clock time, not token count, matters","Faster LLM inference via latency-optimized parallel scaling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLM decoding on the target hardware is memory-bound at small batch sizes, so extra parallel branches and speculative draft tokens add almost no latency; if the workload is compute-bound, the authors' own limitation note concedes that token-count budgets apply and the parallel-branch advantage disappears.","fun_headline_variants_meta":{"raw":{"variants":["Parallel branches beat brute compute for fast LLM answers","Speedy test-time scaling: concurrency is the missing lever","82% on MATH-500 in a minute: latency-aware TTS","Rethink TTS: wall-clock time, not token count, matters","Faster LLM inference via latency-optimized parallel scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1656,"prompt_tokens":916,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":532,"tokens_out":740,"duration_ms":6048,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:10:06.175534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same latency-aware configurations on a compute-heavy accelerator or under a large server batch and measure per-token latency as branch count grows from 1 to 64; if latency rises proportionally with branches even at small batch size — or if a roofline measurement shows the decode workload already sits above the machine's ridge point — the premise that extra branches are nearly free fails and the latency-optimal results become hardware-specific artifacts.","supporting_citations":[],"review_version":1}