Pith. sign in

REVIEW 3 major objections 5 minor 47 references

CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoBa reframes test-time scaling as compute allocation and matches the accuracy of sampling and self-evaluation baselines at roughly half the parameter-weighted tokens.

desk verdict A well-executed local routing policy with a real cost-accuracy payoff, but the hand-set thresholds need holdout validation before the headline savings are taken at face value. read the letter →

arxiv 2608.07424 v2 pith:H4X3672Q submitted 2026-08-07 cs.AI

Yan Zhou , Yue Ouyang , Kaiyang Zheng , Suncheng Xiang This is my paper · ORCID
classification cs.AI
keywords test-timescalingcomputeallocationroutingpolicymodelverificationmajorityvotingmathematicalreasoningcost-accuracyfrontierLLMinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that test-time scaling should be treated as a compute-allocation problem: at each step, a reasoning system must decide whether to sample another candidate, apply cheap verification, invoke a stronger verifier, or stop. It introduces CoBa, a staged routing policy that first gathers a small set of candidates, scores all of them with cheap evidence, and routes only uncertain or high-value candidates to strong verification. On 3,129 example–generator evaluations across five math and symbolic reasoning benchmarks, CoBa-Routed-Strong reaches 85.13% macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85.20% while using 49.1% fewer parameter-weighted tokens, and matching best-of-16 majority voting within 0.01 accuracy points while using 58.9% fewer parameter-weighted tokens. A sympathetic reader cares because this gives local reasoning systems a concrete way to reach high accuracy without paying the uniform sampling or uniform evaluation cost on every example, and it reframes the field's evaluation from accuracy alone to accuracy at controlled cost.

What carries the argument

The central mechanism is compute-balanced routing with a cheap-evidence triage layer: answer agreement and a lightweight judge score decide whether to stop early, sample more candidates, or escalate selected candidates to a strong deep verifier. The final ranking $R(c) = 0.20 f(c) + 0.30 s_1(c) + 0.15 s_2(c) + 0.45 s_3(c)$ fuses answer frequency, lightweight judge score, optional process-verifier score, and strong-verifier score, with weights renormalized when sparse scores are missing. This staged design separates candidate generation from candidate verification and concentrates expensive compute on the examples and candidates where the final decision can still change.

What would settle it

Run CoBa-Routed-Strong on a new benchmark where the cheap Qwen3-8B judge is deliberately reversed or made miscalibrated while keeping the candidate pools identical; if accuracy drops to the level of the early-stopped prefix, the cheap-evidence triage layer is carrying the result rather than the routing structure itself.

Watch

Extended reading notes

Core claim

On the paper's own terms, CoBa shows that a fixed, transparent routing policy over generation, verification, and stopping can reach the accuracy region of much more expensive test-time scaling baselines. The policy obtains a small warm-up candidate pool, scores all candidates with a rule-based answer frequency signal and a lightweight Qwen3-8B judge, and stops when the top answer is stable (answer share at least 0.6) and the lightweight judge score is high (at least 0.7). Otherwise it samples more candidates and routes the top few to a stronger Qwen3-14B deep verifier, selecting the final answer by a fused rank that renormalizes when sparse process-verifier scores are absent. In paired bootstrap tests, CoBa-Routed-Strong significantly outperforms greedy decoding by +3.74 accuracy points, is statistically indistinguishable from best-of-8 and from the self-evaluation weighted-voting proxy, and trails best-of-16 by only 0.70 points while using 2.43x fewer parameter-weighted tokens.

Load-bearing premise

The cheap-evidence stop criterion assumes that a stable answer share and a high lightweight judge score reliably imply that the current top answer is correct, and if these cheap signals are frequently wrong on a new distribution, early stopping will hurt accuracy even though the cost savings are structural.

Editorial extensions

If this is right

  • If the central claim holds, test-time scaling comparisons should report action mix and cost–accuracy frontier alongside accuracy, making routing gains falsifiable rather than hidden behind a single score.
  • Routing helps most when the candidate pool is informative and unevenly difficult: easy examples stop after cheap agreement, while hard contest problems consume additional samples and strong-verifier calls, as shown by the per-dataset action distributions.
  • Uniform best-of-N and always-on evaluators pay the hard-example budget on every example, whereas routed policies concentrate that budget where decisions can still change, yielding the reported frontier movement.
  • The remaining oracle gap separates two failure modes: allocation errors, where better routing could select an existing correct candidate, and generation errors, where the pool lacks a correct answer; this distinction turns oracle gaps into design signals for future systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors left implicit that the same triage idea could transfer to code generation and natural-science reasoning, where public test execution or external tools serve as an even stronger cheap-verification tier before escalating to expensive model-based judges.
  • A testable extension is to recalibrate the stop thresholds (0.6 answer share, 0.7 judge score) on each new distribution, since the paper's fixed thresholds were not tuned per dataset and may be a source of the remaining oracle gap.
  • The learned MLP controller degenerating to a near-greedy policy suggests that offline trajectories from a fixed router may be too narrow to train a robust learned controller, and that denser process signals or online continuation could be needed before learned routing becomes viable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper frames test-time reasoning as a compute-allocation problem in which a system chooses, at each step, among sampling another candidate, applying lightweight verification, invoking a stronger verifier, or stopping. It introduces CoBa, a staged routing policy that first collects a warm-up candidate set, scores all candidates with cheap verification, and routes a subset to a strong verifier. On a shared-pool replay setup over 15 dataset–generator pairs (MATH-500, AIME 2024/2025, AMC 2023, and a Reasoning Gym hard subset), CoBa-Routed-Strong reaches 85.13% macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85.20% while using 49.1% fewer parameter-weighted tokens, and it reaches the same macro-accuracy region as best-of-16 majority voting at 58.9% lower parameter-weighted token cost. The paper reports paired bootstrap tests, per-dataset results, ablations over routing strength, action-distribution evidence, and an explicit pool-oracle upper bound.

Significance. If the headline claims hold, the paper makes a useful contribution: it turns test-time scaling into an explicit allocation question and provides a reproducible, locally deployable policy that approaches the accuracy of much more expensive sampling and self-evaluation baselines. The controlled replay protocol over shared candidate pools is a genuine strength, as it places CoBa and the baselines on identical generated evidence. The paper is also transparent about the remaining best-of-16 paired edge and about the oracle gap, and it reports multiple cost metrics. The main weakness is that the routing hyperparameters are hand-set and no validation protocol is described, which leaves open the possibility that the reported cost–accuracy frontier is partly an artifact of threshold selection on the evaluated benchmarks. This is the load-bearing issue for the paper's central claim and needs to be addressed before the result can be taken as evidence for the general value of compute-balanced routing.

major comments (3)
  1. [§4, Table 1, Eq. (6), Appendix A] The central accuracy–cost claim rests on a single fixed hyperparameter configuration (fusion weights .20/.30/.15/.45, stop thresholds share≥0.6 and s1≥0.7, warm-up k=2, Nmax=8, K=4), but the paper gives no validation protocol for these values. The statement in §5 that the values were set before final aggregation with test labels reserved does not rule out tuning on development runs: Appendix A reveals that earlier 4,096-token pools were used during development and quality checks. If the thresholds were selected, even informally, on related data from the same five benchmarks, the reported 49.1% and 58.9% parameter-token savings could be in-sample. This is load-bearing because the savings are realized by the cheap-evidence stop rule (Algorithm 1, line 11) and by the strong-verification route count K. The learned MLP controller's degeneration to a near-greedy policy under leave-one-dataset-out evaluation is direct evidence that dataset-specific tuning matters. I request a held-out validation split (or leave-one-dataset-out calibration) for the thresholds, plus a sensitivity analysis over the Table 1 parameters showing that the Pareto improvement is not an artifact of the exact values chosen.
  2. [§5, Algorithm 1, line 11] The stop criterion assumes that answer agreement (share≥0.6) plus a high Qwen3-8B judge score (s1≥0.7) reliably implies that the current top answer is correct. The manuscript never reports the calibration of this rule: among examples that stop at line 11, what fraction end with the correct final answer, and among examples that do not stop, how often would an early stop have been wrong? This matters for transfer because the cost savings are structural but the accuracy preservation depends on the cheap signal being a good proxy for correctness. The large AIME 2025 oracle gap (71.1 vs 83.3 in Table 3) suggests that stable cheap evidence is sometimes wrong or that the pool lacks a correct candidate; a precision/recall analysis of the stopping rule and of the subsequent strong-verification selection would clarify how much of the reported frontier is due to routing rather than threshold luck.
  3. [Abstract; §5, 'Significance and Ablations'] The abstract states that CoBa-Routed-Strong 'matches best-of-16 majority voting within 0.01 macro-accuracy points,' but the paper's own paired bootstrap finds best-of-16 significantly more accurate (difference −0.70 points, 95% CI [−1.25,−0.16], p=0.004). Macro accuracy can differ from per-example paired accuracy, but the abstract's wording overstates parity and is inconsistent with the paired conclusion reported later. Please qualify the claim, e.g., 'matches best-of-16 on macro accuracy while the paired test retains a small best-of-16 edge at 2.43× the cost,' and ensure the abstract distinguishes macro-level parity from per-example significance.
minor comments (5)
  1. [Figure 1] The state diagram contains a typo: 'scores + uuncertainly' should read 'scores + uncertainty.'
  2. [§3, Eq. (5)] The objective introduces a cost-sensitivity parameter λ, but the experiments never estimate or report λ; please explain how the fixed replay policies instantiate this objective, or drop λ from the formulation.
  3. [Algorithm 1, lines 17–19] Line 17 says candidates are ranked by 'lightweight score and answer frequency,' while Eq. (6) includes the process-verifier score s2 and the strong-verifier score s3; specify the exact score composition used at each stage so it is clear when V2 and V3 evidence enters the ranking.
  4. [Table 2, oracle row] The cost columns for the pool oracle are not defined in the offline replay protocol; please state explicitly how oracle cost is computed (for example, as the cost of the prefix up to the first correct candidate) so readers do not mistake it for a deployable method's cost.
  5. [Appendix D] The appendix refers to 'released experiment artifacts' but no repository URL or artifact identifier is provided; please include a link or DOI for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoBa's headline results are shared-pool replay measurements of an explicitly fixed routing policy, not derivations from its own definitions.

full rationale

The paper's central claims are empirical replay results over shared candidate pools. The routing policy (Algorithm 1, Eq. 6, Table 1) is a fixed, explicitly specified procedure; the reported accuracy and cost numbers are measurements of that procedure, not quantities derived from the problem formulation. Equation (5) is a framing objective and is not used to solve for the policy, so there is no self-definitional reduction. The final fusion score R(c) weights answer frequency and verifier scores, and the self-evaluation weighted-voting baseline uses related signals, but the paper does not claim CoBa's matching accuracy follows by construction; it is an empirical outcome over 3,129 example-generator evaluations. The stop criterion (Algorithm 1 line 11) could in principle be tuned in-sample, and the paper does not describe a validation split for the Table 1 thresholds; this is a generalizability/overfitting risk, not a circularity, because no equation or definition makes the reported savings equivalent to the inputs. The paper explicitly discloses that a learned MLP controller 'degenerated to a near-greedy policy' under leave-one-dataset-out and that several baselines are local proxies, which further indicates the authors are not hiding the fragility of adaptive control. There are no load-bearing self-citations, imported uniqueness theorems, or renamed known results. Therefore no circular step can be exhibited with the required specificity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a handful of hand-set hyperparameters (fusion weights, stop thresholds, candidate caps) and on domain assumptions about the informativeness of cheap verifier signals and the representativeness of the offline replay pool. No new physical or conceptual entities are introduced. The most fragile items are the stop-threshold assumption and the possibility that the hyperparameters were tuned on the same benchmark distribution.

free parameters (5)
  • Fusion weights in Eq. (6) = 0.20 / 0.30 / 0.15 / 0.45
    Hand-set weights for answer frequency, lightweight judge score, process score, and strong verifier score. Not derived from theory; may have been tuned on development pools from the same benchmarks.
  • Stop criterion thresholds = stable answer share >= 0.6, s1 >= 0.7
    Hand-set thresholds in Table 1 that determine when the policy stops after cheap evidence. Central to the cost savings and accuracy.
  • Warm-up count k = 2
    Fixed number of initial candidates in Algorithm 1, chosen to buy a small amount of diversity before verification.
  • Maximum candidates Nmax for variants = 2 / 4 / 8
    Candidate caps for CoBa-Routed-Light, CoBa-Routed, and CoBa-Routed-Strong respectively.
  • Strong-route count K = 0 / 2 / 4
    Number of candidates sent to the strong verifier in the three variants; the ablation shows increasing accuracy with higher K.
assumptions (4)
  • domain assumption Answer frequency and lightweight judge scores are informative about candidate correctness
    Algorithm 1 ranks candidates and decides stopping using V0 and V1; the method's accuracy depends on these signals correlating with correctness on the evaluation benchmarks.
  • domain assumption The offline replay pool of 16 candidates faithfully represents online sequential generation
    All methods are replayed from fixed pools; the policy's sequential decisions are approximated by subset selection. This is stated in Appendix D as a conservative design choice.
  • domain assumption The Qwen3-14B strong verifier provides a useful correctness signal when applied to a small routed subset
    The final selection uses s3 with weight 0.45; if the strong verifier were uninformative, routing accuracy would not match the self-evaluation and best-of-16 baselines.
  • domain assumption Parameter-weighted token cost is a valid proxy for real inference cost
    The cost metric C_ptok weights tokens by model size in billions. The paper also reports measured latency, which supports the cost ordering, but the metric itself is a modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing." pith.science (2026). https://pith.science/paper/H4X3672Q

@misc{pith2026260807424,
  author       = {Pith},
  title        = {Pith review of: CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4X3672Q}},
  note         = {Machine review of arXiv:2608.07424}
}
read the original abstract

Test-time scaling is often implemented by spending more compute along one axis: sampling more solutions, extending a chain of thought, or applying a stronger evaluator. Under a fixed inference budget, these choices compete. This paper formulates test-time reasoning as a compute-allocation problem in which a system must decide whether the next unit of compute should be spent on generation, verification, or stopping. We introduce CoBa, a compute-balanced routing policy that first obtains a small set of candidates, applies cheap verification broadly, and routes uncertain or high-value candidates to stronger verification. On 3,129 example-generator evaluations spanning MATH-500, AIME 2024/2025, AMC 2023, and procedural symbolic reasoning, CoBa-Routed-Strong reaches 85.13% macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85.20% while using 49.1% fewer parameter-weighted tokens. It also matches best-of-16 majority voting within 0.01 macro-accuracy points while using 58.9% fewer parameter-weighted tokens; paired tests retain a small best-of-16 edge at substantially higher cost. Paired bootstrap tests show significant gains over single-sample decoding, while the remaining gap to the pool oracle exposes headroom for sharper routing. For local reasoning systems, test-time scaling becomes a question of where the next computation is most valuable.

Figures

Figures reproduced from arXiv: 2608.07424 by the authors.

Figure 1
Figure 1. CoBa views test-time scaling as compute allocation. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Accuracy–cost frontier. CoBa reaches the high [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Adaptive action allocation. Harder datasets receive [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Dataset-level Pareto frontiers. Routed policies im [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Evidence grid for compute allocation. The panels [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Cost savings at comparable accuracy. CoBa [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 19 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume =

    Chain-of-thought prompting elicits reasoning in large language models , author =. Advances in neural information processing systems , volume =

  2. [2]

    arXiv preprint arXiv:2203.11171 , year =

    Self-consistency improves chain of thought reasoning in language models , author =. arXiv preprint arXiv:2203.11171 , year =

    arXiv

  3. [3]

    URL https://arxiv

    Training verifiers to solve math word problems, 2021 , author =. URL https://arxiv. org/abs/2110.14168 , volume =

    arXiv 2021

  4. [4]

    International Conference on Learning Representations , volume =

    Let's verify step by step , author =. International Conference on Learning Representations , volume =

    work page

  5. [5]

    arXiv preprint arXiv:2408.03314 , year =

    Scaling llm test-time compute optimally can be more effective than scaling model parameters , author =. arXiv preprint arXiv:2408.03314 , year =

    arXiv

  6. [6]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    s1: Simple test-time scaling , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    work page 2025

  7. [7]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 technical report , author =. arXiv preprint arXiv:2505.09388 , year =

    arXiv

  8. [8]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Genprm: Scaling test-time compute of process reward models via generative reasoning , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    work page

Show all 47 references
  1. [9]

    arXiv preprint arXiv:2503.19877 , year =

    Scaling evaluation-time compute with reasoning models as process evaluators , author =. arXiv preprint arXiv:2503.19877 , year =

    arXiv

  2. [10]

    arXiv preprint arXiv:2505.04842 , year =

    Putting the value back in rl: Better test-time scaling by unifying llm reasoners with verifiers , author =. arXiv preprint arXiv:2505.04842 , year =

    arXiv

  3. [11]

    arXiv preprint arXiv:2602.11202 , year =

    interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification , author =. arXiv preprint arXiv:2602.11202 , year =

    arXiv

  4. [12]

    arXiv preprint arXiv:2504.21318 , year =

    Phi-4-reasoning technical report , author =. arXiv preprint arXiv:2504.21318 , year =

    arXiv

  5. [13]

    arXiv preprint arXiv:2103.03874 , year =

    Measuring mathematical problem solving with the math dataset , author =. arXiv preprint arXiv:2103.03874 , year =

    arXiv

  6. [14]

    2023 , howpublished =

    2023

  7. [15]

    2025 , note =

    AIME 2025 Dataset , author =. 2025 , note =

    2025

  8. [16]

    2024 , howpublished =

    2024

  9. [17]

    Advances in Neural Information Processing Systems , volume =

    Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards , author =. Advances in Neural Information Processing Systems , volume =

  10. [18]

    Proceedings of the 29th symposium on operating systems principles , pages =

    Efficient memory management for large language model serving with pagedattention , author =. Proceedings of the 29th symposium on operating systems principles , pages =

  11. [19]

    Advances in neural information processing systems , volume =

    Language models are few-shot learners , author =. Advances in neural information processing systems , volume =

  12. [20]

    Journal of machine learning research , volume =

    Palm: Scaling language modeling with pathways , author =. Journal of machine learning research , volume =

  13. [21]

    Advances in neural information processing systems , volume =

    Large language models are zero-shot reasoners , author =. Advances in neural information processing systems , volume =

  14. [22]

    Advances in Neural Information Processing Systems , volume =

    Star: Bootstrapping reasoning with reasoning , author =. Advances in Neural Information Processing Systems , volume =

  15. [23]

    arXiv preprint arXiv:2210.03629 , year =

    React: Synergizing reasoning and acting in language models , author =. arXiv preprint arXiv:2210.03629 , year =

    arXiv

  16. [24]

    Advances in neural information processing systems , volume =

    Tree of thoughts: Deliberate problem solving with large language models , author =. Advances in neural information processing systems , volume =

  17. [25]

    International conference on machine learning , pages =

    Pal: Program-aided language models , author =. International conference on machine learning , pages =. 2023 , organization =

    2023

  18. [26]

    arXiv preprint arXiv:2211.12588 , year =

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks , author =. arXiv preprint arXiv:2211.12588 , year =

    arXiv

  19. [27]

    Advances in neural information processing systems , volume =

    Self-refine: Iterative refinement with self-feedback , author =. Advances in neural information processing systems , volume =

  20. [28]

    Advances in neural information processing systems , volume =

    Reflexion: Language agents with verbal reinforcement learning , author =. Advances in neural information processing systems , volume =

  21. [29]

    arXiv preprint arXiv:2407.21787 , year =

    Large language monkeys: Scaling inference compute with repeated sampling , author =. arXiv preprint arXiv:2407.21787 , year =

    arXiv

  22. [30]

    Science , volume =

    Competition-level code generation with alphacode , author =. Science , volume =. 2022 , publisher =

    2022

  23. [31]

    arXiv preprint arXiv:2211.14275 , year =

    Solving math word problems with process-and outcome-based feedback , author =. arXiv preprint arXiv:2211.14275 , year =

    arXiv

  24. [32]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  25. [33]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages =

    Rewardbench: Evaluating reward models for language modeling , author =. Findings of the Association for Computational Linguistics: NAACL 2025 , pages =

    2025

  26. [34]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Processbench: Identifying process errors in mathematical reasoning , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  27. [35]

    arXiv preprint arXiv:1603.08983 , year =

    Adaptive computation time for recurrent neural networks , author =. arXiv preprint arXiv:1603.08983 , year =

    arXiv

  28. [36]

    arXiv preprint arXiv:2305.05176 , year =

    Frugalgpt: How to use large language models while reducing cost and improving performance , author =. arXiv preprint arXiv:2305.05176 , year =

    arXiv

  29. [37]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  30. [38]

    arXiv preprint arXiv:2406.18665 , year =

    Routellm: Learning to route llms with preference data , author =. arXiv preprint arXiv:2406.18665 , year =

    arXiv

  31. [39]

    Advances in neural information processing systems , volume =

    Solving quantitative reasoning problems with language models , author =. Advances in neural information processing systems , volume =

  32. [40]

    arXiv preprint arXiv:2402.03300 , year =

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author =. arXiv preprint arXiv:2402.03300 , year =

    arXiv

  33. [41]

    arXiv preprint arXiv:2311.12022 , year =

    Gpqa: A graduate-level google-proof q&a benchmark , author =. arXiv preprint arXiv:2311.12022 , year =

    arXiv

  34. [42]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  35. [43]

    International Conference on Learning Representations , volume =

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author =. International Conference on Learning Representations , volume =

  36. [44]

    arXiv preprint arXiv:2501.12948 , year =

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author =. arXiv preprint arXiv:2501.12948 , year =

    arXiv

  37. [45]

    arXiv preprint arXiv:2503.24235 , year =

    What, how, where, and how well? a survey on test-time scaling in large language models , author =. arXiv preprint arXiv:2503.24235 , year =

    arXiv

  38. [46]

    arXiv e-prints , pages =

    Efficiently serving llm reasoning programs with certaindex , author =. arXiv e-prints , pages =

  39. [47]

    arXiv preprint arXiv:2509.09864 , year =

    Latency and Token-Aware Test-Time Compute , author =. arXiv preprint arXiv:2509.09864 , year =

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.