Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Re-FORC trains a lightweight adapter on frozen reasoning models to predict the expected reward of additional thinking tokens, enabling early stopping, compute-aware model selection, and test-time scaling that improve accuracy-compute trade-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:13 UTC pith:BB7N3S54

load-bearing objection Promising method for adaptive inference via forecasted reward curves, but reporting inconsistencies and an untested bias in the training labels currently prevent me from trusting the headline numbers. the 4 major comments →

arxiv 2511.02130 v2 pith:BB7N3S54 submitted 2025-11-03 cs.AI cs.LG

Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning

Renos Zabounidis , Aditya Golatkar , Michael Kleinman , Alessandro Achille , Wei Xia , Stefano Soatto This is my paper
classification cs.AI cs.LG
keywords reward predictionchain-of-thoughtearly stoppingtest-time compute scalingmodel selectionGittins indexBeta distributionreasoning efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Re-FORC claims that the expected reward of generating more chain-of-thought tokens, given a query and partial trace, is predictable from the inner activations of a frozen reasoning model. A lightweight adapter trained on traces from existing reasoning models outputs a Beta-distribution forecast of that reward for any additional-token budget. The paper then applies this forecast curve inside a Gittins-index-style greedy policy to decide when to stop, which model to use, and whether to sample another answer. If the forecast is accurate enough, the method saves compute while keeping or improving accuracy: up to 26% token reduction at equal accuracy for early stopping, 4% higher accuracy at equal compute and 55% less compute at equal accuracy for model selection, and 7–11% accuracy gains over confidence-based baselines in test-time scaling. The paper argues the forecast improves with reasoning progress and larger models.

Core claim

The central claim is that the forecasting functional ψ(t|x,z,π) — the expected final reward if a model π continues from state (x,z) for exactly t more thinking tokens — can be learned by a lightweight adapter and used as a proxy for the Gittins index. The adapter predicts Beta parameters for each horizon on a token grid, with the Beta mean as point estimate. The paper claims this predicted curve is accurate enough to drive three decisions: stop reasoning when the expected improvement ψ(t)(1−R̄) no longer covers the token cost λt; route a query to the model whose forecasted net utility is highest; and continue sampling answers in test-time scaling until the forecasted improvement is negative.

What carries the argument

The forecasting functional ψ(t|x,z,π)=E[R(x,y)] over t-token continuations, approximated by a Beta(α,β) predictor from penultimate-layer activations (attention pooling + linear head). The Beta distribution gives bounded support matching rewards in [0,1] and a variance signal; the mean is the point estimate. Decisions are made by a Gittins-index-style greedy threshold: continue or switch only when ψ(t)(1−R̄)−λt>0, where λ is a user-set cost per token. Training uses Monte Carlo targets built by reusing segments of full unconstrained trajectories truncated on a uniform grid. The Gittins index, here approximated by the forecast curve, is a reservation value from optimal search theory that tells

Load-bearing premise

Everything rests on the assumption that answers sampled from truncated full-length reasoning traces give unbiased estimates of the expected reward of continuing from that truncation under the actual (possibly early-stopped) inference process; if truncation changes the future-trace distribution, the forecast curve is biased and all downstream decisions inherit the bias.

What would settle it

On a held-out set of queries, generate full trajectories, truncate at several grid points, and compare Re-FORC's forecasted ψ(t) against the reward empirically obtained by sampling fresh continuations of length t from those exact truncation states (not reusing the original trajectory segments). If the mean absolute error of the forecast is large, or the ordering of stop/continue decisions flips relative to the oracle, the claimed compute savings will not materialize.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Early stopping with Re-FORC reduces compute by up to 26% compared with fixed-budget cutoffs while preserving accuracy, across five math benchmarks.
  • Joint model-and-length selection reaches 4% higher accuracy at equal compute and 55% less compute at equal accuracy compared with the largest model alone.
  • Adaptive test-time scaling improves accuracy by 7% at low compute and 11% at high compute over confidence-based baselines.
  • Users can tune the cost-per-token threshold λ at inference time to trace the whole accuracy-compute frontier without retraining the base model.
  • Forecast quality improves as reasoning progresses and with larger models, so the method is most reliable exactly when decisions matter most.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the forecaster is trained on unconstrained trajectories, an early-stopping policy that changes the distribution of future reasoning traces could bias the forecast; retraining the adapter on traces generated under the stopping policy is a testable fix.
  • The paper never states whether the five evaluation benchmarks were excluded from the math-reasoning training set used for the forecaster; if any overlap exists, the reported gains could be inflated by memorization, and a contamination check on disjoint benchmarks would settle it.
  • Using the Beta variance (which the paper leaves as future work) could tighten stopping thresholds and reduce the overconfidence the authors observe; this is an immediate, low-cost extension.
  • The reward function here is binary correctness on math problems; adapting Re-FORC to continuous or soft rewards (code tests, verifier scores) would test whether the forecast mechanism transfers beyond math.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Re-FORC, a method that trains a lightweight adapter on frozen Qwen3 reasoning models to predict the expected reward of generating t additional thinking tokens from a partial reasoning trace, formalized as ψ(t|x,z,π) in Eq. (3) and approximated by a Beta-mean forecaster in Eq. (4). Using this forecaster, the authors derive greedy stopping, model-routing, and test-time-scaling rules inspired by Gittins-index/Pandora-box theory (Eqs. 8, 10, 12). They report improved accuracy–compute trade-offs on five mathematics datasets: early stopping saves up to 26% compute, model selection achieves 4% higher accuracy at equal compute or 55% less compute at equal accuracy, and adaptive test-time scaling improves accuracy by 7–11% over confidence-based baselines. The main claimed contribution is a continuous, query- and path-dependent forecast of reward as a function of reasoning length, enabling user-controlled cost–accuracy trade-offs at inference time.

Significance. If the forecasting functional is accurate and the reuse-based training estimator is unbiased, Re-FORC is a useful and fairly general mechanism for compute-aware inference: it applies to base or black-box models, provides a tunable λ parameter, and connects an empirically practical adapter to the principled Gittins/Pandora-box framework. The idea of predicting the marginal value of additional thinking tokens as a continuous curve is timely and the three applications are natural and well-motivated. However, the current manuscript's empirical support is substantially thinner than the claims require: there are no error bars or multi-seed results, the headline numbers in the abstract and body disagree, and the key unbiasedness assumption behind the training data is asserted rather than established. The paper does not ship code or data, and it does not report whether evaluation benchmarks were excluded from the training distribution, so the reported gains could be inflated by memorization. The central claim is defensible and worth pursuing, but the present evidence is not yet at the bar for publication.

major comments (4)
  1. [§3.3, Eq. (3)–(4)] The claim that truncating full unconstrained trajectories and reusing trajectory segments yields 'unbiased estimates of ψ(t|x,z,π)' is not established and is questionable. In Eq. (3), ψ conditions on drawing exactly t additional thinking tokens from π(r)(·|x,z,t). But the reuse estimator conditions on the event that a natural rollout did not stop before the truncation point plus t. These are different events; trajectories that would have stopped early are overrepresented in long continuations, and segments that cross a natural stop contain answer tokens rather than thinking tokens. The paper's own difficulty-allocation results suggest that stopping behavior correlates with query difficulty and likely with correctness. The limitation section's admission of 'forecaster overconfidence' is consistent with this label bias. Because Eqs. (8), (10), and (12) all threshold the same forecaster, th
  2. [Figs. 2–4 and Section 6] No error bars, confidence intervals, or multi-seed results appear anywhere in the paper. The central empirical claims are numeric point comparisons — 26% compute savings, 4% accuracy gain, 55% vs. 12% compute reduction, 7–11% scaling gains. On benchmark sets like AIME, with roughly 30 problems per year, a difference of a few percentage points can be noise. Since the paper uses a learned adapter with random initialization and Monte Carlo sampling (N=8), seed variance is non-negligible. The authors should report means and variances over multiple training seeds or subsamples, and state whether the reported gains are statistically significant. Without this, the headline comparisons are not reproducible.
  3. [Abstract vs. Sections 4–6] The reported headline numbers are internally inconsistent. The abstract states: 26% compute reduction, '1.7 percentage points higher peak accuracy,' '12% less compute' to match the largest model, and '9.9 percentage points' scaling gain. The body states (Sections 4.3, 4.4, 6): 4% higher accuracy at equal compute, '55% less compute at equal accuracy,' and '11% ... and 7%' scaling gains, with Section 6 also saying 'approximately 25%' and '50% less compute.' These are not minor wording differences; they change the magnitude of the claimed contribution. The authors must reconcile the abstract, body, and figure captions and state which numbers correspond to which setting (e.g., average over datasets vs. specific model or λ).
  4. [§5.1–5.2] The paper never states whether the five evaluation datasets (AMC 2024, Minerva, MATH500, AIME 2024/25) were excluded from the DeepScaleR-Preview training distribution used to generate forecaster training data. If any of these benchmarks appear in the pretraining or RL data of DeepScaleR/Qwen3, the forecast labels and final-accuracy measurements could be inflated by memorization. At minimum, the authors should report their contamination check or explicitly state the overlap status. This is standard practice for reasoning benchmarks and directly affects the validity of every reported accuracy gain.
minor comments (4)
  1. [§4.4 / Fig. 4] The caption uses 'Re-FoRC' inconsistently with the body's 'Re-FORC.' Also, the figure legend lists 'Avg of k' and 'Majority vote' but the caption text references 'Avg-of-k' and 'Pass-at-k'; unify the naming.
  2. [§5.1] The training details for the adapter are incomplete: no learning rate, number of epochs, optimizer, loss weighting, or early stopping criteria are reported. Without these, the forecaster training is not reproducible.
  3. [§3.3] The sentence 'we sample the model's answer directly from state (x, z)' is ambiguous: it is unclear whether this refers to a t=0 sample or to sampling an answer after a forced continuation. Clarify the data-generation protocol and how the reward R(x,y) is computed (e.g., exact match vs. rubric).
  4. [§4.1, Eq. (6)] In Eq. (6), the term (R(x,y) − R*)+ is dimensionally a reward, but the right-hand side subtracts λt which has units of cost; the authors should clarify the units of λ (cost per token) and how R* is normalized. The later claim that Eq. (7) follows for 'binary reward distribution' deserves a derivation, as the simplification from expectation of positive part to ψ(1−R̄) is not immediate when R is binary and R* is itself a random variable.

Circularity Check

0 steps flagged

No significant circularity: the forecaster is supervised, but downstream claims are validated on external accuracy/compute; only minor self-citation reliance.

full rationale

Re-FORC's forecaster is a fitted supervised predictor (Eq. 4, trained with Eq. 5), but the paper's central empirical claims are not forced by that fit. Early stopping (Eq. 8), model selection (Eqs. 10-11), and test-time scaling (Eq. 12) are evaluated on actual benchmark accuracy and token/compute cost of the base reasoning models, against non-forecaster baselines (fixed budgets, S1, Avg-of-k, majority vote, Pass@k oracle). A forecaster that merely recapitulated its training labels would not produce the reported accuracy-compute frontiers. The strongest concern is not circularity but estimator validity: Section 3.3 asserts that reusing segments of full unconstrained trajectories gives 'unbiased estimates of ψ(t|x,z,π)' without comparing to fresh length-conditioned continuations. If natural stopping is informative, the reuse labels differ from the Eq. 3 quantity, and the forecaster (and downstream thresholds) inherit bias. This is a correctness/bias risk, not a construction-equivalent reduction. The paper's own limitation section concedes 'forecaster overconfidence,' consistent with this risk. The Pandora's-box/Gittins framing leans on the authors' self-citations ([1], [44]), but the underlying optimal-stopping theory is independently sourced (Weitzman [2]; Gittins [46]), so the self-citation is motivational rather than load-bearing. There is also no stated exclusion of the AMC/AIME/MATH500/Minerva benchmarks from DeepScaleR-Preview training data, a contamination risk rather than a construction-equivalent circularity. Overall, no step in the derivation is identical by construction to its inputs; score 1 reflects only the minor self-citation reliance.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on four domain assumptions: binary-reward simplification, unbiased trajectory-reuse estimation, the optimal-stopping framework, and forecaster calibration. The most fragile is trajectory-reuse unbiasedness, since the paper does not prove it and the evaluation is sensitive to it. No new physical entities are introduced.

free parameters (3)
  • λ (cost-per-token) = swept values e.g. 2e-05–3.4e-04
    User-specified threshold controlling all stopping, routing, and scaling decisions; not learned from data, but the headline gains are reported at specific hand-selected λ values.
  • Monte Carlo sample count N = 8
    Number of sampled trajectories used to estimate empirical success rates for ψ targets; directly controls label noise in the training data.
  • Forecast grid resolution Δ and horizon t_max = Δ=512, t_max=8192
    Uniform grid for Beta-parameter outputs with linear interpolation; discretization affects forecast accuracy and is a hand-chosen design point.
axioms (5)
  • domain assumption Reward R is binary correctness, so E[(R−R̄)+] = ψ(1−R̄)
    Used to derive Eq. 7; holds only for binary rewards, while experiments estimate success rates over N=8 samples, making the equality approximate.
  • ad hoc to paper Trajectory reuse yields unbiased Monte Carlo estimates of ψ(t|x,z,π)
    Section 3.3 asserts unbiasedness without proof; reusing segments of one full trajectory for many (z,t) pairs induces correlation and potential bias, especially when trajectories are truncated at max context length.
  • domain assumption Gittins-index / Pandora's-box policy is a valid optimal-stopping framework for the LLM continuation decision
    Sections 3.1–4.2 rely on [1,46,49] for the decision rule; the approximation replaces an index over stopping times with a max over fixed t in Eq. 9.
  • domain assumption The adapter's Beta mean is a sufficiently calibrated estimate of ψ for the proposed decision rules
    Used throughout Section 4; the paper itself notes in Section 7 that the forecaster is occasionally overconfident, so calibration is imperfect.
  • domain assumption Evaluation benchmarks are not seen by the forecaster during training
    Not stated anywhere; if DeepScaleR-Preview overlaps with AMC/AIME/MATH500/Minerva, the forecast gains are partly memorization rather than generalization.

pith-pipeline@v1.3.0-alltime-deepseek · 17687 in / 16355 out tokens · 159201 ms · 2026-08-04T00:13:37.684427+00:00 · methodology

0 comments
read the original abstract

We propose Re-FORC, an adaptive reward prediction method that, given a query, enables prediction of the expected future rewards as a function of the number of future thinking tokens. Re-FORC trains a lightweight adapter on reasoning models, demonstrating improved prediction with longer reasoning and larger models. Re-FORC enables: 1) early stopping of unpromising reasoning chains, reducing compute by up to 26\% compared to fixed-budget cutoffs, while maintaining accuracy, 2) optimized model and thinking length selection that outperforms the largest model alone--- reaching 1.7 percentage points higher peak accuracy while needing up to 12\% less compute to match the largest model's accuracy, 3) adaptive test-time scaling, which increases accuracy by 9.9 percentage points (on average at maximum compute) over confidence-based baselines. Re-FORC allows dynamic reasoning with length control via cost-per-token thresholds while estimating computation time upfront.

Figures

Figures reproduced from arXiv: 2511.02130 by Aditya Golatkar, Alessandro Achille, Michael Kleinman, Renos Zabounidis, Stefano Soatto, Wei Xia.

Figure 1
Figure 1. Figure 1: Forecast performance with reasoning progress. Correlation between Re-FORC (in eq. (4)) and the true reward (in eq. (3)) as CoT progresses for Qwen3 [50] models, averaged across five evaluation datasets (see section 5.2). (left) Pearson correlation (ρ) (higher is better), (middle) mean squared error (MSE) (lower is better), and (right) mean absolute error (MAE) (lower is better). Performance steadily improv… view at source ↗
Figure 2
Figure 2. Figure 2: Early Stopping with Re-FORC. We plot the accuracy vs tokens trade-off for various Qwen-3 models: 1.7B (left), 4B (middle), and 8B (right); averaged across five reasoning datasets (see section 5.2). We show that Re-FORC improves the trade-off frontier over baseline inference-time scaling methods like S1[39] across all model sizes and benchmarks. Results for individual datasets are in Appendix [PITH_FULL_IM… view at source ↗
Figure 3
Figure 3. Figure 3: Model and thinking length selection with Re-FORC. (Left) We plot accuracy-compute trade-off for Qwen-3 models averaged across 5 reasoning benchmarks. We use the method proposed in eq. (10) and eq. (11) to perform routing with model selection, and compare against baselines. We show that the accuracy-compute frontier obtained by Re-FORC outperforms the individual models, including Avg-of-N baseline. (Right) … view at source ↗
Figure 4
Figure 4. Figure 4: Test-time scaling using Re-FORC. We plot the accuracy-token trade off for Pandora’s box greedy search based Re-FORC algorithms for Qwen-3 reasoning models averaged across 3 reasoning benchmarks: AIME 24/25 and AMC24. We also compare our method against repeated-sampling test-time scaling baselines Avg-of-k and majority-vote. Re-FORC based scaling outperforms the baselines only to be outperformed by Pass-at-… view at source ↗
Figure 5
Figure 5. Figure 5: Token distribution and problem difficulty for Qwen3 models averaged across datasets. Problems are ordered by per-model solve rate across datasets (Minerva, MATH500, AMC2024, AIME2024, AIME2025); “easy” problems are solved in ≥90% of trials and “hard” ones in <50%. Each curve shows the cumulative share of total tokens spent versus cumulative problem difficulty, with the dashed diagonal indicating uniform al… view at source ↗
Figure 6
Figure 6. Figure 6: Token distribution and problem difficulty per dataset. We order problems based on the difficulty of a problem (based on the fraction of times the problem was correctly solved) and plot the percentage of tokens needed for solving those problems. A diagonal line indicates equal allocation regardless of problem difficulty and increasingly convex curves indicate fewer token allocation to easier problems. On di… view at source ↗
Figure 7
Figure 7. Figure 7: Forecasting performance for AIME2024 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.4 0.6 0.8 0.40 0.57 0.66 0.73 0.51 0.66 0.76 0.79 0.64 0.64 0.68 Pearson’s ρ 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.08 0.10 0.12 0.13 0.11 0.09 0.09 0.12 0.10 0.09 0.13 0.11 0.11 0.12 MSE 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.225 0.250 0.275 0.300 0.31 … view at source ↗
Figure 8
Figure 8. Figure 8: Forecasting performance for AIME2025 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.4 0.6 0.27 0.37 0.49 0.57 0.44 0.52 0.63 0.65 0.49 0.61 0.67 Pearson’s ρ 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.10 0.15 0.19 0.18 0.17 0.15 0.17 0.16 0.14 0.13 0.16 0.16 0.12 MSE 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.25 0.30 0.35 0.39 0.38 0.35 0.33 0.3… view at source ↗
Figure 9
Figure 9. Figure 9: Forecasting performance for AMC 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.5 0.6 0.49 0.55 0.60 0.62 0.46 0.53 0.61 0.66 0.49 0.48 0.54 0.56 Pearson’s ρ 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.05 0.10 0.15 0.15 0.15 0.14 0.13 0.12 0.11 0.10 0.08 0.09 0.10 0.10 0.09 MSE 1.7B 4B 8B 0 10 20 30 40 50 60 70 80 90 100 Reasoning Progress (%) 0.2 0.3 0.34 0.33 0.32 0.31 … view at source ↗
Figure 10
Figure 10. Figure 10: Forecasting performance for Math500 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Forecasting performance for Minerva A.3 Early Stopping In this section we show the early stopping performance for all the five evaluation datasets considered in the main paper, corresponding to fig. 2. 0 2000 4000 Average Tokens Used 70 80 Accuracy (%) λ = 1λ.3=e−2.040e−05 λ = 2.3e−04 λ = 3.4e−04 Minerva 0 2000 4000 Average Tokens Used 70 80 Accuracy (%) λ = 1λ.3=e−2.040e−05 λ = 2.3e−04 λ = 3.4e−04 MATH50… view at source ↗
Figure 12
Figure 12. Figure 12: Early Stopping performance for 1.7B (top); 4B (middle); and 8B (bottom) Qwen 3 models. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Cost-aware selection comparing accuracy versus token usage, where Pass@k represents [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Test time scaling performance for 1.7B (top); 4B (middle); and 8B (bottom) Qwen 3 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

    cs.LG 2026-07 conditional novelty 7.0

    CAPC combines query-agnostic compression with prompt caching and is reported cheapest in 16/16 LongBench configurations, but the empirical support has unresolved specification and consistency gaps.

  2. ExecTune: Effective Steering of Black-Box LLMs with Guide Models

    cs.LG 2026-04 unverdicted novelty 6.0

    ExecTune trains guide models via acceptance sampling, supervised fine-tuning, and structure-aware RL to boost executability of strategies for black-box LLMs, yielding up to 9.2% higher accuracy and 22.4% lower cost on...

Reference graph

Works this paper leans on

61 extracted references · 8 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Ai agents as universal task solvers.arXiv preprint arXiv:2510.12066, 2025

    Alessandro Achille and Stefano Soatto. Ai agents as universal task solvers.arXiv preprint arXiv:2510.12066, 2025

    arXiv 2025

  2. [2]

    Weitzman

    Martin L. Weitzman. Optimal search for the best alternative.Econometrica, 47(3):641–654, 1979

    1979

  3. [3]

    Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation, 2024

    Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation, 2024

    2024

  4. [4]

    Learning how hard to think: Input-adaptive allocation of lm computation, 2024

    Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of lm computation, 2024

    2024

  5. [5]

    Reasoning models know when they’re right: Probing hidden states for self-verification, 2025

    Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they’re right: Probing hidden states for self-verification, 2025

    2025

  6. [6]

    Reasoning models better express their confidence

    Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, and Minjoon Seo. Reasoning models better express their confidence. arXiv preprint arXiv:2505.14489, 2025

    arXiv 2025

  7. [7]

    Are the hidden states hiding something? testing the limits of factuality-encoding capabilities in llms, 2025

    Giovanni Servedio, Alessandro De Bellis, Dario Di Palma, Vito Walter Anelli, and Tommaso Di Noia. Are the hidden states hiding something? testing the limits of factuality-encoding capabilities in llms, 2025

    2025

  8. [8]

    When do llms admit their mistakes? understanding the role of model belief in retraction, 2025

    Yuqing Yang and Robin Jia. When do llms admit their mistakes? understanding the role of model belief in retraction, 2025

    2025

  9. [9]

    Latts: Locally adaptive test-time scaling.arXiv preprint arXiv:2509.20368, 2025

    Theo Uscidda, Matthew Trager, Michael Kleinman, Aditya Chattopadhyay, Wei Xia, and Stefano Soatto. Latts: Locally adaptive test-time scaling.arXiv preprint arXiv:2509.20368, 2025

    arXiv 2025

  10. [10]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasoning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  13. [13]

    Adaptive test-time reasoning via reward-guided dual-phase search, 2025

    Yingqian Cui, Zhenwei Dai, Pengfei He, Bing He, Hui Liu, Xianfeng Tang, Jingying Zeng, Suhang Wang, Yue Xing, Jiliang Tang, and Benoit Dumoulin. Adaptive test-time reasoning via reward-guided dual-phase search, 2025

    2025

  14. [14]

    Large language model guided tree-of-thought, 2023

    Jieyi Long. Large language model guided tree-of-thought, 2023

    2023

  15. [15]

    Demystifying chains, trees, and graphs of thoughts.IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–20, 2025

    Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwa´sniewski, Jurgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O’Mahony, Onur Mutlu, and Torsten Hoefler. Demystifying chains, trees, and graphs of thoughts.IEEE Transactions on Pattern Analysis and Mac...

    2025

  16. [16]

    Fractured chain-of-thought reasoning, 2025

    Baohao Liao, Hanze Dong, Yuhui Xu, Doyen Sahoo, Christof Monz, Junnan Li, and Caiming Xiong. Fractured chain-of-thought reasoning, 2025

    2025

  17. [17]

    Don’t get lost in the trees: Streamlining llm reasoning by overcoming tree search exploration pitfalls, 2025

    Ante Wang, Linfeng Song, Ye Tian, Dian Yu, Haitao Mi, Xiangyu Duan, Zhaopeng Tu, Jinsong Su, and Dong Yu. Don’t get lost in the trees: Streamlining llm reasoning by overcoming tree search exploration pitfalls, 2025. 11

    2025

  18. [18]

    Bartoldson, Bhavya Kailkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain

    Siddarth Venkatraman, Vineet Jain, Sarthak Mittal, Vedant Shah, Johan Obando-Ceron, Yoshua Bengio, Brian R. Bartoldson, Bhavya Kailkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain. Recursive self-aggregation unlocks deep thinking in large language models, 2025

    2025

  19. [19]

    Self-consistency improves chain of thought reasoning in language models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023

    2023

  20. [20]

    Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning, 2024

    Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning, 2024

    2024

  21. [21]

    Answer convergence as a signal for early stopping in reasoning, 2025

    Xin Liu and Lu Wang. Answer convergence as a signal for early stopping in reasoning, 2025

    2025

  22. [22]

    Confidence improves self-consistency in llms

    Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms. InFindings of the Association for Computational Linguistics: ACL 2025, page 20090–20111. Association for Computational Linguistics, 2025

    2025

  23. [23]

    Best-of-∞ – asymptotic performance of test-time compute, 2025

    Junpei Komiyama, Daisuke Oba, and Masafumi Oyamada. Best-of-∞ – asymptotic performance of test-time compute, 2025

    2025

  24. [24]

    Reasoning at the right length: Adaptive budget forcing for efficient and accu- rate LLM inference

    Anonymous. Reasoning at the right length: Adaptive budget forcing for efficient and accu- rate LLM inference. InSubmitted to The Fourteenth International Conference on Learning Representations, 2025. under review

    2025

  25. [25]

    Stop when enough: Adaptive early-stopping for chain-of-thought reasoning, 2025

    Renliang Sun, Wei Cheng, Dawei Li, Haifeng Chen, and Wei Wang. Stop when enough: Adaptive early-stopping for chain-of-thought reasoning, 2025

    2025

  26. [26]

    Dynamic early exit in reasoning models, 2025

    Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Minghui Chen, Zheng Lin, and Weiping Wang. Dynamic early exit in reasoning models, 2025

    2025

  27. [27]

    Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Tran, Yi Tay, and Donald Metzler

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling, 2022

    2022

  29. [29]

    Learning when to plan: Efficiently allocating test-time compute for llm agents, 2025

    Davide Paglieri, Bartłomiej Cupiał, Jonathan Cook, Ulyana Piterbarg, Jens Tuyls, Edward Grefenstette, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. Learning when to plan: Efficiently allocating test-time compute for llm agents, 2025

    2025

  30. [30]

    Can past experience accelerate llm reasoning?, 2025

    Bo Pan and Liang Zhao. Can past experience accelerate llm reasoning?, 2025

    2025

  31. [31]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  32. [32]

    Least-to-most prompting enables complex reasoning in large language models, 2023

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, 2023

    2023

  33. [33]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    2023

  34. [34]

    Universal model routing for efficient llm inference, 2025

    Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient llm inference, 2025

    2025

  35. [35]

    Chen, Trevor Chow, Ishan S

    Neel Guha, Mayee F. Chen, Trevor Chow, Ishan S. Khare, and Christopher Ré. Smoothie: Label free language model routing, 2024. 12

    2024

  36. [36]

    Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Car- men Hipolito Garcia, Menglin Xia, Laks V . S. Lakshmanan, Qingyun Wu, and Victor Rühle. Best-route: Adaptive llm routing with test-time optimal compute, 2025

    2025

  37. [37]

    Masrouter: Learning to route llms for multi-agent systems, 2025

    Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route llms for multi-agent systems, 2025

    2025

  38. [38]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Principles of metareasoning.Artificial Intelligence, 49(1):361– 395, 1991

    Stuart Russell and Eric Wefald. Principles of metareasoning.Artificial Intelligence, 49(1):361– 395, 1991

    1991

  42. [42]

    The pandora’s box problem with sequential inspections, 2025

    Ali Aouad, Jingwei Ji, and Yaron Shaposhnik. The pandora’s box problem with sequential inspections, 2025

    2025

  43. [43]

    Deep think with confidence, 2025

    Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence, 2025

    2025

  44. [44]

    Ai agents as universal task solvers, 2025

    Alessandro Achille and Stefano Soatto. Ai agents as universal task solvers, 2025

    2025

  45. [45]

    e1: Learning adaptive control of reasoning effort.arXiv preprint arXiv:2510.27042, 2025

    Michael Kleinman, Matthew Trager, Alessandro Achille, Wei Xia, and Stefano Soatto. e1: Learning adaptive control of reasoning effort.arXiv preprint arXiv:2510.27042, 2025

    arXiv 2025

  46. [46]

    The gittins index: A design principle for decision-making under uncertainty.arXiv preprint arXiv:2506.10872, 2025

    Ziv Scully and Alexander Terenin. The gittins index: A design principle for decision-making under uncertainty.arXiv preprint arXiv:2506.10872, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Universal sequential search problems.Problems of information transmission, 9(3):265–266, 1973

    Leonid A Levin. Universal sequential search problems.Problems of information transmission, 9(3):265–266, 1973

    1973

  48. [48]

    Department of Energy, 1978

    Martin Weitzman.Optimal search for the best alternative, volume 78. Department of Energy, 1978

    1978

  49. [49]

    Cost-aware bayesian optimization via the pandora’s box gittins index.Advances in Neural Information Processing Systems, 37:115523–115562, 2024

    Qian Xie, Raul Astudillo, Peter Frazier, Ziv Scully, and Alexander Terenin. Cost-aware bayesian optimization via the pandora’s box gittins index.Advances in Neural Information Processing Systems, 37:115523–115562, 2024

    2024

  50. [50]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2 ,

  52. [52]

    2024 amc 12b — problems and solutions

    Art of Problem Solving. 2024 amc 12b — problems and solutions. https: //artofproblemsolving.com/wiki/index.php/2024_AMC_12B, 2024. Administered November 12, 2024

    2024

  53. [53]

    2024 amc 12a — problems and solutions

    Art of Problem Solving. 2024 amc 12a — problems and solutions. https: //artofproblemsolving.com/wiki/index.php/2024_AMC_12A, 2024. Administered November 6, 2024

    2024

  54. [54]

    2024 amc 10b — problems and solutions

    Art of Problem Solving. 2024 amc 10b — problems and solutions. https: //artofproblemsolving.com/wiki/index.php/2024_AMC_10B, 2024. Administered November 12, 2024. 13

    2024

  55. [55]

    2024 amc 10a — problems and solutions

    Art of Problem Solving. 2024 amc 10a — problems and solutions. https: //artofproblemsolving.com/wiki/index.php/2024_AMC_10A, 2024. Administered November 6, 2024

    2024

  56. [56]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. InProceedings of the 36th International Conference on Neural Information Process...

    2022

  57. [57]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

    2023

  58. [58]

    2024 aime i — problems and solutions

    Art of Problem Solving. 2024 aime i — problems and solutions. https: //artofproblemsolving.com/wiki/index.php/2024_AIME_I, 2024. Held January 31–February 1, 2024

    2024

  59. [59]

    2024 aime ii — problems and solutions

    Art of Problem Solving. 2024 aime ii — problems and solutions. https:// artofproblemsolving.com/wiki/index.php/2024_AIME_II, 2024. Held February 7, 2024

    2024

  60. [60]

    2025 aime i — problems and solutions

    Art of Problem Solving. 2025 aime i — problems and solutions. https:// artofproblemsolving.com/wiki/index.php/2025_AIME_I, 2025. Held February 6, 2025

    2025

  61. [61]

    2025 aime ii — problems and solutions

    Art of Problem Solving. 2025 aime ii — problems and solutions. https:// artofproblemsolving.com/wiki/index.php/2025_AIME_II, 2025. Held February 12, 2025. 14 Appendix A Additional Experiments A.1 Problem Difficulty 0.0 0.5 1.0 Cumulative % of Problems 0 1Cumulative % of Tokens Minerva 0.0 0.5 1.0 Cumulative % of Problems 0 1Cumulative % of Tokens MATH500 ...

    2025