{"id":"bb887ba0-c1ec-4afd-b52a-0d3d5d35e47c","arxiv_id":"2608.04001","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.","lead":"This paper argues that test-time scaling in reasoning models should be treated as a family of budgeted inference algorithms over the model's tree of possible continuations, and it offers a three-regime taxonomy plus evaluation and reproducibility guidelines. A reader might care because accuracy numbers in this field are currently hard to compare across papers when the inference protocol is not reported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claims 'over 2 billion full reasoning traces,' but Table 1 and §5 total 1,948,821 traces; the ≈1.8B figure in §G.5 counts token positions, not traces. The release contribution is misstated by roughly 1000x.","rationale":"The reader correctly flagged the trace-count mismatch in the rationale, and that is the sharpest verifiable defect. The reader's stated weakest assumption, however, is the additivity of cost and the existence of a stable proposal distribution q_pi. I read those sections as already careful: §2.3 defines i.i.d. banks as the canonical case and Appendix B extends to heterogeneous proposals; §3.2 and §3.4 explicitly warn that dependent banks are descriptive summaries; §3.3 says incommensurate costs should be reported separately. Thus the formal machinery is hedged exactly where the reader's concern bites, and the central recommendation—report the full protocol—survives even if costs are non-additive or q_pi is not i.i.d. The trace-count error, by contrast, is unhedged and internally inconsistent: abstract '2 billion traces' vs. body '1,948,821 traces', with the only plausible 2-billion-scale number being completion positions. Since the trace corpus is one of the paper's four stated contributions and the paper itself argues that precise artifact accounting is a reproducibility requirement, this is the most load-bearing concern to settle. The existing CONDITIONAL verdict already requires correction of this overclaim and artifact anchoring, so the verdict need not move.","tokens_in":53649,"tokens_out":6884,"duration_ms":65265,"concrete_test":"Run a release audit: enumerate all trace records in the harimo/scorio dataset (or sum Table 1: 500,661 + 192,000 + 104,160 + 1,152,000 = 1,948,821) and compare with the abstract's '2 billion' claim. Also locate the '1,832,922,842' figure in §G.5 to confirm it labels completion positions rather than traces. If the record count is ≈1.95M, the abstract must be revised to state '≈1.95M traces' or, if intended, '≈1.8B token positions' with the correct units.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central argument—that benchmark numbers depend on the full inference protocol and should be reported together with checkpoint, decoder, controller, reducer, budget, and uncertainty—is well supported and does not require the strong assumption the reader singles out. Additive cost and a stable i.i.d. proposal are used mainly for the canonical leaf-level template; §2.3 and §3.2 explicitly cover heterogeneous banks, search-induced dependence, and descriptive finite-bank statistics, while §3.3 directs that incommensurable costs be reported separately. That weakness is acknowledged and is not the load-bearing issue. The load-bearing defect is the headline release statistic. The abstract states 'assemble over 2 billion full reasoning traces'; the Introduction says 'over 2 million'; Table 1 and §5 give a total of 1,948,821 traces. The only '2 billion' figure in the paper is §G.5's 1,832,922,842 completion positions in one compact snapshot—a different quantity, not traces. A 1000x overstatement in a central contribution, in a paper whose §3.5 argues that artifact counts and hashes are reproducibility requirements, is an internal inconsistency that must be corrected before the release claims are credible. The remaining limitations (unavailable dataset-hub revisions for older blocks, absent commit hash, ordinal verifier calibration caveats, effort/cap confounding) are honestly disclosed and do not threaten the framework.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a systematic account of test-time scaling for reasoning LLMs, formalizing it as budgeted inference over the implicit prefix tree of an autoregressive model and distinguishing single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level search. It proposes evaluation principles that treat the entire inference system as the evaluated object, introduces a discovery–stability profile that unifies repeated-sampling metrics, distinguishes exact replay from distributional reproducibility, and specifies protocol-matched reporting of compute and uncertainty. The framework is instantiated on MMLU-Pro, BBH, competition mathematics, and SuperGPQA with a large released corpus of reasoning traces, and the empirical sections include shared banks, bootstrap intervals, verifier diagnostics, and explicit caveats about token caps and effort confounding.","tokens_in":53945,"tokens_out":5084,"duration_ms":48262,"significance":"If the claims hold, the paper provides a genuinely useful vocabulary and evaluation protocol for test-time scaling: the three-regime taxonomy, the shared-bank versus end-to-end distinction, and the discovery–stability profile are well designed and fill a real gap in the literature. The empirical study is unusually careful: bootstrap intervals are propagated through the actual aggregation protocol, candidate banks are shared across reducers, token-cap rates and length limits are disclosed, verifier limitations (reference-assisted versus reference-free, ordinal rather than calibrated) are stated, and per-candidate seeds and serialization are specified. These practices are exactly the kind the paper argues for, and the release, once its size is stated correctly, will be a reusable resource. The central argument that reported performance depends on the full inference protocol is well supported by the formalization and the experiments.","major_comments":[{"comment":"The Abstract states “assemble over 2 billion full reasoning traces” and the Introduction states “over 2 million reasoning traces,” but Table 1 and Section 5 report a total of 1,948,821 traces. The only 2-billion-scale number in the paper is the 1,832,922,842 “completion positions” in the compact Parquet snapshot described in Section G.5, which is a count of token positions, not full reasoning traces; the Introduction’s “over 2 million” also exceeds the tabulated total of 1,948,821. Because the release is a principal stated contribution and the paper itself (Section 3.5) argues that artifact counts are reproducibility requirements, this internal inconsistency is load-bearing. Please correct both statements (for example, “1,948,821 traces, including over 1.8 billion completion positions in the signal-rich snapshot”) and verify that every occurrence count in the abstract, introduction, body, and conclusion agrees with the release census.","section":"Abstract and Section 1 (release-size claim)"}],"minor_comments":[{"comment":"The default Beta(1,1) prior for the latent success probabilities p_q is a modeling choice; a short sensitivity check (for example, Beta(2,2) or an empirical-Bayes prior derived from an auxiliary bank) would strengthen the claim that the discovery–stability profile comparisons are not prior-driven.","section":"Section 3.4 (Beta prior)"},{"comment":"The decomposition C_total = C_gen + C_eval assumes additive costs; since Section C.3 correctly notes that tree-structured kernels can reuse key–value caches and change the wall-clock frontier, the main text should state explicitly that the decomposition is a reporting convention rather than a physical identity, and that incommensurable costs should be reported separately.","section":"Sections 2.1 and 3.3 (cost accounting)"},{"comment":"For the broad knowledge and symbolic reasoning block, the dataset-hub revisions are unavailable and no chat template is recorded; these omissions are honestly disclosed, but given Section 3.5’s exact-replay requirements, please mark each release block as supporting exact replay, distributional reproducibility, or both, so readers know which artifacts are sufficient for which claim.","section":"Section G.3 (exact replay metadata)"},{"comment":"The 16.12% token-cap rate for gpt-oss high effort is correctly reported, but the cap rate should appear alongside the main accuracy numbers in the main text rather than only in the appendix, since it directly qualifies the effort comparison and is a model of the disclosure the paper recommends.","section":"Section 4.3 and Table 14 (effort confounding)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the empirical care is commendable. The only blocking issue is the release-size misstatement; if the authors correct the abstract and introduction counts and align all occurrences with the release census, I would support acceptance. I have no citation-pattern or novelty concerns beyond what is stated in the public report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper. It gives the field what it actually needs: a vocabulary for test-time scaling as budgeted inference over the implicit prefix tree, a clean separation of end-to-end system performance from candidate-bank diagnostics, and reproducibility standards that distinguish exact replay from distributional reproducibility. The discovery–stability profile is a neat unification of Pass@k, pass_k, Maj@k, and mG-Pass@k, and the energy–gate decoder decomposition cleanly separates support from measure. None of this is earth-shattering alone, but the packaging is genuinely new and likely to be adopted.\n\nThe empirical work is better than most in this area: bootstrap intervals, paired subset replays, shared banks, disclosure of token-cap rates, separate reporting of verifier limitations, and a release with per-row provenance. The authors are honest that three mis-rendered gpt-oss captures are medium-effort repeats, that CompassVerifier is reference-assisted, and that their pointwise score is ordinal, not calibrated. That transparency earns credit.\n\nNow the soft spots, in proportion. The abstract says \"over 2 billion full reasoning traces.\" The Introduction says \"over 2 million,\" and Table 1 totals 1,948,821 traces. The only 2-billion figure in the paper is Section G.5's 1,832,922,842 completion positions in one compact snapshot — positions, not traces. That is a 1000x overstatement of the headline release, in a paper that argues artifact counts and hashes are reproducibility requirements. This is the load-bearing problem. It is fixable with a corrected number and clearer wording, but it is exactly the kind of internal inconsistency that makes reviewers distrust the rest. Also, the release is not anchored with a commit hash of the dataset-hub repository, and older blocks lack dataset-hub revisions; those are disclosed but weaken the exact-replay promise.\n\nThe stress-test note's other concern — that additive cost and i.i.d. proposal assumptions are load-bearing — I disagree with. The paper explicitly covers heterogeneous banks, dependent banks from search, and the need to report incommensurable costs separately. The i.i.d. reading is used for the canonical leaf template and is marked as descriptive for other cases. That is fine.\n\nMy bottom line: send it out. It deserves serious referee time, but the decision should be conditional until the trace count is corrected and artifact versions are pinned. Evaluation and reproducibility researchers in reasoning LLMs will get real value from this paper; the framework and empirical core hold up.","headline":"Careful framework paper whose core argument holds, but the headline release count is wrong by three orders of magnitude and must be corrected before the corpus claims are credible.","tokens_in":54548,"tokens_out":2376,"would_cite":true,"duration_ms":23726,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Test-time scaling results are properties of the full inference protocol, not of a scalar compute budget.","keywords":["test-time scaling","reasoning LLMs","inference protocols","evaluation","reproducibility","discovery-stability profile","repeated-sampling metrics","prefix-tree search"],"falsifier":"For a fixed checkpoint and question set, generate two response banks that differ only in decoder gate order (top-p then top-k versus top-k then top-p) or only in the reducer, then compare accuracy across many prompts under protocol-matched bootstrap intervals; if the protocol changes move reported accuracy by less than the uncertainty in repeated runs, the central claim of protocol dependence is weakened, while systematic shifts beyond the intervals would confirm it.","tokens_in":1976,"feed_emoji":"🧠","tokens_out":2008,"duration_ms":78441,"temperature":0.7,"pith_summary":"This paper argues that \"test-time scaling\" is not a single method but a family of budgeted inference algorithms over an autoregressive model's implicit prefix tree, and that measured performance depends on the entire inference system: checkpoint, prompt, decoder, search or aggregation controller, verifier, budget, and stopping rule. It distinguishes three structural regimes—single-trajectory sequential scaling, leaf-level sampling with terminal reduction, and prefix-level search—and provides an evaluation profile whose coordinates recover or bound common repeated-sampling metrics such as Pass@k and majority-vote consistency. If the paper is right, comparing reasoning systems requires reporting the full inference protocol together with any accuracy number, not just a scalar budget. The paper supports this with fixed response banks across broad knowledge, symbolic reasoning, and competition mathematics, and releases a large corpus of full reasoning traces for reuse.","feed_headline":"A single budget number cannot describe test-time scaling","feed_subtitle":"Formalizes three scaling regimes and makes Pass@k, voting, and verifier metrics coordinates of one evaluation profile with protocol…","key_machinery":"The load-bearing object is the implicit prefix tree $T(x)$ induced by an autoregressive model $p_\\theta$ from prompt $x$, with terminal leaves $L(x)$; every test-time algorithm is viewed as a budgeted procedure $A_B$ that interleaves primitive operations (token generation, prefix expansion, verifier or judge calls, terminal reduction) under a cumulative cost bound $B$, yielding an induced proposal distribution $q_\\pi(y|x)$ over completed leaves. The evaluation machinery is the discovery–stability profile $S_{k,t} = \\frac{1}{Q}\\sum_q \\kappa_{k,t}(p_q)$, where $\\kappa_{k,t}$ is the binomial tail probability that $k$ fresh attempts yield at least $t$ correct candidates; Pass@k, pass_k, majority-vote consistency, and related metrics are coordinates, functionals, or bounds of this profile. The framework also decomposes protocol cost as $C_{\\mathrm{total}} = C_{\\mathrm{gen}} + C_{\\mathrm{eval}}$ with $C_{\\mathrm{eval}} = C_{\\mathrm{signal}} + C_{\\mathrm{control}} + C_{\\mathrm{decision}}$, and an energy–gate decomposition of token decoders defines the proposal distribution induced by truncation and reweighting rules.","core_discovery":"The paper's central claim is that test-time scaling should be formalized as a family of budgeted inference algorithms over the implicit prefix tree of an autoregressive model, with performance treated as a property of the complete inference system rather than of model weights or a single scalar budget. It distinguishes three regimes: single-trajectory sequential scaling, where one active path is extended or revised; leaf-level scaling, where a bank of completed candidates is reduced by voting, verifier selection, or minimum Bayes risk; and prefix-level scaling, where search over unfinished states uses continuation-value estimates and controllers such as beam, best-first, or MCTS-style search. The paper then develops evaluation principles: end-to-end system performance $M_B$ and candidate-bank diagnostics are different estimands; the discovery–stability profile $S_{k,t}$ unifies repeated-sampling metrics; and reproducibility is separated into exact replay versus distributional reproducibility, with uncertainty propagated through the actual aggregation and stopping protocol. Empirically, the paper shows on fixed response banks that candidate discovery outpaces selection: median Pass@80 reaches 82% while the all-correct coordinate falls to 15%, and swapping one reducer for another can move a configuration's accuracy from 75.6% down to 65.8% as the bank grows.","pith_inferences":["This suggests a practical test: if every leaderboard entry carried a protocol fingerprint (prompt hash, decoder gate order, verifier identity, budget decomposition, stopping rule), many apparent ranking disagreements between studies would likely resolve into protocol differences rather than model differences.","The profile's discovery–stability shape also invites a two-axis performance comparison: plotting Pass@k against all-correct consistency would make the trade-off between finding a correct answer and finding it reliably the explicit object of comparison, rather than hiding it in a scalar.","Because the additive-cost premise is the framework's weakest point, a natural extension is to replace token-count budgets with wall-clock budgets under tree-structured inference kernels that share key–value cache; the paper notes such sharing can change the practical compute frontier but does not build it into the formal accounting."],"forward_implications":["Comparing reasoning models by a scalar budget and an accuracy number alone is insufficient; reports should specify the checkpoint, prompt, decoder, controller or reducer, verifier or judge, budget decomposition, and stopping rule together.","Common repeated-sampling metrics are not competing measures but coordinates or bounds of one discovery–stability profile, so they can be reported coherently and related to each other.","Shared-bank comparisons isolate the aggregation stage, end-to-end comparisons measure the deployed system, and compute accounting must include generation, evaluation, control, and decision costs rather than only sample counts.","Released response banks allow new reducers and verifiers to be evaluated without regenerating candidates, supporting exact replay of the paper's analyses.","Distributional reproducibility requires uncertainty estimates that propagate through the aggregation and stopping protocol, such as prompt bootstrap intervals computed after the reducer, rather than a single seeded run."],"supporting_citations":[{"why":"Supplies the implicit prefix tree view and the framing of inference-time algorithms as meta-generation that the three-regime taxonomy extends.","marker":"(Welleck et al., 2024)"},{"why":"Defines self-consistency over multiple samples, which the paper re-derives as empirical MBR and places as a leaf-level reducer.","marker":"(Wang et al., 2022)"},{"why":"Provides the compute-optimal test-time scaling result that the paper's evaluation principles reinterpret as one regime requiring protocol reporting.","marker":"(Snell et al., 2025)"},{"why":"Supplies the budget-forcing single-trajectory method used as the canonical example of sequential scaling.","marker":"(Muennighoff et al., 2025)"},{"why":"Supplies verifier-based selection and the generate-test-select reduction pattern central to leaf-level scaling.","marker":"(Cobbe et al., 2021)"},{"why":"Supplies the Pass@k metric, which the paper recovers as the first coordinate of the discovery–stability profile.","marker":"(Chen et al., 2021)"},{"why":"Supplies process reward models used as canonical prefix-level evaluators in the search formalism.","marker":"(Lightman et al., 2023)"},{"why":"Documents judge biases and protocol sensitivities that support treating the judge as part of the evaluated system.","marker":"(Zheng et al., 2023)"},{"why":"Supplies token-level confidence features and the within-rollout stopping template used in the signal-rich trace release.","marker":"(Fu et al., 2026)"}],"fun_headline_variants":["Test-time scaling: three regimes, not one scalar budget","One budget can't describe test-time scaling's three regimes","Discovery outpaces selection in test-time scaling banks","Reducer choice swings accuracy in test-time scaling","Test-time scaling splits into exact replay and distributional reproducibility"],"cache_read_input_tokens":56576,"weakest_assumption_plain":"The framework assumes that every test-time algorithm can be broken into primitive operations whose costs add up, and that the process gives a stable distribution over completed candidates; if sharing computation between prefixes or stop-when-you-want rules makes costs non-additive or makes the candidates dependent, the quantitative claims need to be reworked.","fun_headline_variants_meta":{"raw":{"variants":["Test-time scaling: three regimes, not one scalar budget","One budget can't describe test-time scaling's three regimes","Discovery outpaces selection in test-time scaling banks","Reducer choice swings accuracy in test-time scaling","Test-time scaling splits into exact replay and distributional reproducibility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3162,"prompt_tokens":1085,"completion_tokens":2077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":701,"completion_tokens_details":{"reasoning_tokens":2001}},"tokens_in":701,"tokens_out":2077,"duration_ms":15463,"temperature":1.0,"reasoning_tokens":2001,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:44:15.450903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a fixed checkpoint and question set, generate two response banks that differ only in decoder gate order (top-p then top-k versus top-k then top-p) or only in the reducer, then compare accuracy across many prompts under protocol-matched bootstrap intervals; if the protocol changes move reported accuracy by less than the uncertainty in repeated runs, the central claim of protocol dependence is weakened, while systematic shifts beyond the intervals would confirm it.","supporting_citations":[],"review_version":1}