{"id":"a8521902-8f3f-434f-90dc-313b24af75a5","arxiv_id":"2501.11651","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A training pipeline called T1 scales RL for LLM reasoning via diverse sampling and rewards, and claims that longer allowed generations directly improve math accuracy without verifiers.","lead":"T1 is a reinforcement learning recipe for open language models that combines heavy oversampling, high temperature, an entropy bonus, and targeted penalties to push math reasoning accuracy up, and to make accuracy improve when the model is allowed to generate longer answers. The authors also propose truncating a model's own chain of thought as a way to measure test-time scaling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The truncation-based inference-scaling measure in Section 4 conflates longer reasoning with increasing visibility of the final answer, so the central causal claim is not yet established.","rationale":"The paper is a serious, reproducible engineering contribution: it releases weights and data, and Figure 7's max-generation-length experiment is a cleaner test-time scaling check than the truncation protocol. Read in good faith, the authors are trying to demonstrate that RL-trained T1, unlike its SFT initializer, turns extra allowed tokens into better answers. For that claim to hold, accuracy must rise with reasoning length even when the final answer is not increasingly visible. The truncation-and-summarization protocol does not ensure this: longer prefixes contain more of the CoT and eventually the answer itself, and the summarizer is a strong base model that can extract the answer. Figure 7 has the same structural issue because short max-length settings truncate responses before the answer. The reader's weakest_assumption identifies exactly this confound, and I agree. The right response is not rejection: the benchmark numbers, the open release, and the qualitative training curves provide independent support, and a masked-answer control could confirm or refute the causal interpretation. Hence the verdict stays CONDITIONAL/UNCHANGED.","tokens_in":28436,"tokens_out":4720,"duration_ms":53772,"concrete_test":"Run a masked-answer control for Figure 6. For each T1 response on AIME and Omni-MATH-500, locate the final-answer block (e.g., text after the last occurrence of an answer marker such as 'Answer:') and truncate at each 10% milestone while excluding that block from the summarizer's input. Recompute the ten accuracy points with the base model as φ. If the accuracy curve flattens or loses its monotone trend once the answer is masked, the observed scaling is largely mechanical; if it still rises, the 'longer thinking helps' claim survives this particular confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim — 'increased inference budgets directly lead to T1's better performance' — rests on the Section 4 truncation-and-summarization protocol, and that protocol does not separate the amount of reasoning from the visibility of the final answer. For each T1 response, the truncated prefixes y_{l_i} grow monotonically toward the full response, which ends with the final answer (see the Appendix examples, where the answer appears after 'Final Answer' or 'Answer'). The summarizer φ is the base Qwen2.5-32B, so as i increases, the prompt handed to φ contains strictly more of T1's chain-of-thought and, once i is large enough, the answer string itself. The monotone accuracy curves in Figure 6 therefore conflate 'more thinking' with 'more of the correct answer is extractable.' The same confound affects Figure 7: increasing max_gen_length primarily lets responses that were previously cut off reach their final answer, so the gain can be explained by completion rather than deeper reasoning. Because the same truncation idea is reused to define 'key steps' in the case study, the confound is load-bearing for the headline claim, not a peripheral evaluation detail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces T1, a reinforcement learning (RL) training recipe for math reasoning built from open base models (GLM-4-9B, Qwen2.5-14B, Qwen2.5-32B). The pipeline first initializes the policy with synthesized chain-of-thought data containing trial-and-error and self-verification, then applies RL with large oversampling (K=64), a token-level entropy bonus, on-policy KL normalization with an EMA reference, and penalties for repetitive, overlong, or garbage outputs. The authors report strong benchmark results, including 92.4 on MATH500, 50.6 on AIME2024, and 49.6 on Omni-MATH-500 for T1 (Qwen2.5-32B), and they propose a truncation-and-summarization protocol in Section 4 to measure inference scaling, claiming that longer thinking directly improves accuracy without external verification.","tokens_in":28675,"tokens_out":6743,"duration_ms":74908,"significance":"If the empirical claims hold, T1 is a valuable open contribution: it provides a concrete, reproducible RL recipe, releases model weights and data, and demonstrates that RL with exploration can substantially improve mathematical reasoning over SFT and base models. The ablations on sampling temperature, number of responses, and penalty rewards are informative. However, the paper's central conceptual claim about inference scaling rests on a measurement protocol that conflates longer reasoning with increasing visibility of the final answer, so the causal interpretation of Figure 6 is not yet established. The benchmark results are plausible and likely useful to the community, but their strength is limited by test-set hyperparameter selection and the absence of error bars.","major_comments":[{"comment":"The inference-scaling protocol truncates each T1 response y at l_i = i×10% of its length and asks the base model φ to summarize the prefix y_{:l_i} into an answer. Because the full response ends with a final answer, as shown in the Appendix examples, the prefixes are monotonically increasing in how much of the correct answer is visible to φ. The monotone accuracy curves in Figure 6 are therefore compatible with a mechanical explanation: longer prefixes contain more of the chain-of-thought and eventually the answer string itself, so the summarizer's accuracy rises even without additional 'thinking.' The paper's headline claim that increased inference budgets 'directly lead' to better performance is not established by this protocol. The authors should add controls such as truncating from the end, masking or removing the final-answer segment, or comparing random prefixes of the same length.","section":"Section 4, Figure 6"},{"comment":"Figure 7 is a cleaner test than Figure 6 because the policy must generate a complete response under each budget, but it still partially conflates 'more reasoning' with 'response completion': increasing max_gen_length mainly lets responses that would previously have been cut off before the final answer finish, and the reported accuracy is averaged over all responses including truncated ones. The paper does not report the truncation rate at each budget. To support the causal claim, the authors should report the fraction of truncated responses at each max_gen_length and re-analyze accuracy on the subset of responses that complete with their final answer within the budget.","section":"Section 4, Figure 7"},{"comment":"The hyperparameters (temperature, min-p, and K) are selected using MATH500, AIME, and Omni-MATH-500, which are the same test benchmarks reported in Table 1. Appendix A.1 describes no held-out validation split for this selection. The reported advantage over QwQ-32B-Preview may therefore be partly the result of optimizing on the evaluation data, and the benchmark comparisons should be interpreted with caution. The authors should perform hyperparameter selection on a separate validation set or explicitly quantify the overfitting risk.","section":"Section 3.2, Table 2"},{"comment":"The comparison of K=4, 16, and 64 is made for a fixed number of prompts and training steps, so K=64 consumes 16 times more rollout tokens per update than K=4. The faster reward growth and higher accuracy in Figure 3 can therefore be attributed to increased compute rather than to sampling diversity. An ablation that matches total sampled responses—for example, K=4 with correspondingly more prompts or training steps—is needed to support the claim that oversampling 'encourages exploration' is the causal factor behind the improvements.","section":"Section 3.2, Figure 3"},{"comment":"Table 1 reports a single pass@1 number for most benchmarks, with AIME averaged over 32 samples but no variance. Given that AIME2024 contains only 30 problems and that the standard error of a 90% accuracy on 500 problems is about 1.3%, the differences between T1 and QwQ-32B-Preview are not shown to be statistically significant. Reporting standard errors or confidence intervals would strengthen the empirical claims.","section":"Table 1 and Appendix A.1"}],"minor_comments":[{"comment":"The abstract and Section 4 state that 'T1 with open LLMs as its base' exhibits inference scaling, but Figure 6 and Figure 7 present inference-scaling results only for Qwen2.5-32B; the authors should clarify whether the claim applies to all base models or only to the 32B variant.","section":"Abstract and Section 4"},{"comment":"There is a typo in 'performance gainss' near the discussion of high sampling temperatures.","section":"Section 2.2.2"},{"comment":"The notation uses K for the number of sampled responses in the text but k in the leave-one-out normalization formulas; this should be made consistent.","section":"Equation (2) and Equation (4)"},{"comment":"The construction of Omni-MATH-500 by sampling 500 examples is not described with a seed or sampling procedure, which limits reproducibility of that benchmark subset.","section":"Appendix A.1"},{"comment":"The bottom panel labels such as 'Select the Best One as Answer -2' and 'Answer-2' are difficult to parse; a clearer diagram or caption would help.","section":"Figure 5"},{"comment":"The inference engine is called 'SGLANG' in the text but is commonly referred to as 'SGLang'; this should be corrected.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The training recipe and benchmark gains are likely to be of interest to the community, and the public release of weights and data is a real strength. However, the paper's central inference-scaling claim is confounded in the proposed measurement, and the requested controls (end-truncation, masking final answers, truncation-rate analysis, validation-set hyperparameter selection) are feasible within the manuscript's scope. I would not reject the paper, but the revision needs to substantially deconfound the inference-scaling evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The RL recipe is the real contribution here. SFT on synthesized trial-and-error CoT, K=64 oversampling at temperature 1.2, entropy bonus, leave-one-out reward/KL normalization, EMA reference, and the penalty rules form a practical pipeline that yields a genuinely strong open 32B model: 92.4 on MATH500, 50.6 on AIME2024, 49.6 on Omni-MATH-500, beating QwQ-32B-Preview on those sets. The authors ship weights and training data, and the ablations on K, temperature, and penalties are useful. That alone makes the paper worth engaging with.\n\nThe soft spot is exactly where the stress-test lands. The Section 4 truncation protocol cuts a single response at i×10% of its length and asks the base model to summarize the prefix into an answer. As the prefix lengthens, it contains strictly more of the chain-of-thought and eventually the final answer string itself (the appendix examples confirm the answer appears near the end after 'Answer'). So the monotone curves in Figure 6 are partly mechanical: the summarizer is increasingly likely to see or reconstruct the correct answer. That confound is load-bearing for the abstract's claim that \"increased inference budgets directly lead to better performance.\" Figure 7's max-generation-length experiment is cleaner—it actually varies how much the model is allowed to generate—and it does show a real effect that grows with RL training. But the paper leans on the truncation analysis for its headline causal statement, and that statement is not established by the evidence.\n\nTwo smaller issues: most reported numbers have no error bars (AIME is averaged over 32 runs, which is good, but MATH500 and Omni-MATH-500 are single greedy passes), and hyperparameters like τ, K, min-p, and the penalty thresholds appear to be selected on the evaluation benchmarks themselves, with no explicit held-out validation split. That inflates the apparent gains somewhat, though the margins are large enough that overfitting alone is unlikely to explain them.\n\nThis is a solid empirical paper with an overstated interpretation. It deserves peer review: the recipe is reproducible, the model is open, and the inference-scaling question is important even if the current measurement does not answer it cleanly. Send it to referees, but require a control that isolates reasoning length from answer visibility—e.g., withholding the final answer from the summarizer or using a separate answer extractor—and ask for error bars or a validation split. The training results will stand; the causal claim needs fixing.","headline":"The T1 RL recipe is real and the open 32B model posts impressive math numbers, but the headline inference-scaling claim rests on a truncation protocol that partly just reveals the already-written final answer.","tokens_in":29244,"tokens_out":2113,"would_cite":true,"duration_ms":24952,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"T1, an open-weights model trained with exploration-heavy reinforcement learning, demonstrates that allowing longer generation directly improves math reasoning accuracy with no extra verifier, and tops prior open models on MATH500…","keywords":["reinforcement learning","inference scaling","test-time scaling","LLM reasoning","chain-of-thought","oversampling","math reasoning","exploration"],"falsifier":"Run a length-matched control in which the middle of each truncated thinking prefix is replaced by the same number of tokens of unrelated or repeated text before summarization; if accuracy still rises with prefix length, the effect is token quantity rather than reasoning content, and if it collapses, longer thought is what drives the curve. A second check is to strip the last 10% of every response before truncating, so the final answer can never appear in the prefix, and see whether the monotone accuracy increase persists.","tokens_in":28210,"feed_emoji":"🧠","tokens_out":8769,"duration_ms":84720,"temperature":0.7,"pith_summary":"T1 is a training recipe that makes a large language model's reasoning accuracy improve as it is allowed to generate more tokens, with no external verifier involved. The recipe's first move is supervised fine-tuning on chain-of-thought data that deliberately contains wrong attempts, reflections, and self-verification, widening the space of reasoning paths the model can produce. The second move is reinforcement learning with heavy exploration: 64 sampled responses per prompt, high sampling temperature, an entropy bonus, and a slackened, on-policy KL penalty, plus a penalty for repetitive or garbled output. The authors' central claim is that this exploration-heavy RL, rather than imitation of correct steps, is what creates inference scaling, and they support it by showing that accuracy rises monotonically with the length of truncated thinking prefixes on AIME2024, Omni-MATH-500, and MATH500. On those benchmarks, the 32B T1 model also surpasses prior open models, including QwQ-32B-Preview, which would make inference scaling an open-model property rather than a closed-system one.","feed_headline":"Longer thinking directly lifts math scores after RL scaling","feed_subtitle":"A 32B open model trained with exploration-heavy RL beats QwQ and improves as generation budget grows.","key_machinery":"The machinery is the T1 reinforcement-learning pipeline together with a truncate-and-summarize inference-scaling protocol. The pipeline's load-bearing parts are: SFT initialization on synthesized chain-of-thought that includes trial-and-error, errors, and verification; $K=64$ oversampling per prompt with temperature around 1.2; leave-one-out reward normalization following RLOO; a token-level entropy bonus in the loss; KL normalization with an EMA-updated reference model; and a $-1$ reward for repetitive, overlong, or garbled responses. The protocol cuts each completed response at $i\\times 10\\%$ of its length and asks the base model to summarize the truncated prefix into an answer, making the average token count of the prefix the independent variable. The identity that carries the argument is the monotone accuracy curve against prefix length after enough RL training, which the paper reads as evidence that more thinking tokens directly buy better answers.","core_discovery":"On the paper's own terms, the discovery is that test-time scaling can be trained into an open LLM by scaling reinforcement learning, not by repeated sampling with a verifier. T1's accuracy on AIME2024 climbs from about 24% to 50% as the average number of thinking tokens in a truncated single response grows, with similar monotone trends on Omni-MATH-500 and MATH500, and the slope of that accuracy-versus-length curve steepens as RL training progresses. The measurement protocol is the basis of the claim: each full response is cut at 10% increments of its length and the base model summarizes the prefix into an answer, so longer prefixes contain more of the reasoning process. The paper reports that the same protocol shows almost no inference scaling for the SFT model or for a policy stopped at 30% of RL steps, while fully trained policies gain substantially from longer generation, and that these gains require no additional verification signal.","pith_inferences":["A testable extension: re-run the truncation protocol with a summarizer asked to output only the currently established conclusion, not the final answer, to separate reasoning content from answer visibility; the monotone curve should flatten if the measured effect is partly answer leakage.","The same exploration-heavy recipe could apply to non-math domains if a dense reward signal existed, but the paper only demonstrates out-of-domain transfer to GPQA, where its training signal is math correctness, so transfer to open-ended reasoning remains undemonstrated.","For serving economics, T1's result implies that part of the accuracy budget can be shifted from verifier calls to raw generation tokens, changing the compute trade-off for deployed reasoning models; this is an economic consequence the paper does not discuss."],"forward_implications":["With a fixed inference budget, accuracy increases monotonically as RL training proceeds, so training compute and inference compute are complementary inputs to T1's reasoning performance.","A single long generation replaces the repeated-sampling-plus-verifier pattern: the measured scaling requires no external reward model or verifier at test time.","Harder problems benefit more from extra thinking: AIME2024 shows the largest relative gain as maximum generation length grows, while MATH500 gains the least.","Inference scaling appears only after sufficient RL training, so the scaling property is acquired through the training recipe rather than inherited from the base model."],"supporting_citations":[{"why":"This work defines the o1-style RL-plus-inference-scaling target that T1 aims to reproduce in open models.","marker":"(OpenAI, 2024)"},{"why":"This work provides the repeated-sampling view of test-time scaling that T1's single-long-generation analysis contrasts with.","marker":"(Snell et al., 2024)"},{"why":"This work shows the repeated-sampling baseline for scaling inference compute that T1 says does not update the policy.","marker":"(Brown et al., 2024)"},{"why":"This work documents prior RLHF scaling limits that motivate T1's exploration-focused RL design.","marker":"(Hou et al., 2024)"},{"why":"This work supplies a representative RL-on-math baseline whose modest gains T1 aims to beat.","marker":"(Shao et al., 2024)"},{"why":"This work is the source of the leave-one-out reward normalization used in Equation 2.","marker":"(Ahmadian et al., 2024)"},{"why":"This work supports the claim that models already contain strong sampling spaces, motivating $K=64$ oversampling.","marker":"(Li et al., 2024a)"},{"why":"This work represents the imitation and CoT approach whose trial-and-error elements T1's SFT initialization extends.","marker":"(Zelikman et al., 2022)"}],"fun_headline_variants":["RL scaling trains test-time thinking gains in open LLM","Thinking longer boosts math: RL scaling does it","Bigger inference budget, better math after RL","Open LLM learns to scale reasoning at test time","Trained inference scaling: more thinking, more math accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the truncate-and-summarize protocol measures how much thinking helps, not how much of the final answer is already visible in the prefix, so the rising accuracy with longer prefixes is interpreted as inference scaling rather than a mechanical artifact.","fun_headline_variants_meta":{"raw":{"variants":["RL scaling trains test-time thinking gains in open LLM","Thinking longer boosts math: RL scaling does it","Bigger inference budget, better math after RL","Open LLM learns to scale reasoning at test time","Trained inference scaling: more thinking, more math accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1314,"prompt_tokens":876,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":492,"tokens_out":438,"duration_ms":5329,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:00:36.852042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a length-matched control in which the middle of each truncated thinking prefix is replaced by the same number of tokens of unrelated or repeated text before summarization; if accuracy still rises with prefix length, the effect is token quantity rather than reasoning content, and if it collapses, longer thought is what drives the curve. A second check is to strip the last 10% of every response before truncating, so the final answer can never appear in the prefix, and see whether the monotone accuracy increase persists.","supporting_citations":[{"cited_title":"Learning to reason with llms","cited_arxiv_id":null,"evidence_quote":"This work defines the o1-style RL-plus-inference-scaling target that T1 aims to reproduce in open models."},{"cited_title":"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms","cited_arxiv_id":null,"evidence_quote":"This work is the source of the leave-one-out reward normalization used in Equation 2."}],"review_version":1}