{"id":"74e600cf-1c16-4355-ab03-c5b0fb18fa5e","arxiv_id":"2505.24273","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.","lead":"Researchers tested whether and how many backtracking steps should appear in the data used to warm up reinforcement learning for AI reasoning. They found harder tasks need more backtracking examples, and whether the example answers are correct matters less than the reasoning structure.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The difficulty–backtrack-count correlation rests on three tasks that differ in solver type, trace length, and reward structure, not on a controlled difficulty axis; the observed ordering may reflect task-specific trace length rather than task difficulty.","rationale":"Reader's weakest assumption named synthetic detour operationalization and the three-task difficulty ranking. I focus on the latter because it is the one that, if wrong, directly invalidates the abstract's second finding and the practical recipe. The synthetic backtrack construction is actually a strength: it is a controlled, search-derived intervention that isolates structure from content, and the paper separately shows shuffled data destroys RL, so the detour manipulation is not vacuous. The gap is inferential: with N=3 tasks, all differing along multiple dimensions, the claimed monotone relation between difficulty and backtrack count is underdetermined. This is a correctness-risk issue rather than an internal inconsistency. The paper's own limitations section acknowledges toy datasets and single model family, but does not flag the three-task extrapolation. A within-task difficulty sweep would settle it. If it fails, the paper's strongest claim should be downgraded to a task-specific observation; if it passes, the practical recipe gains real support. Separately, missing error bars and seeds mean even the existing three-task optima are point estimates, but the difficulty-controlled test is the decisive check. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":15148,"tokens_out":3566,"duration_ms":48653,"concrete_test":"Run the same backtrack sweep on a single task with controlled difficulty levels. For example, generate Sudoku puzzles with 25, 40, and 55 empty cells (or Countdown puzzles with 4, 5, and 6 numbers), keep the solver and trace template fixed, and train RL from SFT warm-ups with 0, 1, 5, and 10 backtracks at each difficulty level. If the optimal backtrack count increases monotonically with difficulty within the same task, the correlation claim is supported; if it stays flat or varies non-monotonically, the three-task pattern is task-specific. This test also controls for solver type and trace length differences that currently confound the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most generalizable claim—'more challenging problems with larger search space tend to need higher numbers of backtracks during the SFT stage'—is inferred from exactly three tasks (Countdown, Arc 1D, Sudoku) whose optimal backtrack counts are 1, 0, and 5. These tasks differ simultaneously in solver type (DFS vs heuristic search), trace length (a Sudoku solution fills 30–60 cells even with zero backtracks), branching factor, reward parsing, and baseline accuracy. The paper provides no within-task difficulty manipulation, so 'difficulty' is not isolated from task identity. Low baseline accuracy (e.g., Sudoku 0.000) is used as a difficulty proxy, but it measures the model's prior competence, not properties of the search space that would predict how many backtracks a warm-up should contain. The pattern could equally be explained by trace length or by the number of decision points in the optimal solution: Sudoku's optimal trace is much longer than Arc 1D's, so more inserted detours may simply be needed to make the demonstration long enough for RL to learn from, not because 'harder problems need more backtracking.' The central practical recipe—match backtrack count to task difficulty—therefore overreaches the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how the choice of SFT warm-up data affects subsequent RL post-training for LLM reasoning. It compares cold-start RL against several SFT initializations (self-sampled, distilled, synthetic backtracking, and shuffled) across eight reasoning tasks. The main findings are: (1) short self-sampled CoT warm-ups give moderate gains over cold-start RL on most tasks; (2) the correctness of distilled or self-sampled trajectories has little effect on final RL performance; (3) synthetic backtracks injected into SFT data improve RL training, with the optimal number of backtracks apparently scaling with task difficulty (Arc 1D optimal 0, Countdown optimal 1, Sudoku optimal 5); and (4) shuffled (inconsistent) SFT data severely degrades RL. The paper concludes with a practical recipe: match the backtrack count in SFT warm-up to the difficulty of the task, while content correctness is secondary.","tokens_in":15362,"tokens_out":4494,"duration_ms":55903,"significance":"If the central claims hold, the paper provides actionable guidance for designing SFT warm-up data for verifiable-reward RL: task-matched backtrack density matters more than trajectory correctness. The study is unusually controlled: it uses rule-based rewards, multiple reasoning tasks, and openly released code, and it explicitly contrasts synthetic search-generated traces with distilled traces. The three-task correlation between task difficulty and optimal backtrack count is a striking and potentially useful empirical pattern. However, the evidence for that pattern is currently thin (three tasks, confounded axes, single-run curves), so the generality of the recipe is not yet established. The paper also makes the provocative claim that RL is largely insensitive to the correctness of long CoT traces, which deserves further scrutiny because it is based on comparisons without statistical grounding.","major_comments":[{"comment":"The central claim that more challenging problems need more backtracks is inferred from exactly three tasks (Arc 1D, Countdown, Sudoku) that differ simultaneously in solver type (heuristic search vs DFS), trace length (Sudoku solutions fill 30–60 cells even at zero backtracks), branching factor, reward parsing, and baseline accuracy. The paper uses baseline accuracy as a difficulty proxy, but that measures the model's prior competence, not a property of the search space that would predict backtracking needs. No within-task difficulty manipulation is performed, so the observed ordering (0, 1, 5 backtracks) could equally be explained by trace length or by the number of decision points in the optimal solution. The abstract's claim that 'more challenging problems with larger search space tend to need higher numbers of backtracks' therefore overreaches the evidence. A within-task difficulty sweep (e.g., Countdown with different numbers of input numbers, or Sudoku with different numbers of givens) would be needed to isolate difficulty from task identity.","section":"Section 4.3, 'Optimal number of backtracks scales with problem difficulty'"},{"comment":"The reward trajectories in Figure 4 appear to be single runs, and the text reports no seeds, error bars, or variance. The selection of the 'optimal' backtrack counts (Countdown 1, Arc 1D 0, Sudoku 5) is made by visual inspection of these single curves. The claim that certain backtrack counts yield 'more stable' training or that too many backtracks lead to 'model degeneration' is not supported by any measure of run-to-run variability. Given that the optimal counts are load-bearing for the paper's central recipe, the authors should either report multiple seeds with mean±std, or explicitly temper the stability and optimality claims to single-run observations. At minimum, the number of evaluation and training examples per condition should be stated.","section":"Figure 4(a-c) and Section 4.3 'Optimal number of backtracks'"},{"comment":"The paper operationalizes 'backtracking' as injected incorrect branches in DFS/heuristic search trajectories. The title and abstract speak of backtracking as a general reasoning behavior, and the introduction motivates the work with LLMs' natural 'wait' and 'verify' tokens. However, the controlled experiments only vary the count of synthetic detours, not the form of natural backtracking (e.g., token-level self-correction, partial rewrites, nested retries). If these synthetic traces do not resemble how LLMs naturally backtrack, the measured relationship between backtrack count and RL performance is an artifact of the data construction. The authors should acknowledge this validity gap and, ideally, compare against a condition using naturally occurring backtracking traces or analyze whether the trained models actually produce 'wait'-type tokens after warm-up.","section":"Section 4.3, 'Building synthetic datasets'"},{"comment":"The abstract states that 'longer CoT with backtracks generally induce better and more stable RL training.' This is not supported across the three tasks studied: Arc 1D shows the opposite trend, with the zero-backtrack model achieving 90.8% accuracy and performance declining as backtrack count increases. The paper's own analysis says Arc 1D is 'the easier task among three' and that 'performance declines as the number of backtracks increases.' The general statement should be qualified to reflect the task-dependent pattern, e.g., 'for sufficiently difficult tasks, longer CoT with backtracks can improve RL training, but for easier tasks backtracking may be unnecessary or harmful.' Without this qualification, the abstract misrepresents the findings.","section":"Abstract and Section 4.3, Figure 4d"}],"minor_comments":[{"comment":"The sentence 'It it also worthwhile noting that short CoTs enable two distinct scaling patterns...' contains a typo ('It it' should be 'It is').","section":"Section 4.1"},{"comment":"The text says 'A vanilla The Qwen2.5-3B-Instruct baseline solves none of the test instances.' The phrase 'A vanilla The' should be corrected to 'The vanilla Qwen2.5-3B-Instruct baseline.'","section":"Section 4.3"},{"comment":"The QwQ-32B baseline table lists only six tasks (AG, CD, ARC, SDK, CCR, LF) and omits Zebra Puzzles and Self Reference; if these were not evaluated, this should be stated explicitly, otherwise the table should include all eight tasks.","section":"Table 3"},{"comment":"The rollout lengths for RL are described only as '4k to 8k tokens' or '1k to 2k tokens, depending on the tasks.' Exact values for each task and each SFT condition should be listed, otherwise the experiments are not fully reproducible.","section":"Section 3.3 and Appendix B"},{"comment":"When comparing against QwQ-32B, the paper cites reference [18] (Qwen2.5 technical report) for the Countdown accuracy of 51.5%; the correct citation for QwQ-32B is reference [26] (the QwQ blog).","section":"Section 4.3"},{"comment":"The claim that correct and incorrect SFT lead to 'similar behaviors' and that 'RL prioritizes structural patterns over content correctness' is based on visual inspection of reward curves and point evaluations without error bars or statistical tests; the authors should either add variance estimates or soften the causal language.","section":"Section 4.2, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-motivated empirical study with a valuable central idea, but the evidence for the main generalized claim (difficulty–backtrack-count correlation) is currently too thin: three tasks with multiple confounds and single-run curves. I would encourage the editor to treat the requested revisions as substantive rather than cosmetic. If the authors add within-task difficulty manipulations, multi-seed results, and explicitly scope their claims, the paper could become a solid contribution to the SFT-for-RL literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper asks a practical question that most of us in the RLVR space haven't isolated: how much backtracking should the SFT warm-up contain, and does the optimal amount scale with task difficulty? The authors run a controlled sweep over synthetic backtrack count on three tasks and find a tidy ordering—Countdown wants one, Arc 1D wants zero, Sudoku wants five. The correct-versus-incorrect distillation comparison is also a nice experiment: RL post-training seems indifferent to whether the warm-up trajectories had the right final answer, at least on Countdown. There is real value here for anyone curating SFT data for reasoning models, and the code is promised, which is a good sign.\n\nAnother solid piece is the eight-task comparison showing that short self-sampled CoTs help RL warm-up beyond cold-start, and the shuffled-SFT result—destroying the prompt-completion pairing makes RL fail—gives a clean contrast with the correctness-insensitivity finding.\n\nThat said, the central generalization is not supported by the experiments as written. The difficulty–backtrack-count correlation rests on three tasks that differ in solver type, trace length, and reward parsing. Arc 1D's optimal traces are short; Sudoku's solutions are long even with zero backtracks. The stress-test note is on target: low baseline accuracy is a proxy for the model's prior competence, not for the search-space size that would predict how many detours a warm-up should contain. Without a within-task difficulty manipulation, the ordering could just be reflecting trace length or decision-point count. The abstract also overstates the case when it says longer CoT with backtracks 'generally induce better and more stable RL training'—Arc 1D's own data contradicts that.\n\nThere are smaller issues. The reward curves in Figure 4 look like single runs; there are no variance bands or seed repeats, so we can't tell whether the differences between one and two backtracks are real. The correctness-insensitivity claim leans heavily on Countdown, and the self-sampled results show exceptions (Self Reference). The synthetic backtracking construction is clean but narrow: injected wrong branches in a search tree are not the same as a model's natural 'wait, let me reconsider' behavior, so the external validity is limited.\n\nOn balance, the paper is a serious empirical study with a clear falsifiable design. It deserves a proper peer review, but I'd want the authors to add multiple seeds, report variance, and reframe the difficulty claim as a hypothesis about three specific tasks rather than a general law. If you're building data pipelines for reasoning RL, it's worth reading; just don't update your recipe on the correlation yet.","headline":"A plausible and useful empirical recipe, but the difficulty–backtrack-count law outruns the evidence; worth refereeing with a demand for seeds and a narrower claim.","tokens_in":15899,"tokens_out":3412,"would_cite":true,"duration_ms":39417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that backtracking detours in SFT warm-up data improve RL training, with harder tasks needing more detours, while trajectory correctness barely matters.","keywords":["SFT warm-up","reinforcement learning","chain-of-thought","backtracking","verifiable rewards","reasoning tasks","synthetic data","RLVR"],"falsifier":"Train the same SFT-plus-RL pipeline on additional tasks with known search-space sizes, using backtracks sampled from the model's own 'wait'-style behavior rather than DFS detours; if optimal backtrack count does not rise with search-space size, or if natural and synthetic backtracks diverge, the paper's central difficulty–backtrack relationship fails.","tokens_in":14916,"feed_emoji":"🧩","tokens_out":7433,"duration_ms":77492,"temperature":0.7,"pith_summary":"This paper asks how much backtracking should be built into the supervised fine-tuning (SFT) data that primes a language model before reinforcement learning (RL). Across eight verifiable reasoning tasks, it finds that short chain-of-thought warm-ups help RL moderately, and that this help shrinks as tasks get harder. Using search-generated demonstrations with controlled numbers of injected false branches, it reports that longer trajectories containing backtracks generally lead to better and more stable RL training, and that harder problems need more backtracks. It also reports that whether those warm-up trajectories are correct or incorrect has little effect on final RL performance, while mismatching a prompt with another problem's completion destroys training. If the paper is right, the practical recipe for SFT warm-up data is difficulty-matched backtracking structure rather than trajectory correctness.","feed_headline":"Backtracking in SFT data lifts RL reasoning, and harder tasks need more","feed_subtitle":"Injected synthetic wrong branches speed RL training; harder puzzles need deeper search detours.","key_machinery":"The central object is the synthetic backtracking dataset, constructed by linearizing a search-tree solution and inserting a controlled number of wrong branches as detours: for Countdown and Sudoku, a depth-first-search solver supplies the tree and each backtrack is an incorrect operation or grid fill that is later abandoned; for Arc 1D, a hand-crafted heuristic search creates a detour by choosing an incorrect transformation and retrying. Varying the number of backtracks (0, 1, 2, 3 for Countdown and Arc 1D; 0, 1, 5, 10 for Sudoku) yields SFT warm-up sets that are matched in content correctness but differ in structure. The experimental pipeline then runs PPO or GRPO with rule-based verifiable rewards from each initialization, so differences in reward trajectory and final accuracy isolate the effect of backtrack frequency.","core_discovery":"On its own terms, the paper establishes that backtracking in SFT warm-up data is a tunable, difficulty-dependent resource for RL with verifiable rewards. Cold-start RL already lifts the base Qwen2.5-3B-Instruct model, and self-sampled short CoT warm-ups add moderate further gains, but the gains fall off on harder tasks. When synthetic SFT datasets are built with depth-first search or heuristic search and a varying number of injected detour branches, RL training follows the backtrack count: for Arc 1D (easiest) zero backtracks is optimal at 90.8% accuracy; for Countdown (moderate) one backtrack is optimal at 69.7% accuracy versus 38.9% with zero; for Sudoku (hardest) five backtracks are optimal at 28.9% versus 14.4% with zero, while one or ten backtracks cause degeneration. The paper further claims that correct and incorrect distilled trajectories converge after RL, so content correctness is not the driver, whereas shuffling prompts against completions inside the backtracking data makes RL collapse to near-zero accuracy. In short, RL amplifies the search structure it is initialized with, and the right amount of structure scales with the size of the problem's search space.","pith_inferences":["If the difficulty–backtrack-count relationship holds beyond the three synthetic tasks, SFT curricula could be generated automatically by estimating search-space size and injecting a proportional number of detours, without running expensive RL sweeps.","The paper's correctness-indifference result is specific to rule-based final-answer rewards; a natural test is whether process rewards that score intermediate steps would make trajectory correctness matter again.","Natural LLM backtracks are finer-grained (token-level 'wait' events, partial corrections, nested retries) than whole-branch detours; comparing trace granularity directly would show whether the measured optimum is an artifact of DFS-style detours.","Shuffled-SFT collapse suggests SFT binds a reasoning template to a problem class; a testable extension is shuffling within a difficulty level to see whether binding or difficulty matching drives the effect."],"forward_implications":["SFT warm-up data for RL should be chosen by expected search difficulty: easy tasks are best served by near-optimal traces, while hard tasks gain from several injected backtracking detours.","Trajectory correctness is a second-order property: suboptimal or even incorrect warm-up traces can yield RL performance comparable to correct ones, so data collection effort can be redirected from filtering to structuring.","Internal consistency between prompt and completion is load-bearing: shuffled SFT data with the right structure in the wrong place drives RL accuracy to zero on the controlled tasks.","Backtrack-initialized small models can exceed much larger reasoning baselines on the tested tasks (Countdown 69.7% vs 51.5%, Sudoku 28.9% vs 0.0%, Arc 1D 90.8% vs 24.0% for QwQ-32B).","Too few or too many backtracks are both harmful on hard tasks, so the optimal SFT mix is not simply more search but a difficulty-matched level of search structure."],"supporting_citations":[{"why":"Supplies the prior result that CoT warm-up induces backtracking and larger RL gains; Section 4.1 tests and qualifies that claim across more tasks.","marker":"[6]"},{"why":"Provides the prior study of explicit backtracking traces on Countdown and Sudoku that the synthetic-data experiments extend and control more tightly.","marker":"[16]"},{"why":"Establishes the RL-with-verifiable-rewards setting in which backtracking emerges, motivating the question of how much backtracking SFT should supply.","marker":"[3]"},{"why":"Supplies the eight reasoning task datasets and their evaluation formats used throughout the controlled comparisons.","marker":"[27]"},{"why":"Contributes the 'SFT memorizes, RL generalizes' framing that the correctness-ablation results directly nuance.","marker":"[2]"},{"why":"Provides QwQ-32B as the teacher for distilled SFT trajectories and as the strong baseline that backtracking-initialized models are compared against.","marker":"[26]"}],"fun_headline_variants":["Backtracking in SFT data: more helps harder reasoning tasks, up to a point","RL inherits backtracking structure from SFT: harder puzzles need more detours","Optimal backtrack count in SFT scales with task difficulty for RL","For RL, SFT backtracking is a dial: harder tasks need more turns","Backtracking in SFT: RL learns the search shape, not the answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on treating an injected wrong branch in a search-derived trace as a faithful 'backtrack,' so the measured difficulty–backtrack curve could be an artifact of synthetic data if natural LLM backtracking behaves differently.","fun_headline_variants_meta":{"raw":{"variants":["Backtracking in SFT data: more helps harder reasoning tasks, up to a point","RL inherits backtracking structure from SFT: harder puzzles need more detours","Optimal backtrack count in SFT scales with task difficulty for RL","For RL, SFT backtracking is a dial: harder tasks need more turns","Backtracking in SFT: RL learns the search shape, not the answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00102,"raw_usage":{"total_tokens":4396,"prompt_tokens":1131,"completion_tokens":3265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":747,"completion_tokens_details":{"reasoning_tokens":3160}},"tokens_in":747,"tokens_out":3265,"duration_ms":28021,"temperature":1.0,"reasoning_tokens":3160,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:27:21.972626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same SFT-plus-RL pipeline on additional tasks with known search-space sizes, using backtracks sampled from the model's own 'wait'-style behavior rather than DFS detours; if optimal backtrack count does not rise with search-space size, or if natural and synthetic backtracks diverge, the paper's central difficulty–backtrack relationship fails.","supporting_citations":[{"cited_title":"Reasoning gym","cited_arxiv_id":null,"evidence_quote":"Supplies the eight reasoning task datasets and their evaluation formats used throughout the controlled comparisons."}],"review_version":1}