{"id":"a7e31c38-fb74-4e72-995b-abb2a6d14898","arxiv_id":"2412.12544","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Token-level MCTS with CoT prompting lifts Qwen2.5-Coder-32B-Instruct to a 0.351 pass rate on LiveCodeBench-Hard, approaching O1-Mini's pass@1.","lead":"A ByteDance team shows that token-level Monte Carlo Tree Search, combined with chain-of-thought prompting, lets a 32-billion-parameter open-source coding model match the competition performance of much larger proprietary models. The method reaches a 0.351 pass rate on LiveCodeBench-Hard, close to OpenAI's O1-Mini, but the comparison is complicated by the search's use of public test cases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison to pass@k is confounded: MCTS's reward is public-test pass rate (Eq. 9-10), a signal pass@k baselines never receive, and the 0.305/0.351 results use ~242/229 mean generations (Table 4), so the claimed advantage may be test-time filtering plus a larger budget, not tree search.","rationale":"The central claim depends on comparing MCTS's pass rate to pass@k of larger models. For that comparison to be fair, the baselines must have access to the same information and compute. The paper violates both: public tests are used as rewards in every simulation (Sec 3.2 Simulation), and the specific numbers advertised in the abstract use more than 200 mean generations (Table 4) while being compared to pass@100. The problem is not that the method is invalid—using execution feedback at test time is a legitimate algorithm—but that the paper's evidence does not separate that effect from the structure of the search. Section 4.3's best-path prompt experiment gives partial support that search paths matter, but those paths are found with public-test feedback, so it does not resolve the confound. The reader's conditional verdict is appropriate: a matched filtering baseline and compute accounting would settle the question. No formal verification or released code is provided, so the independent support is limited; the paper's transparency about mean generations is a point in its favor but does not fix the missing controls.","tokens_in":14824,"tokens_out":8963,"duration_ms":80027,"concrete_test":"On LiveCodeBench-Hard with Qwen2.5-Coder-32B-Instruct, run a matched filtering baseline: sample N solutions per problem (N=242 for direct, N=229 for CoT, plus N=100), with temperature 0.7, top_p 0.8, repetition_penalty 1.05; keep the solution with the highest public-test pass rate (first one passing all public tests if any); measure its private-test pass rate. If this baseline meets or exceeds 0.305/0.351 at N≈242, or exceeds the MCTS curve at matched N, then the public-test reward and sample budget, not the tree search structure, explain the headline result. Also record total sampled tokens and number of LLM forward passes for MCTS and the baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"1) The Simulation reward (Eq. 9/10) is computed by running generated code on the public test cases. pass@k baselines in Tables 1 and 3 generate k independent programs and evaluate only on private tests; they never see public-test feedback. Thus MCTS is effectively selecting among generated programs using an oracle that the baselines lack. A trivial 'Best-of-N with public-test filtering' baseline would likely raise pass@k substantially, so the gap may not come from the tree structure. 2) Even under the paper's own generation count, the headline direct and CoT runs on LiveCodeBench-Hard use mean generations 241.9 and 228.7 (Table 4), more than twice the 100 samples in the pass@100 values they are compared against; max_rollouts=16, where mean generations <100, gives 0.212 directly (below GPT4o pass@100 0.245), so the comparison is not budget-matched. 3) Token-level expansion/selection also consumes additional LLM forward passes (one per expanded node, P-UCB priors), which are not counted in 'mean generations.' Section 4.3's prompt-with-best-path result is suggestive but does not control for the public-test oracle used to find those paths.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Seed-CTS, a token-level Monte Carlo Tree Search (MCTS) method for competition-level code generation. The policy is an open-weight instruct model, expansion uses the top-k most probable next tokens, selection uses a P-UCB score, and simulation rewards are computed by running generated code on public test cases (hard reward and partial reward, Eqs. 9--10). On LiveCodeBench-Hard, direct prompting with Qwen2.5-Coder-32B-Instruct reports a pass rate of 0.305, and CoT prompting raises this to 0.351, compared with pass@100 = 0.245 for GPT4o-0513 and pass@1 = 0.366 for O1-Mini. The paper also reports model-agnostic results across four model sizes and a CodeContest-Test experiment. The authors explicitly report mean numbers of generations per problem as a reproducibility measure.","tokens_in":15073,"tokens_out":3957,"duration_ms":33130,"significance":"If the reported gains survive a matched-budget comparison against a Best-of-N baseline that receives the same public-test feedback, the result would be significant: it would demonstrate that a 32B open-weight model combined with token-level tree search can rival much larger proprietary systems on hard competitive coding, and it would suggest a practical route for synthesizing SFT data from a target model itself. The paper is transparent in reporting mean generation counts and evaluates across multiple base models and two benchmarks, which are welcome practices. The current evidence, however, does not yet isolate the contribution of tree search from the public-test reward signal or from the larger effective sample budget, so the central claim requires additional controlled experiments.","major_comments":[{"comment":"The reported gains of MCTS over pass@k baselines are confounded by the public-test reward. Equations (9) and (10) define the simulation reward using the public test set, while the pass@k baselines in Tables 1 and 3 generate k independent programs and evaluate only on private tests. A Best-of-N baseline that generates N samples and filters or selects them using the same public-test signal must be included before the improvement can be attributed to tree search rather than to test-time filtering. Without this control, the central claim in the abstract is not supported.","section":"§4.1, Tables 1--4; §3.2, Eqs. (9)--(10)"},{"comment":"The headline comparisons are not budget-matched. In Table 4, the direct-prompting pass rate of 0.305 at max_rollouts=64 uses mean generations 241.9, and the CoT pass rate of 0.351 uses 228.7; both exceed the 100 samples used for the pass@100 baselines. At max_rollouts=16, where mean generations are 68.2 and the direct-prompting pass rate is 0.212, the method is below GPT4o-0513's pass@100 of 0.245. The cross-model comparisons should be restated at matched sample counts or with an explicit sample-efficiency curve.","section":"§4.1, Table 4; Figure 3"},{"comment":"The reported 'mean generations' metric counts complete program simulations only. Token-level expansion (Eq. 7) and P-UCB selection (Eq. 5) require additional LLM forward passes per expanded node, so the total LLM compute is undercounted in Figures 2(d), 3(d), and the efficiency claims in Section 4.4. Please report total forward passes or provide a bound on the expansion overhead; otherwise the efficiency comparison against pass@k sampling is unfair.","section":"§3.2, Eqs. (5)--(7); §4.1, Figure 3(d)"},{"comment":"The interpretive experiment in Section 4.3 does not isolate the contribution of the selection phase. The 'best paths' inserted into the prompt were discovered using the public-test reward during MCTS, so the improved pass@k rates in Figure 6 may reflect leakage of public-test information into the prompt rather than the value of the P-UCB path structure. A control that inserts a path sampled without public-test feedback is needed before concluding that the selection and expansion phases are responsible for the gains.","section":"§4.3, Figure 6"}],"minor_comments":[{"comment":"The Figure 7 caption reports max_rollouts=16 for the CodeContest experiment, while Section 4.4 states that max_rollouts=32; please correct this inconsistency.","section":"Figure 7 caption vs. §4.4"},{"comment":"The text says both DeepSeekCoder-6.7B-Instruct and Qwen2.5-7B-Instruct exhibit pass@100 rates below 10; since the numbers in Table 3 are 0.080 and 0.099, this should read 'below 0.10' or 'below 10%'.","section":"§4.1, Hard Level paragraph"},{"comment":"The reward equations use T to denote the test set, which is ambiguous because Eq. (2) defines T = Tpub ∪ Tpriv; the text should state explicitly that only Tpub is used during search.","section":"§3.2, Eqs. (9)--(10) and Eq. (2)"},{"comment":"The caption contains a typo: 'it's own pass@100 rates' should be 'its own pass@100 rates'.","section":"Figure 1 caption"},{"comment":"The contribution bullet says 'Cot Prompting' but should be 'CoT Prompting' for consistency with the rest of the paper.","section":"Contributions, bullet 1"},{"comment":"The reference list has inconsistent formatting (e.g., some entries lack venues or year disambiguation, and two distinct Shinn et al. entries are nearly identical); please unify the bibliography style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The missing Best-of-N-with-public-test-filtering baseline is fixable and central; the paper's claimed superiority over pass@k is not yet established. The budget-matching and compute-accounting issues are also addressable. I see no reason to suspect the empirical results are fabricated, but the current presentation oversells the tree-search contribution. If the authors add the control and adjust the claims, the paper could be suitable; in its present form it is not."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nThe thing to know: the headline comparison is confounded. The MCTS reward is the pass rate on public test cases (Eq. 9-10), while the pass@k baselines never see that signal. So the 0.305 vs GPT4o’s pass@100 0.245 is largely test-time filtering, not tree search. The right control is Best-of-N with the same public-test filter, and it’s absent.\n\nWhat’s new here: token-level MCTS with P-UCB priors for competition code generation, with CoT prompting, tested across several open models. That’s a reasonable configuration of known ideas. The paper also reports mean generations per problem, which is a step toward honest cost reporting, and the “best path as prompt” ablation (Section 4.3) is a nice probe into where the gains come from, though it inherits the same oracle issue.\n\nThe soft spots are real. First, the reward uses public tests, and the baselines don’t. Second, the budget isn’t matched: at max_rollouts=64 on Hard, mean generations are 241.9 (direct) and 228.7 (CoT), more than twice the 100 samples in the GPT4o pass@100 number they’re compared against. At max_rollouts=16, where generations are <100, MCTS gets 0.212, below GPT4o’s 0.245. Third, the “mean generations” undercounts compute because each expanded node runs top-k selection, which is extra LLM forward passes. Fourth, there are no error bars; pass rates on 151 problems have substantial variance. Finally, the SFT data synthesis is a stated motivation but no training experiment is run.\n\nNone of this makes the method useless. It might still be an efficient way to find good plans and generate SFT data. But as it stands, the central claim of surpassing GPT4o at pass@100 is not supported. The paper would need a matched Best-of-N-with-public-tests baseline, an honest compute accounting, and ideally repeated runs.\n\nWho should read it: people working on inference-time search for code and on open-model code agents. It’s a worthwhile data point and the ablation is interesting. Worth sending to peer review, but with a clear ask: fix the baseline and the budget before accepting.","headline":"The headline result is confounded by public-test reward and an unmatched generation budget, though the method and ablations are worth a serious look.","tokens_in":15657,"tokens_out":3138,"would_cite":true,"duration_ms":28217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Seed-CTS claims that token-level Monte Carlo Tree Search plus Chain-of-Thought prompting lets a 32B open-source code model reach 0.351 on LiveCodeBench-Hard, beating GPT-4o's pass@100 and approaching O1-Mini's pass@1.","keywords":["token-level Monte Carlo Tree Search","competitive programming","code generation","chain-of-thought prompting","LiveCodeBench","open-source language models","test-time search","pass@k"],"falsifier":"Run the same token-level MCTS on LiveCodeBench-Hard with the public test outputs hidden from the reward (e.g., scoring rollouts on a separate validation split), or count every LLM forward pass used in expansion and rollout; if the pass rate falls to the generator's pass@k level or the per-pass advantage disappears, the reported gains come from test-case filtering rather than from tree search structure.","tokens_in":14582,"feed_emoji":"💻","tokens_out":9426,"duration_ms":67594,"temperature":0.7,"pith_summary":"The paper sets out to show that a mid-sized open-source code model, when paired with token-level tree search, can match or beat much larger proprietary models on competition-level coding problems. Its method, Seed-CTS, grows a search tree one token at a time, scores complete programs against the problem's public test cases, and folds Chain-of-Thought planning into the rollout. On LiveCodeBench-Hard, Qwen2.5-Coder-32B-Instruct reaches 0.305, above GPT-4o's pass@100 of 0.245, and with CoT reaches 0.351, near O1-Mini's pass@1 of 0.366. If these comparisons hold under a fair cost accounting, the result suggests that search can substitute for model scale in hard coding tasks, and that the searched solutions could become training data for the same model.","feed_headline":"Tree search lifts a 32B coder past GPT-4o on hard tests","feed_subtitle":"Same 32B open model hits 0.351 on LiveCodeBench-Hard, approaching O1-Mini's pass@1","key_machinery":"The machinery is a token-level Monte Carlo Tree Search: each tree node is a candidate token, expansion creates the top-k next tokens from the model's distribution, and a rollout continues to a complete program whose score is either the hard pass/fail on the public tests or the partial pass rate. Selection follows a P-UCB formula that balances the average reward, visit-count exploration, and the model's prior token probability. The CoT variant inserts a planning phase before code generation in the rollout, so the search refines both the plan and the code.","core_discovery":"The central discovery is that treating code generation as a token-level search problem converts repeated sampling into structured exploration that outperforms best-of-N from the same generator. With max_rollouts=64, MCTS with Qwen2.5-Coder-32B-Instruct scores 0.305 on LiveCodeBench-Hard, against the model's own pass@100 of 0.197 and GPT-4o's 0.245; adding a two-step CoT prompt (plan then code) raises the score to 0.351, approaching O1-Mini's pass@1 of 0.366. The paper also reports that the improvement appears across four model sizes (6.7B to 32B) and that the best search paths, when appended to the prompt, improve ordinary sampling, indicating the search is finding reusable reasoning plans.","pith_inferences":["The efficiency story depends on how costs are counted: the paper compares mean complete generations, while token-level expansion and rollouts consume many additional forward passes, and the search sees public test outputs during search while pass@k baselines do not; a stricter budget or a reward that hides test outputs would test whether the search structure itself, rather than test-case filtering","If the gains survive a fairer accounting, token-level tree search could shift the practical frontier of coding models toward smaller open weights, since 32B-class models would need no proprietary API for hard problems.","The prompt-amplification result suggests an amortization strategy the paper does not develop: run the search once per problem to extract a plan path, then sample code from that plan cheaply; this is testable and could cut search cost by an order of magnitude."],"forward_implications":["A 32B open-source model with token-level MCTS can exceed the pass@100 of GPT-4o on LiveCodeBench-Hard (0.305 vs 0.245), and with CoT reaches 0.351, just 0.015 below O1-Mini's pass@1.","The search is model-agnostic: at max_rollouts=32, DeepSeekCoder-6.7B-Instruct reaches 0.205 on the hard subset, matching Qwen2.5-72B-Instruct-api's pass@100 of 0.212.","MCTS uses fewer complete generations than the pass@k baselines it beats (for example, about 68 mean generations at rollouts=16 on hard), suggesting efficiency rather than brute-force sampling.","Best paths found by search improve plain pass@k when inserted into the prompt (up to 25.5% relative at k=1), so the search discovers reusable plans, not just code.","The method can generate high-quality solutions directly from the target model, making it a source of SFT data for competition-level problems."],"supporting_citations":[{"why":"Supplies the LiveCodeBench-Medium and LiveCodeBench-Hard datasets and the state-of-the-art pass@k rates used as baselines.","marker":"[Naman et al., 2024]"},{"why":"Technical report for the Qwen2.5-Coder family, the open-source generating models at the core of the experiments.","marker":"[Hui et al., 2024]"},{"why":"Defines pass@k evaluation and motivates the repeated-sampling baseline that tree search is compared against.","marker":"Chen et al. [2021]"},{"why":"Introduces Chain-of-Thought prompting, the planning step that the method adds on top of pure MCTS.","marker":"[Wei et al., 2022]"}],"fun_headline_variants":["Token-level tree search beats GPT-4o on hard code tests","32B model's tree search outpaces GPT-4o on coding","Token-level tree search lifts 32B coder past GPT-4o","Beating GPT-4o with tree search on a 32B model","Tree search helps a 32B coder approach O1-Mini"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is fair only if counting complete program generations captures the true computational cost, and only if the public test cases used as search rewards do not give MCTS an information advantage that pass@k baselines lack.","fun_headline_variants_meta":{"raw":{"variants":["Token-level tree search beats GPT-4o on hard code tests","32B model's tree search outpaces GPT-4o on coding","Token-level tree search lifts 32B coder past GPT-4o","Beating GPT-4o with tree search on a 32B model","Tree search helps a 32B coder approach O1-Mini"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3350,"prompt_tokens":972,"completion_tokens":2378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2281}},"tokens_in":588,"tokens_out":2378,"duration_ms":15385,"temperature":1.0,"reasoning_tokens":2281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:58:05.682078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same token-level MCTS on LiveCodeBench-Hard with the public test outputs hidden from the reward (e.g., scoring rollouts on a separate validation split), or count every LLM forward pass used in expansion and rollout; if the pass rate falls to the generator's pass@k level or the per-pass advantage disappears, the reported gains come from test-case filtering rather than from tree search structure.","supporting_citations":[{"cited_title":"V., Zhou, D., et al","cited_arxiv_id":null,"evidence_quote":"Introduces Chain-of-Thought prompting, the planning step that the method adds on top of pure MCTS."}],"review_version":1}