{"id":"db28b829-2473-4f4e-97cd-cde2178d3f6b","arxiv_id":"2412.14212","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Tree-of-Code augments CodeAct-style code generation with tree-structured exploration and majority voting, reporting 81.6% accuracy on M3ToolEval versus 74.4% for the CodeAct baseline.","lead":"Tree-of-Code combines tree search over generated Python code with majority voting, aiming to make LLM agents more reliable on complex tasks. The authors report higher accuracy than CodeAct on the M3ToolEval benchmark, but the evidence is a single table without error bars, code, or the Tree-of-Thought comparison promised in the abstract.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gap over CodeAct is confounded: ToC uses a multi-model voting ensemble while CodeAct is a single GPT-4 run; the tree structure is not isolated.","rationale":"The paper's contribution is an engineering combination of CodeAct and ToT, and the evidence for it rests on Table 1. But Table 1 compares ToC with 'Mix-modal sampling' (GPT-4, ERNIE-4.0-Turbo, DeepSeek Coder, Claude 3.5) against CodeAct's best single GPT-4 mode. Any observed gain could come from ensembling stronger models, from generating and voting over many candidate solutions, or from the BFS tree structure. The last is the claimed cause; the experiment does not separate it. The missing error bars make the 7.2% difference even harder to interpret. The same lack of control affects the turns metric: parallel BFS is expected to reduce interaction turns regardless of accuracy gain. The reader's weakest assumption correctly identifies this. The checklist's 'Yes' answers on error bars and code release conflict with the actual manuscript, which strengthens the need for verification. No single correction fixes the paper; it needs either a controlled single-model ablation or matched-compute comparison. Therefore the reject verdict stands.","tokens_in":9963,"tokens_out":3419,"duration_ms":30952,"concrete_test":"Run an ablation on M3ToolEval: (1) ToC restricted to GPT-4 only, keeping BFS and voting; (2) CodeAct code_as_reasoning augmented with majority voting over a matched number of candidate trajectories and roughly equal per-task LLM calls/tokens; (3) report bootstrap 95% CIs over tasks. If single-model ToC drops toward 74.40% or CodeAct with voting reaches 81.60%, the tree structure is not the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that ToC achieves 81.60% vs CodeAct's 74.40% on M3ToolEval (Table 1). This comparison is confounded. In Section 4, ToC uses 'Mix-modal sampling' over GPT-4, ERNIE-4.0-Turbo, DeepSeek Coder, and Claude 3.5, plus temperature and prompt variation, and a breadth-first tree with majority voting. The CodeAct baseline is a single GPT-4 model with iterative correction. The treatment and control thus differ in at least two dimensions: the tree/voting procedure and the model portfolio/inference budget. The 7.2% gap could be produced entirely by the stronger model mix or by the larger number of generated candidates per task; it does not isolate the contribution of tree-structured exploration. Table 1 contains no error bars, and the text reports none, despite the NeurIPS checklist answering 'Yes' for statistical significance, so the gap may not be reproducible. The abstract also claims 'more stable results compared to Tree-of-Thought,' but no ToT baseline appears in the experiments. The paper does not report per-task token or API call costs, so 'generation depth set at 3' is not a compute control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Tree-of-Code (ToC), a method that combines Tree-of-Thought-style structured search with CodeAct-style code generation and execution for LLM-based agents. In ToC, each node is a generated code snippet together with its execution result; a breadth-first expansion explores alternative nodes by varying prompts, temperatures, and base LLMs, and a majority vote over successfully executed nodes produces the final answer. The authors evaluate ToC on the M3ToolEval benchmark and report in Table 1 that ToC achieves 81.60% accuracy with an average of 2.3 turns, compared with 74.40% accuracy and 5.5 turns for the best CodeAct (GPT-4) configuration. The paper argues that this demonstrates improved robustness and accuracy from tree-structured exploration and voting. The manuscript includes no ablations, no statistical significance measures, no Tree-of-Thought baseline, and no released code.","tokens_in":10266,"tokens_out":3197,"duration_ms":28309,"significance":"The idea of combining tree-structured exploration with executable code actions is a reasonable and potentially useful direction for LLM agent research. If the claimed gains were rigorously established through controlled comparisons, the method would be a meaningful contribution to the growing literature on code-as-action agents. However, the paper as written does not provide such evidence: the central comparison in Section 4, Table 1 conflates the tree/voting mechanism with a four-model ensemble, so the 7.2% accuracy gap cannot be attributed to the proposed architecture. The paper also makes an unsupported claim in the abstract about superiority over Tree-of-Thought without reporting any ToT baseline. On the positive side, the paper uses a standard benchmark (M3ToolEval) and directly compares against a CodeAct baseline, which gives the empirical claim a concrete reference point; nevertheless, the absence of error bars, compute controls, and reproducibility artifacts prevents the results from being verified. If the authors provide a same-model ablation, a ToT baseline, significance tests, and code, the contribution could become solid, but these are currently missing.","major_comments":[{"comment":"The headline comparison is confounded: ToC is evaluated with 'Mix-modal sampling' across GPT-4, ERNIE-4.0-Turbo, DeepSeek Coder, and Claude 3.5, plus temperature and prompt variation, while the CodeAct baseline is a single GPT-4 model. The treatment and control therefore differ in at least two dimensions: the tree-search/voting mechanism and the model portfolio/inference budget. The reported 7.2% accuracy gain and the reduction in turns could be entirely due to the stronger model mix or the larger number of sampled candidates per task. Please provide a controlled ablation: ToC with GPT-4 only, and CodeAct with an equally strong multi-model ensemble, with matched inference budget and the same number of generated candidates.","section":"Section 4, Table 1"},{"comment":"The abstract claims that ToC provides 'more stable results compared to Tree-of-Thought,' but no Tree-of-Thought baseline appears anywhere in the experiments. Table 1 only compares CodeAct variants. Unless a ToT baseline is run on M3ToolEval and reported with variance or error bars, this claim is unsupported and should be removed.","section":"Abstract and Section 4"},{"comment":"The paper reports no error bars, confidence intervals, or significance tests for the accuracy numbers in Table 1, yet the checklist answers 'Yes' to the question on statistical significance, with the justification that 'the paper reported error bars properly.' This is factually inconsistent with the manuscript. Please provide the number of evaluation tasks, standard errors or bootstrap intervals, and significance tests (e.g., paired tests across tasks) so that the 7.2% gap can be assessed.","section":"Section 4 and NeurIPS checklist item 7"},{"comment":"The checklist states that code will be open-sourced and that all computational resources are specified, but the manuscript contains no code repository, no prompt templates, no model version identifiers, no branching factor, no exact voting procedure, and no token/API cost breakdown. The phrase 'generation depth was set at 3' is not a compute control. Please provide the missing reproducibility details, including the exact prompts, model APIs and dates, tree-expansion parameters, and total inference cost per method.","section":"NeurIPS checklist items 4, 5, and 8"}],"minor_comments":[{"comment":"Equation (1) writes 'Execution(i) = Code(...)' but the right-hand side appears to be code generation, not execution; the notation should be clarified, for example by distinguishing a generation function from an execution environment.","section":"Section 3.2, Eq. (1)"},{"comment":"The phrase 'majority vote and tragic summarization process' appears to contain a typo; presumably 'strategic summarization' or 'tragic' is a word-substitution error.","section":"Section 3.4"},{"comment":"The caption contains the typo 'CodAct' instead of 'CodeAct'.","section":"Figure 1 caption"},{"comment":"Several references are duplicated: [1] and [2] are the same GPT-4 technical report, [8] and [9] are the same DeepSeek-Coder paper, and [22] and [23] appear to be the same paper in different venues. These should be consolidated.","section":"References"},{"comment":"Several checklist items contain boilerplate justifications that do not match the paper content; for example, item 2 answers 'Yes' that limitations are discussed, but the manuscript has no Limitations section, and item 8 claims all computational resources are specified, but none are listed.","section":"NeurIPS checklist"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently unsupported because the comparison in Table 1 does not isolate the proposed tree/voting mechanism from the use of a four-model ensemble. The checklist also contains several 'Yes' answers that are contradicted by the manuscript. I would be willing to reconsider if the authors supply controlled ablations, a Tree-of-Thought baseline, error bars with significance tests, and a reproducibility package; these are substantial but feasible additions rather than scope-level flaws."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does a clear, readable job of combining two known ideas: Tree-of-Thought style breadth-first exploration and CodeAct-style executable code actions. The llm-function tool, the tree over executed code nodes, and the majority vote are a natural extension of the cited frameworks, and the writing is straightforward. That is the extent of the novelty, but it is honest novelty.\n\nThe problem is the evidence. The headline result—81.60% vs 74.40% on M3ToolEval—compares a four-model ensemble (GPT-4, ERNIE, DeepSeek Coder, Claude 3.5) with temperature and prompt variation to a single GPT-4 run of CodeAct. That is not isolating the tree structure; it is comparing a bigger inference budget to a smaller one. The reader's stress-test is right. Also, the abstract promises \"more stable results compared to Tree-of-Thought,\" but no ToT baseline appears in Table 1 or anywhere else. There are no error bars, no significance tests, and no released code. The NeurIPS checklist answers \"Yes\" to statistical significance and to compute-resource reporting, but the paper contains neither; that is a factual mismatch, not a minor omission.\n\nWhat the paper does well is set up a reasonable engineering hypothesis: that exploring multiple code execution paths and voting can improve robustness over iterative correction. That hypothesis is plausible, and the turn reduction (2.3 vs 5.5) is worth investigating. But as it stands, the central empirical claim could be entirely an artifact of the model mix or the number of generated candidates per task.\n\nWho gets value from this? Someone working on agentic code generation who wants a compact summary of a plausible variant and who is prepared to treat the numbers as indicative rather than demonstrative. A serious referee should engage with it, but only if the paper is substantially revised: same-model ablations, a real ToT baseline, error bars, compute cost per method, and either code or a very detailed protocol. I would not cite it in its current form, and the missing ToT baseline makes the abstract misleading overall.","headline":"A sensible ToT+CodeAct mashup whose central 7.2% claim is untrustworthy as presented because the comparison is confounded and the paper lacks the basics.","tokens_in":10722,"tokens_out":1427,"would_cite":false,"duration_ms":14367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tree-of-Code treats each executed program as a node in a decision tree and votes among successful branches, beating CodeAct on M3ToolEval by 7.2 percentage points.","keywords":["Tree-of-Code","LLM agents","code-as-reasoning","tree search","majority voting","task planning","M3ToolEval","CodeAct"],"falsifier":"Run ToC with a single model, such as GPT-4 only, and an execution budget equal to CodeAct's, including matched temperatures, prompt counts, and total code executions per task; if accuracy no longer exceeds CodeAct's 74.40%, the tree structure and voting are not the source of the gain.","tokens_in":9800,"feed_emoji":"🌳","tokens_out":8908,"duration_ms":73894,"temperature":0.7,"pith_summary":"Large language model agents often act by writing and executing Python code in a step-by-step loop, an approach the paper calls CodeAct. The paper argues that this incremental repair loop is brittle on complex tasks because errors and hallucinations accumulate across rounds. Tree-of-Code instead generates end-to-end code solutions, treats each executed result as a node in a decision tree, expands failed branches using different prompts, models, and temperatures, and majority-votes over successful nodes. On M3ToolEval the paper reports 81.60% accuracy versus 74.40% for CodeAct's best mode, with average interaction turns falling from 5.5 to 2.3. The method matters because it points to a route to more stable agents that pools diverse LLMs without fine-tuning.","feed_headline":"Tree-of-Code lifts LLM agent accuracy to 81.6 percent","feed_subtitle":"Voting over executed code branches beats CodeAct's 74.4 percent and cuts task turns from 5.5 to 2.3.","key_machinery":"The carrying object is the execution tree: each node is the pair of generated Python code and its execution result, rather than a thought string as in Tree-of-Thought. Reflection between nodes is modeled as the lines of the tree and comes in two forms: system-level reflection, which samples from different LLMs such as GPT-4, ERNIE-4.0-Turbo, DeepSeek Coder, and Claude 3.5, and operation-level reflection, which varies prompts, temperature, and evaluation strategies. The llm-function tool lets the code itself request an LLM-generated summary, making reasoning explicit inside executable code. The mechanism does its work by combining breadth-first exploration of diverse solution branches with execution feedback, so failed branches get retried from a new angle while successful branches accumulate candidates for a final majority vote.","core_discovery":"Tree-of-Code claims that robustness in LLM agents comes from searching over complete code executions rather than repairing code incrementally. In this framework the tree $T=(N, L)$ has nodes $N$ defined as a generated code solution paired with its execution result, and lines $L$ are reflection strategies that expand failed nodes: system-level reflection samples different LLMs, while operation-level reflection varies prompts and evaluation. A special llm-function tool lets generated code call an LLM to produce final answers, so a complete solution is produced end-to-end. Breadth-first expansion continues until all leaves execute successfully or a maximum depth is reached, and successful node outputs are collected and majority-voted. On M3ToolEval the paper reports 81.60% accuracy for ToC versus 74.40% for CodeAct's code_as_reasoning mode, 52.40% for its json_as_reasoning mode, and 53.70% for its text_as_reasoning mode, with average turns reduced from 5.5 to 2.3.","pith_inferences":["Editorial extension: a controlled ablation that runs ToC with one LLM and the same total execution budget as CodeAct would determine whether the tree and voting, rather than model diversity and extra compute, are responsible for the reported gap.","Editorial extension: the same scheme should transfer to any executable action language beyond Python, such as shell scripts or API call sequences, making ToC a general search-over-actions planner.","Editorial extension: replacing one-model-one-vote with execution-derived confidence scores or a learned verifier is a natural next step the paper does not explore."],"forward_implications":["ToC raises accuracy on M3ToolEval to 81.60% from 74.40% for CodeAct's code reasoning mode while using 2.3 average turns instead of 5.5.","ToC also exceeds CodeAct's JSON and text action modes, which score 52.40% and 53.70%, so the reported advantage holds across action formats, not only code.","Because ToC can combine several commercial and open LLMs without fine-tuning, agent builders can pool heterogeneous models behind a single tree-search procedure.","Bounded depth plus majority voting keeps exploration finite and produces a single voted answer, supporting stable degradation when some branches fail."],"supporting_citations":[{"why":"Defines the CodeAct baseline and supplies the M3ToolEval benchmark on which ToC is evaluated.","marker":"[20]"},{"why":"Contributes the tree-structured search and reflection paradigm that ToC adapts from thought steps to code execution.","marker":"[25]"},{"why":"GPT-4 is both the CodeAct baseline model and one of the models sampled in ToC's diversity strategy.","marker":"[2]"},{"why":"DeepSeek Coder is one of the models used in ToC's model-diversity sampling.","marker":"[8]"},{"why":"ERNIE-4.0-Turbo is another model used in ToC's model-diversity sampling.","marker":"[15]"},{"why":"Claude 3.5 is another model used in ToC's model-diversity sampling.","marker":"[3]"}],"fun_headline_variants":["Tree-of-Code lifts LLM agent accuracy to 81.6%","Voting over executed code branches yields 81.6% accuracy","Tree-of-Code trims task turns to 2.3, hits 81.6% accuracy","Tree-of-Thought meets CodeAct: Tree-of-Code hits 81.6%","Tree-of-Code: robust planning via BFS over code executions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed gain rests on the assumption that the accuracy gap comes from the tree search and voting, not from ToC getting more compute and a stronger set of models than the CodeAct baseline.","fun_headline_variants_meta":{"raw":{"variants":["Tree-of-Code lifts LLM agent accuracy to 81.6%","Voting over executed code branches yields 81.6% accuracy","Tree-of-Code trims task turns to 2.3, hits 81.6% accuracy","Tree-of-Thought meets CodeAct: Tree-of-Code hits 81.6%","Tree-of-Code: robust planning via BFS over code executions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2825,"prompt_tokens":942,"completion_tokens":1883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1777}},"tokens_in":558,"tokens_out":1883,"duration_ms":12063,"temperature":1.0,"reasoning_tokens":1777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:56:42.721010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ToC with a single model, such as GPT-4 only, and an execution budget equal to CodeAct's, including matched temperatures, prompt counts, and total code executions per task; if accuracy no longer exceeds CodeAct's 74.40%, the tree structure and voting are not the source of the gain.","supporting_citations":[{"cited_title":"Evaluation on agi/gpt based on the dikwp for ernie bot","cited_arxiv_id":null,"evidence_quote":"ERNIE-4.0-Turbo is another model used in ToC's model-diversity sampling."},{"cited_title":"Claude 3.5, 2023","cited_arxiv_id":null,"evidence_quote":"Claude 3.5 is another model used in ToC's model-diversity sampling."}],"review_version":1}