{"id":"64fae36a-b05c-46fa-9216-9e916b17f209","arxiv_id":"2412.17061","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TOA uses Monte Carlo Tree Search to dynamically orchestrate multiple LLMs during best-of-N sampling, and the authors report compute-efficiency gains over single-agent and fixed-workflow baselines on alignment, translation, and math.","lead":"This paper introduces TOA, a system that uses tree search to decide which of several language models should refine which previous answer, producing better training examples than sampling from a single model. The authors report that TOA beats single-model sampling and fixed multi-agent workflows on AlpacaEval and WMT while using similar compute per useful sample.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward-model overfitting in TOA's search makes compute-optimality unverified: Figure 4 shows optimizing KIWI degrades KIWI-XXL at high compute, and the same mechanism is used in all tasks.","rationale":"The reader's weakest assumption identifies the reward model as a reliable proxy; the paper's own Figure 4 demonstrates that this assumption fails in the MT setting. I agree this is the single most load-bearing concern because the mechanism that generates TOA's advantage is exactly the reward-guided search, and the only external-evaluator scaling curve provided (KIWI-XXL) shows the advantage decaying at high compute. Other concerns—missing error bars, downsampled evaluation, the MATH result where combining all four models underperforms the top two—are real but secondary. The MATH result is explicitly discussed and bounded by the authors, and the lack of error bars weakens significance but does not pinpoint a mechanism of failure. The reward-hacking evidence points to a specific mechanism that could invalidate the central claim across all three tasks. The paper is transparent about the issue, which is a point in its favor, but transparency does not establish that the compute-optimality conclusion survives. A conditional verdict remains appropriate because the method may still be useful for data synthesis or for moderate compute budgets, but the scaling claim needs the proposed external check before it can be accepted. Therefore the reader's verdict should not change.","tokens_in":26606,"tokens_out":5533,"duration_ms":54707,"concrete_test":"Extend the WMT zh→en scaling experiment in Figure 4 (right) to larger budgets, e.g., N=2560 and 5120 responses per prompt, and plot KIWI-XXL for all methods over the full range without truncating the moving average. If TOA's KIWI-XXL score falls below Parallel Ensemble or Sequential Refinement at these budgets, then the 'most compute-optimal across scales' claim is falsified in the high-compute regime, confirming reward hacking as the driver of TOA's apparent edge. As a complementary check, run TOA on the same data using a different reference-free reward model (e.g., XCOMET-XL) for search guidance and compare the external KIWI-XXL curves; if the advantage is reward-model-specific, it will shrink or vanish.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TOA is the most compute-optimal method as inference compute scales depends on the reward model being a faithful proxy for response quality, because TOA's MCTS selects actions by maximizing that reward (Section 5.1, Eq. 2-5). The paper itself gives direct evidence against this precondition in Section 6.2 and Figure 4: the left panel shows steady improvement in the KIWI-guided reward with more compute, while the right panel shows the external KIWI-XXL score first improves then declines at high compute—a clear signature of reward hacking. The paper acknowledges this and asserts that 'TOA remains the best overall approach,' but the plotted compute range stops before the decline fully plays out, so that assertion is untested at larger budgets. The same reward-guided mechanism is used for alignment (ArmoRM, Figure 3) and math (Qwen2.5-Math-RM-72B, Figure 5), yet no external-evaluator scaling curves are provided for those tasks; reward hacking may therefore be present there too, undetected. If the reward is gameable, then TOA's apparent gains over parallel ensemble or sequential refinement at a fixed FLOP budget do not necessarily translate to genuine quality. This undermines the headline compute-optimality claim, notwithstanding the paper's useful admission of the limitation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Tree Search-based Orchestrated Agents (TOA) for multi-agent sampling, in which the coordination of several language models is framed as a Markov decision process and solved with Monte Carlo Tree Search guided by a reward model. The method is evaluated on alignment (AlpacaEval), machine translation (WMT'21/'22), and mathematical reasoning (MATH), where the authors report scaling curves as a function of inference FLOPs and claim that TOA is the most compute-optimal method compared with parallel ensemble, sequential refinement, and mixture-of-agents baselines. The paper also demonstrates that fine-tuning with TOA-synthesized alignment data improves AlpacaEval and Arena-Hard scores over several preference-learning baselines.","tokens_in":26854,"tokens_out":4259,"duration_ms":38722,"significance":"If the compute-optimality claim holds, TOA would be a practical and timely contribution to inference-time scaling and synthetic data generation. The formalization of existing fixed workflows as special cases of a general multi-agent coordination framework is useful, and the release of code supports reproducibility. The paper is also transparent in acknowledging reward hacking in the machine-translation experiments. However, the demonstrated reward hacking raises a load-bearing concern: because the same reward model guides the search and is used to report the internal-quality scaling curves, the observed gains may partly reflect overfitting to that reward model. The absence of external-evaluator scaling curves for alignment and math prevents the reader from resolving this concern, and the central claim as stated is therefore stronger than the evidence currently supports.","major_comments":[{"comment":"The paper's own reward-hacking result directly undercuts the headline compute-optimality claim. The left panel of Figure 4 evaluates translations with the same KIWI reward model that guides TOA's MCTS (Section 5.1, Eq. (5)), while the right panel shows that when the selected best translation is re-scored by the external KIWI-XXL model, the score declines after an initial improvement at high compute budgets. Because the same reward-guided selection mechanism is used for alignment (ArmoRM, Figure 3) and math (Qwen2.5-Math-RM-72B, Figure 5), and no external-evaluator scaling curves are provided for those tasks, the claim that TOA remains compute-optimal across all tasks is not established. Moreover, the plotted compute range in Figure 4 stops before the decline fully plays out, so the assertion that TOA remains the best overall approach is untested at larger budgets.","section":"Section 6.2, Figure 4"},{"comment":"Because the reward model directly determines the MCTS selection, expansion, simulation, and backpropagation steps, the compute-optimality of TOA is conditional on the reward model being a faithful proxy for true quality. The paper provides no evidence that ArmoRM or Qwen2.5-Math-RM-72B maintains fidelity to external judges (GPT-4, KIWI-XXL, ground-truth answers) across the full compute range considered. I ask for per-task external-evaluator scaling curves, or at least a demonstration that the internal reward remains monotonically related to external quality over the tested budgets, to support the transferability of the method.","section":"Section 5.1, Eq. (2)-(5)"},{"comment":"The absence of error bars, the downsampling of AlpacaEval to 200 prompts (Figure 3 caption), and the moving-average smoothing applied in Figures 4 and 5 make it difficult to assess whether the observed ordering among TOA, sequential refinement, and mixture-of-agents is statistically reliable, especially near crossing points of the scaling curves. The compute-optimality conclusion should be supported by confidence intervals or multiple random seeds at each compute budget, and the raw unsmoothed curves should be reported alongside the smoothed versions.","section":"Figures 3-5"},{"comment":"The fitted scaling law R = a·log10(C)^2 + b·log10(C) + c is presented as a 'reliable means to predict the reward', but it is based on only seven points, has no uncertainty estimates, and uses the internal ArmoRM reward rather than an external quality measure. The extrapolative claim should be validated on held-out compute budgets and against external evaluators before it is described as reliable.","section":"Section 6.2"}],"minor_comments":[{"comment":"The sentence 'The above stages of a −d repeat' should read 'a–d' or 'a to d'.","section":"Section 5.1"},{"comment":"The text says 'the left figure illustrates' and 'the right figure reveals'; for clarity, use 'left panel' and 'right panel' when referring to subfigures of Figure 4.","section":"Section 6.2"},{"comment":"The statement 'For MoA, we use the default prompt for generation when alignment is involved' should specify what the default prompt is and where it appears in the appendix.","section":"Appendix B.3"},{"comment":"The legend entries such as 'Par. Ensemble_2' and 'TOA_4' are not explained in the caption; please state that the subscript denotes the number of models combined.","section":"Figure 5"},{"comment":"The MATH evaluation uses only 100 problems sampled from MATH500 (Table 4); this small test set should be listed as a limitation or at least mentioned explicitly in the main text.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The central claim is defensible but currently overreaches the evidence because reward hacking is demonstrated in the one task with an external scaling curve, and no such curves are provided for the other tasks. If the authors add external-evaluator scaling curves, error bars, and a more cautious reading of the reward-hacking result, the paper could become publishable. The scope fits a computational linguistics venue; the conceptual contribution is clear and the experiments are extensive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about 2412.17061: it applies MCTS to the problem of coordinating several LLMs for best-of-N sampling, and it gives scaling curves across alignment, translation, and math. The core idea is simple and sensible—treat model coordination as a sequential decision process, use a reward model to guide the tree, and let the workflow adapt per prompt. The paper also formalizes parallel ensemble, sequential refinement, and Mixture-of-Agents under one framework, which is a nice organizing move.\n\nThe empirical work is broad: three tasks, two sets of models, and headline numbers on WMT (average KIWI-XXL 84.05, above CPO and GPT-4) and AlpacaEval (72.2 LC win rate) from external evaluators. The fine-tuning experiments with TOA-synthesized data on Ultrafeedback and Magpie prompts give a concrete downstream use. The authors also deserve credit for printing the reward-hacking result: Figure 4 shows KIWI-guided search improving the internal reward while the external KIWI-XXL score turns over at high compute. And Section 6.2 admits that combining four models on MATH does not beat the top two.\n\nNow the soft spots. The stress-test note claims no external-evaluator scaling curves exist for alignment and math. That is wrong—Figure 3 right is GPT-4 LC win rate and Figure 5 right is majority-vote accuracy. The real issue is narrower but still live: the compute-optimality claim rests on the plotted FLOP range, and Figure 4's reward-hacking turnover means we do not know what happens at larger budgets. The scaling curves have no error bars, AlpacaEval and WMT are downsampled to 200 prompts, two figures use moving averages, and the free parameters (α, depth, sample counts) lack sensitivity analysis. A GitHub link appears in the footnotes, but I could not verify code or data from a static read. None of these are disqualifying; they keep the central claim at 'plausible, not demonstrated.'\n\nWho benefits: people doing data synthesis, best-of-N inference, or multi-agent orchestration with LLMs. This is an engineering contribution rather than a conceptual shift, but a useful one. A serious referee should engage with it—the reward-hacking section alone is worth a careful look—and the revision should ask for error bars, sensitivity analysis, and an extended compute range or a second reward model to test robustness.\n\nRecommendation: send to peer review. Desk-rejecting it would be a disservice.","headline":"Applying MCTS to multi-agent sampling is a sensible engineering step, and the paper is honest about its reward-hacking limitation; the compute-optimality claim is plausible but the current evidence does not fully nail it down.","tokens_in":27437,"tokens_out":4319,"would_cite":true,"duration_ms":33985,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tree-search coordinator lets several LLMs share inference compute more efficiently than any fixed workflow, the paper claims, and the synthetic data it produces transfers to fine-tuning.","keywords":["multi-agent sampling","inference compute scaling","Monte Carlo Tree Search","model coordination","data synthesis","best-of-N sampling","reward model","alignment"],"falsifier":"Re-run TOA and the baselines at matched FLOPs while evaluating the selected outputs with a reward model different from the one guiding the search, and check whether TOA's advantage persists. The paper already reports a decline on KIWI-XXL at high compute for translation; if that decline is large enough to eliminate TOA's edge over parallel ensemble under an independent judge, the compute-optimality claim would be an artifact of reward hacking.","tokens_in":26322,"feed_emoji":"🤖","tokens_out":3716,"duration_ms":30785,"temperature":0.7,"pith_summary":"This paper tries to establish that when inference compute is spent on generating multiple outputs from several different language models, how you coordinate those models matters as much as how many samples you draw. It proposes TOA, a Monte Carlo Tree Search that decides, for each input question, which model should refine which earlier response, guided by a reward model. Across alignment, machine translation, and math, the paper reports that this dynamic coordination beats random sampling from a single model, parallel ensembles, sequential refinement, and a fixed mixture-of-agents workflow at equal FLOPs. If true, multi-agent sampling becomes a practical way to buy better performance and better synthetic training data with compute rather than with larger models.","feed_headline":"Tree search makes multi-agent sampling compute-optimal","feed_subtitle":"TOA coordinates several LLMs per question, beating fixed workflows on AlpacaEval, WMT, and MATH.","key_machinery":"The central object is TOA, a Monte Carlo Tree Search whose tree alternates between model nodes and response nodes. Each action is a pair (model, previous response), the transition draws a new response from that model conditioned on that response, and the reward is a reward-model score for the new response. UCB balances visiting low-count branches against high-reward branches; response nodes are pruned to keep the best-scoring children while model nodes are not. This lets the search discover, per question, which refinement path among the K models yields the highest rewards, instead of following a fixed pipeline.","core_discovery":"The paper's central claim is that optimal multi-agent coordination is instance-specific, so a fixed workflow wastes compute, and that viewing coordination as sequential decision making and searching it with MCTS yields the most compute-efficient data synthesis. TOA formalizes best-of-N sampling from K models as an MDP: each action picks a model and a previously generated response to refine, and a reward model scores each new response. The search tree alternates model layers and response layers, uses UCB to trade exploration against reward, prunes low-scoring responses, and backpropagates rewards. The paper reports that this method is the most compute-optimal across AlpacaEval, WMT, and MATH, reaches a 72.2% length-controlled win rate on AlpacaEval, sets new SOTA results on WMT, and that fine-tuning on TOA-synthesized alignment data beats SimPO on AlpacaEval and Arena-Hard.","pith_inferences":["If the reward-model dependence is tamed — for example with a more robust or multi-objective reward — the same MCTS coordination could be applied to other multi-agent settings such as code generation, tool use, or retrieval-augmented generation, where instance-specific workflows likely matter too.","The finding that successor models along optimal paths are usually different from predecessors suggests that diversity, not just quality, is what makes multi-agent refinement effective; a testable extension is whether deliberately maximizing pairwise model diversity in the MCTS action space improves results further.","The paper's own reward-hacking observation implies that compute-efficiency comparisons of search-based agents should be re-run with held-out reward models or external judges before treating FLOPs-versus-quality curves as stable."],"forward_implications":["Multi-agent best-of-N sampling is more compute-efficient than single-model sampling at the same FLOPs, across alignment, translation, and math.","A tree-search coordinator can beat parallel ensemble, sequential refinement, and mixture-of-agents baselines while using no more inference FLOPs.","Synthetic alignment data generated by TOA transfers to fine-tuning, outperforming preference-learning baselines like SimPO on AlpacaEval and Arena-Hard.","TOA sets a new state of the art on WMT translation benchmarks, above GPT-4 and CPO, at lower total FLOPs than the nearest competitor MoA."],"supporting_citations":[{"why":"Supplies the Monte Carlo Tree Search algorithm that TOA adapts for model coordination.","marker":"Browne et al. (2012)"},{"why":"Defines the Mixture-of-Agents baseline, the representative fixed workflow that TOA is compared against.","marker":"Wang et al. (2024b)"},{"why":"Provides the test-time compute scaling context and the sequential refinement baseline in the multi-agent setting.","marker":"Snell et al. (2024)"},{"why":"Provides PRS, the strong single-agent tree-search sampling baseline that multi-agent methods must beat.","marker":"Ye and Ng (2024)"},{"why":"Supplies SimPO, the preference-learning baseline that TOA-synthesized data outperforms after fine-tuning.","marker":"Meng et al. (2024)"},{"why":"Provides CPO and the WMT winner/GPT-4 numbers that TOA surpasses on translation benchmarks.","marker":"Xu et al. (2024b)"},{"why":"Provides the KIWI reward model that guides and evaluates TOA's machine translation search.","marker":"Rei et al. (2022)"},{"why":"Provides ArmoRM, the reward model used to guide and rank outputs for alignment and data synthesis.","marker":"Wang et al. (2024a)"},{"why":"Supplies DPO, the fine-tuning method applied to TOA-generated synthetic alignment data.","marker":"Rafailov et al. (2023)"}],"fun_headline_variants":["Tree search finds optimal LLM coordination per question","MCTS-based multi-agent sampling scales better than fixed workflows","TOA: compute-efficient data synthesis via tree search agent collaboration","Search-based agentics beats static pipelines on AlpacaEval and WMT","Dynamic coordination via MCTS most compute-optimal for multi-agent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the reward model that guides the search — ArmoRM for alignment, KIWI for translation, Qwen2.5-Math-RM for math — is a trustworthy measure of response quality, and its own translation experiments show that optimizing KIWI eventually lowers scores on the independent KIWI-XXL judge.","fun_headline_variants_meta":{"raw":{"variants":["Tree search finds optimal LLM coordination per question","MCTS-based multi-agent sampling scales better than fixed workflows","TOA: compute-efficient data synthesis via tree search agent collaboration","Search-based agentics beats static pipelines on AlpacaEval and WMT","Dynamic coordination via MCTS most compute-optimal for multi-agent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000445,"raw_usage":{"total_tokens":2249,"prompt_tokens":944,"completion_tokens":1305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1219}},"tokens_in":560,"tokens_out":1305,"duration_ms":9096,"temperature":1.0,"reasoning_tokens":1219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:49:22.963864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run TOA and the baselines at matched FLOPs while evaluating the selected outputs with a reward model different from the one guiding the search, and check whether TOA's advantage persists. The paper already reports a decline on KIWI-XXL at high compute for translation; if that decline is large enough to eliminate TOA's edge over parallel ensemble under an independent judge, the compute-optimality claim would be an artifact of reward hacking.","supporting_citations":[{"cited_title":"Treviso, Nuno Miguel Guerreiro, Chrysoula Zerva, Ana C","cited_arxiv_id":null,"evidence_quote":"Provides the KIWI reward model that guides and evaluates TOA's machine translation search."},{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"Supplies DPO, the fine-tuning method applied to TOA-generated synthetic alignment data."}],"review_version":1}