{"id":"bd369ebf-b18c-4f24-ba8a-7c84558cf580","arxiv_id":"2508.13167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp, HLE, AIME25, and LiveCodeBench.","lead":"This paper trains one large language model to imitate the work of a team of cooperating AI agents, converting the team's step-by-step behavior into a single end-to-end reasoning format and then fine-tuning it with reinforcement learning. The resulting 32B models post new high scores on web browsing, coding, and math benchmarks, and the authors say all weights, code, and data are released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Web-agent SOTA rests on an uncalibrated LLM-as-judge used in both RL reward (Eq. 10) and evaluation (§4.1.1); without human-graded agreement the GAIA/BrowseComp/HLE gains may be judge-alignment, not capability.","rationale":"The reader's weakest assumption—that the LLM-as-judge is unvalidated and potentially biased—is exactly the load-bearing concern I identify. The paper's web-agent numbers are the primary evidence for the abstract's broad SOTA claim, and they are obtained with the same uncalibrated judge used as the RL reward. This is not a disagreement with consensus; it is a correctness risk in the evaluation metric itself. The code and math evaluations are rule-based and provide independent support for the core training recipe, so I do not recommend rejection. The overclaim regarding OAgents still outperforming AFM on GAIA/BrowseComp/HLE is a secondary reporting issue and does not change the verdict. A concrete human-scoring check would decisively settle whether the judge inflates results, and the paper's current evidence is incomplete without it. Thus the reader's CONDITIONAL verdict remains appropriate, with no adjustment needed.","tokens_in":54497,"tokens_out":3979,"duration_ms":48036,"concrete_test":"Re-score AFM-RL-32B outputs on the 103 GAIA text-only validation examples and a random 100-example subset of BrowseComp and HLE using human annotators (or the official benchmark's deterministic ground-truth scoring). Compare the resulting pass@1 with the Qwen-2.5-72B judge pass@1. Also run the same judge prompt on OAgents' outputs on the same subsets to test for family bias. If the human-graded pass@1 differs by more than 5 points absolute from the judge-based pass@1, or if the judge shows a systematic >5-point advantage for Qwen-family outputs, the web SOTA claim is not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AFM establishes new state-of-the-art web-agent performance depends on an LLM-as-judge for both the RL reward (Eq. 10, Section 3.3.2) and final evaluation (Section 4.1.1, Metrics; prompt in D.4), with Qwen-2.5-72B as judge. No calibration, human agreement, or comparison against the official benchmark scoring is reported. GAIA has known answers, BrowseComp has verifiable short answers, and HLE has deterministic grading; all could be scored objectively, but the paper instead uses a binary judge that is also the reward model during training. This creates a feedback loop: the model can optimize for answers the judge accepts, potentially overfitting to the judge's biases (e.g., family bias, leniency on semantically loose but plausible outputs) rather than genuine task success. If the judge inflates scores, the web-side numbers in Tables 7 and 13 are not trustworthy, undercutting the abstract and headline 'new state-of-the-art' in web-agent settings. The code/math results (Tables 11, 12) are test-case based and remain intact, so the training recipe itself is not falsified, but the web pillar of the central claim is unsupported until judge validity is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chain-of-Agents (CoA), a single-model paradigm that simulates multi-agent collaboration through dynamic role and tool agents, and trains 'Agent Foundation Models' (AFMs) by (i) distilling OAgents trajectories into CoA-format supervised fine-tuning data and (ii) applying DAPO-style reinforcement learning with verifiable rewards for code/math and an LLM-as-judge reward for web tasks. On MHQA, GAIA, BrowseComp, HLE, LiveCodeBench, CodeContests, and math benchmarks, the authors report state-of-the-art results at the 7B/32B scale, together with an efficiency analysis and an open-source release of models, code, and data.","tokens_in":54912,"tokens_out":4726,"duration_ms":60573,"significance":"The training recipe is a plausible and timely way to turn multi-agent collaboration into an end-to-end trainable policy, and the code/math results are supported by test-case and exact-answer verification. The temporal split on LiveCodeBench v4-v5, the avg@16 protocol for small math sets, and the explicit data-filtering stages are good methodological choices. The claimed web-agent superiority, however, rests on an LLM-as-judge signal that is used both as the RL reward and as the evaluation metric, with no human calibration; this is the main barrier to accepting the web-side headline claims. If the judge concern is resolved with objective scoring or calibration, the contribution is substantial and the open-sourced assets would be valuable to the community.","major_comments":[{"comment":"The web-agent reward and the final evaluation both use Qwen-2.5-72B-Instruct as an LLM-as-judge, but no human agreement, no cross-judge consistency, and no comparison to the official GAIA/BrowseComp/HLE answer keys is reported. GAIA has known answers, BrowseComp uses verifiable short answers, and HLE has deterministic grading; these could be scored objectively. As written, the GAIA 55.3, BrowseComp 11.1, and HLE 18.0 results may partly reflect alignment with the judge's biases (including family bias, since the trained model and judge belong to the same Qwen family) rather than genuine task success. The code/math results are unaffected, but this issue is load-bearing for the abstract's 'new state-of-the-art' claim in web-agent settings.","section":"§3.3.2 Eq. (10), §4.1.1 Metrics, §D.4"},{"comment":"GAIA is evaluated on only 103 text-only examples. The claimed SOTA margins are small: AFM-RL at 55.3 vs. WebSailor at 53.2 and WebShaper at 53.3 is a difference of about two examples, and even the 3.8-point margin vs. WebDancer is about four examples. No confidence intervals, bootstraps, or repeated-run variance are reported. The 'new state-of-the-art' phrasing on GAIA is therefore fragile and should be qualified with uncertainty estimates or additional evaluation trials.","section":"§4.1.2, Table 7"},{"comment":"The MHQA results in Table 6 are also evaluated with the Qwen-2.5-72B judge. It is not clear which baseline numbers were re-scored with this judge and which were taken from original papers using possibly different metrics. This can make the comparisons inconsistent, especially against methods whose published scores use exact match or other graders. Please state explicitly which baselines were reproduced, and report exact-match accuracy alongside judge-based accuracy for at least the main MHQA tables.","section":"§4.1.1 Metrics, Table 6"}],"minor_comments":[{"comment":"The AFM-SFT and AFM-RL rows appear twice without 7B/32B labels, so the reader cannot tell which row corresponds to which scale. Please add explicit model-size columns or row labels.","section":"Table 12"},{"comment":"The 'Generalization on Unseen Agents' claim is supported only by a single case study. This is interesting as an illustration, but it should be framed as anecdotal or supplemented with quantitative success rates on unseen-tool invocations.","section":"§5.2"},{"comment":"The efficiency comparison uses only 10 GAIA instances, and the metric 'prompt engineering cost' is not precisely defined. Please state the measurement procedure and, ideally, report a confidence interval or a larger evaluation set.","section":"§5.1"},{"comment":"Typos and unclear phrasing: 'progressive progressive filtering', 'sophiscated', 'contentional TIR', and the Figure 1 caption says 'Chain-of-Action' instead of 'Chain-of-Agents'. Also, Table 6's column headings are inconsistent with the dataset descriptions in §4.1.1 (e.g., TQ appears in the text but not in the table, while TriviaQA is placed under 'Single-Hop QA' in the table but described as multi-hop elsewhere).","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The core training recipe is interesting and the code/math evaluation is solid. The main risk is the LLM-as-judge feedback loop for web tasks; I would ask the authors to add objective re-scoring or judge calibration before the web-agent SOTA claims can be accepted. I also note that the teacher system OAgents and the task-generation pipeline TaskCraft appear to be from the same research group; this is not improper if disclosed, but the independence of the distilled trajectories from the evaluation benchmarks should be verified during the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part of this paper worth taking seriously is the code and math side. The recipe—multi-agent distillation from OAgents trajectories into role-tagged Chain-of-Agents sequences, observation masking, then DAPO RL on verifiable tasks—is coherent and the numbers are grounded: LiveCodeBench and CodeContests use real test cases, math uses Math-Verify, and the temporal split on v4/v5 plus avg@16 on small sets are sensible. The SFT-to-RL gains are consistent and the ablations are useful. If the open-source release actually ships, this is a solid starting point for agentic RL.\n\nThe web half is where the load-bearing weakness sits, and the stress-test note is right. GAIA, BrowseComp, and HLE all have objective answers, but the paper uses Qwen-2.5-72B as an LLM judge for both the RL reward (Eq. 10) and the final evaluation (Section 4.1.1, D.4). No human agreement, no calibration, no comparison against official scoring. The paper itself flags this in Section 3.2.1—QA/search correctness is judged by LLM—but never addresses the implication. That is not a minor gap: the abstract's 'new state-of-the-art' in web-agent settings could be partly judge-alignment, especially since the judge is from the same model family as the trained model.\n\nOther soft spots are proportionally smaller. The SOTA claim is narrower than stated: the teacher OAgents still leads on several web benchmarks, and the 103-example GAIA subset is a small sample. The name 'Chain-of-Agents' collides with an existing, uncited line of work. The open-source promise is unverifiable from the manuscript itself. None of these sink the code/math contribution, but they do reinforce that the paper's headline is overreaching.\n\nBottom line: the training recipe is real and the code/math results are trustworthy. The web-agent claim needs either human-graded calibration of the judge or re-evaluation with the benchmarks' official scorers before it can be accepted. This paper deserves a serious referee—it is a large, coherent empirical study with reproducible grounding on the code side—but the referee should push hard on the web evaluation. I would cite it for the code/math recipe, not for the web-agent SOTA.","headline":"The code/math half of this paper is a genuinely useful training recipe, but the web-agent SOTA headline leans on an uncalibrated LLM judge used in both RL and evaluation, so the central claim is only half-supported.","tokens_in":55474,"tokens_out":1420,"would_cite":true,"duration_ms":18545,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that multi-agent collaboration can be distilled end-to-end into a single LLM's weights, producing a model that outperforms prompt-engineered multi-agent systems and tool-integrated-reasoning baselines on web and code benchm","keywords":["chain-of-agents","multi-agent distillation","agentic reinforcement learning","agent foundation models","tool-integrated reasoning","web agents","code agents","LLM-as-judge"],"falsifier":"Take a random sample of AFM's predictions on GAIA and HLE (say 100 each), grade them by human annotators, and compare pass/fail agreement against the 72B judge model's scores; if agreement is near chance or biased toward the judge's own output style, the headline web-agent numbers would not survive independent verification. For the code/math side, the falsifier is a counter-example question where the model's trajectory plans and reflects correctly but its final code fails all hidden test cases, checking whether the RL reward actually tracks execution success rather than format.","tokens_in":54413,"feed_emoji":"🤖","tokens_out":6219,"duration_ms":67089,"temperature":0.7,"pith_summary":"This paper introduces Chain-of-Agents, a way to make one LLM behave like an entire multi-agent system during a single decoding pass: the model dynamically activates role-playing and tool agents, and its output trajectory is trained end-to-end. To create training data, the authors run a state-of-the-art multi-agent system on thousands of tasks and transcribe its successful executions into chain-of-agents trajectories, then fine-tune on those and apply reinforcement learning with binary rewards (an LLM judge for open-ended web tasks, test-case pass/fail for code and math). The resulting Agent Foundation Models set new state-of-the-art numbers at the 32B scale on GAIA (55.3%), BrowseComp (11.1%), HLE (18.0%), AIME2025 (59.8%), LiveCodeBench v5 (47.9%), and CodeContests (32.7%), while cutting inference token cost by roughly 85% compared with a traditional multi-agent framework. A sympathetic reader would take away that the coordination policy of a multi-agent system can live in weights rather than in prompts, making agentic ability trainable and scalable.","feed_headline":"Multi-agent teamwork distilled into a single 32B model","feed_subtitle":"One trained model tops GAIA, BrowseComp, HLE and AIME2025, beating prompt-built agent teams at far lower token cost.","key_machinery":"The chain-of-agents trajectory is the load-bearing object: a single model emits interleaved <think>, <plan>, <tool>, <observation>, <reflection>, and <answer> tokens, with role and tool selection expressed as token choices. Multi-agent distillation supplies the training signal by transcribing a working multi-agent system's execution into that format; progressive filtering keeps only long (5-20 hop), reflective, correct trajectories; and agentic RL sharpens the policy with sparse binary rewards. Observation masking during SFT keeps the model from memorizing noisy tool outputs.","core_discovery":"The central claim is that the reasoning pattern of a multi-agent system—planning, searching, crawling, reflecting, verifying, then answering—can be re-expressed as a single autoregressive trajectory, and that this trajectory can be taught to a base LLM in two stages. First, multi-agent distillation turns successful executions of an open-source multi-agent system into sequences of the form <think><plan><tools><observation><reflection>...<answer>, filtered for complexity, quality, reflection presence, and error-correction value, and trained with observation masking. Second, agentic RL on verifiable tasks uses outcome rewards—an LLM judge's binary verdict for web agents, sandbox-executed test c","pith_inferences":["Because the web-agent reward and evaluation both come from the same judge family, a separate human-graded pass on GAIA/HLE outputs would be the cleanest test of whether the reported gains are genuine task competence or alignment with the judge's preferences; the code/math numbers are immune to this concern since they are test-case based.","The recipe implies a rising-ceiling property: whenever a better multi-agent teacher appears, its successful trajectories can be distilled into the same backbone, so AFM performance can track the state of the art in multi-agent design without a new architecture.","The reflection-enrichment and error-correction upsampling steps suggest that what matters most in the curriculum is exposure to self-corrected failures; ablating these filters would isolate how much of the SFT gain comes from reflection tokens versus trajectory length.","One can test whether CoA is a general inductive bias rather than a distillation artifact by training from scratch on synthesized trajectories without a teacher system; if performance holds, the paradigm itself, not the teacher, is doing the work."],"forward_implications":["Multi-agent frameworks can be replaced by a single trained model, eliminating per-task prompt/workflow engineering and inter-agent token overhead.","Agentic capability becomes data-centric: collecting more and better multi-agent trajectories can improve the model without redesigning the framework.","RL on verifiable agentic tasks (test cases, exact answers) transfers to open-ended web tasks when paired with an LLM-judge reward, extending TIR-style training to multi-tool coordination.","Test-time scaling composes with the paradigm: Pass@3 with best-of-N selection raises GAIA from 55.3 to 69.9, implying the model's trajectory distribution is useful as a proposal sampler.","The paper's generalization experiment indicates that a code-trained CoA model can operate unseen tools from prompt schemas, suggesting tool-use skill transfers across agent domains."],"supporting_citations":[{"why":"Supplies the teacher multi-agent system whose successful execution is transcribed into chain-of-agents trajectories for SFT.","marker":"[82]"},{"why":"Provides the search-RL baseline that AFM must beat, and the RL data setup reused for the MHQA training set.","marker":"[21]"},{"why":"Supplies the clip-based policy optimization algorithm used for agentic reinforcement learning.","marker":"[69]"},{"why":"Establishes the LLM-as-judge methodology used for web-agent rewards and for scoring GAIA, BrowseComp, and HLE.","marker":"[76]"},{"why":"The GAIA benchmark is the primary web-agent evaluation set where the paper reports 55.3% at 32B scale.","marker":"[37]"},{"why":"The BrowseComp benchmark measures advanced web navigation, where the paper reports 11.1% at 32B scale.","marker":"[63]"},{"why":"The HLE text-only subset evaluates frontier academic problem solving, where the paper reports 18.0%.","marker":"[41]"},{"why":"The LiveCodeBench benchmark (v4-v5) is the main contamination-free code evaluation, where AFM reports 47.9% on v5.","marker":"[20]"}],"fun_headline_variants":["One model replays multi-agent reasoning to beat agent teams","AFM: single model distilled from multi-agent systems, tops SOTA","Open-sourced AFM mimics agent teams, cuts token cost","Train one model to simulate multi-agent teamwork, win benchmarks","32B model with agentic RL outshines prompt-built agent teams"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The web-agent results rest on the assumption that a 72B language-model judge's binary 'correct/incorrect' verdict is a valid oracle for open-ended task success; no human-graded calibration is reported, so part of the GAIA/HLE/BrowseComp gains could be alignment with the judge rather than with ground truth.","fun_headline_variants_meta":{"raw":{"variants":["One model replays multi-agent reasoning to beat agent teams","AFM: single model distilled from multi-agent systems, tops SOTA","Open-sourced AFM mimics agent teams, cuts token cost","Train one model to simulate multi-agent teamwork, win benchmarks","32B model with agentic RL outshines prompt-built agent teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2750,"prompt_tokens":833,"completion_tokens":1917,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1829}},"tokens_in":577,"tokens_out":1917,"duration_ms":14927,"temperature":1.0,"reasoning_tokens":1829,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:51:06.309916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of AFM's predictions on GAIA and HLE (say 100 each), grade them by human annotators, and compare pass/fail agreement against the 72B judge model's scores; if agreement is near chance or biased toward the judge's own output style, the headline web-agent numbers would not survive independent verification. For the code/math side, the falsifier is a counter-example question where the model's trajectory plans and reflects correctly but its final code fails all hidden test cases, checking whether the RL reward actually tracks execution success rather than format.","supporting_citations":[],"review_version":1}