{"id":"b581f354-323d-4708-85e7-121976b61b28","arxiv_id":"2507.22606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-based framework that generates a tool-enabled, traceback-capable multi-agent system as a finite state machine from a task description, outperforming other automatic design methods on four benchmarks.","lead":"MetaAgent automatically builds a multi-agent LLM system from a short task description, organizing the agents as a finite state machine. In tests on writing, science questions, ML pipelines, and software tasks, its generated systems beat other automatic design methods and roughly match human-designed ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix H.1 shows agents emitting '<STATE_TRANS>' tokens themselves, indicating the separate Condition Verifier described in Section 3.2.2 may be bypassed; the FSM control claim needs implementation-level confirmation.","rationale":"The reader's weakest_assumption already flagged the H.1 token mismatch, and I agree it is the most load-bearing issue. The central claim has two parts: (1) the generated multi-agent system performs well, and (2) the FSM, with its per-state condition verifier, is what controls actions and transitions. The empirical tables could support part (1) if reproduced, but part (2) is directly contradicted by the appendix example, which shows the task-solving agent itself outputting '<STATE_TRANS>' tokens. This matters because the paper's novelty argument—traceback, null-transition, and a verifier that checks natural-language conditions—distinguishes MetaAgent from earlier hard-coded FSM approaches. If transitions are actually self-declared by the agent, the system is more like a prompt-engineered agent with structured output, and the claimed generality over linear/debate/orchestrator structures is not demonstrated. I also note a secondary internal inconsistency: Algorithm 2 inserts the output into the listeners of the new state after 's <- starget', whereas Section 3.3 says the output of the current state is inserted into the listeners of that current state; the appendix example indicates the origin listeners are intended. This is a minor reproducibility bug, but the verifier mismatch is the deeper concern. The verdict should remain CONDITIONAL because the concern is addressable by code inspection and clarification, not by a demonstrated flaw in the reported numbers. Credit is due for releasing code, setting temperature to 0, and using objective checkpoints, which make the implementation test concrete and feasible.","tokens_in":23869,"tokens_out":8488,"duration_ms":98130,"concrete_test":"Clone the released repository (https://github.com/SaFoLab-WISC/MetaAgent) and instrument the deployment loop for one ML Bench run (e.g., Titanic) to log every LLM API call. If each state transition is decided by the task-solving agent's own response (one agent call per state, no second call whose system prompt is the agent prompt plus transition conditions), the verifier described in Section 3.2.2 is not implemented. Also search the actual system prompts used in the code for '<STATE_TRANS>'; if present, the agent is self-declaring transitions. This settles whether the FSM-versus-verifier architecture matches the evaluated system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is that a designed FSM controls agent actions and transitions through a per-state Condition Verifier (Section 3.2.2, Algorithm 2). The appendix example H.1 contradicts this: each task-solving agent's system prompt instructs it to output '<STATE_TRANS>: 2' when a condition is met, and the deployment transcript shows the agent itself emitting '<STATE_TRANS>: 2' after its tool result, with no separate verifier call. If the transition decision is made by the task-solving agent or by parsing its emitted token, the described independent verifier is bypassed. The claimed novelty and the advantages over linear/debate/orchestrator structures rest on verifier-enforced FSM transitions, but the only implementation evidence in the paper indicates self-declared state tokens. This is closer to the hard-coded string-detection transition methods the paper criticizes in related works. Without implementation-level confirmation that a separate LLM verifier decides transitions, the empirical results do not establish the FSM control mechanism described in the abstract and Section 3.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MetaAgent proposes to automatically construct a task-level multi-agent system from a general task description. An LLM designer first generates a set of agents and then an FSM whose states bind a task-solving agent, a state instruction, listener agents, and a condition verifier; transitions are expressed as natural-language conditions. An optimization stage uses an LLM 'adaptor' to merge states that are deemed redundant. At deployment, Algorithm 2 runs the current state's agent and uses the condition verifier to select the next state, including null-transitions and tracebacks. The paper reports experiments on Trivial Creative Writing, GPQA, ML Bench, and five software development tasks, comparing against prompt-based, auto-designed, and human-designed baselines, plus ablations on tools, traceback, optimization, and foundation-model quality.","tokens_in":24082,"tokens_out":8215,"duration_ms":86248,"significance":"If the described mechanism is realized as implemented, the paper would make a useful empirical contribution: it shows that a task-level description can cheaply produce a tool-enabled multi-agent system that is competitive with human-designed frameworks (e.g., 0.83 vs 0.86 average on ML Bench, 0.85 vs 0.35 on software checkpoints), and the design-stage token costs in Table 7 are low. The ablations are well-chosen and the distinction between per-case and per-task design is practically important. The provision of a code link is also a strength for reproducibility. However, the significance currently hinges on two unresolved points: whether the condition verifier is actually used in deployment, and whether the auto-designed baselines were evaluated fairly. The manuscript's own appendix evidence raises the first concern directly, so the empirical results do not yet establish the mechanism claimed in the abstract and Section 3.","major_comments":[{"comment":"The described mechanism relies on an independent Condition Verifier (s.Ver) that evaluates each state's output against natural-language transition conditions and returns the destination state. The only end-to-end artifact in the paper contradicts this: in Appendix H.1, each generated agent's system prompt instructs the task-solving agent itself to emit literal tokens such as '<STATE_TRANS>: 2' or '<STATE_TRANS>: None', and the deployment transcript shows DataAndModelAgent emitting '<STATE_TRANS>: 2' directly, with no separate verifier invocation. If transitions are decided by the task-solving agent or by parsing that emitted token, the independent verifier is bypassed and the method degenerates to the hard-coded string-detection transitions that Section 2.1 criticizes in prior work. Please provide implementation-level confirmation of a distinct verifier (e.g., code or logs showing s.Ver called on the agent output), or revise the method description and the novelty claims accordingly.","section":"§3.2.2, Algorithm 2, Appendix H.1"},{"comment":"The comparison set for 'auto-designed methods' is incomplete. EvoAgent, ADAS, and Symbolic-Learning are cited in Section 2.1 as the closest prior work on automatic multi-agent design, and Table 1 marks EvoAgent as auto-designed and tool-enabled, but none of them is evaluated in the experiments. The abstract's claim of surpassing 'other auto-designed methods' therefore cannot be substantiated beyond SPP and AutoAgents. At minimum, EvoAgent, which is directly comparable in not requiring external training data, should be run on the same benchmarks.","section":"§4.1–4.2.1, Tables 2–4"},{"comment":"AutoAgents obtains 0.00 on all five ML Bench datasets while obtaining nonzero results on software development (Table 4). The paper says the adaptation 'extract[s] the generated code and get[s] the execution result' (§4.2.1). A uniform zero across all datasets is more consistent with a harness failure (e.g., no code generated, wrong interface, or execution error) than with genuinely poor model performance. Without sample outputs or a working adaptation script, the claim that MetaAgent surpasses AutoAgents on ML Bench is unsupported.","section":"Table 3, §4.2.1"},{"comment":"All results are single numbers with no error bars, repeated runs, or statistical tests, despite language-model nondeterminism even at temperature 0 in practice. The key comparisons include small margins (e.g., 0.86 vs 0.79 on Trivial Creative Writing; 0.83 vs 0.86 average on ML Bench versus DataInterpreter), so without variance information the reported advantages cannot be evaluated. Please report at least 3–5 runs per condition, or a clearly documented deterministic execution protocol.","section":"§4.2–4.2.1, Tables 2–4"}],"minor_comments":[{"comment":"The text says 'Table 5 presents the results for five different software development tasks', but Table 5 in the manuscript is the foundation-model transfer table; the software development results are in Table 4.","section":"§4.2.1, Tables 4 and 5"},{"comment":"The text says '6 tasks in software development', but Appendix B lists five software tasks (2048, Snake, Brick Breaker, Excel, Weather) and the cost table also says '6 Software Development Tasks'; the count should be corrected.","section":"§4.3, Table 7, Appendix B"},{"comment":"There are typos in the method and in the generated prompts, including 'null-trastion' for 'null-transition' and 'Thses enironment' in the prompt text; these should be cleaned up if the prompts are part of the public artifact.","section":"§3.2.2, Appendix H.1"},{"comment":"The header uses 'GPT3.5-Turbo' inconsistently; standardize as GPT-3.5-Turbo.","section":"Table 5"},{"comment":"The claim that linear, debate, and orchestrator structures are special cases of FSM is conceptually useful but largely definitional; consider moving the broader philosophical claim to the discussion and keeping the concrete comparison in Figure 3.","section":"§3.5"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the Appendix H.1 implementation mismatch: the described condition verifier is not visible in the only end-to-end example. I would ask the authors for the actual deployment code and traces before making a final decision. If the verifier is indeed bypassed, the submission should be revised with the claims scaled back to what is actually implemented. The missing EvoAgent comparison also matters for the scope of the central claim. The idea is promising and the task-level design cost is attractive, so I would not reject outright, but the current evidence is not sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate engineering contribution with a real gap between the description and the appendix. The idea of auto-designing a finite state machine to organize LLM agents, with tools and a state-merging optimizer, is new as a combination, and the paper shows it works reasonably on ML, software, and text tasks.\n\nThe good parts are concrete. The optimization step is cheap and data-free, the ablations for tool-using, traceback, and optimization all show meaningful performance drops, and the cost analysis is a useful addition. If the system works as described, it would give practitioners a practical way to go from a task paragraph to a tool-enabled multi-agent system.\n\nNow the soft spots, in order of importance.\n\nThe biggest one is the verifier. Section 3.2.2 and Algorithm 2 describe a separate condition verifier that checks the agent's output against transition conditions. But the running example in Appendix H.1 shows each task-solving agent's system prompt instructing it to output '<STATE_TRANS>: 2' when its own condition is met, and the deployment transcript shows the agent emitting that token itself. There is no separate verifier call shown. If that token is parsed directly, then the transition decision is made by the same agent that performed the work, and the claimed independent control mechanism is not what is implemented. That is close to the hard-coded string detection the paper criticizes in prior FSM work. The paper needs to clarify whether the verifier is actually called in deployment, and if so, why the agent prompts contain those tokens.\n\nOther issues are more standard. No repeated runs or variance info, despite temperature 0 being set. AutoAgents scores 0.00 on every ML bench task, which smells like a broken adaptation; it should be fixed or dropped. EvoAgent and ADAS are cited but not evaluated, which weakens the 'surpasses other auto-designed methods' claim. The FSM generalization argument in Section 3.5 is definitional but harmless.\n\nOverall, the central empirical claim—that the generated system beats the auto-designed baselines tested and is competitive with human-designed ones—is plausible with the tested baselines, but it is not fully established until the verifier question is resolved. This is not a desk-reject paper; it is worth a careful review, and the authors can likely answer the questions. I would send it out.","headline":"A practically useful auto-design framework for FSM-based multi-agent systems that deserves review, but the appendix suggests the implemented transition mechanism differs from the described condition verifier.","tokens_in":24614,"tokens_out":2905,"would_cite":true,"duration_ms":30818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MetaAgent claims that one task description is enough to auto-generate a tool-using, traceback-capable multi-agent system, and that the resulting systems beat other auto-designed methods and nearly match task-specialized human designs.","keywords":["multi-agent systems","finite state machine","automatic agent design","LLM agents","tool use","state traceback","condition verifier","task-level generalization"],"falsifier":"Instrument the deployment stage so every state transition is logged with its cause. If the majority of transitions are triggered by a '<STATE_TRANS>' string that the task-solving agent itself was prompted to output, rather than by an independent condition-verifier call, then the separate verifier described in Algorithm 2 is not the actual controller, and the system's claimed behavior reduces to self-reporting. A second check would measure verifier accuracy on held-out transitions across new task domains.","tokens_in":23655,"feed_emoji":"🤖","tokens_out":9488,"duration_ms":97480,"temperature":0.7,"pith_summary":"MetaAgent claims that a single natural-language description of a task type is enough to automatically build a working multi-agent system. The generated system is organized as a finite state machine in which each state pairs a task-solving agent with a condition verifier, listener agents, and transition conditions written in plain language, and an optimization pass merges redundant states before deployment. Across text-based benchmarks, machine-learning tasks, and software-development tasks, the generated systems outperform other auto-designed approaches and come close to, or exceed, human-designed systems tuned for those tasks. If true, this would turn multi-agent system construction from hand-crafted engineering into a prompting step plus automated polishing, at lower token cost than per-case automatic design. The paper also argues that the FSM subsumes existing linear, debate, and orchestrator-based multi-agent structures.","feed_headline":"Auto-built agent workflows rival hand-built multi-agent systems","feed_subtitle":"MetaAgent turns one task sentence into a finite-state machine of agents with tools, feedback loops, and backtracking","key_machinery":"The load-bearing object is the tuple $M = (\\Sigma, S, s_0, F, \\delta)$, the standard finite state machine, re-read so that each state $s$ is a work situation rather than an abstract node. A state carries an agent $s.\\text{Agent}$, an instruction $s.\\text{Ins}$, a condition verifier $s.\\text{Ver}$, and listeners $s.\\text{Lis}$; the verifier maps agent output to natural-language transition conditions, choosing the next state, a traceback, or a null-transition with feedback. The construction stage uses a designer LLM to produce agents first and then the state/transition graph, and an adaptor LLM to merge pairwise-equivalent states until the structure stabilizes. This object carries the argument because it is simultaneously the design output, the optimization target, and the runtime controller: the same JSON states define which agent acts, when control moves, and when the system loops back.","core_discovery":"MetaAgent's central claim is that a finite state machine is the right universal skeleton for an automatically designed multi-agent system, and that an LLM can write that skeleton from a task description alone. In the generated FSM, each state bundles an assigned agent, a natural-language state instruction, a condition verifier that chooses the next transition, and listener agents that receive the state's output; a null-transition keeps the same agent working when no condition fires, and transitions back to earlier states provide traceback. The paper reports that the generated systems reach 0.86 on Trivial Creative Writing and 0.60 on GPQA, outperform all compared auto-design systems on the machine-learning benchmark with an average normalized score of 0.83, and pass 85% of software-development checkpoints on average versus 35% for the human-designed baseline. It also argues that linear, debate, and orchestrator-structured multi-agent systems are all restricted cases of an FSM, which is why the structure fits auto-design so well.","pith_inferences":["Inference: the same generation loop should transfer to workflow domains the paper did not test, such as document processing or customer support, because nothing in the designer prompt is tied to machine learning, writing, or software.","Inference: deployment traces could be used as a free training signal: whenever a null-transition or traceback fires, the condition text that triggered it could be logged and later used to tighten the verifier, turning the FSM into a self-improving structure.","Inference: the design-stage cost numbers suggest that task-level FSM design, rather than per-case design, is what makes auto-design practical; a direct comparison of design cost per unit of generalization would make that economy explicit."],"forward_implications":["A task-type description, not a per-case prompt, is enough to build a reusable multi-agent system that generalizes across cases within the same task domain.","Tool use, traceback, and state merging are the active ingredients: the ablations show performance drops on every benchmark when any of the three is removed.","Because linear, debate, and orchestrator structures are special cases of an FSM, any improvement to FSM-based auto-design carries over to those existing structures rather than competing with them.","An automatically generated system can come close to, and in the software tasks exceed, systems that were hand-built and hand-tuned for those specific benchmarks."],"supporting_citations":[{"why":"Supplies the formal tuple definition of a finite state machine that MetaAgent adopts as the structure of a multi-agent system.","marker":"Hopcroft et al., 2001"},{"why":"Provides the human-designed software-development multi-agent baseline and the message-passing and listener ideas that MetaAgent's state design adapts.","marker":"Hong et al., 2024b"},{"why":"Supplies the Machine Learning Bench dataset, the DataInterpreter baseline, and the normalized performance score used to compare generated systems.","marker":"Hong et al., 2024a"},{"why":"Supplies the Solo-Performance-Prompting auto-design baseline and the Trivial Creative Writing benchmark used in the text-task comparison.","marker":"Wang et al., 2024d"},{"why":"Supplies the AutoAgents auto-design baseline that MetaAgent is compared against on machine-learning and software tasks.","marker":"Chen et al., 2024a"},{"why":"Provides the Symbolic Learning auto-design approach, which needs external data, and the software-development task set used for evaluation.","marker":"Zhou et al., 2024"},{"why":"Supplies the GPQA(Diamond) benchmark used to measure MetaAgent's reasoning-task performance.","marker":"Rein et al., 2023"}],"fun_headline_variants":["MetaAgent turns a task description into a finite-state multi-agent system","FSM-based auto-design creates multi-agent systems that rival hand-built","One sentence to a full agent team: MetaAgent uses finite state machines","Auto-designed finite-state agent systems match human-crafted performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the designer LLM produces a correct, complete finite state machine from the task description and that, during deployment, the condition verifier reliably maps agent outputs to the natural-language transition conditions.","fun_headline_variants_meta":{"raw":{"variants":["MetaAgent turns a task description into a finite-state multi-agent system","FSM-based auto-design creates multi-agent systems that rival hand-built","One sentence to a full agent team: MetaAgent uses finite state machines","Auto-designed finite-state agent systems match human-crafted performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3870,"prompt_tokens":915,"completion_tokens":2955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":2881}},"tokens_in":531,"tokens_out":2955,"duration_ms":25783,"temperature":1.0,"reasoning_tokens":2881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:28:14.382414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the deployment stage so every state transition is logged with its cause. If the majority of transitions are triggered by a '<STATE_TRANS>' string that the task-solving agent itself was prompted to output, rather than by an independent condition-verifier call, then the separate verifier described in Algorithm 2 is not the actual controller, and the system's claimed behavior reduces to self-reporting. A second check would measure verifier accuracy on held-out transitions across new task domains.","supporting_citations":[],"review_version":1}