{"id":"7cdaac86-6625-43b5-ab7d-7094e1722b2c","arxiv_id":"2608.07196","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EMAS evolves a multi-agent system's prompts and topology from its own execution traces, accepting a revision only after it recurs across samples and passes a paired validation check.","lead":"EMAS is a system that lets a team of AI agents improve its own prompts and connections as it works, without retraining the underlying model. It only makes a change when the same problem appears repeatedly and a test run shows the change helps, and it reports large accuracy and token savings on coding and math benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy gains are Test-selected best checkpoints, not the output of a deployable policy; EMAS's claimed 'attains' requires a Test-free checkpoint rule that the paper does not provide.","rationale":"The reader's weakest assumption correctly identifies the Test-selected checkpoint problem, and my reading of the full text confirms it is the most load-bearing issue. The paper is unusually candid: Section 5 admits the headline results are retrospective Test-best checkpoints, Section 3.6 admits Validation acceptance does not imply Test improvement, and Table 9 quantifies that 39 of 93 committed revisions regress on Test. Yet the Abstract and Section 4.1 phrase the results as 'EMAS attains' the highest accuracy, which implies a user running EMAS would obtain those numbers. Because the reported numbers are selected over all Versions using Test labels, the central claim is an upper-bound reachability claim rather than a statement about the deployed algorithm. This is not an internal inconsistency or a fatal flaw in the method; it is a reporting-protocol gap that can be closed by reporting final-Version or Validation-selected results. The suggested test would settle whether the headline is merely optimistic or actually unreachable: if final-Version and Validation-best numbers are close to the reported Test-best numbers and still beat the baselines under the same rule, the central claim stands; if they degrade substantially or fall below AFlow/ADAS, the paper should be reframed as showing what EMAS can reach with oracle checkpoint selection, not what it attains. I agree with the reader that the appropriate verdict is CONDITIONAL pending this analysis, so no change to the verdict is needed from my pass.","tokens_in":19346,"tokens_out":2823,"duration_ms":29360,"concrete_test":"Recompute the EMAS rows of Tables 1 and 4 under two deployment-feasible policies: (i) the final Version after two epochs, without any checkpoint selection; and (ii) the Version with highest Validation aggregate correct count at the end of Epoch 2, breaking ties by fewer tokens then earlier Version. Compare these against the same AFlow and ADAS numbers under the same reporting rule on Test. In particular, check Qwen-MBPP: does the final-version or Validation-best accuracy remain near 89.12%, or does it fall toward the Initial MAS value of 55.09% or below AFlow's 93.09%? Also report the fraction of categories where the Test-best Version differs from the Validation-best Version; if that fraction is high, the headline gains are unattainable without Test peeking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EMAS 'attains the highest task-weighted overall accuracy for both backbones' with relative gains of 6.30% and 20.10% within two epochs (Abstract; §4.1). But the paper's own Conclusion states that 'the headline results use the best checkpoints observed retrospectively on Test,' and Table 5 shows that each reported result is a specific Version (V0, V5, V6, V18, etc.) reached during the trajectory. No deployment-time rule is given for choosing among committed Versions without peeking at Test. Section 3.6 explicitly warns that Validation acceptance 'does not guarantee improvement on held-out Test tasks,' and Table 9 confirms 39 of 93 committed revisions regress on Test, with an average loss of 3.03 percentage points. Therefore the reported checkpoints are not what a user running EMAS would obtain in practice unless that user can see Test answers. The comparison is also asymmetric: AFlow and ADAS are static pipeline results averaged over repeats, while EMAS is the best of all evaluated Versions selected post hoc on Test. The headline 'attains' is thus an oracle-reachability statement, not a property of the deployed algorithm. The authors deserve credit for disclosing this limitation, but the disclosure does not repair the Abstract's wording or the comparison protocol; the load-bearing question is whether a fixed, Test-free checkpoint rule preserves the claimed gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EMAS, a method for evolving multi-agent system (MAS) prompts and topology while keeping LLM weights frozen. EMAS represents an MAS as a graph of fine-grained Steps, converts execution traces into structured revision hypotheses, triggers candidate construction only when a hypothesis recurs across samples, and commits the candidate only if paired Validation shows it improves aggregate accuracy or reduces tokens without lowering accuracy. The method is evaluated on Math, MBPP, PlanBench, and Game24 with Qwen3.6-27B and Kimi-K2-6, reporting that EMAS attains the highest task-weighted accuracy for both backbones, with relative gains of 6.30% and 20.10% within two epochs, and a large MBPP improvement for Qwen. The paper also includes ablations on recurrence thresholds and the Validation gate, as well as long-horizon Game24 experiments.","tokens_in":63,"tokens_out":4878,"duration_ms":134692,"significance":"If the central claims held as stated, this would be a useful contribution: it demonstrates a concrete mechanism for converting post-deployment experience into persistent, reusable system-level updates without weight updates, and it ships unusually thorough empirical apparatus, including confidence intervals, full version trajectories, reproducibility statements, provenance bundles, and explicit disclosures of non-monotonic evolution. The ablations provide genuine evidence that recurrence and paired Validation stabilize the revision process. However, the headline results are not yet established as properties of a deployable algorithm because the reported checkpoints are selected retrospectively on Test; the paper itself concedes this in the Conclusion. The significance of the contribution hinges on whether a fixed, Test-free checkpoint rule preserves a meaningful fraction of the reported gains.","major_comments":[{"comment":"The headline claim that EMAS \"attains the highest task-weighted overall accuracy\" is not currently a property of a deployable policy. Table 5 shows that the reported category-level results are obtained at Versions V1–V18 chosen from each trajectory, and §5 states explicitly that \"the headline results use the best checkpoints observed retrospectively on Test\" and that no deployment-time rule for choosing among Versions is provided. A user running EMAS therefore cannot realize the Table 1 gains without peeking at Test. This issue is load-bearing because the Abstract and §4.1 present the gains as achieved by EMAS. I ask the authors to define and evaluate a fixed, Test-free checkpoint rule (e.g., the last committed Version, the best-on-Validation Version, or a rule based on a separate holdout) and to re-report all headline numbers under that rule; if no such rule preserves the gains, the claims should be reframed as reachability results rather than as deployed performance.","section":"§4.1 / Table 1 / §5 / Table 5"},{"comment":"Equation (3) accept/reject on Validation only, and the paper correctly notes that Validation acceptance \"does not guarantee improvement on held-out Test tasks.\" Table 9 quantifies the gap: 39 of 93 E2 revisions regress on Test, with an average loss of 3.03 percentage points. This is not by itself a flaw, but it becomes one when Table 5's reported best Versions are selected using Test outcomes. The evaluation protocol therefore conflates the algorithm's online output with an oracle-selected trajectory. Please provide the distribution of end-of-epoch or Validation-selected performance across runs, not only the best checkpoint, so the reader can assess what a deployed EMAS run would actually deliver.","section":"§3.6 / Eq. (3) / Table 9"},{"comment":"The comparison with AFlow and ADAS is asymmetric: those baselines are reported as means over repeats 1–3, while the EMAS entries are the best observed Versions selected retrospectively on Test (per Table 5), averaged over three runs. This selection gives EMAS an advantage that is not available to the baselines and can inflate the reported margins. The comparison should be re-run with EMAS's chosen Version determined by a Test-free rule, or the baselines should be allowed the same retrospective best-Version selection and the associated selection bias should be reported and discussed.","section":"Table 1 / Table 4"}],"minor_comments":[{"comment":"The first sentence, \"MAS is the orchestration of LLM, this allows the system to learn with fixed language model parameters,\" is grammatically incomplete and should be rewritten.","section":"§2.1"},{"comment":"The support-threshold table is not legible: entries such as \"PROMPT-ONLY6\" and \"REMOVE-EDGE 106\" run together, obscuring the operation-specific κ values. Please reformat the table.","section":"Table 7"},{"comment":"The Introduction claims the validation set is \"balanced, containing equal numbers of correct and incorrect samples,\" but Table 3 shows, for example, a 47-correct/3-incorrect split for Kimi on PlanBench blocks. This inconsistency should be resolved either by qualifying the claim or by explaining why such imbalanced splits are acceptable for the acceptance rule in Eq. (3).","section":"Introduction / Table 3"},{"comment":"The caption says red stars mark where each trajectory reaches its highest observed accuracy, but Table 5 shows that the reported Version is not always the final Version. Please clarify whether the red stars correspond to the Table 5 reported Versions or to the best Version over the full trajectory.","section":"§4.2 / Figure 3"},{"comment":"The text mentions a \"checkpoint mechanism\" but Section 3 describes only accept/reject transitions and does not define checkpoints as an algorithmic component. Please clarify whether checkpoints are part of the EMAS procedure or solely an evaluation artifact used for reporting.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The authors are unusually transparent about the oracle-selection problem, and the reproducibility artifacts are a real strength. The core issue is not circularity or internal inconsistency; it is that the headline numbers are not yet attributable to the algorithm as deployed. If the authors can supply a Test-free checkpoint rule and re-report the main results under it, this could be a strong paper. I do not see a need to reject outright, because the method itself is plausible and the ablations are informative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core mechanism is a sensible engineering contribution, but the headline numbers are best checkpoints picked on Test, not what a user running EMAS would get. The authors disclose this in the conclusion; the abstract still says 'attains.' That mismatch is the thing to know.\n\nWhat is actually new is the combination of structured diagnosis keys h=(objective, defect, operation, location), operation-specific recurrence thresholds, and paired validation with distinct acceptance rules for accuracy and cost. GPTSwarm, Semantic Backpropagation, GEPA, and MASPO each have pieces of this, but not the whole package. The cost-directed evolution with an accuracy-preserving gate is a useful addition, and the ablations on Game24 are informative: dropping the recurrence threshold roughly doubles cumulative regression, and removing the validation gate drops best accuracy from 94.7% to 65.6%. The paper ships code, result bundles, provenance manifests, and it reports Table 9's 39/93 Test regressions. That is honest.\n\nThe soft spot is load-bearing. The reported gains—6.30% on Kimi, 20.10% on Qwen, MBPP going from 55.09% to 89.12%—are the best Version observed during evolution, selected retrospectively on Test. No deployment-time rule picks those Versions without Test access. Validation acceptance does not guarantee Test improvement, and 41.9% of committed E2 revisions regress on Test. Comparing a Test-picked oracle to static AFlow/ADAS averages is not a fair protocol. The paper's own conclusion says the checkpoints 'do not provide a deployment-time rule,' so the abstract's 'attains' overstates what the algorithm delivers.\n\nSecondary issues: the fixed Validation set is reused across revisions, creating adaptive selection pressure the authors mention but do not quantify; and token savings are not general—Kimi EMAS averages 17.1k tokens/task versus AFlow's 6.8k, so the cost story lives mostly in MBPP and long-horizon Game24.\n\nWho should read it: anyone working on automated MAS design, self-improving agents, or evaluation of evolving systems. The failure-attribution and gating machinery is worth studying even though the protocol needs fixing. It deserves a serious referee round, provided the authors report final-version or fixed-checkpoint-policy results and compare all methods under the same rule. If a Test-free rule preserves most of the gain, this is a solid contribution; as is, the headline is not proven.\n\nRecommendation: send it to review with the Test-free reporting as a condition.","headline":"A useful mechanism for MAS evolution with an evaluation protocol that overstates deployability—the headline gains are Test-selected checkpoints, not a fixed policy.","tokens_in":20145,"tokens_out":3783,"would_cite":true,"duration_ms":33729,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a deployed multi-agent system can keep improving after launch by converting recurring execution failures into validated, persistent edits to its own prompts and topology, while the underlying language-model weights…","keywords":["multi-agent systems","system-level evolution","prompt optimization","topology revision","recursive self-improvement","execution traces","validation gating","token cost reduction"],"falsifier":"Deploy EMAS with a hard rule that always promotes the most recently accepted revision, with no retrospective checkpoint lookup, and measure accuracy on the same held-out test tasks; if that accuracy falls far below the reported best-checkpoint numbers, as the 39 regressing revisions among 93 suggest it would, then the headline gains are not attainable without a selection rule the paper does not supply.","tokens_in":19127,"feed_emoji":"🤖","tokens_out":5766,"duration_ms":49256,"temperature":0.7,"pith_summary":"This paper proposes EMAS, a procedure that lets a deployed multi-agent system revise its own prompts and topology using the traces of its own executions, while the underlying LLM stays frozen. The claim is that recurring, sample-specific failures can be turned into persistent, validated system edits: a mistake must repeat across several tasks before a candidate change is generated, and the change is committed only if it beats the current system on a paired validation set. Across four benchmarks and two frozen LLMs, EMAS reports task-weighted accuracy gains of 6.30% and 20.10% within two evolution epochs, and a 62.2% token reduction on MBPP with Qwen3.6-27B. The paper's point is that system-level evolution, editing structure and instructions rather than weights, is a viable route to continued improvement after deployment.","feed_headline":"Self-evolving agent teams gain up to 20% accuracy","feed_subtitle":"Recurring mistakes trigger validated rewrites of agent topology; one code benchmark gains 34 points accuracy and cuts tokens 62%.","key_machinery":"The central object is the step-level graph of the MAS, where each node is one atomic LLM call and each edge transfers an artifact, together with the diagnosis record $h=(o,d,a,\\ell)$ that maps a trace outcome to an objective, a defect class, a permitted operation, and a canonical location. Revision candidates are gated by an operation-specific support threshold $\\kappa_a$, with additions requiring fewer supporting samples than removals because removals risk destroying needed computation, and every candidate must clear a paired validation rule on a fixed, balanced validation set before becoming version $v+1$. This converts evolution into a revision-authorization problem: the system decides which edits are allowed to shape future behavior.","core_discovery":"EMAS claims that the executable layer of a multi-agent system, the graph of LLM-call steps, the edges that carry artifacts between them, and the prompts that govern each step, is a body of reusable state that can be improved from experience just as weights are improved by training. Execution traces are converted into structured diagnoses that name an objective (accuracy or cost), a defect class, a revision operation, and a canonical location; a candidate revision is proposed only when the same diagnosis recurs across enough distinct samples, and is committed only when paired validation against the current system satisfies the objective-specific acceptance rule: strictly more correct answers for accuracy revisions, or no fewer correct answers at strictly lower token cost for cost revisions. Under a two-epoch budget, the evolved systems attain the highest task-weighted accuracy among the compared methods with both backbones and are best or tied in six of eight model, benchmark settings, with the largest gain on Qwen, MBPP, where accuracy rises from 55.09% to 89.12% while tokens per task fall by 62.2%. The paper further claims that evidence accumulation and validation gating, rather than the generation of edits, are what make evolution stable.","pith_inferences":["The checkpoint mechanism is retrospective: the paper itself reports that 39 of 93 committed revisions regress on Test and that headline numbers use the best Test checkpoint, so a deployable EMAS still needs an online rule for when to promote a new version, and rolling validation or retention-aware acceptance would be the natural thing to try.","The structured diagnosis key is itself an audit artifact: recording which defect, operation, and location produced each committed revision gives an inspectable lineage for system changes, which could support safety review of self-modifying agent systems.","Because evolution runs per task category, EMAS effectively builds a Pareto family of versions per category, so a user could select an operating point by token budget rather than only by peak accuracy.","A direct stress test of the causal story would vary the recurrence threshold continuously and measure the Test regression rate; if the regression rate does not fall as the threshold rises, the recurrence gate is not doing the stabilizing work the ablations attribute to it."],"forward_implications":["A deployed agent team can keep improving without weight updates, as long as it can observe its own failures recur and test its own proposed fixes on a validation set.","Accuracy repair and cost reduction are expressed by the same mechanism: swap the diagnosis objective and the acceptance criterion, and the same set of graph operations applies.","Near-saturation categories shift naturally from accuracy-directed to cost-directed revisions, so one evolution procedure covers both regimes.","The stability of evolution depends mainly on the two gates, recurrence and validation, rather than on the cleverness of the generated edits; removing the validation gate on Game24 with Qwen drops the best accuracy from 94.73% to 65.56%.","The reported two-epoch gains are not a ceiling: extending Game24 to 15 epochs raises Qwen accuracy from 79.90% to 94.73% while lowering tokens per task."],"supporting_citations":[{"why":"Supplies the computational-graph representation of agentic systems that EMAS adopts and refines into step-level atomic nodes.","marker":"Zhuge et al., 2024"},{"why":"Establishes the validation gate and cross-query aggregation that EMAS's paired validation builds on.","marker":"Wang et al., 2024"},{"why":"Shows execution traces and output feedback can drive revision of program parameters, motivating trace-based diagnosis.","marker":"Cheng et al., 2024"},{"why":"Automated design of agentic systems; one of the strongest baselines EMAS must beat on accuracy.","marker":"Hu et al., 2025"},{"why":"AFlow automates agentic workflow generation and serves as the main accuracy-oriented comparison baseline.","marker":"Zhang et al., 2025b"},{"why":"MASS optimizes prompts and topologies jointly, the direct lineage of EMAS's dual objective.","marker":"Zhou et al., 2026"},{"why":"Frames persistent self-improvement through empirical evaluation, the setting EMAS situates itself against.","marker":"Zhang et al., 2026"},{"why":"Trajectory-conditioned prompt revision evaluated along a Pareto frontier, close to EMAS's accuracy, cost view.","marker":"Agrawal et al., 2026"}],"fun_headline_variants":["Evidence-guided agent evolution improves accuracy 20% and reduces token cost 62%","Recurring mistakes trigger validated rewrites that improve accuracy up to 20%","Agent teams evolve from traces, raising accuracy 20% and cutting token use 62%","EMAS uses repeated diagnoses to safely rewrite agent teams, gaining 20% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on selecting the best version in hindsight on the held-out test set, and the paper provides no deployment-time rule that would let a running system identify that version.","fun_headline_variants_meta":{"raw":{"variants":["Evidence-guided agent evolution improves accuracy 20% and reduces token cost 62%","Recurring mistakes trigger validated rewrites that improve accuracy up to 20%","Agent teams evolve from traces, raising accuracy 20% and cutting token use 62%","EMAS uses repeated diagnoses to safely rewrite agent teams, gaining 20% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":4020,"prompt_tokens":1036,"completion_tokens":2984,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":2895}},"tokens_in":652,"tokens_out":2984,"duration_ms":19773,"temperature":1.0,"reasoning_tokens":2895,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:03:06.992643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy EMAS with a hard rule that always promotes the most recently accepted revision, with no retrospective checkpoint lookup, and measure accuracy on the same held-out test tasks; if that accuracy falls far below the reported best-checkpoint numbers, as the 39 regressing revisions among 93 suggest it would, then the headline gains are not attainable without a selection rule the paper does not supply.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MASS optimizes prompts and topologies jointly, the direct lineage of EMAS's dual objective."},{"cited_title":"and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J","cited_arxiv_id":null,"evidence_quote":"Trajectory-conditioned prompt revision evaluated along a Pareto frontier, close to EMAS's accuracy, cost view."}],"review_version":1}