{"id":"8e199683-775c-4561-968c-ea023386fdc3","arxiv_id":"2505.14569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Structured agent communication protocols with persistent execution graphs improve multi-agent task accuracy and report quality.","lead":"This paper introduces Agent Context Protocols (ACPs), a structured messaging and error-handling system for coordinating multiple AI agents through a shared task graph. It reports state-of-the-art scores on a long-horizon web assistance benchmark and human-preferred long-form reports.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AssistantBench SOTA claim conflates ACP with privileged domain-specific APIs and is within sampling noise; the causal case for ACPs rests on the synthetic dashboard ablation alone.","rationale":"The reader's verdict is CONDITIONAL, and the tool confound identified in the reader's weakest_assumption is the same load-bearing concern I find. The paper's central claim is that ACPs enhance collective inference; the strongest quantitative evidence for that is the AssistantBench SOTA number, but that number is not a clean test of ACP because the ACP system receives privileged, task-specific APIs. The base ACP is not SOTA, and the difference between the SOTA ACP and the next best baseline is tiny relative to the number of tasks. I would not reject the paper: the protocol is clearly specified, code is released, and the dashboard ablation directly compares ACP against weaker forms of coordination and shows a substantial gain. Those facts justify a conditional acceptance, but the conditions should include a controlled AssistantBench comparison and statistical significance testing. Since my conclusion matches the reader's CONDITIONAL verdict, no adjustment is needed.","tokens_in":42193,"tokens_out":5254,"duration_ms":48515,"concrete_test":"Run the strongest non-ACP baselines (e.g., Magentic-One, SPA-CB) on the same 214 AssistantBench tasks with exactly the same domain-specific API set provided to the ACP system, and apply a McNemar test on paired per-task outcomes between ACP+domain and each baseline. If a non-ACP system with the same tools reaches or exceeds 28.3%, or if the 28.3% vs 27.7% difference is not statistically significant, the SOTA claim should be qualified or withdrawn. Also report ACP+domain with the domain tools removed as a fully matched comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that ACPs achieve state-of-the-art results on AssistantBench is not supported by the comparison in Table 1. The 28.30% result is obtained by adding domain-specific APIs (TripadvisorSearchHotels, SkyScrapperFlightSearch, Goodreads, Google Maps, etc.) that no baseline was given. The paper's own base ACP with generic tools scores 24.80%, below Magentic-One (25.30%) and Magentic-One with o1 (27.70%); adding the domain tools accounts for a 3.5-point jump. No baseline runs the same tool set without ACP, so the protocol's contribution to the SOTA number is not isolated. Moreover, the SOTA margin over Magentic-One (o1) is 0.6 percentage points, roughly 1.3 tasks out of 214, which is within sampling noise; no confidence intervals or repeated runs are reported. The central causal claim that ACPs enhance collective inference therefore rests on the synthetic dashboard ablation, not on the headline AssistantBench result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Agent Context Protocols (ACPs), a structured protocol family for inter-agent communication, coordination, and error handling in multi-agent LLM systems. ACPs represent a task as a persistent execution blueprint (a directed acyclic graph of tool invocations), use standardized request/response/assistance message schemas, and define HTTP-like status codes for fault tolerance. The authors evaluate ACP-powered systems on AssistantBench, multimodal report generation, and a self-created dashboard-creation task, reporting 28.3% accuracy on AssistantBench, human-rated superiority of generated reports over Perplexity Deep Research and Gemini Deep Research, and ablation gains for the dashboard task.","tokens_in":42512,"tokens_out":4268,"duration_ms":42254,"significance":"If the reported results hold, ACPs would be a practically useful contribution: they offer a domain-agnostic coordination layer, standardized error handling, modular integration of domain-specific tools, and a public code release. The use of an external benchmark (AssistantBench) and human-preference evaluation are appropriate evaluation strategies, and the dashboard ablation is a sensible attempt to isolate coordination and fault tolerance. However, the headline claims are currently weakened by a confounded AssistantBench comparison and small, unreplicated human evaluations. The strongest surviving evidence for the central mechanism is the dashboard ablation, not the AssistantBench SOTA claim.","major_comments":[{"comment":"The 28.30% result that is used as the headline SOTA claim is obtained by adding a set of domain-specific tools (TripadvisorSearchHotels, SkyScrapperFlightSearch, Goodreads, Google Maps, etc.) that none of the baselines were given. The paper's own base ACP configuration with minimal tools scores 24.80%, below Magentic-One (o1) at 27.70%, so the 3.5-point gap between the two ACP variants cannot be attributed to ACPs without a control that gives the same tool set to a non-ACP orchestrator. Please report such a control, or re-frame the SOTA claim as a property of the ACP plus tool-set package.","section":"§4, Table 1"},{"comment":"Even if the tool-set confound were removed, the margin over Magentic-One (o1) is 0.6 percentage points, which is roughly 1.3 tasks out of 214. The paper does not report confidence intervals, multiple runs, or task-level significance tests, so the 'state-of-the-art' claim is within sampling noise. Please provide variance estimates or repeated-run results, or soften the claim accordingly.","section":"§4, Table 1"},{"comment":"The claim that ACP-generated reports 'significantly beat' commercial systems is not backed by any statistical analysis: 15 raters scored only five reports per system, the reported means are over rater–report pairs, and there are no error bars, per-rater models, or pairwise significance tests. The evaluation also has no inter-rater agreement measure. Please add appropriate analysis or change 'significantly' to 'on average'.","section":"§5, Fig. 3"},{"comment":"The central causal evidence for ACPs rests on the dashboard ablation, but the dataset is self-created and only described by a brief seed-generation procedure in Appendix D.2, and the evaluation uses 10 raters with no error bars or inter-rater agreement. The rubric in Table D.1 is reasonable, but without releasing the full dataset, prompts, and dashboard outputs, or providing an independent validation set, the 3.95 vs. 2.94 vs. 1.96 result cannot be independently assessed. Please make the evaluation artifacts available and add variance information.","section":"§6, Table 2 and Appendix D"}],"minor_comments":[{"comment":"The abstract states that ACPs 'enables' practitioners; the verb should be 'enable', and the phrase 'across ≈ 85% of the dimensions' is vague because no denominator or statistical basis is given.","section":"Abstract"},{"comment":"The column header 'EM' is not defined in the text or table caption, so the reader cannot tell whether it stands for exact match, entity matching, or another metric.","section":"Table 1"},{"comment":"The execution-timeline figure uses notations such as o(1)1 and segmented agent names that are not explained in the caption; please add a legend or a fuller axis description.","section":"§6.2, Figure 4"},{"comment":"The dataset-generation statement that all queries 'underwent manual review' would be more convincing if the criteria for feasibility and the number of queries per level were reported.","section":"Appendix D.2"},{"comment":"Several references list only 'Descriptive Link' or 'Web Search Summary (2025)' without a stable URL or author; these are not independently verifiable citations.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The AssistantBench confound is the main correctness risk: the headline SOTA number cannot be attributed to ACPs as submitted. The dashboard ablation is genuinely informative, but it needs statistical support and artifact release. With a re-analysis and additional controls, the paper could become acceptable; I would not reject on circularity grounds, since the evaluations use external benchmarks or human preference rather than fitting the protocol to the data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a legit systems paper, but its headline SOTA claim doesn't survive its own Table 1. The 28.3% on AssistantBench comes from a version with a pile of domain-specific APIs (Tripadvisor, SkyScrapper, Goodreads, Google Maps) that no baseline was given. The base ACP with generic tools gets 24.8%, below Magentic-One+o1 at 27.7%. So \"ACPs achieve state-of-the-art\" conflates the protocol with the tools. The margin over Magentic-One+o1 is 0.6 points — about 1.3 tasks out of 214 — sampling noise without confidence intervals.\n\nWhat's genuinely new: the ACP package — a persistent execution DAG storing intermediate outputs, structured message schemas, and HTTP-like error codes with assistance requests for recovery. Each piece exists in prior work, but this combination for inter-agent communication is a reasonable synthesis. The modularity story is real: adding domain tools without retraining jumps 24.8 to 28.3, and the paper is transparent that the jump is tool-driven. The cleanest evidence is the dashboard ablation: Single Agent 1.96, No Assistance 2.94, ACP 3.95. That isolates decomposition and then structured coordination, and it supports the core claim. The appendix example of an assistance request (missing latitude/longitude) is concrete and shows the mechanism working. Code release is a plus.\n\nSoft spots, in proportion. The abstract and intro oversell 28.3% as an ACP result. The human report evaluation is thin: 15 raters for 5 reports, no error bars, one prompt per domain. The dashboard dataset is synthetic and the evaluation criteria are authored by the team, so treat it as a case study, not a benchmark. None of this is fatal to the protocol idea; it just makes the empirical support weaker than the prose suggests.\n\nBottom line: this deserves a serious referee. The protocol design is worth engaging with, and the ablation is a genuine attempt to isolate the mechanism. The authors need to fix the comparison — give the same tools to baselines, run repeated trials or report intervals — and rewrite the headline so the protocol contribution is separate from the tool contribution. I'd be happy to see a revised version.\n\nWho's it for: multi-agent researchers, protocol designers, and anyone working on long-horizon tool use. Not a field-changer, but a useful contribution to the standardization conversation.","headline":"The ACP protocol design is a real synthesis worth engaging with, but the AssistantBench SOTA claim is a tool confound and the dashboard ablation is the only clean evidence for the protocol's causal effect.","tokens_in":42933,"tokens_out":3964,"would_cite":true,"duration_ms":36175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Agent Context Protocols—structured coordination via persistent execution blueprints and standardized messages—let teams of specialized AI agents reach state-of-the-art performance on long-horizon web assistance and…","keywords":["multi-agent systems","agent context protocols","collective inference","execution blueprint","fault tolerance","AssistantBench","multimodal report generation","LLM agents"],"falsifier":"Give those same domain-specific lookup APIs to the strongest baseline that lacked them and rerun AssistantBench; if the baseline matches or exceeds 28.3% accuracy, the protocol itself is not the decisive ingredient.","tokens_in":42021,"feed_emoji":"🤖","tokens_out":6064,"duration_ms":55662,"temperature":0.7,"pith_summary":"The paper tries to establish that the bottleneck in multi-agent AI systems is not individual model capability but the coordination protocol, and that structured protocols can replace ad-hoc natural-language chatter. It introduces Agent Context Protocols (ACPs): a persistent task graph called the Execution Blueprint plus standardized request, response, and assistance messages with descriptive error codes. If correct, this gives practitioners a modular recipe for assembling top-tier generalist systems from specialized agents and domain tools without retraining. The evidence includes 28.3% accuracy on AssistantBench, human-evaluated multimodal reports rated above two commercial deep-research systems, and ablation results attributing large gains to structured coordination and fault tolerance. The top AssistantBench configuration also layers several domain-specific lookup APIs onto the protocol, and the paper credits the protocol's modularity for the resulting improvement.","feed_headline":"Structured agent protocols hit 28.3% on AssistantBench","feed_subtitle":"Persistent task graphs and standardized error handling let specialized AI agents cooperate over long tasks.","key_machinery":"The Execution Blueprint is the central object: a persistent directed acyclic graph whose nodes are individual tool invocations, whose edges are data dependencies, and which doubles as shared memory for intermediate agent outputs. Around it, ACPs define standardized message schemas—AGENT REQUEST, AGENT RESPONSE, ASSISTANCE REQUEST—and descriptive error codes (601–607) modeled on HTTP status codes. This machinery makes every tool call a structured, validated, recoverable step, and localizes failures so a failed sub-task can be rerouted or abandoned without halting unrelated branches of the graph.","core_discovery":"The paper's central claim is that collective inference among LLM-based agents becomes reliable and long-horizon when communication is governed by structured, agent-agnostic protocols rather than unstructured natural language. In an ACP system, a complex task is decomposed into sub-tasks and compiled into a persistent directed acyclic graph of individual tool invocations, the Execution Blueprint, which stores intermediate outputs for downstream steps. Agents exchange messages through fixed schemas—AGENT REQUEST, AGENT RESPONSE, and ASSISTANCE REQUEST—and failures are localized with standardized status codes and suggested resolutions. With this machinery, the authors report state-of-the-art accuracy of 28.3% on AssistantBench, surpassing generalist and planning baselines, and multimodal technical reports that human reviewers scored higher than those of two commercial deep-research systems across five of six dimensions and on overall average. A dashboard-creation ablation further shows that adding structured coordination and assistance requests to multi-agent decomposition lifts human quality scores from 2.94 to 3.95 on a 0–5 scale.","pith_inferences":["The AssistantBench comparison would be cleaner if the domain-specific lookup APIs given to the ACP system were also given to at least one baseline; the paper does not report that, so the protocol's marginal contribution over tool access is not yet isolated.","Because the error codes and message schemas are agent-agnostic, a natural next test is whether agents built by unrelated providers can be dropped into one Execution Blueprint and cooperate successfully without shared training.","The design suggests a measurable recovery-rate metric: the fraction of ASSISTANCE REQUESTs that lead to successful re-planning or rerouting, which would directly stress-test the fault-tolerance claim.","A single-agent variant using the same structured request and response schemas could separate the benefit of structured tool interaction from the benefit of multi-agent coordination itself."],"forward_implications":["Domain-specialized tools can be added to a working ACP system without retraining; on AssistantBench the paper reports a gain from 24.8% to 28.3% accuracy from this kind of modular extension.","Fault tolerance through assistance requests and error codes lets long-horizon workflows survive individual tool failures; in the dashboard ablation, half of ACP-generated outputs received the maximum human quality score.","The same protocol stack transfers across objectives, since web assistance, multimodal report generation, and dashboard creation all use the same Execution Blueprint and message schemas.","ACP-generated reports required close to 150 inter-agent messages, indicating the framework can sustain coordination across many interdependent agents without cascading failure.","If the central claim holds, practitioners can build competitive generalist systems by composing off-the-shelf specialized agents and tools under a structured protocol instead of training one monolithic model."],"supporting_citations":[{"why":"Provides AssistantBench, the 214-task benchmark, and the closed-book, retrieval, and planning baselines the paper compares against.","marker":"Yoran et al., 2024"},{"why":"Supplies Magentic-One, the generalist multi-agent baseline whose AssistantBench score the ACP system surpasses.","marker":"Fourney et al., 2024"},{"why":"Defines the ReAct single-agent paradigm used as one ablation baseline in the dashboard study.","marker":"Yao et al., 2023"},{"why":"Introduces the Model Context Protocol, the single-agent standardization precedent that ACPs extend to inter-agent communication.","marker":"Anthropic, 2025"},{"why":"Exemplifies the conversational multi-agent collaboration style that ACPs argue is imprecise and ad-hoc.","marker":"Hong et al., 2023"},{"why":"One of the commercial deep-research systems whose reports were rated by human evaluators against ACP-generated reports.","marker":"Perplexity, 2025"},{"why":"The other commercial deep-research system whose reports were rated by human evaluators against ACP-generated reports.","marker":"Google, 2025"}],"fun_headline_variants":["Agent Context Protocols: structured coordination for superior AI","ACPs enable state-of-the-art collective inference at 28.3%","Protocol-driven AI agents beat commercial systems on complex tasks","Agent Context Protocols: the key to reliable multi-agent AI","Protocols not chatter: how ACPs lift multi-agent AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The AssistantBench state-of-the-art result assumes that the extra domain-specific APIs given to the ACP system (trip, flight, hotel, and map lookups) were not the real reason it beat baselines that lacked those tools.","fun_headline_variants_meta":{"raw":{"variants":["Agent Context Protocols: structured coordination for superior AI","ACPs enable state-of-the-art collective inference at 28.3%","Protocol-driven AI agents beat commercial systems on complex tasks","Agent Context Protocols: the key to reliable multi-agent AI","Protocols not chatter: how ACPs lift multi-agent AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001258,"raw_usage":{"total_tokens":5153,"prompt_tokens":942,"completion_tokens":4211,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":4127}},"tokens_in":558,"tokens_out":4211,"duration_ms":38287,"temperature":1.0,"reasoning_tokens":4127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:31:10.818007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give those same domain-specific lookup APIs to the strongest baseline that lacked them and rerun AssistantBench; if the baseline matches or exceeds 28.3% accuracy, the protocol itself is not the decisive ingredient.","supporting_citations":[],"review_version":1}