{"id":"2cc527ba-93e1-4b53-801b-2798c1912ed3","arxiv_id":"2507.08616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark built from five distributed computing problems shows that frontier LLM agent networks solve small coordination tasks but break down as the network scales to 100 agents.","lead":"AgentsNet is a new benchmark that tests how well teams of AI agents coordinate on classic distributed computing tasks like coloring a network, electing a leader, and reaching consensus while talking only to their neighbors. Early results show frontier models handle small teams, but coordination collapses as the network grows to 100 agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-communication baseline leaves open that AgentsNet scores largely reflect per-agent priors (e.g., unanimous '0' on Consensus) rather than emergent coordination, so the central construct-validity claim is not yet established.","rationale":"The reader's weakest_assumption identifies the same construct-validity issue, and I agree that it is the most load-bearing concern. The paper's strongest evidence for coordination is the qualitative transcript analysis (Section 5.4), which demonstrates genuine multi-agent behaviors such as strategy negotiation and conflict resolution. However, the quantitative benchmark score is the central empirical output, and it lacks the control needed to attribute performance to coordination rather than to individual model priors or task-specific response biases. A no-communication baseline is cheap to implement and would directly test whether communication and collaboration are what the benchmark measures. The single-model 100-agent scaling and the 'practically unlimited' phrasing are secondary issues; they do not change the conditional verdict. The benchmark remains valuable as a scalable distributed-task stress test, but the construct-validity claim should be treated as conditional pending this control.","tokens_in":20866,"tokens_out":4855,"duration_ms":66119,"concrete_test":"Run the identical 4/8/16-node graph instances with T=0 message-passing rounds: each agent is given the final-response prompt immediately after the system prompt, with no messages from neighbors, and the same strict scoring is applied. If the aggregate no-communication solve rate is within, say, 20 percentage points of the full-protocol rate—especially on Consensus and Coloring—then the benchmark does not isolate coordination; if it is near zero across tasks, the construct-validity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AgentsNet measures the ability of multi-agent systems to collaboratively form strategies, self-organize, and communicate (Abstract; Section 3.1). This requires that observed solve rates be attributable to the network communication protocol and not to properties of an individual agent. The paper reports no no-communication ablation and no single-agent baseline. This is not a minor omission: for Consensus, an agent with no neighbors' messages can still answer a fixed value, and if all agents share an LLM prior, the network can 'solve' the task without any coordination. The high Consensus scores in Table 2 (e.g., 0.99-1.00 for several models) are consistent with this possibility. Similarly, Coloring and Vertex Cover can be partially solved by degree- or name-based heuristics. Without comparing the T=0 (zero message-passing) condition to the full protocol, the benchmark cannot distinguish 'coordination capability' from 'individual response biases' or 'prompt following.' The qualitative transcripts in Section 5.4 show real coordination behavior in some runs, but the headline quantitative claim is the aggregate solved fraction, which is the part lacking the baseline. Section 6 and Appendix G list limitations such as synchronous communication and homogeneous agents but do not acknowledge this missing control. This concern does not reject the benchmark's usefulness as a challenging distributed-task suite, but it blocks the interpretative claim about measuring collaboration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentsNet, a benchmark for multi-agent LLM coordination built from five distributed-computing problems: (Delta+1)-Coloring, Minimal Vertex Cover, Maximal Matching, Leader Election, and Consensus. Agents communicate only with graph neighbors through synchronous message-passing rounds, and the benchmark reports both strict solved-instance fractions and soft scores. The authors evaluate ten frontier and open-weight LLMs on graphs of 4, 8, and 16 nodes, and add a scaling study up to 100 agents using one model. They report that performance degrades as network size grows, with strong small-network results for several frontier models, and they provide qualitative transcript analyses of coordination successes and failures.","tokens_in":21083,"tokens_out":4148,"duration_ms":51273,"significance":"If its construct validity can be established, AgentsNet is a valuable complement to existing multi-agent benchmarks: the tasks have formal correctness criteria, the evaluation does not fit model outputs to the data, and the benchmark is open-source and scalable in principle. The qualitative transcript analysis is a genuine strength, as it documents concrete coordination behaviors such as self-initiated conflict resolution and strategy propagation. However, the central interpretative claim that the benchmark measures coordination and collaboration rather than individual response biases is not yet supported by the experiments, so the significance of the contribution is currently conditional on additional controls.","major_comments":[{"comment":"The headline claim that AgentsNet measures the ability of multi-agent systems to collaboratively form strategies, self-organize, and communicate (Abstract; Section 3.1) is not yet established because the paper reports no no-communication or single-agent baseline. On Consensus, an agent that receives no neighbor messages can simply answer '0', and if all agents share the same LLM prior, the network can solve the task without any coordination; the near-perfect Consensus fractions in Table 2 (e.g., Claude 3.7 Sonnet 1.00, Gemini 2.5 Flash 1.00) are compatible with this alternative explanation. Similarly, Coloring and Vertex Cover can be partially addressed by degree- or name-based heuristics. I ask the authors to add a T=0 no-message condition using the same prompts, graphs, and final-response protocol, and to report solved fractions per task per model against that control. Without this comparison, the quantitative results cannot be attributed to the message-passing protocol rather than to individual response biases or prompt following.","section":"Section 5.2, Table 2 and Section 3.1"},{"comment":"The scaling experiment rests on a single model (Gemini 2.0 Flash) and reports average solved fractions without standard errors, repeat counts, or per-instance variance. The statement that 'performance smoothly decreases as the network grows' and that difficulty can be gradually increased is therefore only weakly supported. The authors should either add at least one additional model and repeated runs with error bars, or explicitly frame Figure 5 as an exploratory single-model observation rather than a benchmark property.","section":"Section 5.3, Figure 5"},{"comment":"The limitations sections list synchronous communication, JSON parsing, and homogeneous agents, but do not acknowledge the missing no-communication baseline, which is the main threat to the paper's central claim. This should be discussed explicitly, and the proposed control should be listed as necessary future work if it is not added in the revision.","section":"Section 6 and Appendix G"}],"minor_comments":[{"comment":"The axis label 'T ask' contains a typo and should read 'Task'.","section":"Figure 5"},{"comment":"The soft-score discussion notes that random baselines can already exceed 50% for Coloring, but the paper does not provide random or trivial baselines for the binary solved-fraction scores in Table 2; adding such baselines would help interpret the absolute numbers.","section":"Appendix B, Table 4"},{"comment":"The text says 'we run at least one repeat per graph' and Appendix C defines a general formula with ni runs, but the number of repeats actually used is not reported anywhere; please state the exact number of runs per graph instance and configuration for reproducibility.","section":"Section 5.1 and Appendix C"},{"comment":"For Consensus, the task has no initial input values, so any fixed output is a valid solution; the paper should clarify that this variant tests agreement only and discuss how this choice affects the interpretation of high Consensus scores.","section":"Section 3.1 and Appendix B"},{"comment":"Figure 4 is visually dense because of the many small numeric labels and color categories; consider simplifying it or moving the per-size breakdown entirely to a table, as the main text already references Table 2.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The missing baseline is a substantive but fixable gap rather than a fatal flaw. The benchmark design, formal scoring, and qualitative analysis are solid enough that a revision adding a no-communication control and strengthening the scaling evidence could make the central claim defensible. I do not see a circularity problem or an obvious fit issue with the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AgentsNet is worth a look. It fills a real gap: existing multi-agent LLM benchmarks top out at a handful of agents and mostly use open-ended tasks, while this one scales to 100 agents and evaluates on five classic distributed computing problems with formal correctness criteria. The protocol is clearly specified, the graph generators are standard, the authors report standard errors, and they ship code and data. The qualitative analysis is also a genuine strength—the transcripts show real coordination behaviors like agents correcting each other's color assignments and adapting to stale information.\n\nThe main soft spot is the missing no-communication baseline. The abstract and Section 3.1 claim AgentsNet measures collaborative strategy formation, self-organization, and coordination. But without a T=0 condition or a single-agent control, the aggregate solve rates could largely reflect per-agent priors rather than anything the network does. The Consensus numbers are the clearest warning: several models hit 0.99–1.00, which is exactly what you would expect if every agent just outputs 0. Coloring and Vertex Cover can also be partially solved by simple heuristics like picking a degree-based response. This is not a minor omission—it directly blocks the interpretative claim that the benchmark isolates coordination from individual reasoning. The Limitations sections (Section 6 and Appendix G) list several valid caveats but do not mention this missing control, which should have been the first thing on the list.\n\nTwo smaller issues. The 100-agent scaling result rests on a single model (Gemini 2.0 Flash), so the claim that performance degrades with network size is supported for that model but not as a general statement. And the abstract's \"practically unlimited\" phrasing is too strong given that current models collapse at 100 agents and the cost grows quickly; \"scalable in principle\" would be accurate.\n\nNone of this sinks the paper. The benchmark itself is well designed, the tasks are appropriate, and the missing baseline is easily added in a revision. The paper demonstrates clear thinking and honest engagement with the literature. I would send this to peer review, with a strong request for the baseline as a condition of acceptance. Readers interested in multi-agent evaluation or LLM coordination will get value from the task suite even if they do not yet accept the construct-validity claim.","headline":"AgentsNet is a genuinely useful benchmark for multi-agent LLM coordination, but the missing no-communication baseline means the central claim about measuring coordination is not yet established.","tokens_in":21630,"tokens_out":1561,"would_cite":true,"duration_ms":19740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentsNet claims to measure multi-agent LLM coordination through five distributed-computing tasks solved under synchronous neighbor-only message-passing, and reports that frontier models coordinate on small graphs but collapse as networks…","keywords":["multi-agent LLM systems","coordination benchmark","distributed computing","LOCAL model","message passing","graph reasoning","scalability","leader election"],"falsifier":"Run a no-communication baseline on the same 27 graph instances: each agent is shown its own system prompt and final-response query but receives no messages from neighbors, and scores are computed exactly as in the paper. If that baseline matches or exceeds the message-passing scores on any of the five tasks, the benchmark's claim that it measures coordination rather than individual reasoning would be contradicted.","tokens_in":20651,"feed_emoji":"🤝","tokens_out":7456,"duration_ms":77492,"temperature":0.7,"pith_summary":"AgentsNet is a benchmark that asks networks of LLM-based agents to solve five problems borrowed from distributed computing—graph coloring, minimal vertex cover, maximal matching, leader election, and consensus—using only synchronous message-passing with their immediate neighbors. The paper's central claim is that these tasks measure whether a multi-agent system can genuinely coordinate, self-organize, and communicate under a given network topology, rather than merely answer a reasoning question. On graphs of 4, 8, and 16 nodes, the paper finds that frontier models solve many instances, but performance drops steadily as the network grows, and at 100 agents most tasks are essentially unsolved. The purpose of the benchmark is to create a difficulty scale that can grow with future models, since existing multi-agent benchmarks stop at 2–5 agents. If the claim is right, AgentsNet provides a way to compare agent architectures on their coordination ability, not just their reasoning strength.","feed_headline":"LLM agent networks lose coordination as they scale to 100 agents","feed_subtitle":"Five classic distributed-computing tasks show which frontier models can still coordinate at 100 agents — few can.","key_machinery":"The load-bearing object is the benchmark itself, instantiated as a synchronous message-passing protocol in the style of the LOCAL model: agents sit on the nodes of a graph, exchange text messages only with neighbors in rounds, and finally output a structured answer; the graph families—small-world, scale-free, and geometric (Delaunay) triangulations—and the five tasks fix a difficulty ladder from local coloring to global consensus. The protocol's role is to make the network topology the only channel of information, so that solving a task requires the agents to actually coordinate. Named machinery includes the five problems—$\\Delta+1$-coloring, minimal vertex cover, maximal matching, leader election, and consensus—each with a known distributed round complexity, and the binary 'fully solved' scoring rule that requires the whole network's output to be correct.","core_discovery":"The paper's central discovery is a measurement protocol: by grounding agentic tasks in the LOCAL model of distributed computing, the performance of an LLM agent network on these five problems becomes a proxy for its capacity to coordinate. Each agent is a separate LLM instance with a system prompt describing the task, a list of neighbors, and a fixed number of synchronous rounds; agents exchange JSON messages only with neighbors and must give a final parseable answer. The benchmark counts an instance as solved only when the entire network's answers satisfy the task's global constraints, e.g., exactly one leader or a proper coloring. Under this protocol, the strongest tested models reach aggregate scores around 0.70–0.80 on small graphs, while a mid-tier model falls from roughly 85 percent solved consensus instances at 20 agents to 15 percent at 100 agents and near zero for most other tasks. The authors read this as evidence that frontier LLMs can coordinate on small networks but lose coherent global strategy as network size and communication history grow.","pith_inferences":["Inference: the same protocol could be run with heterogeneous agents (different models, different prompts) or with faulty or adversarial agents, which the paper explicitly leaves out; those settings might change scores substantially and would test robustness rather than cooperative ability.","Inference: because the paper lacks a no-communication or single-agent baseline, part of the observed performance on small graphs may reflect individual reasoning rather than emergent coordination; a direct comparison would separate the two.","Inference: the qualitative finding that agents uncritically accept neighbor messages suggests a testable extension—measuring how often a deliberately corrupted message propagates through the network—which would quantify trust dynamics in LLM agent systems.","Inference: the soft scores in the appendix show partial progress even when strict solve rates are low, so a continuous metric might be more sensitive for comparing models near the scaling cliff; the paper itself notes this tension."],"forward_implications":["If AgentsNet measures coordination as claimed, then increasing the number of agents is a tunable difficulty knob: the benchmark can be made harder without redesign, simply by sampling larger graphs.","Models already differ sharply on the benchmark (e.g., aggregate scores from 0.26 to 0.80), so the protocol can discriminate between architectures on coordination ability, not just reasoning.","The scaling study implies that current frontier LLMs are not yet able to maintain coherent global strategies beyond roughly 20–30 agents, so the practical ceiling on decentralized LLM agent networks is near that size.","Because the tasks have known distributed lower bounds, benchmark performance can be compared against what a correct distributed algorithm would need, giving a principled floor for what coordination ought to require.","The failures documented in transcripts—late strategy agreement, uncritical acceptance of neighbor messages, and reliance on outdated information—point to concrete mechanisms that limit coordination and could be targeted by future training or prompting."],"supporting_citations":[{"why":"Defines the LOCAL model of distributed computing—synchronous rounds and immediate-neighbor message-passing—which the benchmark's communication protocol is built on.","marker":"[25]"},{"why":"Supplies the O(D) round complexity and formal treatment for leader election and consensus, used to set the number of message-passing rounds for global tasks.","marker":"[28]"},{"why":"Introduces leader election as a canonical coordination problem, grounding the selection of that task in the benchmark.","marker":"[2]"},{"why":"Provides the deterministic (Δ+1)-coloring round complexity used to justify coloring as a bounded-round local task.","marker":"[5]"},{"why":"Supplies the randomized round complexity for maximal matching, used for the matching task's complexity statement.","marker":"[37]"},{"why":"Supplies the small-world graph model used as one of the three topology families in the benchmark.","marker":"[45]"},{"why":"Supplies the scale-free preferential attachment graph model used as another topology family.","marker":"[4]"},{"why":"Reports human-subject network coloring experiments showing that topology and size influence coordination success, motivating the benchmark's focus on network structure.","marker":"[22]"},{"why":"Supplies the standard-error reporting methodology applied to all benchmark results.","marker":"[32]"}],"fun_headline_variants":["Scale breaks LLM agent coordination in new benchmark","LLM agents fail to coordinate as networks scale to 100","New test shows LLM coordination collapses at scale","Agent networks lose coordination beyond small size","Scaling LLM agents exposes coordination limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is construct validity: solving these five distributed-computing puzzles under the fixed synchronous message-passing protocol is treated as a genuine measure of multi-agent coordination, even though the paper reports no single-agent or no-communication baseline to separate coordination from individual reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Scale breaks LLM agent coordination in new benchmark","LLM agents fail to coordinate as networks scale to 100","New test shows LLM coordination collapses at scale","Agent networks lose coordination beyond small size","Scaling LLM agents exposes coordination limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1266,"prompt_tokens":965,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":581,"tokens_out":301,"duration_ms":3812,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:13:57.820012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a no-communication baseline on the same 27 graph instances: each agent is shown its own system prompt and final-response query but receives no messages from neighbors, and scores are computed exactly as in the paper. If that baseline matches or exceeds the message-passing scores on any of the five tasks, the benchmark's claim that it measures coordination rather than individual reasoning would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LOCAL model of distributed computing—synchronous rounds and immediate-neighbor message-passing—which the benchmark's communication protocol is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the O(D) round complexity and formal treatment for leader election and consensus, used to set the number of message-passing rounds for global tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces leader election as a canonical coordination problem, grounding the selection of that task in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the deterministic (Δ+1)-coloring round complexity used to justify coloring as a bounded-round local task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the randomized round complexity for maximal matching, used for the matching task's complexity statement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the small-world graph model used as one of the three topology families in the benchmark."},{"cited_title":"and Albert, R","cited_arxiv_id":null,"evidence_quote":"Supplies the scale-free preferential attachment graph model used as another topology family."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports human-subject network coloring experiments showing that topology and size influence coordination success, motivating the benchmark's focus on network structure."}],"review_version":1}