REVIEW 3 major objections 6 minor 1 cited by
Inaugural MOASEI Competition at AAMAS'2025: A Technical Report
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The inaugural MOASEI competition benchmarked multi-agent policies under agent and task openness, finding the best learned policies statistically tied with a simple greedy baseline and an LLM-based submission crashing in the hardest…
desk verdict A credible competition report whose empirical claims about openness outrun the evidence; the headline result is that learned policies did not beat a trivial baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Partially observable stochastic games (POSGs) formalize each track, and the free-range-zoo environment suite provides three domain implementations—wildfire, cybersecurity, and rideshare—with mechanisms for agent openness and task openness. The evaluation pipeline runs 256 fixed-seed episodes per policy on three held-out configurations and applies Wilcoxon signed-rank tests to determine statistical significance. This machinery carries the argument by giving openness a concrete, per-domain definition and by making the comparison between learned and heuristic policies statistically explicit.
What would settle it
Re-running the evaluation on configurations sampled from a wider distribution of openness parameters, rather than the three fixed held-out scenarios, and observing that the statistical ties between learned and greedy policies disappear would refute the paper's generalization claims.
Extended reading notes
Core claim
The paper's central discovery is that a formal competition evaluating open-agent systems is feasible and yields reproducible evidence about policy classes: learned relational and convolutional policies match, but do not statistically exceed, a greedy baseline in the wildfire domain, and the LLM-augmented policy crashes in the most open configuration. In the cybersecurity domain, the winning weighted-scoring policy also failed to achieve a statistically significant advantage over heuristic baselines. The authors interpret this as evidence that open-world benchmarking can identify which architectural families generalize under openness, while also showing that the current testbed does not yet separate sophisticated learning from simple heuristics.
Load-bearing premise
The evaluation treats three held-out configurations drawn from the same environment family as a sufficient measure of generalization, without analyzing how well they cover the space of possible openness levels.
Editorial extensions
If this is right
- The competition infrastructure—public environments, fixed-seed evaluation, and leaderboards—can be reused as a shared benchmark for open-agent systems.
- Because the top learned policies statistically tied the greedy baseline, the current wildfire and cybersecurity testbeds do not yet reward sophisticated learning as strongly as intended.
- The comparable performance of CNN and GNN policies in the wildfire track suggests lightweight spatial architectures are a viable alternative to relational ones in these domains.
- The LLM-based meta-optimizer, despite not winning, demonstrates a route to policy adaptation that does not require end-to-end reinforcement learning.
- With no rideshare submissions, the track design as it stood failed to attract participation, informing a needed redesign for future iterations.
Reading between the lines
- If the three held-out configurations are representative of the openness spectrum, the statistical ties suggest the benchmark tasks may need larger-magnitude or higher-frequency openness to separate learning-based policies from heuristics.
- A concrete testable extension would combine the LLM meta-optimizer with a GNN base policy and compare against both components alone on out-of-distribution configurations.
- The paper's claims about GNN and CNN robustness would be strengthened or undercut by rerunning the evaluation on environment families outside free-range-zoo with structurally different openness.
- Reporting variance across draws of the evaluation configurations, rather than only across the 256 seeded episodes, would provide a more direct measure of generalization under openness.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This technical report documents the organization and results of the inaugural MOASEI Competition at AAMAS 2025, a multi-agent benchmarking event built on the free-range-zoo environment suite. Three tracks were designed—Wildfire, Rideshare, and Cybersecurity—with formalizations as partially observable stochastic games. Eleven teams registered and four submitted solutions: three for Wildfire (an LLM-based entry, a GNN-based entry, and a CNN-based entry) and one for Cybersecurity (a weighted-scoring entry). Each policy was evaluated with 256 runs per configuration on three held-out configurations per domain, and results were compared against naïve baselines using Wilcoxon signed-rank tests. The report's main empirical observations are that the top Wildfire submissions performed similarly to each other and to the simple "smallest" baseline, that the Cybersecurity winner's advantage over heuristic baselines was not statistically significant, and that no submissions were received for Rideshare. The paper also presents qualitative findings about GNN, CNN, LLM, and predictor-augmented approaches and outlines future competition plans.
Significance. If the descriptive claims are taken at face value, the paper documents a useful, reproducible open-agent benchmarking infrastructure and transparently reports participation and outcomes. The public codebase, fixed-seed evaluation pipeline, and candid reporting of crashes and non-significant comparisons are strengths. However, the central empirical insight is limited by the paper's own admission that the top Wildfire and Cybersecurity submissions were statistically indistinguishable from simple greedy baselines. Consequently, the stronger claims in the abstract and Section 5 about architecture-specific robustness to openness and 'promising strategies' are not established by the reported evidence. The paper's value as a competition infrastructure report is credible, but its empirical contribution requires substantial reframing or additional analysis.
major comments (3)
- [Section 4.1, Table 1; Section 5] The paper's own statistical comparison undercuts its central empirical claim. In Table 1, markov_mayhem (11.36±0.61) and university_of_tehran (11.41±0.63) have total Wildfire rewards essentially identical to the 'smallest' baseline (11.35±0.57), and the text in Section 4.1 states that these submissions 'did not have statistically significant differences between each other and smallest.' Section 5's claims that GNN-based policies 'demonstrated high adaptability to agent dropout and dynamic tasks' and that CNN policies 'matched the peak performance' therefore go beyond the evidence; the submissions matched, rather than outperformed, a trivial greedy heuristic. The authors should either provide additional analyses (e.g., task-specific metrics, temporal dynamics, or direct comparisons not confounded by configuration difficulty) or substantially temper the conclusions in the abstract, Section 4.1, and Section 5.
- [Section 4, first paragraph; Section 4.1; Section 5] The robustness-to-openness claims rest on an unquantified and unvalidated openness gradient. Section 4.1 attributes the WS1-to-WS3 differences to 'significantly more openness,' and Section 5 draws conclusions about 'robustness against openness' for GNN and CNN methods. However, the paper defines no openness metric, reports no values for agent/task entry and exit rates or other openness parameters, and does not describe how the three held-out configurations were sampled from the environment family. The observed performance differences could equally be due to changes in fire spread rate, world size, agent count, or other environment parameters. The authors should operationalize openness, report the concrete parameter settings for WS1–WS3 and CS1–CS3, or restrict all such claims to 'performance on the three tested configurations.'
- [Section 4, first paragraph; Figures 3–5 and 13–15] The statistical methodology for the significance claims is underspecified. The text states that n=256 independent simulation runs per policy were executed with fixed seeds, and that Wilcoxon signed-rank comparisons were used. A signed-rank test requires paired observations, so the pairing structure must be clarified: if the 256 runs are paired across policies by shared seed, describing them as 'independent' is misleading; if they are not paired, the Wilcoxon signed-rank test is inappropriate. Additionally, no multiple-comparison correction is reported for the many pairwise tests across baselines, configurations, and metrics. Because the paper's only comparative statements are negative (no statistically significant differences), the validity of the test procedure is load-bearing and should be fixed or explicitly justified.
minor comments (6)
- [Section 3.2, Section 4, Section 4.2, Section 7] The manuscript contains multiple typos and grammatical errors, including 'questions questions' (Section 3.2), 'signficance' (Section 4), 'recieved' and 'signficant' (Section 4.2), 'thourougly' (Section 7), and 'these modification will allow use' (Section 6). A careful proofreading pass is needed.
- [Table 1] The bit_student row reports 'crash' for WS3, but the Total column (7.71) appears to be the sum of WS1 and WS2 only. The table should state explicitly that the total excludes WS3, and the text should explain how crashed runs were handled in the leaderboard.
- [Section 2.2] The reward specification 'rewards them by 2agents_required' is mathematically ambiguous; it should be written as 2 × agents_required or an equivalent explicit formula.
- [Section 4.1 and 4.2, Figures 3–5, 13–15] The figures labeled 'Wilcoxon-signed-rank comparisons' are never described in the surrounding text—readers are not told what is being compared (e.g., which pairs of policies, whether the display shows test statistics or p-values). Adding a sentence or a detailed caption would make the figures interpretable.
- [Tables 5–8] The cybersecurity action-proportion tables contain rendering artifacts such as '𝑎𝑚𝑜𝑣𝑒', '𝑎𝑛𝑜𝑜𝑝', '𝑎𝑝𝑎𝑡𝑐ℎ', and '𝑎𝑚𝑜𝑛𝑖𝑡𝑜𝑟' in table headers. These should be replaced with plain-text labels such as 'move', 'noop', 'patch', and 'monitor' for readability.
- [Section 4.1] The sentence 'We observe that neither number of burnouts nor simulation length are correlated with cumulative episodic rewards' reports visual correlations from Figures 6–8 without any coefficient or test. Either provide a quantitative correlation statistic or soften the claim to an observation from the plotted data.
Circularity Check
No circular derivation; the report is an empirical evaluation of independently submitted policies against baselines, with no fitted parameter presented as a prediction.
full rationale
The report contains no derivation chain of the kind that could be circular. Its load-bearing results are measured cumulative episodic rewards from 256 seeded simulator runs per policy on held-out configurations (Section 4), plus Wilcoxon signed-rank comparisons. These are direct empirical observations of externally submitted policies (Markov Mayhem, University of Tehran, BIT Student, Zana Cyber) and fixed baselines; no parameter is fitted to a subset of data and then reported as a prediction. The mention of the authors' prior work [3] appears only to establish the POSG/openness background for the competition design, and the reported rankings are not derived from that citation. The Section 5 statements that GNN and CNN policies are robust to openness are summaries of observed performance, not consequences of an equation that assumes the conclusion. Concerns that the three held-out configurations may not operationalize the WS1-WS3 openness gradient, or that the paper does not quantify openness, are validity or correctness concerns, not circularity: they do not show that an output is equivalent to an input by construction. Accordingly, no circular step can be exhibited, and the score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Each competition domain is formally modeled as a partially observable stochastic game (POSG) with the tuple {S,A,T,R,Z,O}.
- domain assumption The free-range-zoo environments accurately instantiate the intended forms of agent and task openness.
- domain assumption The three held-out evaluation configurations (WS1-WS3, CS1-CS3) are representative of different levels of openness.
Cite this review
Pith. "Pith review of Inaugural MOASEI Competition at AAMAS'2025: A Technical Report." pith.science (2026). https://pith.science/paper/TLMC4CES
@misc{pith2026250705469,
author = {Pith},
title = {Pith review of: Inaugural MOASEI Competition at AAMAS'2025: A Technical Report},
year = {2026},
howpublished = {\url{https://pith.science/paper/TLMC4CES}},
note = {Machine review of arXiv:2507.05469}
}
read the original abstract
We present the Methods for Open Agent Systems Evaluation Initiative (MOASEI) Competition, a multi-agent AI benchmarking event designed to evaluate decision-making under open-world conditions. Built on the free-range-zoo environment suite, MOASEI introduced dynamic, partially observable domains with agent and task openness--settings where entities may appear, disappear, or change behavior over time. The 2025 competition featured three tracks--Wildfire, Rideshare, and Cybersecurity--each highlighting distinct dimensions of openness and coordination complexity. Eleven teams from international institutions participated, with four of those teams submitting diverse solutions including graph neural networks, convolutional architectures, predictive modeling, and large language model--driven meta--optimization. Evaluation metrics centered on expected utility, robustness to perturbations, and responsiveness to environmental change. The results reveal promising strategies for generalization and adaptation in open environments, offering both empirical insight and infrastructure for future research. This report details the competition's design, findings, and contributions to the open-agent systems research community.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
PLATO: Pointer Learner for Agent and Task Openness
A pointer-network actor plus GNN critic jointly handles agent and task openness in MARL without fixed bounds, with proofs of well-definedness and strong wildfire results.
Reference graph
Works this paper leans on
-
[1]
free-range-zoo GitHub Repository
2024. free-range-zoo GitHub Repository. Website. Retrieved June 24, 2025 from https://github.com/oasys-mas/free-range-zoo
work page 2024
-
[2]
Methods for Open Agent Systems Evaluation Initiative (MOASEI) 2025 Competition
2025. Methods for Open Agent Systems Evaluation Initiative (MOASEI) 2025 Competition. Website. Retrieved June 24, 2025 from https://oasys-mas.github.io/ moasei.html
work page 2025
-
[3]
Adam Eck, Leen-Kiat Soh, and Prashant Doshi. 2023. Decision making in open agent systems. AI Magazine 44, 4 (2023), 508–523. Table 1: Wildfire Expected Rewards Across Submitted and Baseline Policies Team / Policy WS1 WS2 WS3 Total noop −2.54± 0.05 −4.20± 0.06 −6.00± 0.00 −12.73± 0.12 random −1.23± 0.15 −2.38± 0.17 −4.27± 0.12 −7.88± 0.44 smallest 5.23± 0....
work page 2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.