{"id":"5daf3aad-b217-4751-8484-57376a5f7827","arxiv_id":"2501.13727","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"SS-MARL combines graph-neural-network message passing with a trust-region constrained update and a new multi-constraint recovery step, reporting better reward and safety trade-offs than MACPO, MAPPO, and InforMARL in cooperative navigation with up to 96 agents.","lead":"This paper proposes SS-MARL, a multi-agent reinforcement learning method that combines graph neural network message passing with constrained policy optimization to improve safety and scalability. It reports zero-shot transfer from 3-agent training to 96-agent cooperative navigation in a modified particle environment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SS-MARL's own Table 2 falsifies its headline safety claim: with c=1, reported per-agent episode costs are 2.06, 4.11, and 7.00 for n=3, 6, 9, and Appendix §2.3 confirms c is treated as an undiscounted per-episode bound; 'satisfies the cost constraint' is therefore not supported.","rationale":"The paper's strongest claim is that SS-MARL achieves a better optimality-safety trade-off than baselines and satisfies cost constraints, with §5.2 explicitly asserting that at n=9 SS-MARL 'not only obtains positive rewards but also satisfies the cost constraint.' For this claim to hold, the final policy must satisfy Eq. (1) with c=1. The paper's own Table 2 reports per-agent episode costs of 2.06, 4.11, and 7.00 for n=3, 6, and 9, all above c=1. Appendix §2.3 removes the main ambiguity: it says that with c=1 and episode length 100, the expected average step cost per agent is 0.01, meaning c is enforced as an undiscounted per-episode bound. Thus the reported numbers are not compatible with the stated safety constraint under the paper's own convention. The theoretical appendix does not bridge this gap: Lemma 1.2 and Theorem 1 rely on exact value functions and delegate the cost-feasibility argument to [Gu et al., 2023], whereas the implementation replaces those with learned critics and local observations; the recovery step in §4.2 is a heuristic weighted objective without a proof that the approximate update remains feasible. Consequently the central safety claim is contradicted by the paper's own evidence, and the reader's REJECT verdict is appropriate. I do not see a need to change that verdict, so I mark the recommendation as UNCHANGED.","tokens_in":22427,"tokens_out":4890,"duration_ms":42110,"concrete_test":"Re-run the final SS-MARL(PS) checkpoints (or retrain with the reported hyperparameters, c=1) on n=3, 6, 9 for 100 episodes. For each agent, record both the undiscounted per-episode collision count and the discounted cumulative cost sum gamma_c^t C with gamma_c=0.99. Report the mean and 95th percentile per-agent per-episode value in both metrics, and state explicitly which metric Table 2's 'Cost' column reports and which metric is used in the Eq. (5) constraint. If the mean per-agent value in the metric used to enforce c exceeds 1, the 'satisfies the cost constraint' claim is false and the safety guarantee is not demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: SS-MARL satisfies safety constraints while improving optimality and scalability (Abstract; §5.2). This requires the final policy to satisfy the cost bound c in Eq. (1). The paper's own evaluation contradicts that requirement. In §5.2, c=1 for all agents, and the text says the expected cost for an agent within one episode should not exceed 1; Appendix §2.3 restates this as an average per-step cost of 0.01 over 100 steps, i.e., an undiscounted per-episode bound. Table 2 then reports SS-MARL(PS) test costs of 2.06 (n=3), 4.11 (n=6), and 7.00 (n=9), all exceeding c=1 by factors of 2–7. The sentence in §5.2 claiming that for n=9 SS-MARL 'not only obtains positive rewards but also satisfies the cost constraint' is therefore inconsistent with the table immediately above it. The theoretical support does not repair this: Appendix Lemma 1.2 and Theorem 1 prove cost feasibility only under the exact value-function assumption of [Gu et al., 2023] (as the paper itself notes in §2.1), while Eq. (5) is implemented with learned critics, GAE estimates, and local observations, and the multi-constraint recovery step (§4.2, Eqs. 7–9) is a weighted heuristic with no feasibility proof under these approximations. Even if one tried to reinterpret Table 2's costs as undiscounted while Eq. (1) is discounted, the appendix's own convention treats c=1 as the undiscounted per-episode bound, so the contradiction stands unless the authors re-report the discounted metric. Because the stated safety guarantee is the paper's main distinguishing contribution, this is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SS-MARL, a framework for safe and scalable multi-agent reinforcement learning. It combines a graph neural network backbone with message passing over agents, obstacles, and goals to obtain policies that accept variable-size observations, and a constrained joint policy optimization step derived from MACPO/CPO that includes a weighted recovery rule intended to handle multiple cost constraints. The authors evaluate the method on a modified MPE environment (Safe MPE) in cooperative navigation, compare it with RMAPPO, RMACPO, and InforMARL, and demonstrate zero-shot transfer from n=3 to up to n=96 agents, including hardware experiments.","tokens_in":22810,"tokens_out":7251,"duration_ms":61890,"significance":"If the safety guarantees were valid, the paper would be a useful contribution: it offers an end-to-end architecture for variable-size multi-agent systems, a trust-region recovery step for multiple constraints, and evidence of zero-shot scalability, all in a safety-constrained setting. The paper also ships an environment modification (Safe MPE) and hardware validation, which are practical strengths. However, the central claim that SS-MARL 'satisfies the cost constraint' is contradicted by the paper's own experimental table, and the theoretical feasibility proof is largely imported from prior work and is not shown to survive the approximations used in implementation. As a result, the main advertised advantage over reward-shaping baselines is not established by the evidence.","major_comments":[{"comment":"The central safety claim is contradicted by the paper's own evaluation. With c = 1 defined in §5.2 as the expected per-episode cost per agent, and Appendix §2.3 confirming that c = 1 corresponds to an undiscounted episode bound, Table 2 reports SS-MARL(PS) test costs of 2.06, 4.11, and 7.00 for n = 3, 6, and 9. These exceed the bound by factors of 2–7. The sentence in §5.2 stating that at n = 9 SS-MARL 'not only obtains positive rewards but also satisfies the cost constraint' is therefore directly inconsistent with the table. If the table is instead reporting a different metric (e.g., discounted episode cost), the units must be stated and the comparison to c must be re-derived; as written, the paper does not support its headline safety claim.","section":"§5.2, Table 2"},{"comment":"The theoretical feasibility guarantee is not established for the algorithm actually implemented. Theorem 1 relies on Lemma 1.2, which is stated as a bound from [Gu et al., 2023] under the assumption that exact state value functions are available, as the paper itself notes in §2.1. In the implemented method, the advantages in Eq. (5) are computed with learned critics, GAE estimates, and local observations over a communication graph. No error bound, pessimism correction, or robustness argument is provided to show that the constrained update and the recovery step preserve feasibility under these approximations. The empirical violation in Table 2 is consistent with this gap, so the theory cannot be used to repair the safety claim.","section":"Appendix §1.1, Theorem 1; §2.1"},{"comment":"The proposed multi-constraint recovery step is a heuristic rather than a proven mechanism. The update in Eq. (9) moves along a weighted combination of the constraint gradients, with weights defined in Eq. (8) for currently violated constraints. The paper does not prove that a backtracking line search along this direction decreases every violated constraint; with multiple constraints sharing a Hessian, reducing a weighted sum does not guarantee reduction of each individual cost. Moreover, all experiments use a single cost per agent (collisions), so the multi-constraint recovery is never empirically tested. Either a proof or an experiment with m_i >= 2 is needed to support the claim that SS-MARL handles multiple constraints.","section":"§4.2, Eq. (9); Appendix §1.2"},{"comment":"The scalability results are presented without reference to the safety constraint used during training. The text states that SS-MARL trained with n = 3 can be scaled to 96 agents while 'maintaining collision times per agent below 5', but it does not state the cost bound c for these transfer experiments. If the same c = 1 applies, the reported average episode costs per agent below 5 also violate the constraint; if a larger bound is allowed, the safety claim needs to be qualified accordingly. Please report the constraint value and the fraction of episodes satisfying it for the transferred policies.","section":"§5.3, Figure 6"}],"minor_comments":[{"comment":"'tranfer' should be 'transfer'.","section":"§2.3"},{"comment":"'cosntraints' should be 'constraints'.","section":"§4.2"},{"comment":"The caption does not specify whether 'Cost' is per episode, per step, discounted, or undiscounted; please define the metric explicitly and align it with Eq. (1).","section":"Table 2"},{"comment":"The text refers to 'average costs per step' in Figure 4 and 'average episode costs per agent' in Figure 6; use consistent terminology and units across all cost plots and tables.","section":"Figures 4 and 6"}],"recommendation":"reject","confidential_remarks":"The paper's own Table 2 refutes the headline safety claim, and the theoretical support is delegated to prior work without showing that it holds under the implemented approximations. The issues are load-bearing and would require substantially new experimental and theoretical work to resolve, so I recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central safety claim is contradicted by the paper's own Table 2. With c=1, SS-MARL(PS) test costs are 2.06, 4.11, and 7.00 for n=3, 6, 9, all 2–7x over the stated bound. The sentence in Section 5.2 claiming that SS-MARL 'satisfies the cost constraint' at n=9 sits directly above a table that says otherwise. The appendix closes the only escape hatch: c is treated as an undiscounted per-episode bound (average step cost 0.01 over 100 steps), so a discounting reinterpretation won't fix it. The load-bearing claim is not supported.\n\nWhat is actually new: the multi-constraint recovery step (Eq. 7–9) for mi≥1 extends CPO/MACPO, which only handle a single constraint. That is a legitimate small contribution. The GNN message-passing architecture with agent/graph aggregation for variable-size observations is a reasonable design, and zero-shot transfer to 96 agents is a solid empirical result, backed by hardware experiments on 3 and 6 rovers. I also credit the authors for being explicit that the cost-satisfaction part of the theory comes from Gu et al. (2023) under an exact-value-function assumption.\n\nThe soft spot is the gap between that theory and the implementation. The guarantee assumes exact value functions; the implementation uses learned critics, GAE, and local observations, and the recovery step is a heuristic with no feasibility proof under those approximations. The paper states the assumption in Section 2.1 and in the appendix, so this is not a hidden flaw, but it means the experiments need to do the work of showing feasibility, and they don't. The scalability comparison is interesting, but it cannot carry the safety claim.\n\nWho this is for: anyone working on safe MARL evaluation or scalable policy architectures. I would send it to peer review because the flaw is fixable and the core idea has merit, but I would not accept it as is. A responsible handling would ask for a corrected cost metric, a demonstration that learned critics preserve feasibility, or a substantially softened safety claim. My own verdict is skeptical, but the paper is serious and worth engaging.","headline":"The paper's central safety claim is falsified by its own Table 2, but the multi-constraint recovery step and GNN architecture are worth a second look.","tokens_in":23388,"tokens_out":3070,"would_cite":false,"duration_ms":24717,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SS-MARL claims safe multi-agent reinforcement learning that scales zero-shot from 3 to 96 agents by combining graph message passing with constrained joint policy optimization.","keywords":["safe multi-agent reinforcement learning","constrained Markov game","graph neural network","zero-shot transfer","scalability","trust-region policy optimization","cooperative navigation","multi-agent systems"],"falsifier":"Run the final SS-MARL(PS) policy from Table 2 for 100 test episodes at n=3, 6, and 9 and record the distribution of per-agent per-episode costs; if mean costs stay above the declared c=1 bound (2.06, 4.11, and 7.00 respectively), the central safety claim is refuted as stated.","tokens_in":22198,"feed_emoji":"🤖","tokens_out":6304,"duration_ms":52172,"temperature":0.7,"pith_summary":"The paper sets out to solve two problems at once: multi-agent reinforcement learning that respects hard safety constraints instead of soft reward penalties, and policies that transfer to far more agents than they were trained with. It claims that both can be achieved in one framework, called SS-MARL, by combining a graph neural network that aggregates variable-size local observations and communications with a constrained joint policy optimization that keeps expected collision costs under a user-set bound. On cooperative navigation tasks, the paper reports that SS-MARL reaches goals while keeping collision costs lower than reward-shaping baselines, and that a model trained with three agents keeps a high success rate when tested with up to 96 agents. A sympathetic reader would take the central claim to be that safety constraints and scalability are not in conflict, and that both can be delivered by one architecture plus one constrained updating rule.","feed_headline":"SS-MARL scales safe multi-agent RL from 3 to 96 agents","feed_subtitle":"Three-agent training transfers to 96-agent swarms with collision costs kept low.","key_machinery":"The load-bearing mechanism is a graph-attention message-passing backbone with two aggregation modes. The actor uses agent aggregation, selecting the learned embedding of the agent's own vertex after several message-passing rounds, so its input dimension never depends on agent count; the critic and cost critic use graph aggregation that attention-pools all agent vertices into fixed-length reward and cost value estimates. On the optimization side, the paper adapts trust-region policy optimization to the multi-agent setting and adds a weighted recovery step for the case where a policy update violates several cost constraints at once.","core_discovery":"SS-MARL's central proposal is a combination: encode the multi-agent system as a graph whose vertices are agents, goals, and obstacles; pass messages along perception and communication edges with attention; aggregate per-agent to form the actor's local representation and graph-wide to form the critic and cost critic; then update policies with a trust-region constrained optimization that maximizes a reward surrogate subject to each agent's cost constraints. The paper argues that this yields monotonic reward improvement and cost constraint satisfaction in the idealized setting where value functions are known, and that the network's fixed-length representations let the same policy act in environments with different numbers of agents. Empirically it claims SS-MARL outperforms fixed-input safe MARL and reward-shaping baselines in both reward and collision cost, and demonstrates zero-shot transfer from three agents to 96 agents.","pith_inferences":["Editorial extension: The safety guarantee is only as strong as the learned cost critics that replace the exact value functions used in the theory; the paper's own Table 2 reports average costs of 2.06, 4.11, and 7.00 against a declared bound of c=1, so in the reported test setting the constraint is not actually met.","Editorial extension: The attention weights over communication edges could be inspected to see whether agents learn to ignore far-away or irrelevant entities; if they do, the method might scale to even larger swarms with bounded communication load.","Editorial extension: A natural next test is whether zero-shot transfer holds under shifted dynamics, such as denser obstacles or heterogeneous agents, where the graph structure changes more dramatically than simply adding more identical agents."],"forward_implications":["A single trained policy can be deployed in environments with many more agents than seen in training, since the actor's input and output sizes do not depend on the number of agents.","Safety becomes an adjustable upper bound on expected per-episode cost, so the same algorithm can be tuned between conservative and reward-seeking behavior by changing one parameter.","The recovery step extends constrained multi-agent updates to settings with more than one cost constraint per agent, which earlier trust-region safe MARL methods do not handle.","Because policy sharing among homogeneous agents speeds convergence while retaining safety, the method is practical for large homogeneous fleets."],"supporting_citations":[{"why":"Supplies the theory of constrained multi-agent policy updates under known value functions, including the cost-satisfaction part of the monotonic-improvement theorem, and provides the RMACPO baseline.","marker":"[Gu et al., 2023]"},{"why":"Provides the sequential multi-agent trust-region update and the monotonic-improvement framework that SS-MARL extends to constraints.","marker":"[Kuba et al., 2022]"},{"why":"Provides the trust-region constrained update and the single-constraint recovery step that SS-MARL generalizes to multiple cost constraints.","marker":"[Achiam et al., 2017]"},{"why":"Supplies the TRPO bound and trust-region step used in the constrained optimization and in the recovery update.","marker":"[Schulman, 2015]"},{"why":"Supplies generalized advantage estimation, used to compute reward and cost advantages from the critics.","marker":"[Schulman et al., 2015]"},{"why":"Serves as the scalability baseline (InforMARL) that aggregates local observations with attention and is compared against in zero-shot transfer.","marker":"[Nayak et al., 2023]"},{"why":"Provides the Multi-Agent Particle Environment that the paper extends into Safe MPE for experiments.","marker":"[Lowe et al., 2017]"},{"why":"Serves as the transformer-based communication scalability baseline (TEM) in the zero-shot transfer comparison.","marker":"[Guo et al., 2023]"}],"fun_headline_variants":["SS-MARL: safe multi-agent RL that scales from 3 to 96 agents","Graph-based safe MARL scales to 96 agents with zero-shot transfer","Safe RL for multi-agent systems scales via graph message passing","SS-MARL transfers safe policies from 3 to 96 agents without retraining","Scalable safe MARL: train on 3 agents, deploy on 96"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The safety guarantee assumes the value functions used in the constrained update are exact; when they are approximated by learned critics operating on local observations and communication, the guarantee may break.","fun_headline_variants_meta":{"raw":{"variants":["SS-MARL: safe multi-agent RL that scales from 3 to 96 agents","Graph-based safe MARL scales to 96 agents with zero-shot transfer","Safe RL for multi-agent systems scales via graph message passing","SS-MARL transfers safe policies from 3 to 96 agents without retraining","Scalable safe MARL: train on 3 agents, deploy on 96"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1395,"prompt_tokens":856,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":472,"tokens_out":539,"duration_ms":4365,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:39:02.274007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final SS-MARL(PS) policy from Table 2 for 100 test episodes at n=3, 6, and 9 and record the distribution of per-agent per-episode costs; if mean costs stay above the declared c=1 bound (2.06, 4.11, and 7.00 respectively), the central safety claim is refuted as stated.","supporting_citations":[{"cited_title":"Trust region policy optimisation in multi-agent reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the sequential multi-agent trust-region update and the monotonic-improvement framework that SS-MARL extends to constraints."},{"cited_title":"Constrained policy optimization","cited_arxiv_id":null,"evidence_quote":"Provides the trust-region constrained update and the single-constraint recovery step that SS-MARL generalizes to multiple cost constraints."},{"cited_title":"Scalable multi-agent reinforcement learning through intelligent information aggregation","cited_arxiv_id":null,"evidence_quote":"Serves as the scalability baseline (InforMARL) that aggregates local observations with attention and is compared against in zero-shot transfer."},{"cited_title":"Multi-agent actor-critic for mixed cooperative-competitive environments","cited_arxiv_id":null,"evidence_quote":"Provides the Multi-Agent Particle Environment that the paper extends into Safe MPE for experiments."},{"cited_title":"Scalable communication for multi-agent reinforcement learning via transformer-based email mechanism","cited_arxiv_id":null,"evidence_quote":"Serves as the transformer-based communication scalability baseline (TEM) in the zero-shot transfer comparison."}],"review_version":1}