{"id":"9299e833-f235-4bf2-ae02-c0a5a365e91e","arxiv_id":"2504.15425","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Def-MARL is a CTDE safe MARL algorithm that solves the multi-agent safe optimal control problem via a distributed epigraph form, achieving near-100% safety rates in simulation and hardware.","lead":"This paper introduces Def-MARL, a multi-agent reinforcement learning algorithm that uses the epigraph form of a constrained optimal control problem to enforce zero constraint violation. It proves that the centralized problem can be solved in a distributed way during execution, and shows strong safety and performance in simulations and on Crazyflie drones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's injectivity assumption is generically false (flat z-tail), so Lemma 3's monotonicity and the distributed z=max_i z_i guarantee are unproven as stated.","rationale":"The reader identified the injectivity assumption of Theorem 1 as the weakest point; this stress-test agrees and sharpens it: the assumption is not just unverified but provably false for large z in any problem with finite cost, because the inner problem becomes minimization of Vh alone, making Vl constant over an interval. Lemma 3's proof is the only place monotonicity of Vh in z is established, and it relies on injectivity in Case 2; with the assumption false, the proof does not go through. As a result, the central theoretical contribution — that the centralized epigraph outer problem can be solved distributively as z = max_i z_i — is not proven for the general problem class the paper claims. The practical method may still work (the empirical evaluations show near-100% safety and good performance), and the theorem's conclusion might be salvageable by proving monotonicity directly or by a weaker assumption, so the appropriate verdict remains CONDITIONAL rather than REJECT. This matches the reader's CONDITIONAL verdict, so no change is recommended. The paper should be asked to either repair Lemma 3 with a valid proof of monotonicity, replace the injectivity assumption with a testable or weaker condition, or explicitly restrict the theorem's scope to settings where the assumption can be guaranteed. The proposed MDP test would settle whether the theorem's conclusion actually holds when its assumption fails.","tokens_in":29641,"tokens_out":24415,"duration_ms":206225,"concrete_test":"On a small finite MDP (e.g., two states, two actions, with a collision constraint and a scalar cost), compute the exact optimal policy pi(.,z) for a dense grid of z values via dynamic programming on the augmented state [x,z] per Proposition 1. Plot Vl(x0; pi(.,z)) and Vh_i(o_i; pi(.,z)) versus z. Check (i) whether any two distinct z values yield the same Vl (injectivity); (ii) whether Vh is non-increasing in z; and (iii) whether z* equals max_i z_i with z_i = min{z : Vh_i(pi(.,z)) <= 0}. If Vl has flat regions while Vh is monotone and the equality still holds, the theorem's conclusion survives but the stated assumption is false and must be weakened; if Vh is non-monotone or the equality fails, the distributed safety guarantee in Theorem 1 is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-C states Theorem 1 under 'Assume no two unique values of z achieves the same unique cost', i.e., the map z -> Vl(x; pi(.,z)) is injective. This is not merely unverified; it is generically false for the paper's own setting. For all sufficiently large z, the inner-problem objective max{Vh, Vl-z} is dominated by Vh (the cost constraint is inactive), so the optimal policy pi(.,z) minimizes Vh alone and is independent of z over an interval; hence Vl(pi(.,z)) is constant and injectivity fails whenever a safest policy has finite cost, which holds for all bounded-cost MDPs including the MPE and MuJoCo tasks. Consequently, Lemma 3 (Appendix B) — which uses injectivity to prove the crucial monotonicity Vh(pi_{z*+epsilon}) <= Vh(pi_{z*}) — does not apply in these regimes. Without that monotonicity, the double-inequality proof of z* = max_i z_i collapses: z_distr may fall in a region where some Vh_i(pi(.,z_distr)) > 0 even though each z_i is individually the first safe z. The zero-violation guarantee for distributed execution is therefore not established by the proof as written; the empirical safety rates near but not exactly 100% are consistent with this gap and with the acknowledged NN estimation errors, but the paper's theoretical claim that the centralized epigraph outer problem is equivalently computed as max_i z_i remains unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Def-MARL, a centralized-training/distributed-execution MARL algorithm for the multi-agent safe optimal control problem (MASOCP) with zero constraint violation. The method reformulates the constrained problem in epigraph form with an auxiliary cost-upper-bound variable z, trains a z-conditioned policy and value functions with PPO and GNNs, and during execution solves a distributed outer problem in which each agent computes the smallest z_i making its local constraint-value function Vh_i non-positive, with the joint z taken as max_i z_i. The central theoretical claim (Theorem 1) is that this distributed computation is equivalent to the centralized epigraph outer problem under an injectivity assumption on the map from z to the optimal cost. Experiments on six MPE tasks and two Safe Multi-agent MuJoCo tasks compare Def-MARL against penalty and Lagrangian baselines, reporting near-100% safety with lower cost and more stable training using fixed hyperparameters, plus Crazyflie hardware demonstrations against MPC baselines. The paper also reports generalization to MAS sizes up to 512 agents and ablations on z-communication and the safety margin parameter xi.","tokens_in":29977,"tokens_out":13585,"duration_ms":131806,"significance":"If the theoretical decomposition is correct, this is a valuable and non-obvious contribution: it extends single-agent epigraph-form safe RL to the multi-agent CTDE setting, provides a distributed execution rule based on local constraint-value functions, and avoids the well-documented training instability of Lagrangian methods at zero violation threshold. The empirical package is strong: eight simulated tasks across two simulators with consistent hyperparameters, comparison against multiple penalty and Lagrangian baselines, ablations, hardware experiments on Crazyflie drones, and a public code project. The generalization experiments to hundreds of agents and the hardware success rates are particularly interesting. The main qualification is that the zero-violation guarantee is not achieved in simulation (safety rates are often 93-100%), and the proof of Theorem 1 rests on an injectivity assumption that is unverified and appears to be violated in the paper's own settings. The paper would be a solid contribution after the theoretical claim is repaired or appropriately weakened.","major_comments":[{"comment":"Theorem 1 is stated under the assumption that no two distinct values of z yield the same unique optimal cost, i.e., the map z -> Vl(x; pi(.,z)) is injective. This assumption is load-bearing: the proof of Lemma 3 uses it in Case 2 to conclude from Vl(pi_{z*}) = Vl(pi_{z*+epsilon}) that pi_{z*} = pi_{z*+epsilon}, which yields the monotonicity Vh(pi_{z*+epsilon}) <= Vh(pi_{z*}) needed for the second half of the double-inequality proof of z* = max_i z_i. Without that conclusion, the proof only shows consistency inequalities and does not establish that Vh_i(pi(.,z_distr)) <= 0 for all i, so the distributed outer problem can return a z at which some agent is unsafe even though each individual z_i is the first safe z for that agent. The assumption is also in direct tension with Remark 1: for all sufficiently large z, the inner objective max{Vh, Vl - z} is dominated by Vh, so the optimal policy minimizes Vh alone and is independent of z over an entire interval whenever a least-violating policy has finite cost. All environments in the paper have bounded finite-horizon costs, so such a flat tail exists and this generic failure of injectivity applies. The theorem as stated therefore does not cover the paper's own experimental settings, and the claimed equivalence between the centralized epigraph outer problem and z = max_i z_i is not established by the proof as written. Please either prove monotonicity of Vh_i(pi(.,z)) under weaker assumptions, state and empirically verify an explicit nonincreasing property of Vh_i in z, or weaken the claim to a conservative bound with a margin.","section":"Section IV-C, Appendix B (Lemma 3), Eq. (15)"},{"comment":"The paper defines MASOCP (2) as requiring zero constraint violation, and the abstract states that Def-MARL \"satisfies safety constraints\", but the simulation results do not achieve zero violation: Table II reports safety rates of 96.9-99.0%, Table III reports 93.4-100%, Table I reports 99.5-99.8%, and Fig. 4 shows similar near-but-not-perfect values. The Limitations section correctly concedes that guarantees do not hold under inexact minimization of the losses or with learned value functions. The empirical claims should be reworded to \"near-100% safety\" in the abstract and results, and the gap between the idealized zero-violation guarantee and the learned-function-approximation setting should be presented as a central caveat rather than relegated to the final limitations paragraph. This does not diminish the empirical comparison, but the current wording overstates the extent to which problem (2) is solved in practice.","section":"Section V-B, Tables I-III, Fig. 4, Section VIII"},{"comment":"The main experiments disable z-communication and use the approximation z <- z_i, while Theorem 1 and Algorithm 2 describe the communicated variant z = max_i z_i. The paper states this discrepancy and provides an ablation (Table II), but the reported headline results are all for the approximate variant. Since the approximation is not covered by the theorem, the paper should make this separation prominent in Section V-A and in the abstract, and should not imply that the experimental safety/cost numbers validate the exact distributed equivalence of Theorem 1. The ablation shows the two variants are close in these environments, which is useful evidence, but it is not a substitute for verifying the theorem's assumptions or for a theoretical statement about the approximate variant.","section":"Section V-C and Section IV-C"}],"minor_comments":[{"comment":"In the proof of Lemma 3, after considering the case Vh(x; pi_{z*}) >= Vl(x; pi_{z*}) - (z* + epsilon), the text says \"We thus suppose that Vh(x; pi_{z*}) >= Vl(x; pi_{z*}) - (z* + epsilon)\", which should be the complementary strict inequality \"<\" for the rest of the proof.","section":"Appendix B, Lemma 3"},{"comment":"The phrase \"Assume no two unique values of z achieves the same unique cost\" is grammatically unclear and should be restated as a precise injectivity condition, e.g., \"Assume the map z -> Vl(x; pi(.,z)) is injective for the relevant range of z.\"","section":"Section IV-C"},{"comment":"The claim that Def-MARL uses constant hyperparameters across all environments is stronger than the implementation: zmax is computed per environment (Eqs. 63-64), the number of Vh GNN layers differs for ConnectSpread (Table V), and training steps vary by environment. Please clarify which hyperparameters are fixed and which are environment-specific.","section":"Appendix E-C, Table IV"},{"comment":"There is a typo in the hyperparameter table: \"massage passing dimension\" should be \"message passing dimension\".","section":"Table IV"},{"comment":"The abstract's claim that Def-MARL \"satisfies safety constraints\" should be qualified as \"near-100% safety\" or \"satisfies constraints up to value-function estimation error\", consistent with the numbers in Fig. 4 and Tables I-III.","section":"Abstract and Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is solid and the paper is well within scope for an RSS venue, but the central decomposition theorem currently has a gap that is likely to be reproducible by reviewers: the injectivity assumption is not tested, is contradicted by the flat-tail behavior noted in Remark 1, and is used essentially in the proof of Lemma 3. I would encourage the editor to request a major revision rather than reject, because the gap appears fixable by either proving monotonicity under weaker conditions, adding a conservative safety margin to the distributed z rule, or explicitly reframing Theorem 1 as a heuristic with empirical validation. The zero-violation overclaim in the abstract should also be corrected in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it actually delivers what it promises: a distributed version of the epigraph safe-RL idea, with new theory (Theorem 1) showing the outer problem reduces to max_i z_i, plus careful experiments across eight simulated tasks and real Crazyflie flights. Second, the theory rests on an injectivity assumption that is stated in one line and never checked, and the paper's headline claim of zero constraint violation is not matched by its own numbers (safety rates run 97–100%, not 100%). Neither issue sinks the work, but both need airing.\n\nThe genuinely new piece is Theorem 1. The single-agent epigraph method is prior work; the extension to multi-agent with per-agent value functions and a consensus step is a real, non-obvious step. The proof is nontrivial and mostly coherent. On the empirical side, the paper does a good job: it compares against penalty and Lagrangian baselines with multiple hyperparameters, shows Def-MARL is closest to the top-left safety/cost corner in all environments, and demonstrates stable training with one hyperparameter set. The hardware experiments are a plus, especially the corridor task where Def-MARL beats centralized and decentralized MPC.\n\nNow the soft spots, in order of seriousness. The injectivity assumption in Theorem 1 ('no two unique values of z achieves the same unique cost') is not just unverified; it is questionable for the authors' own setting. For large z, the inner objective max{V_h, V_l - z} is dominated by V_h, so the policy becomes independent of z over an interval—injectivity fails. If that is right, Lemma 3's monotonicity step and the distributed equality z* = max_i z_i are not proven as stated. The paper never tests injectivity or discusses this regime. This is a load-bearing gap for the theory, though the empirical success suggests the practical method still works. Second, the paper calls the problem 'zero constraint violation' but reports 97–100% safety rates in simulation; that overstates the guarantee, and the limitations section does acknowledge NN estimation error. Third, the proof of Lemma 3 has typos in the case split (one branch states the wrong inequality), which made me reread it several times.\n\nWho is this for? Anyone working on safe MARL or multi-robot control will find the algorithm useful and the empirical comparison informative. It deserves a serious referee: the contribution is meaningful, the evaluation is thorough, and the theoretical gap is fixable. I'd send it to review, asking the authors to address the injectivity assumption directly or drop the theorem to a conditional statement.","headline":"A solid, honestly evaluated extension of the authors' epigraph RL to multi-agent safe control, with a genuine distributed-decomposition theorem; the unverified injectivity assumption and the gap between 'zero violation' and the 97-100% empirical safety rates are the main things I'd want fixed before trusting the theory.","tokens_in":81,"tokens_out":890,"would_cite":true,"duration_ms":36579,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that the centralized epigraph-form safe optimal control problem can be solved distributedly, with each agent finding its own minimal cost bound and the team taking the maximum.","keywords":["multi-agent safe optimal control","zero constraint violation","epigraph form","constrained Markov decision process","centralized training distributed execution","proximal policy optimization","graph neural network","Crazyflie"],"falsifier":"Run Def-MARL on a task engineered to make the learned policy yield equal cumulative cost for two different $z$ values (for instance, symmetric goals with identical distances), record $z_i$ and $z=\\max_i z_i$ during execution, and compare with the centralized outer solution found by exhaustive search over $z$; if they differ, Theorem 1's decomposition fails in that regime. A simpler probe: evaluate the learned $V_i^h(o_i;\\pi(\\cdot,z))$ over a fine $z$-grid and check monotonicity.","tokens_in":29447,"feed_emoji":"🤖","tokens_out":5852,"duration_ms":50935,"temperature":0.7,"pith_summary":"Def-MARL tackles the multi-agent safe optimal control problem, where safety means zero constraint violation at every step. The paper rewrites the constrained problem in epigraph form, turning it into an outer one-dimensional search over a cost bound $z$ and an unconstrained inner policy optimization. It then proves (Theorem 1) that the centralized outer problem is equivalent to $z=\\max_i z_i$, where each agent solves a local problem using only its own constraint-value function. A sympathetic reading of the evidence: across eight simulated tasks and Crazyflie hardware experiments, Def-MARL keeps near-100% safety with costs matching the aggressive baselines, while training is stable with constant hyperparameters. This matters because Lagrangian methods become unstable when the violation threshold is zero, and Def-MARL removes that instability without sacrificing the safety constraint.","feed_headline":"Epigraph form gives multi-agent RL hard safety with stable training","feed_subtitle":"Def-MARL splits the safe-control problem into per-agent root-finds, matching cost baselines while keeping near-100% safety.","key_machinery":"The epigraph form is the central object: a constrained optimization $\\min_\\pi J(\\pi)$ subject to $h(\\pi)\\le 0$ is replaced by $\\min_z z$ subject to $\\min_\\pi \\max\\{h(\\pi), J(\\pi)-z\\}\\le 0$. The paper extends this to multi-agent systems by defining the total value function $V(x,z;\\pi)=\\max_i \\max\\{V_i^h(o_i;\\pi), V^l(x;\\pi)-z\\}$ and showing it satisfies dynamic programming in the augmented state with $z_{k+1}=z_k-l(x_k,u_k)$. This dynamic programming property makes the inner policy problem solvable by PPO with GNN value functions, and Theorem 1 then decomposes the outer problem across agents, replacing a centralized optimization with local root-finding plus a max.","core_discovery":"The paper's central claim is that the multi-agent safe optimal control problem with hard safety constraints can be solved by a centralized-training, distributed-execution algorithm built on the epigraph form. Concretely, for the joint policy $\\pi$ and per-agent constraint values $V_i^h$, the safety constraint $V^h(x;\\pi)\\le 0$ is equivalent to $\\max_i V_i^h(o_i;\\pi)\\le 0$, and the outer epigraph objective becomes $z=\\max_i z_i$ with $z_i=\\min\\{z' : V_i^h(o_i;\\pi(\\cdot,z'))\\le 0\\}$ (Theorem 1, under an injectivity assumption). During execution each agent solves its own one-dimensional root-finding problem for $z_i$ using its learned local constraint-value function, then takes the maximum, so no centralized cost-value function is needed online. Simulations on modified MPE and Safe Multi-agent MuJoCo, plus Crazyflie hardware experiments, show near-100% safety and costs competitive with the best unsafe baselines.","pith_inferences":["The injectivity assumption behind Theorem 1 is never verified; a natural testbed is to construct a symmetric goal configuration where two different cost bounds yield equal cumulative cost and compare the distributed $z$ with the centralized epigraph optimum.","The paper's observation that omitting $z$-communication barely hurts performance suggests a stronger statement may hold, but it also means the practical algorithm operates outside the theorem's conditions; quantifying the gap would turn an empirical finding into a characterization.","The same epigraph machinery could be combined with a certified constraint-value function, such as one trained with formal reachability margins, to turn the near-100% safety into a provable guarantee, which the paper itself lists as an open issue.","The homogeneous-MAS assumption and GNN backbone are conveniences, not requirements; Theorem 1 only needs per-agent constraint-value functions, so heterogeneous-agent extensions are a plausible next step."],"forward_implications":["With Def-MARL, zero-constraint-violation safety is achieved without the usual penalty-performance trade-off: safety matches the most conservative baselines while cost matches the least safe ones.","Distributed execution needs only a local constraint-value function and a one-dimensional root-find per agent; no centralized value network is required at run time.","Because the epigraph objective adds $z$ instead of multiplying by a Lagrange multiplier, policy gradients do not blow up with constraint severity, giving smoother training curves than Lagrangian baselines.","Increasing the safety buffer $\\xi$ in the outer problem (toward $\\nu$) trades a small cost increase for higher safety rates, giving a practical dial for robustness to estimation error.","A policy trained with 8 agents generalizes to 512 agents at constant density with safety rates near 99.5%, and hardware trials on Crazyflie quadcopters show 100% success in corridor and inspect tasks."],"supporting_citations":[{"why":"Supplies the single-agent epigraph-form method and the equivalence proof that Def-MARL extends to multi-agent systems.","marker":"[66]"},{"why":"Defines the epigraph form of a constrained optimization problem that underlies the whole reformulation.","marker":"[10]"},{"why":"Provides the Safe Multi-agent MuJoCo benchmark and the MAPPO-Lagrangian baseline compared in the experiments.","marker":"[32]"},{"why":"Provides the MPE environments and the InforMARL penalty-method baseline used for comparison.","marker":"[51]"},{"why":"Supplies the MAPPO training scheme that Def-MARL's centralized inner-loop PPO update follows.","marker":"[80]"},{"why":"Supplies PPO, the policy optimization algorithm used to train the z-conditioned policy.","marker":"[63]"},{"why":"Supplies Chandrupatla's root-finding method used to solve each agent's one-dimensional outer problem with z.","marker":"[12]"},{"why":"Supplies the graph-transformer architecture used for the policy and value networks and for message passing among agents.","marker":"[65]"}],"fun_headline_variants":["Epigraph form powers distributed safe MARL with hard constraints","Def-MARL: epigraph form yields stable, safe multi-agent RL","Zero-constraint violation in multi-agent RL via epigraph form","Distributed epigraph MARL achieves near-100% safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that no two distinct values of the cost bound $z$ produce the same cumulative cost under the learned policy, so the constraint-value function is monotone in $z$; the paper assumes this in Theorem 1 but does not test it.","fun_headline_variants_meta":{"raw":{"variants":["Epigraph form powers distributed safe MARL with hard constraints","Def-MARL: epigraph form yields stable, safe multi-agent RL","Zero-constraint violation in multi-agent RL via epigraph form","Distributed epigraph MARL achieves near-100% safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1540,"prompt_tokens":966,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":582,"tokens_out":574,"duration_ms":4966,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:27:09.213850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Def-MARL on a task engineered to make the learned policy yield equal cumulative cost for two different $z$ values (for instance, symmetric goals with identical distances), record $z_i$ and $z=\\max_i z_i$ during execution, and compare with the centralized outer solution found by exhaustive search over $z$; if they differ, Theorem 1's decomposition fails in that regime. A simpler probe: evaluate the learned $V_i^h(o_i;\\pi(\\cdot,z))$ over a fine $z$-grid and check monotonicity.","supporting_citations":[{"cited_title":"Solving stabilize-avoid optimal control via epigraph form and deep reinforce- ment learning","cited_arxiv_id":null,"evidence_quote":"Supplies the single-agent epigraph-form method and the equivalence proof that Def-MARL extends to multi-agent systems."},{"cited_title":"Safe multi-agent reinforcement learning for multi-robot control","cited_arxiv_id":null,"evidence_quote":"Provides the Safe Multi-agent MuJoCo benchmark and the MAPPO-Lagrangian baseline compared in the experiments."},{"cited_title":"Scalable multi-agent reinforcement learning through intelligent information aggregation","cited_arxiv_id":null,"evidence_quote":"Provides the MPE environments and the InforMARL penalty-method baseline used for comparison."},{"cited_title":"The surprising effectiveness of ppo in cooperative multi-agent games","cited_arxiv_id":null,"evidence_quote":"Supplies the MAPPO training scheme that Def-MARL's centralized inner-loop PPO update follows."}],"review_version":1}