{"id":"20e6ca08-0a2d-4faa-a8c6-dbdda724e379","arxiv_id":"2511.15053","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Coupled-policy constrained MARL is solved by a distributed primal-dual method that exchanges only local estimates and converges to an ε-first-order stationary point with geometrically decaying truncation error.","lead":"Agents that must satisfy safety constraints while cooperating with coupled, neighbor-dependent policies are handled by a new distributed primal-dual algorithm, which keeps sensitive policy parameters and Lagrange multipliers local. The paper proves approximate convergence and illustrates the method on a small GridWorld benchmark against an independent-policy baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gradient estimator in (24) uses each agent's local parameter estimates θ̂^i in the score, while trajectories are sampled from a different joint policy θ̂^j; Lemma 4's proof silently switches to θ̂^j in eq. (96), so the unbiasedness step underlying Theorem 3 does not apply to Algorithm 1 as writt","rationale":"The reader's weakest assumption (Assumption 4) is indeed strong, but I found a more immediate and specific flaw in the proof chain. The central claim is Theorem 3, which depends on Lemma 4's unbiasedness. Algorithm 1's estimator (24) and the proof's target (42) use agent i's estimates, while the sampling distribution uses agent j's estimates. Lemma 4's proof switches notation in eq. (96) to \\hat θ^j, so it proves a different statement. This is an internal inconsistency in the proof, not a matter of consensus. Theorem 2's asymptotic convergence does not fix the per-iteration bias. The positive aspects—exponential decay (Lemma 2), push-sum consensus, and the overall skeleton—are coherent, but the main theorem's proof has a gap. I would reject the current version, or require a major revision that either changes (24) to use \\hat θ^j with corresponding communication assumptions, or rigorously bounds the extra bias. The reader's other concerns (abstract mismatch, missing Corollary 3 proof, weak simulations) remain valid.","tokens_in":37264,"tokens_out":14356,"duration_ms":150041,"concrete_test":"Re-derive Lemma 4(ii) directly from (24), keeping \\hat θ^i_{j,m} in the score. The expectation over T2,T3 and over the executed policy \\hat π_{\\hat θ_m} equals (42) only if the score's parameter equals the sampling policy's parameter; a two-agent softmax example with \\hat θ^1 ≠ \\hat θ^2 shows the equality fails. Alternatively, implement the two-agent tabular softmax MDP of Section VI with κ=κp=1 and compare the sample mean of (24) against g^{...}_{app,i} computed with the actual sampling distribution; a nonzero discrepancy of order ‖\\hat θ^i − \\hat θ^j‖ confirms the bias. Additionally, check whether Section VIII-G eq. (96) should read \\hat θ^i; if the authors intended \\hat θ^j, then Assumption 2 must be revised to give agent i access to agent j's local estimates.","verdict_should_be":"REJECT","load_bearing_attack":"Algorithm 1's policy-gradient estimator (24) is computed with agent i's private estimates \\hat θ^i_{j,m}, \\hat θ^i_{N^E_{j,-j},m} in the score function. But sample trajectories are generated by the executed joint policy \\hat π_{\\hat θ_m}, whose j-th component is π_j(·|\\hat θ^j_{j,m},\\hat θ^j_{N^E_{j,-j},m}) (Section IV-C). Lemma 4(ii) claims \\hat g^{...}_{app,i}(k) is unbiased for g^{...}_{app,i} in (42). In the proof (Section VIII-G), eq. (96) evaluates the score at \\hat θ^j_{j,m} and \\hat θ^j_{N^E_{j,-j},m}, not at \\hat θ^i. Thus the proof establishes unbiasedness only for a modified estimator that uses the executing agent's own estimates. Since \\hat θ^i_{j,m} ≠ \\hat θ^j_{j,m} at finite m (they are different agents' local estimates, converging only asymptotically), the estimator actually implemented in (24) has an unanalyzed bias. Corollary 1 and the proof of Theorem 3 rely on Lemma 4's unbiasedness; without it, the ε-FOSP guarantee is unproven for Algorithm 1 as stated. This is a correctness gap in the main theorem, independent of whether Assumption 4's positivity condition is reasonable.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper considers constrained multi-agent reinforcement learning (CMARL) under coupled policies, where each agent’s policy depends on the states and parameters of its κ_p-hop neighbors. It proposes a distributed primal-dual (DSPD) algorithm in which agents maintain push-sum estimates of other agents’ policy parameters and Lagrange multipliers, exchange true private parameters only with immediate neighbors over a separate time-varying network, and update primal and dual variables from local trajectory samples using a truncated approximation of the coupled policy gradient. The main theoretical claim is that, under Assumptions 1–7 and with epoch count M = exp(1/ε) and batch sizes K_θ = K_μ = log(2/δ)/2ε², Algorithm 1 finds an ε-FOSP with high probability up to an approximation error ε₀(κ,κ_p) determined by the truncation distance κ. The proof proceeds through an exponential-decay lemma, a consensus theorem for private-parameter estimates, and a descent argument on the Lagrangian.","tokens_in":37645,"tokens_out":6646,"duration_ms":71256,"significance":"If the result is correct, this would be the first distributed CMARL algorithm with coupled policies and a formal high-probability ε-FOSP guarantee while avoiding exchange of true policy parameters and Lagrange multipliers. The paper has several strengths: the proof architecture is explicit, the exponential-decay property for coupled policies is proved in detail, the consensus analysis is nontrivial, and assumptions are stated rather than hidden. The central claim is plausible but currently rests on a specific unbiasedness assertion that does not match the algorithm as written, and on an omitted proof of a smoothness property that is load-bearing in the main theorem. These issues require careful correction before the main contribution can be accepted.","major_comments":[{"comment":"Corollary 3 states that ∇_θ L(θ,μ) is L_{θθ}-Lipschitz in θ, and its proof is omitted with the remark that it is similar to Lemma 5. This is not a cosmetic omission: the L_{θθ}-smoothness is used directly in Eq. (120) of the proof of Theorem 3 to lower-bound the Lagrangian difference. A central theorem cannot rely on an unproved, load-bearing statement. The authors should provide the complete proof of Corollary 3, or explicitly derive the smoothness inequality used in (120) from Assumptions 3–5.","section":"Section V-C, Corollary 3"},{"comment":"Assumption 4 requires ξ^{π_θ}_ρ(s,a) > 0 for every joint state-action pair and every policy θ. This is load-bearing because the truncated Q-function in (34) is defined via conditional weights ξ^{π_θ}_ρ(·|·) in (35); if the denominator in (35) is zero, the weights are undefined and the exponential-decay argument, Theorem 1, and the approximation error ε₀(κ,κ_p) all break down. The assumption is stronger than a standard exploration assumption, as it must hold even for policies that attempt to satisfy safety constraints and may steer the system away from some joint state-action pairs. The paper should justify that this assumption is satisfied in the intended applications, or state clearly that the theoretical result applies only under this additional regularity condition and explain what can fail without it.","section":"Assumption 4; Eqs. (34)–(35)"}],"minor_comments":[{"comment":"The abstract claims an approximation error O(γ^{(κ+1)/κ_p}), while Theorem 1 and Theorem 3 prove a bound with exponent h(κ,κ_p)+1, where h is defined in (30). For non-integer (κ+1)/κ_p, h+1 equals the ceiling, which gives a smaller error than the abstract’s expression. To be precise, the abstract and conclusion should state the bound in terms of h(κ,κ_p).","section":"Abstract and Section VII"},{"comment":"The abstract states that simulations are performed on a wireless access-control network, but the simulation section uses a 5×5 GridWorld environment with four agents. The abstract should be corrected to match the experimental setup.","section":"Abstract and Section VI"},{"comment":"The proof of Theorem 2 provides convergence of the Lagrange-multiplier estimates in detail, but the corresponding convergence of the policy-parameter estimates θ̂^i_{j,m} is omitted with the comment that the proof is analogous. Since Theorem 3’s rate depends on this statement, a complete proof should be included rather than cited by analogy.","section":"Theorem 2"},{"comment":"In the paragraph before Algorithm 1, the pseudocode is referred to as the “DSCP algorithm”; this should be “DSPD algorithm.” There are also minor grammatical issues such as “time-varying networks” in Assumption 2 and “the the upper bounds” in Section V-C.","section":"Algorithm 1 text"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the mismatch between the implemented gradient estimator and the unbiasedness proof in Lemma 4(ii). This is not merely a typographical detail; it sits at the foundation of Theorem 3. If the authors can either revise Algorithm 1 so that the score is evaluated at the parameters used to generate the trajectories, or provide a rigorous bias analysis for the current estimator, and also supply the omitted proof of Corollary 3, the main result is likely salvageable. In its current form, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front: this is the first CMARL paper I have seen that combines coupled policies with privacy-preserving parameter exchange, and the main convergence theorem is not proven as written because the policy-gradient estimator in Algorithm 1 does not match the one analyzed in Lemma 4.\n\nWhat is genuinely new and good: the coupled-policy model is a natural generalization of the independent-policy setting in [21], and the idea of hiding true parameters and Lagrange multipliers behind push-sum consensus is a worthwhile extension. The exponential-decay analysis in Lemma 2 and Theorem 1 is a real adaptation of the truncation argument, and the proof is mostly detailed, with explicit constants and a clear FOSP definition. If the technical gaps were fixed, this would be a useful contribution to scalable safe MARL.\n\nThe soft spots are not cosmetic. First, the stress-test note is correct: in Algorithm 1, eq. (24) evaluates the score at agent i's local estimates \\hat\\theta^i_{j,m}, but the trajectories are sampled from a joint policy whose j-th component uses \\hat\\theta^j_{j,m}. Lemma 4(ii)'s proof silently switches to \\hat\\theta^j in eq. (96), so the unbiasedness claim does not apply to the algorithm that is actually run. The bias is not shown to be small, and Theorem 3 leans on it directly. That is a load-bearing correctness gap. Second, Corollary 3 — the L-smoothness of the Lagrangian gradient — is used in Theorem 3 but explicitly left unproved with only a note that the proof is omitted. Third, the abstract promises a wireless access-control experiment and an approximation error O(\\gamma^{(\\kappa+1)/\\kappa_p}), but the experiments are a single toy GridWorld and Theorem 1 gives the h(\\kappa,\\kappa_p) exponent, which can differ from the abstract's formula. The simulation also compares against only one baseline, with no code or data, so it adds little credibility.\n\nNone of these seem impossible to fix — the estimator bias could be addressed by changing the algorithm or adding a decaying bias term to the analysis — but as submitted the main guarantee does not follow. This is worth a serious referee because the setting is new and the authors clearly know the literature, but it is not ready for acceptance.","headline":"New setting, but the main convergence theorem has an unproven estimator bias; the paper needs major revision before it can be trusted.","tokens_in":38175,"tokens_out":4256,"would_cite":false,"duration_ms":45422,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A distributed primal-dual algorithm brings coupled-policy constrained multi-agent reinforcement learning to an approximate first-order stationary point while keeping private parameters local.","keywords":["constrained multi-agent reinforcement learning","coupled policies","distributed primal-dual algorithm","epsilon-first-order stationary convergence","policy gradient","push-sum consensus","privacy-preserving distributed optimization","exponential decay of influence"],"falsifier":"Take a two-agent CMARL instance on a line graph with κ=κ_p=1 and choose a constraint-satisfying policy that assigns zero probability to one joint action at a reachable state. Compute the denominator in the visitation-weight definition for that state-action pair: it is zero, so the truncated Q-function is undefined and the paper's Theorem 1 bound cannot be stated. Demonstrating a successful run of the algorithm on such an instance — where the zero-visitation state has positive probability under the unconstrained optimum — would directly contradict Assumption 4 and the derived error bound.","tokens_in":37099,"feed_emoji":"🤖","tokens_out":7059,"duration_ms":71193,"temperature":0.7,"pith_summary":"The paper tackles constrained multi-agent reinforcement learning in which each agent's policy depends not only on its own state and parameters but also on those of its κ_p-hop neighbors. It proposes a distributed primal-dual algorithm (DSPD) where agents use only local state-action and reward information within fixed hop distances, and exchange only estimates of other agents' policy parameters and Lagrange multipliers over a separate time-varying network. The central theoretical claim is that, with high probability, the algorithm reaches an ε-first-order stationary point of the Lagrangian, with an approximation error of O(γ^{(κ+1)/κ_p}) stemming from truncating reward information at distance κ. The result matters because it is the first formal convergence guarantee for coupled-policy constrained MARL, and it shows that private learned variables need not be globally shared. Simulations in a stochastic GridWorld confirm faster objective improvement than an independent-policy baseline while constraints are satisfied.","feed_headline":"Coupled-policy RL hits epsilon-accuracy with private parameters","feed_subtitle":"A distributed primal-dual method gives the first epsilon-FOSP guarantee for coupled-policy constrained MARL using only local information.","key_machinery":"The load-bearing objects are: (1) the neighbors' averaged Lagrangian Q-function, which sums objective and constraint rewards only over agent i's κ+2κ_p-hop neighborhood and replaces the global Q in the policy gradient; (2) the truncated Lagrangian Q-function, built by averaging the local Q-function over all outside-neighbor states and actions using visitation weights, which Lemma 3 shows is exactly equal to the approximated gradient; (3) the exponential decay property, which bounds how far a reward signal can travel in κ_p steps and yields the γ^{h(κ,κ_p)+1} error rate; and (4) the push-sum protocol over a time-varying learning network, which lets each agent maintain provably convergent esti","core_discovery":"On the paper's own terms, the discovery is that the global Lagrangian Q-function in the coupled policy gradient can be replaced by a neighbor-averaged Lagrangian Q-function computed from rewards within κ+2κ_p hops, and the resulting approximation error decays exponentially in the truncation distance. The key bound is that the approximated gradient differs from the exact gradient by at most a constant times γ^{h(κ,κ_p)+1}, where h(κ,κ_p) grows roughly as (κ+1)/κ_p. Each agent's local estimates of other agents' policy parameters and Lagrange multipliers, maintained via a push-sum protocol over a time-varying learning network, converge to the true values at rate O(1/m). Theorem 3 then asserts t","pith_inferences":["The bound suggests a scaling law: to keep a fixed accuracy, the truncation radius κ may need to grow roughly linearly with the coupling distance κ_p, since h(κ,κ_p) ≈ (κ+1)/κ_p. A testable extension is to measure, in a ring network, the minimal κ that achieves a target gradient error for increasing κ_p.","The privacy guarantee is spatial rather than cryptographic: agents still transmit their true policy parameters and Lagrange multipliers to immediate neighbors over the learning network, so the scheme protects against global eavesdropping but assumes neighboring agents are trustworthy.","Because the main theorem controls the stationarity measure E(θ,μ), it does not assert global optimality or constraint satisfaction at every iterate; an interesting extension would be to combine the local-estimate push-sum construction with a constraint-violation bound to obtain a sample-complexity guarantee for feasible policies.","The same machinery of neighbor-averaged Q-functions plus consensus over estimates could be adapted to actor-critic or deep-RL variants, but the O(1/m) estimate-convergence rate and the Lipschitz constants in Lemma 5 would need to be re-established for neural policies."],"forward_implications":["Each agent can execute and improve a coupled policy using only state-action pairs from its 2κ_p-hop neighbors and rewards from its κ+2κ_p-hop neighbors, so per-agent computation and communication stay independent of the total number of agents.","True policy parameters and Lagrange multipliers never need to be broadcast beyond immediate neighbors; because the local estimates converge to the true values, privacy-preserving execution does not sacrifice the stationarity guarantee.","The approximation error decays as γ^{h(κ,κ_p)+1}; choosing the truncation radius κ somewhat larger than the coupling distance κ_p makes the residual term negligible, so near-exact gradient information is achievable locally.","With an exponential iteration count M=exp(1/ε) and sample batches K=log(2/δ)/2ε², the algorithm returns an ε-FOSP with probability at least 1−δ, up to the truncation error — the first such guarantee for coupled-policy constrained MARL.","In the GridWorld experiments, the coupled-policy update improves objective value faster than an independent-policy primal-dual baseline while keeping constraints satisfied throughout learning."],"fun_headline_variants":["Coupled-policy MARL: private epsilon-FOSP via local info","Distributed primal-dual: epsilon-FOSP with neighbor-only exchange","Neighbor-averaged Lagrangian gives epsilon-FOSP in coupled RL","DSPD: coupled-policy RL converges to epsilon-FOSP privately","Local parameters only: epsilon-FOSP for constrained multi-agent RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Assumption 4 — every joint state and action is visited with positive probability under every joint policy parameter θ — is load-bearing: the truncated-Q construction divides by the visitation weight of the inside-neighborhood states and actions, so if a feasible constrained policy makes some joint state-action pair unreachable, the conditional weights are undefined and the exponential-decay argument collapses.","fun_headline_variants_meta":{"raw":{"variants":["Coupled-policy MARL: private epsilon-FOSP via local info","Distributed primal-dual: epsilon-FOSP with neighbor-only exchange","Neighbor-averaged Lagrangian gives epsilon-FOSP in coupled RL","DSPD: coupled-policy RL converges to epsilon-FOSP privately","Local parameters only: epsilon-FOSP for constrained multi-agent RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1105,"prompt_tokens":781,"completion_tokens":324,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":525,"tokens_out":324,"duration_ms":4651,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:28:49.993096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a two-agent CMARL instance on a line graph with κ=κ_p=1 and choose a constraint-satisfying policy that assigns zero probability to one joint action at a reachable state. Compute the denominator in the visitation-weight definition for that state-action pair: it is zero, so the truncated Q-function is undefined and the paper's Theorem 1 bound cannot be stated. Demonstrating a successful run of the algorithm on such an instance — where the zero-visitation state has positive probability under the unconstrained optimum — would directly contradict Assumption 4 and the derived error bound.","supporting_citations":[],"review_version":1}