Pith. sign in

REVIEW 2 major objections 4 minor 31 references

Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

T0 review · 2 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A distributed primal-dual algorithm brings coupled-policy constrained multi-agent reinforcement learning to an approximate first-order stationary point while keeping private parameters local.

desk verdict New setting, but the main convergence theorem has an unproven estimator bias; the paper needs major revision before it can be trusted. read the letter →

arxiv 2511.15053 v3 pith:V4TXID5A submitted 2025-11-19 cs.MA

Pengcheng Dai , He Wang , Dongming Wang , Wenwu Yu This is my paper · ORCID
classification cs.MA
keywords constrainedmulti-agentreinforcementlearningcoupledpoliciesdistributedprimal-dualalgorithmepsilon-first-orderstationaryconvergencepolicygradientpush-sumconsensusprivacy-preservingoptimizationexponentialdecayofinfluence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles constrained multi-agent reinforcement learning in which each agent's policy depends not only on its own state and parameters but also on those of its κ_p-hop neighbors. It proposes a distributed primal-dual algorithm (DSPD) where agents use only local state-action and reward information within fixed hop distances, and exchange only estimates of other agents' policy parameters and Lagrange multipliers over a separate time-varying network. The central theoretical claim is that, with high probability, the algorithm reaches an ε-first-order stationary point of the Lagrangian, with an approximation error of O(γ^{(κ+1)/κ_p}) stemming from truncating reward information at distance κ. The result matters because it is the first formal convergence guarantee for coupled-policy constrained MARL, and it shows that private learned variables need not be globally shared. Simulations in a stochastic GridWorld confirm faster objective improvement than an independent-policy baseline while constraints are satisfied.

What carries the argument

The load-bearing objects are: (1) the neighbors' averaged Lagrangian Q-function, which sums objective and constraint rewards only over agent i's κ+2κ_p-hop neighborhood and replaces the global Q in the policy gradient; (2) the truncated Lagrangian Q-function, built by averaging the local Q-function over all outside-neighbor states and actions using visitation weights, which Lemma 3 shows is exactly equal to the approximated gradient; (3) the exponential decay property, which bounds how far a reward signal can travel in κ_p steps and yields the γ^{h(κ,κ_p)+1} error rate; and (4) the push-sum protocol over a time-varying learning network, which lets each agent maintain provably convergent esti

What would settle it

Take a two-agent CMARL instance on a line graph with κ=κ_p=1 and choose a constraint-satisfying policy that assigns zero probability to one joint action at a reachable state. Compute the denominator in the visitation-weight definition for that state-action pair: it is zero, so the truncated Q-function is undefined and the paper's Theorem 1 bound cannot be stated. Demonstrating a successful run of the algorithm on such an instance — where the zero-visitation state has positive probability under the unconstrained optimum — would directly contradict Assumption 4 and the derived error bound.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the global Lagrangian Q-function in the coupled policy gradient can be replaced by a neighbor-averaged Lagrangian Q-function computed from rewards within κ+2κ_p hops, and the resulting approximation error decays exponentially in the truncation distance. The key bound is that the approximated gradient differs from the exact gradient by at most a constant times γ^{h(κ,κ_p)+1}, where h(κ,κ_p) grows roughly as (κ+1)/κ_p. Each agent's local estimates of other agents' policy parameters and Lagrange multipliers, maintained via a push-sum protocol over a time-varying learning network, converge to the true values at rate O(1/m). Theorem 3 then asserts t

Load-bearing premise

Assumption 4 — every joint state and action is visited with positive probability under every joint policy parameter θ — is load-bearing: the truncated-Q construction divides by the visitation weight of the inside-neighborhood states and actions, so if a feasible constrained policy makes some joint state-action pair unreachable, the conditional weights are undefined and the exponential-decay argument collapses.

Editorial extensions

If this is right

  • Each agent can execute and improve a coupled policy using only state-action pairs from its 2κ_p-hop neighbors and rewards from its κ+2κ_p-hop neighbors, so per-agent computation and communication stay independent of the total number of agents.
  • True policy parameters and Lagrange multipliers never need to be broadcast beyond immediate neighbors; because the local estimates converge to the true values, privacy-preserving execution does not sacrifice the stationarity guarantee.
  • The approximation error decays as γ^{h(κ,κ_p)+1}; choosing the truncation radius κ somewhat larger than the coupling distance κ_p makes the residual term negligible, so near-exact gradient information is achievable locally.
  • With an exponential iteration count M=exp(1/ε) and sample batches K=log(2/δ)/2ε², the algorithm returns an ε-FOSP with probability at least 1−δ, up to the truncation error — the first such guarantee for coupled-policy constrained MARL.
  • In the GridWorld experiments, the coupled-policy update improves objective value faster than an independent-policy primal-dual baseline while keeping constraints satisfied throughout learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bound suggests a scaling law: to keep a fixed accuracy, the truncation radius κ may need to grow roughly linearly with the coupling distance κ_p, since h(κ,κ_p) ≈ (κ+1)/κ_p. A testable extension is to measure, in a ring network, the minimal κ that achieves a target gradient error for increasing κ_p.
  • The privacy guarantee is spatial rather than cryptographic: agents still transmit their true policy parameters and Lagrange multipliers to immediate neighbors over the learning network, so the scheme protects against global eavesdropping but assumes neighboring agents are trustworthy.
  • Because the main theorem controls the stationarity measure E(θ,μ), it does not assert global optimality or constraint satisfaction at every iterate; an interesting extension would be to combine the local-estimate push-sum construction with a constraint-violation bound to obtain a sample-complexity guarantee for feasible policies.
  • The same machinery of neighbor-averaged Q-functions plus consensus over estimates could be adapted to actor-critic or deep-RL variants, but the O(1/m) estimate-convergence rate and the Lipschitz constants in Lemma 5 would need to be re-established for neural policies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper considers constrained multi-agent reinforcement learning (CMARL) under coupled policies, where each agent’s policy depends on the states and parameters of its κ_p-hop neighbors. It proposes a distributed primal-dual (DSPD) algorithm in which agents maintain push-sum estimates of other agents’ policy parameters and Lagrange multipliers, exchange true private parameters only with immediate neighbors over a separate time-varying network, and update primal and dual variables from local trajectory samples using a truncated approximation of the coupled policy gradient. The main theoretical claim is that, under Assumptions 1–7 and with epoch count M = exp(1/ε) and batch sizes K_θ = K_μ = log(2/δ)/2ε², Algorithm 1 finds an ε-FOSP with high probability up to an approximation error ε₀(κ,κ_p) determined by the truncation distance κ. The proof proceeds through an exponential-decay lemma, a consensus theorem for private-parameter estimates, and a descent argument on the Lagrangian.

Significance. If the result is correct, this would be the first distributed CMARL algorithm with coupled policies and a formal high-probability ε-FOSP guarantee while avoiding exchange of true policy parameters and Lagrange multipliers. The paper has several strengths: the proof architecture is explicit, the exponential-decay property for coupled policies is proved in detail, the consensus analysis is nontrivial, and assumptions are stated rather than hidden. The central claim is plausible but currently rests on a specific unbiasedness assertion that does not match the algorithm as written, and on an omitted proof of a smoothness property that is load-bearing in the main theorem. These issues require careful correction before the main contribution can be accepted.

major comments (2)
  1. [Section V-C, Corollary 3] Corollary 3 states that ∇_θ L(θ,μ) is L_{θθ}-Lipschitz in θ, and its proof is omitted with the remark that it is similar to Lemma 5. This is not a cosmetic omission: the L_{θθ}-smoothness is used directly in Eq. (120) of the proof of Theorem 3 to lower-bound the Lagrangian difference. A central theorem cannot rely on an unproved, load-bearing statement. The authors should provide the complete proof of Corollary 3, or explicitly derive the smoothness inequality used in (120) from Assumptions 3–5.
  2. [Assumption 4; Eqs. (34)–(35)] Assumption 4 requires ξ^{π_θ}_ρ(s,a) > 0 for every joint state-action pair and every policy θ. This is load-bearing because the truncated Q-function in (34) is defined via conditional weights ξ^{π_θ}_ρ(·|·) in (35); if the denominator in (35) is zero, the weights are undefined and the exponential-decay argument, Theorem 1, and the approximation error ε₀(κ,κ_p) all break down. The assumption is stronger than a standard exploration assumption, as it must hold even for policies that attempt to satisfy safety constraints and may steer the system away from some joint state-action pairs. The paper should justify that this assumption is satisfied in the intended applications, or state clearly that the theoretical result applies only under this additional regularity condition and explain what can fail without it.
minor comments (4)
  1. [Abstract and Section VII] The abstract claims an approximation error O(γ^{(κ+1)/κ_p}), while Theorem 1 and Theorem 3 prove a bound with exponent h(κ,κ_p)+1, where h is defined in (30). For non-integer (κ+1)/κ_p, h+1 equals the ceiling, which gives a smaller error than the abstract’s expression. To be precise, the abstract and conclusion should state the bound in terms of h(κ,κ_p).
  2. [Abstract and Section VI] The abstract states that simulations are performed on a wireless access-control network, but the simulation section uses a 5×5 GridWorld environment with four agents. The abstract should be corrected to match the experimental setup.
  3. [Theorem 2] The proof of Theorem 2 provides convergence of the Lagrange-multiplier estimates in detail, but the corresponding convergence of the policy-parameter estimates θ̂^i_{j,m} is omitted with the comment that the proof is analogous. Since Theorem 3’s rate depends on this statement, a complete proof should be included rather than cited by analogy.
  4. [Algorithm 1 text] In the paragraph before Algorithm 1, the pseudocode is referred to as the “DSCP algorithm”; this should be “DSPD algorithm.” There are also minor grammatical issues such as “time-varying networks” in Assumption 2 and “the the upper bounds” in Section V-C.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the convergence guarantee is derived from explicit lemmas and standard external results, with no fitted inputs or load-bearing self-citations.

full rationale

The paper's derivation chain is self-contained rather than circular. Theorem 3's bound E(θ_m*, μ_m*+1)=O(ε)+O(ε0(κ,κp)) is obtained from Assumptions 1–7 through an explicit chain of lemmas: Lemma 2 derives the exponential-decay property from the local transition kernels and κp-coupled policies (Section VIII-C); Lemma 3 proves the identity g_app,i = g_tru,i by score-function cancellation (Section VIII-D); Theorem 1 then bounds the approximation error as 2(R_f+μ_maxR_g)BN γ^{h(κ,κp)+1}/(1−γ)^2; Theorem 2 establishes consensus of θ̂ and μ̂ using the external push-sum result [25, Lemma 6] and [28, Lemma 3.1]; Lemma 4 plus Corollary 1 give unbiasedness and Hoeffding concentration; Lemma 5/Corollary 2 propagate the consensus and truncation errors into the gradient error; and the weighted-descent argument in Section VIII-J converts these into the stated rate with the specified M, η, K. No fitted parameter from the GridWorld simulation enters the proof, and no load-bearing step reduces to the authors' own prior work; refs [3],[8],[10] are introductory application citations only. The omitted Corollary 3 proof, and any estimator/score mismatch in Lemma 4, would be correctness/completeness defects, not instances in which a predicted quantity equals its input by construction.

Assumptions & free parameters 8 free parameters · 10 assumptions · 0 invented entities

The framework introduces algorithmic constructs—coupled policies and local estimates—but no new physical entities, forces, or dimensions. The coupled policy is a modeling choice within the stated information-local assumption, and the local estimates are consensus variables. The central theoretical claims rest on the seven stated assumptions plus standard background results; no additional hidden entities are needed.

free parameters (8)
  • κ (truncation distance) = 1 (simulation)
    Controls the radius of reward/state truncation and the error exponent; hand-set in simulation, not fitted.
  • κ_p (policy coupling distance) = 1 (simulation)
    Controls how many neighbors' states/parameters enter each agent's policy; hand-set in simulation.
  • K_μ, K_θ (batch sizes) = 20/40 (simulation); log(2/δ)/(2ε²) in Theorem 3
    Number of sample trajectories used to estimate gradients; prescribed in theory and chosen by hand in experiments.
  • M (number of iterations) = 2500 (simulation); exp(1/ε) in Theorem 3
    Iteration budget; part of the convergence statement and simulation configuration.
  • γ (discount factor) = 0.9 (simulation)
    Discount factor; a problem input, but its value in the experiment is chosen by hand and directly affects the error exponent.
  • Softmax mixing weights (0.9/0.1) = 0.9/0.1
    Mixing weights in the coupled softmax policy (49); chosen by hand for the simulation, not derived.
  • χ_max, χ_min (perturbation bounds) = 0.1, 0.02
    Bounds on the GridWorld transition perturbation; chosen by hand for the simulation.
  • Learning-rate schedules = η_μ=1/2m, η_θ=1/(2m+Lθθ)
    Set in Theorem 3; they are required for the descent argument but are not fitted to data.
assumptions (10)
  • domain assumption Assumption 1: there exists a FOSP (θ*, μ*) with μ*_i ≤ μ_max
    Bounds the dual feasible set; standard in primal-dual constrained optimization, used throughout the proofs.
  • domain assumption Assumption 2: each agent receives state-action pairs from 2κ_p-hop neighbors and rewards from (κ+2κ_p)-hop neighbors, and exchanges parameters only over a separate time-varying network
    Defines the information architecture; the distributed algorithm and privacy claims are built on this exact access pattern.
  • domain assumption Assumption 3: objective and constraint rewards are bounded by R_f and R_g
    Used to bound Q-functions, gradient estimates, and the consensus error terms.
  • domain assumption Assumption 4: positive discounted visitation probability for every state and state-action pair under every policy
    Needed for conditional weights in the truncated Q construction (eq. 35); if false, the exponential-decay argument collapses.
  • domain assumption Assumption 5: log-policy gradients are bounded by B and L-Lipschitz
    Used to bound policy-gradient norms and to establish smoothness of the Lagrangian.
  • domain assumption Assumption 6: time-varying learning networks are uniformly strongly connected
    Required for push-sum consensus convergence (Lemma 6 from [25]); essential for Theorem 2.
  • domain assumption Assumption 7: learning rates η_θ,m and η_μ,m are O(1/m)
    Standard diminishing step-size condition; Theorem 3 instantiates specific schedules.
  • standard math Policy gradient theorem and score-function identities
    Used to derive the coupled policy gradient expression in Lemma 1.
  • standard math Nedic-Olshevsky consensus lemma for time-varying directed graphs
    Provides the geometric convergence of the push-sum protocol used in Theorem 2.
  • standard math Azuma-Hoeffding and measure-theoretic convergence theorems
    Used for the high-probability concentration bounds and unbiasedness arguments in Corollary 1/Lemma 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies." pith.science (2026). https://pith.science/paper/V4TXID5A

@misc{pith2026251115053,
  author       = {Pith},
  title        = {Pith review of: Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4TXID5A}},
  note         = {Machine review of arXiv:2511.15053}
}
abstract

This paper investigates constrained multi-agent reinforcement learning (CMARL) in coupled environments, where agents collaboratively maximize the sum of local objectives while satisfying individual safety constraints. Existing studies face two limitations: (1) most rely on independent policies that fail to capture complex interactions in coupled environments; and (2) agents require the global Lagrange multipliers, which are sensitive learned variables whose global sharing risks exposing private agent-specific information. To overcome these issues, we propose a framework where agents adopt coupled policies that depend on both the states and policy parameters of their $\kappa_p$-hop neighbors, where $\kappa_p>0$ denotes the coupling distance, and develop a distributed and scalable primal-dual (DSPD) algorithm wherein each agent accesses only information within a prescribed local neighborhood. In the proposed algorithm, agents exchange sensitive parameters only with immediate neighbors over a separate time-varying network, while maintaining local estimates to execute the coupled policy. We establish that the proposed algorithm achieves $\epsilon$-first-order stationary convergence with approximation error $\mathcal{O}(\gamma^{\frac{\kappa+1}{\kappa_{p}}})$, where $\kappa>0$ is the truncated distance and $\gamma\in(0,1)$ is discount factor. Simulations on a wireless access-control network demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms, validating its effectiveness.

Figures

Figures reproduced from arXiv: 2511.15053 by the authors.

Figure 1
Figure 1. Performance of Algorithm 1 and SPDAC on the objective function. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Constraint performance of Algorithm 1 and SPDAC. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. The evolution of the estimation errors for agents with respect to the [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 2 linked inside Pith

  1. [21]

    Scalable primal- dual actor-critic method for safe multi-agent RL with general utilities,

    D. Ying, Y . Zhang, Y . Ding, A. Koppel, and J. Lavaei, “Scalable primal- dual actor-critic method for safe multi-agent RL with general utilities,” inproc. Adv. Neural Inf. Process. Syst., vol. 36, 2024

    2024

  2. [1]

    Multiagent deep reinforcement learning for large-scale traffic signal control,

    T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multiagent deep reinforcement learning for large-scale traffic signal control,”IEEE Trans. Intell. Transp. Syst., vol. 21, no. 3, pp. 1086-1095, Mar. 2020

    2020

  3. [2]

    Large-Scale traffic signal control using a novel multiagent reinforcement learning,

    X. Wang, L. Ke, Z. Qiao, and X. Chai, “Large-Scale traffic signal control using a novel multiagent reinforcement learning,”IEEE Trans. Cybern., vol. 51, no. 1, pp. 174-187, Jan. 2021

    2021

  4. [3]

    Applications in traffic signal control: a distributed policy gradient decomposition algorithm,

    P. Dai, W. Yu, H. Wang, and J. Jiang, “Applications in traffic signal control: a distributed policy gradient decomposition algorithm,”IEEE Trans. Ind. Inform., vol. 20, no. 2, pp. 2762-2775, Feb. 2024

    2024

  5. [4]

    Deep reinforcement learning for joint channel selection and power control in D2D networks,

    J. Tan, Y .-C. Liang, L. Zhang, and G. Feng, “Deep reinforcement learning for joint channel selection and power control in D2D networks,”IEEE Trans. Wireless Commun., vol. 20, no. 2, pp. 1363-1378, Feb. 2021

    2021

  6. [5]

    Power allocation in multiuser cellular networks: Deep reinforcement learning approaches,

    F. Meng, P. Chen, L. Wu, and J. Cheng, “Power allocation in multiuser cellular networks: Deep reinforcement learning approaches,”IEEE Trans. Wireless Commun., vol. 19, no. 10, pp. 6255-6267, Oct. 2020

    2020

  7. [6]

    Reinforcement learning based recommender systems: a survey,

    M. M. Afsar, T. Crump, and B. Far, “Reinforcement learning based recommender systems: a survey,”ACM Comput. Surv., vol. 55, no. 7, pp. 1-38, 2022

    2022

  8. [7]

    A survey on reinforcement learning for recommender systems,

    Y . Lin, Y . Liu, F. Lin, L. Zou, P. Wu, and W. Zeng, “A survey on reinforcement learning for recommender systems,”IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 10, pp. 13164-13184, Oct. 2024

    2024

Show all 31 references
  1. [8]

    Distributed reinforcement learning algorithm for dynamic economic dispatch with unknown generation cost functions,

    P. Dai, W. Yu, G. Wen, and S. Baldi, “Distributed reinforcement learning algorithm for dynamic economic dispatch with unknown generation cost functions,”IEEE Trans. Ind. Inform., vol. 16, no. 4, pp. 2258-2267, Apr. 2020

    2020

  2. [9]

    DistributedQ-learning-based online optimization algorithm for unit commitment and dispatch in smart grid,

    F. Li, J. Qin, and W. Zheng, “DistributedQ-learning-based online optimization algorithm for unit commitment and dispatch in smart grid,” IEEE Trans. Cybern., vol. 50, no. 9, pp. 4146-4156, Sept. 2020

    2020

  3. [10]

    Distributed Q-learning algorithm for dynamic resource allocation with unknown objective functions and appli- cation to microgrid,

    P. Dai, W. Yu, and D. Chen, “Distributed Q-learning algorithm for dynamic resource allocation with unknown objective functions and appli- cation to microgrid,”IEEE Trans. Cybern., vol. 52, no. 11, pp. 12340- 12350, Nov. 2022

    2022

  4. [11]

    Safe, multi- agent, reinforcement learning for autonomous driving,

    S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi- agent, reinforcement learning for autonomous driving,”arXiv preprint arXiv:1610.03295, 2016

    2016 arXiv

  5. [12]

    Con strained reinforcement learning has zero duality gap,

    S. Paternain, L. F. Chamon, M. Calvo-Fullana, and A. Ribeiro, “Con strained reinforcement learning has zero duality gap,” inproc. Adv. Neural Inf. Process. Syst., 2019, pp. 7553-7563

    2019

  6. [13]

    Safe policies for reinforcement learning via primal-dual meth- ods,

    Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, Alejandro Ribeiro, “Safe policies for reinforcement learning via primal-dual meth- ods,”IEEE Trans. Autom. Control, vol. 68, no. 3, pp. 1321-1336, Mar. 2022

    2022

  7. [14]

    State augmented constrained reinforcement learning: overcoming the limita- tions of learning with rewards,

    M. Calvo-Fullana, S. Paternain, L. F. Chamon, and A. Ribeiro, “State augmented constrained reinforcement learning: overcoming the limita- tions of learning with rewards,”IEEE Trans. Autom. Control, vol. 69, no. 7, pp. 4275-4290, Jul. 2024

    2024

  8. [15]

    Probabilistic constraint for safety-critical reinforcement learning,

    W. Chen, D. Subramanian, S. Paternain, “Probabilistic constraint for safety-critical reinforcement learning,”IEEE Trans. Autom. Control, vol. 69, no. 10, pp. 6789-6804, Oct. 2024

    2024

  9. [16]

    A review of safe reinforcement learning: methods, theories, and applications,

    S. Gu, L. Yang, Y . Du, G. Chen, F. Walter, and Jun Wang, “A review of safe reinforcement learning: methods, theories, and applications,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 11216-11235, Dec. 2024

    2024

  10. [17]

    Provably efficient safe exploration via primal-dual policy optimization,

    D. Ding, X. Wei, Z. Yang, Z. Wang, and M. Jovanovic, “Provably efficient safe exploration via primal-dual policy optimization,” inProc. Int. Conf. Artif. Intell. Stat., 2021, pp. 3304-3312

    2021

  11. [18]

    A dual approach to constrained markov decision processes with entropy regularization,

    D. Ying, Y . Ding, and J. Lavaei, “A dual approach to constrained markov decision processes with entropy regularization,” inProc. Int. Conf. Artif. Intell. Stat., 2022, pp. 1887-1909

    2022

  12. [19]

    Policy-based primal-dual methods for convex constrained markov decision processes,

    D. Ying, M. A. Guo, Y . Ding, J. Lavaei, and Z.-J. Shen, “Policy-based primal-dual methods for convex constrained markov decision processes,” inProc. AAAI Conf. Artif. Intell., 2023, pp. 10963-10971

    2023

  13. [20]

    Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning,

    S. Lu, K. Zhang, T. Chen, T. Bas ¸ar, and L. Horesh, “Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., 2021, pp. 8767-8775

    2021

  14. [22]

    Scalable reinforcement learning of localized policies for multi-agent networked systems,

    G. Qu, A. Wierman, and N. Li, “Scalable reinforcement learning of localized policies for multi-agent networked systems,” inProc. Learn. Dyn. Control, 2020, pp. 256-266

    2020

  15. [23]

    Solving a class of non-convex min-max games using iterative first order methods,

    M. Nouiehed, M. Sanjabi, T. Huang, J. D. Lee, and M. Razaviyayn. “Solving a class of non-convex min-max games using iterative first order methods,” inproc. Adv. Neural Inf. Process. Syst., V ol. 32, 2019

    2019

  16. [24]

    Policy gra- dient methods for reinforcement learning with function approximation,

    R. S. Sutton, D. A. McAllester, S. P. Singh, and Y . Mansour, “Policy gra- dient methods for reinforcement learning with function approximation,” inproc. Adv. Neural Inf. Process. Syst., 2000, pp. 1057-1063. MANUSCRIPT FOR REVIEW 12

    2000

  17. [25]

    Distributed optimization over time-varying directed graphs,

    A. Nedi ´c and A. Olshevsky, “Distributed optimization over time-varying directed graphs,”IEEE Trans. Autom. Control, vol. 60, no. 3, pp. 601-615, Mar. 2015

    2015

  18. [26]

    Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference,

    Q. Zhang and L. Ying, “Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference,”arXiv preprint arXiv:2409.17401, 2024

    2024 arXiv

  19. [27]

    ϕ-update: a class of policy update methods with policy convergence guarantee,

    W. Li, J. Liu, and K. Wei, “ϕ-update: a class of policy update methods with policy convergence guarantee,” inproc. Int. Conf. Learn. Represent., 2025

    2025

  20. [28]

    Distributed stochastic subgra- dient projection algorithms for convex optimization,

    S. S. Ram, A. Nedi ´c, and V . V . Veeravalli, “Distributed stochastic subgra- dient projection algorithms for convex optimization,”J. Optim. Theory Appl., vol. 147, no. 3, pp. 516-545, 2010

    2010

  21. [29]

    Yeh,Real Analysis: Theory of Measure and Integration

    J. Yeh,Real Analysis: Theory of Measure and Integration. Singapore: World Scientific, 2006

    2006

  22. [30]

    Global convergence of policy gradient methods to (almost) locally optimal policies,

    K. Zhang, A. Koppel, H. Zhu, and T. Bas ¸ar, “Global convergence of policy gradient methods to (almost) locally optimal policies,”SIAM J. Control Optim., vol. 58, no. 6, pp. 3586-3612, Jan. 2020. VIII. APPENDIX In this section, we give rigorous, detailed proofs of the theorems...

    2020

  23. [31]

    Given that the proof procedure is strikingly analogous to that of Case (i), it is omitted for brevity.2 G

    Similar to the proof of Case (i), by employing (89), we can obtainlim m→∞ ˆθi j,m =θ j,m for alli, j∈ N. Given that the proof procedure is strikingly analogous to that of Case (i), it is omitted for brevity.2 G. Proof of Lemma 4 Proof.For Case (i), by the definition of ˆh ˆπˆθ...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.