Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a distributed neural actor-critic algorithm for networked multi-agent reinforcement learning converges to the globally optimal joint policy, with average suboptimality decaying as $O(1/\sqrt{K})$ up to vanishing…

desk verdict A plausible first distributed neural actor-critic global convergence result, but the advertised global optimality rests on an unverified Barron-space condition and one shaky lemma. read the letter →

arxiv 2505.24113 v1 pith:URITOVHW submitted 2025-05-30 cs.MA

Pengcheng Dai , Yuanqiu Mo , Wenwu Yu , Wei Ren This is my paper
classification cs.MA MSC 68T0768T0593A1690C40
keywords networkedmulti-agentreinforcementlearningneuralpolicygradientdistributedcriticglobalconvergencerandom-featurefunctionclasstime-varyingcommunicationnetworkactor-criticconsensus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to prove that networked multi-agent reinforcement learning can be solved by an actor-critic algorithm that uses neural networks rather than linear function approximation and needs no central controller. In the proposed algorithm, agents jointly evaluate the current joint policy by exchanging their Q-network parameters with neighbors over a time-varying directed communication graph, using only their own reward functions; each agent then updates its own policy from its local Q estimate. The main result is a global optimality bound: the average suboptimality gap $(1-\gamma)/K \sum_{k=0}^{K-1} \mathbb{E}[J(\pi^*) - J(\pi_{\theta(k)})]$ is $O(1/\sqrt{K})$ plus terms that vanish as the critic horizon, network width, and batch size grow, provided the true Q-functions and an auxiliary function stay inside a bounded random-feature function class. A sympathetic reader would care because this is the kind of global convergence guarantee that linear-approximation distributed methods lack, and it indicates that communication constraints do not destroy the rate achievable by centralized neural policy gradient.

What carries the argument

The load-bearing machinery is a pair of two-layer ReLU networks with fixed random signs, feature scaling $1/(mN)^p$ with $1/2 < p < 1$, and parameter projection onto $\ell^2$-balls of radius $B/\sqrt{N}$. The approximate Q-function $\hat{Q}_i(z; W_i)$ and the policy feature function $f_i(s,a_i;\theta_i)$ share the same random initialization, which yields the compatibility identity $\nabla_{W_i} \hat{A}_i(z;W_i) = \nabla_\theta \log \pi_\theta(a|s)$ (Lemma 2); this is what makes the critic gradient a valid surrogate for the joint policy gradient. The convergence argument relies on three mechanisms: average consensus over a uniformly strongly connected time-varying graph, the approximate stationary point of the mean-squared projected temporal-difference error obtained by linearizing the ReLU network at initialization, and the function class $F_{B,\infty} = \{f_0(z) + \int \mathbf{1}\{W^\top x > 0\}\, x^\top \iota(W)\, d\mu(W) : \|\iota\|_\infty \le B/\sqrt{d}\}$, which lets the random-feature networks approximate both the true Q-function and the auxiliary function $u_{\theta(k)}$ well enough for the global gap to vanish.

What would settle it

On a small tabular multi-agent MDP with a nonsmooth reward (for example, an indicator of a half-space in the joint state-action space), compute along the training trajectory the projection residual $\min_{f \in F_{B,m}} \mathbb{E}[(f(z)-Q^{\pi_{\theta(k)}}(z))^2]$ and the analogous residual for $u_{\theta(k)}$. If either residual fails to shrink as $m$ grows, the premise of Theorem 3 fails; if the average suboptimality nevertheless vanishes, then the true mechanism of convergence is something other than the assumed function-class membership, and that should be identified.

Watch

Extended reading notes

Core claim

The central claim is that a consensus-based distributed critic feeding a decentralized neural actor converges to the globally optimal joint policy, not merely to a stationary point. Theorem 3 states that, under Assumptions 1-5, if the true Q-function $Q^{\pi_{\theta(k)}}$ of every policy visited and the auxiliary function $u_{\theta(k)}$ defined in (63) both lie in $F_{B,\infty}$, then the average suboptimality in (70) is bounded by the sum of an $O(1/\sqrt{\eta_a K})$ policy-improvement term, an $O(1/\sqrt{|B|})$ gradient-estimation term, and approximation terms of order $\tilde{O}(N^{1/4} T_c^{-3/8})$ and $O(m^{(1-2p)/8} N^{(3-2p)/8})$; the last two vanish as critic iterations $T_c$ and network width $m$ grow. The proof first shows the distributed critic evaluates the joint policy correctly, then establishes an $O(1/K)$ stationarity rate, then converts stationarity into global optimality through the auxiliary-function assumption. In the robot path-planning simulations, the objective value of the distributed algorithm tracks the centralized benchmark closely while requiring much less time per iteration.

Load-bearing premise

The load-bearing premise is that the true Q-function of every policy encountered during training, and the auxiliary function $u_{\theta(k)}$ built from visitation ratios, both belong to the fixed function class $F_{B,\infty}$; the paper gives no conditions on the MDP, rewards, or communication graph that would guarantee this.

Editorial extensions

If this is right

  • The algorithm achieves global optimality for networked multi-agent RL with no central controller and no sharing of reward functions, only Q-parameter exchange between neighbors.
  • The $O(1/\sqrt{K})$ average suboptimality preserves the rate of centralized neural policy gradient, so distributed communication does not change the convergence order.
  • The distributed critic step is globally convergent on its own: each agent's Q-estimate approaches the true global Q-function as $T_c$ and $m$ grow.
  • Larger sample batches shrink the policy-gradient variance term, matching the paper's ablation results.
  • Because of the compatible-approximation identity, the architecture is positioned for extension to natural policy gradient updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: the fixed-class assumption on $Q^{\pi_{\theta(k)}}$ and $u_{\theta(k)}$ is the part most likely to fail in practice, since the paper offers no MDP-level conditions that imply it; identifying such conditions is the natural next theorem.
  • An empirical audit could compute the projection residual of $Q^{\pi_{\theta(k)}}$ onto $F_{B,m}$ along the training trajectory; if the residual stops decreasing with $m$, the theory's premise is violated even if the algorithm still works.
  • The exponents $T_c^{-3/8}$ and $m^{(1-2p)/8}$ suggest that very wide networks and long critic phases are needed to make approximation error negligible, so in resource-limited deployments the critic may dominate the cost.
  • A testable extension: replace the random-feature critic with a backprop-trained shallow network and compare the suboptimality gap; the current theorem does not cover that training scheme, so the experiment would show whether the guarantee is specific to the random-feature architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Algorithm 1, a distributed neural policy gradient method for networked multi-agent reinforcement learning. Agents run a distributed critic that averages neighbor parameters with local TD errors, then update their policy parameters in a decentralized actor step using locally approximated Q-functions. The main theoretical claims are: (i) Theorem 1 and Corollary 1 establish convergence of the distributed critic to the true Q-function as T_c and m grow, under a Barron-type representability condition; (ii) Theorem 2 establishes stationarity convergence of the actor; (iii) Theorem 3 claims global optimality in average suboptimality, again under the condition that every intermediate Q-function and an auxiliary function u_{θ(k)} lie in a fixed-radius Barron class F_{B,∞}. Simulations on robot path planning networks compare the method against a centralized softmax policy gradient baseline.

Significance. If the main theorems are correct, the paper would be a valuable extension of neural policy gradient global-convergence theory from single-agent settings to networked multi-agent settings with a distributed critic and a decentralized actor. The paper is transparent about its proof strategy, uses an external centralized baseline for comparison, and provides explicit rates rather than fitted constants. However, the advertised global optimality rests on an unverified Barron-space condition for all iterates, and the proof of Theorem 3 contains a gap when moving from stationary policies to nonstationary iterates. These issues make the central claim conditional rather than established, so the paper needs substantial revision before the global-convergence contribution can be accepted as stated.

major comments (4)
  1. [Section VI.B, Theorem 3 and Eq. (70)] The global optimality bound is conditional on the hypothesis that Q^{π_{θ(k)}} ∈ F_{B,∞} and u_{θ(k)} ∈ F_{B,∞} for every iterate k. This hypothesis is not derived from Assumptions 1–5 or from primitive MDP conditions. The function u_{θ(k)} defined in Eq. (63) contains Radon–Nikodym derivatives dσ^{π*}/dσ^{π_{θ(k)}} and dν^{π*}/dν^{π_{θ(k)}}; no argument is given that these derivatives are bounded or that the Barron norm of u_{θ(k)} is uniformly O(B) along the random updates. If either inclusion fails, Corollary 1's projection bias and Lemma 8's bound in Eq. (68) need not vanish, and only the stationarity result of Theorem 2 remains. The simulations in Section VII do not measure B or the projection error, so they do not provide empirical evidence for the condition. The abstract and the statement of Theorem 3 should be reworded as a conditional result, or the condition should be replaced by verifiable MDP-level assumptions.
  2. [Section VI.B, proof of Theorem 3, Eq. (73)] The passage from Eq. (72) to Eq. (73) invokes Lemma 8, but Lemma 8 is stated and proved only for a stationary policy π_{\hat θ} satisfying u_{\hat θ} ∈ F_{B,∞}. The average in Eq. (73) is over nonstationary iterates θ(k). No lemma is provided that bounds J(π*) − J(π_{θ(k)}) for nonstationary θ(k) in terms of the gradient mapping ρ(k) plus the Barron approximation error. As written, Theorem 3 does not follow from the displayed equations; the proof needs an explicit domination argument for nonstationary policies or a different route from Theorem 2 to the global gap.
  3. [Section IV.A, Lemma 2 and Eq. (27)] The claimed equality ∇_{W_i} \hat A_i(z; W_i) = ∇_θ log π_θ(a|s) is not established by the proof. After initialization, W_i(t) is updated in the distributed critic step (28) while θ_j(k) is updated in the decentralized actor step (31), so the blocks of W_i do not remain equal to θ_j at later times. The proof only observes equality of initial values and adds 'with m sufficiently large,' which does not address the discrepancy caused by the separate updates. Since Remark 3 uses Lemma 2 to claim a compatible-function-approximation property, Lemma 2 must either be proved under the actual coupling of Algorithm 1 or removed from the paper's claims.
  4. [Section V.A and proof of Theorem 2] Fact 1 and Lemma 3 are stated without proofs and are used repeatedly in Lemma 5, Lemma 6, and Theorem 1. If these are direct imports from [23], the manuscript should state the precise adaptation and provide proofs or a explicit reference covering each statement, because the constants and the expectations over random initialization directly affect the rate in Eq. (36). In addition, the final step of the proof of Theorem 2 says the bound follows from (60) and Theorem 1, but Eq. (60) contains E‖Q^{π_{θ(k)}} − \hat Q^{π_{θ(k)}}_{i,out}‖_{ς_k}; that term is controlled by Corollary 1, not by Theorem 1, and the proof should cite Corollary 1 explicitly.
minor comments (4)
  1. [Section VI.B, Eq. (63) and surrounding text] In the sentence after Eq. (63), ν^{π*} and ν^{π_{\hat θ}} are described as 'stationary state-action distribution' even though ν was defined in Eq. (5) as a state visitation measure; please correct the terminology.
  2. [Throughout] There are several typos and inconsistent spellings, including 'distribiuted' in the proof of Lemma 5, 'Algorithhm' in the heading of Section VI.B, 'objectice' in Assumption 4, and 'Algotithm 1' in the caption of Fig. 8; these should be corrected.
  3. [Section VII, Figs. 5–11] Some figures plot objective values against running time, while ablation figures plot against iterations; please label axes consistently and, for the comparisons against the centralized algorithm, report the number of independent runs and show error bars or confidence intervals so that the claimed closeness to the centralized baseline is statistically supported.
  4. [Section V.A, Fact 1] Fact 1 and Lemma 3 are stated to hold 'almost everywhere'; the nondifferentiability of ReLU at zero should be handled explicitly in the statements or by referring to a standard subgradient convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 3 is a conditional convergence bound; the Barron-class hypothesis is an explicit assumption, not a disguised conclusion.

full rationale

The paper's derivation chain is non-circular. The main global-convergence result (Theorem 3, Eq. (70)) is proved rather than fitted: the critic error is bounded through a signal-noise/telescoping argument (Lemma 6, Theorem 1), the actor gradient error through Lemma 7, and the stationarity-to-global-optimality step through Lemma 8. Lemma 8 imports Theorem 4.8 from [18] (Wang et al., ICLR 2021), and Corollary 1 imports the gamma-contraction property of the projected Bellman operator from Lemma 4.2 of [23]; both are external, not authored by the present paper, and are therefore independent support rather than circular self-citation. The only self-citation, [17], appears in the introduction to contextualize prior distributed linear-approximation actor-critic algorithms and is not load-bearing for any theorem. The Barron-type condition Q^{pi_theta(k)}, u_theta(k) in F_{B,infty} is an explicit hypothesis of Corollary 1 and Theorem 3; it is not manufactured by the proof. In particular, u_theta(k) in Eq. (63) contains Radon-Nikodym derivatives of visitation measures, which are not automatically elements of F_{B,infty}, so the assumption is substantive and unverified rather than true by construction. If the assumption fails, Theorem 2 still provides stationarity, but the advertised global optimality does not follow; this is a limitation of a conditional theorem, not a circular step. No parameters are fitted to data and then renamed predictions, and the simulation is benchmarked against an external centralized algorithm [30]. The omitted proofs (Fact 1 and some lemmas) are either straightforward or delegated to external references, and they do not introduce circularity. Score 0.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

The central convergence theorem rests on five technical assumptions about the network, rewards, smoothness, and visitations, plus two ad hoc representability conditions on the function class. The algorithm also assumes access to the global state-action and to stationary distribution samples. There are no new physical or mathematical entities introduced; the only hand-chosen numbers are the scaling exponent p and the radius B.

free parameters (2)
  • p = p ∈ (1/2, 1)
    Hand-chosen scaling exponent in the Q-network (17) and policy network (18). Remark 2 states the range is 'specifically designed' to make error terms vanish in Lemma 5 and the final bounds.
  • B = B > 0
    Radius of the parameter space S_W^B and of the function class F_{B,∞}. It controls expressiveness and the size of the error terms in Theorems 1-3, but is not derived from the MDP and is not specified in the simulations.
assumptions (8)
  • domain assumption Assumption 1: time-varying communication graphs are uniformly strongly connected with doubly stochastic weight matrices
    Required for the consensus result in Lemma 4; standard in distributed optimization and NETMARL literature.
  • domain assumption Assumption 2: rewards are uniformly bounded by R_0
    Used throughout to bound Q-functions and TD errors.
  • domain assumption Assumption 3: local linearization regularity condition (c_0 bound on the probability that the random feature activation changes)
    Imported from Cai et al. [23]; controls the error between the original and locally linearized neural network, used in Lemma 3 and Theorem 1.
  • domain assumption Assumption 4: ∇J(π_θ) is L-Lipschitz and the policy gradient estimator has bounded variance
    Used in the descent lemma for the actor step in the proof of Theorem 2.
  • domain assumption Assumption 5: bounded Radon-Nikodym derivative dσ_k/dς_k ≤ κ_a for all iterates
    Concentrability condition needed to relate the state-action visitation measure to the stationary measure in Lemmas 7 and 8.
  • domain assumption Each agent can access the global state-action pair z = (s,a)
    Stated in Section III.A. The Q-network (17) and policy (19) depend on the global state s and on all agents' features.
  • ad hoc to paper Q^{π_θ(k)} ∈ F_{B,∞} and u_θ(k) ∈ F_{B,∞} for all k
    Conditions imposed in Lemma 8 and Theorem 3. The true Q-functions of all intermediate policies are required to lie in a Barron-type random feature class; the paper does not derive this from MDP properties or verify it.
  • domain assumption Sampling from stationary distributions of z and (z,z')
    Algorithm 1 lines 5 and 9 require samples from the stationary distribution of the current policy, a common but strong theoretical idealization in RL convergence analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/URITOVHW

@misc{pith2026250524113,
  author       = {Pith},
  title        = {Pith review of: Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/URITOVHW}},
  note         = {Machine review of arXiv:2505.24113}
}
read the original abstract

This paper studies the networked multi-agent reinforcement learning (NMARL) problem, where the objective of agents is to collaboratively maximize the discounted average cumulative rewards. Different from the existing methods that suffer from poor expression due to linear function approximation, we propose a distributed neural policy gradient algorithm that features two innovatively designed neural networks, specifically for the approximate Q-functions and policy functions of agents. This distributed neural policy gradient algorithm consists of two key components: the distributed critic step and the decentralized actor step. In the distributed critic step, agents receive the approximate Q-function parameters from their neighboring agents via a time-varying communication networks to collaboratively evaluate the joint policy. In contrast, in the decentralized actor step, each agent updates its local policy parameter solely based on its own approximate Q-function. In the convergence analysis, we first establish the global convergence of agents for the joint policy evaluation in the distributed critic step. Subsequently, we rigorously demonstrate the global convergence of the overall distributed neural policy gradient algorithm with respect to the objective function. Finally, the effectiveness of the proposed algorithm is demonstrated by comparing it with a centralized algorithm through simulation in the robot path planning environment.

Figures

Figures reproduced from arXiv: 2505.24113 by the authors.

Figure 1
Figure 1. Two novel neural networks for the approximate function and policy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The flow diagram of distributed policy gradient algorithm. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The main analytical flowchart of proof process. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Acyclic path networks and communication networks in the robots [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 7
Figure 7. Figure 7: Performances of Algorithm 1 with |B| = 4, 6, 8 in path network 1-1. C. Path network 3-2-1 This subsection considers the path planning problem on path network 3-2-1, where the number of agents is N = 6, the discount factor is γ = 0.9, and the initial positions are b1, b…
Figure 5
Figure 5. Figure 5: Performances of the centralized algorithm and our Algorithm 1 in [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 8
Figure 8. Figure 8: Performances of the centralized algorithm and our Algorithm 1 in [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 6
Figure 6. Figure 6: Policies of agents at location b1 generated by Algorithm 1 in path network 1-1. To evaluate the convergence performance of Algorithm 1 under varying conditions, an ablation study is conducted by adjusting the size of the sample batch B. The discounted average cumulativ…
Figure 9
Figure 9. Figure 9: Policies of agent 1, 3, 4, and 6 at initial location generated by [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Performances of Algorithm 1 with |B| = 8, 10, 20 in path network 3- 2-1. To investigate the impact of isolated agents in communi￾cation networks on Algorithm 1, we consider two scenarios: (i) agents 1 and 4 are isolated; (ii) agents 2 and 5 are isolated. As depicted i…
Figure 11
Figure 11. Figure 11: Performances of Algorithm 1 with isolated agents in path network 3- [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

  1. [18]

    Neural policy gradient methods: Global optimality and rates of convergence,

    L. Wang, Q. Cai, Z. Yang, and Z. Wang, “Neural policy gradient methods: Global optimality and rates of convergence,” inProc. Int. Conf. Learn. Represent., 2021

    work page 2021

  2. [19]

    Finite-time analysis of dis- tributed TD(0) with linear function approximation on multi-agent rein- forcement learning,

    T. Doan, S. Maguluri, and J. Romberg. “Finite-time analysis of dis- tributed TD(0) with linear function approximation on multi-agent rein- forcement learning,” inProc. Int. Conf. Mach. Learn., pp. 1626-1635, 2019

    work page 2019

  3. [23]

    Neural temporal-difference learning converges to global optima,

    Q. Cai, Z. Yang, J. D. Lee, and Z. Wang, “Neural temporal-difference learning converges to global optima,” inProc. Adv. Neural Inf. Process. Syst., vol 32, 2019

    work page 2019

  4. [17]

    Distributed actor-critic algorithms for multiagent reinforcement learning over directed graphs,

    P. Dai, W. Yu, H. Wang, and S. Baldi, “Distributed actor-critic algorithms for multiagent reinforcement learning over directed graphs,”IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 10, pp: 7210-7221, Oct. 2023

    work page 2023

  5. [1]

    R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998

    work page 1998

  6. [2]

    Distributed reinforcement learning algorithm for dynamic economic dispatch with unknown generation cost functions,

    P. Dai, W. Yu, G. Wen, and S. Baldi, “Distributed reinforcement learning algorithm for dynamic economic dispatch with unknown generation cost functions,”IEEE Trans. Ind. Informat., vol. 16, no. 4, pp. 2258-2267, Apr. 2020

    work page 2020

  7. [3]

    Distributed Q-learning algorithm for dynamic resource allocation with unknown objective functions and ap- plication to microgrid,

    P. Dai, W. Yu, and D. Chen, “Distributed Q-learning algorithm for dynamic resource allocation with unknown objective functions and ap- plication to microgrid,”IEEE Trans. Cybern.,IEEE Trans. Cybern., vol. 52, no. 11, pp. 12340-12350, Nov. 2022

    work page 2022

  8. [4]

    Multi-agent deep reinforcement learning for large-scale traffic signal control,

    T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,”IEEE Trans. Intell. Transp. Syst., vol. 21, no. 3, pp. 1086-1095, Mar. 2020

    work page 2020

Show all 31 references
  1. [5]

    Large-Scale traffic signal control using a novel multiagent reinforcement learning,

    X. Wang, L. Ke, Z. Qiao, and X. Chai, “Large-Scale traffic signal control using a novel multiagent reinforcement learning,”IEEE Trans. Cybern., vol. 51, no. 1, pp. 174-187, Jan. 2021

    2021

  2. [6]

    A multi-channel transmission schedule for remote state estimation under DoS attacks,

    K. Ding, Y . Li, D. E. Quevedo, S. Dey, and L. Shi, “A multi-channel transmission schedule for remote state estimation under DoS attacks,” Automatica, vol. 78, pp. 194-201, Apr. 2017

    2017

  3. [7]

    Distributed reinforcement learning for cyber-physical system with multiple remote state estimation under DoS attacker,

    P. Dai, W. Yu, H. Wang, G. Wen, and Y . Lv, “Distributed reinforcement learning for cyber-physical system with multiple remote state estimation under DoS attacker,”IEEE Trans. Netw. Sci. Eng., vol. 7, no. 4, pp. 3212- 3222, Oct. 2020

    2020

  4. [8]

    Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks,

    Y . S. Nasir and D. Guo, “Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks,”IEEE J. Sel. Areas Commun., vol. 37, no. 10, pp. 2239-2250, Oct. 2019

    2019

  5. [9]

    Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,

    N. Zhao, Y .-C. Liang, D. Niyato, Y . Pei, M. Wu, and Y . Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,”IEEE Trans. Wireless Commun., vol. 18, no. 11, pp. 5141-5152, Nov. 2019. MANUSCRIPT FOR REVIEW 16

    2019

  6. [10]

    Intellilight: A reinforcement learning approach for intelligent traffic light control,

    H. Wei, G. Zheng, H. Yao, and Z. Li, “Intellilight: A reinforcement learning approach for intelligent traffic light control,” inProc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min., pp. 2496-2505, 2018

    2018

  7. [11]

    Multi-agent reinforcement learning: Independent vs. coopera- tive agents,

    M. Tan, “Multi-agent reinforcement learning: Independent vs. coopera- tive agents,” inProc. Int. Conf. Mach. Learn., pp. 330-337, 1993

    1993

  8. [12]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” inProc. Int. Conf. Mach. Learn., pp. 6846-6859, 2018

    2018

  9. [13]

    QPLEX: Duplex dueling multi-agent Q-Learning,

    J. Wang, Z. Ren, T. Liu, Y . Yu, and C. Zhang, “QPLEX: Duplex dueling multi-agent Q-Learning,” inProc. Int. Conf. Learn. Represent., 2021

    2021

  10. [14]

    Counterfactual multi-agent policy gradients,

    J. N. Foerster, G. Farquhar, T. Afoura, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” inProc. AAAI Conf. Artif. Intell., pp. 2974-2982, 2018

    2018

  11. [15]

    Off-policy multi-agent decomposed policy gradients,

    Y . Wang, B. Han, T. N. Wang, H. Dong, and C. Zhang, “Off-policy multi-agent decomposed policy gradients,”arXiv:2007.12322, 2020

    2007 arXiv

  12. [16]

    Fully decentralized multi-agent reinforcement learning with networked agents,

    K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Bas ¸ar, “Fully decentralized multi-agent reinforcement learning with networked agents,” inProc. Int. Conf. Mach. Learn., pp. 5872-5881, 2018

    2018

  13. [20]

    A communication-efficient multi-agent actor-critic algorithm for distributed reinforcement learning,

    Y . Lin, K. Zhang, Z. Yang, Z. Wang, T. Bas ¸ar, R. Sandhu, and J. Liu, “A communication-efficient multi-agent actor-critic algorithm for distributed reinforcement learning,” inProc. IEEE Conf. Decis. Control, pp. 5562- 5567, 2019

    2019

  14. [21]

    A multiagent off-policy actor-critic algorithm for distributed reinforcement learning,

    W. Suttle, Z. Yang, K. Zhang, Z. Wang, T. Bas ¸ar, and J. Liu, “A multiagent off-policy actor-critic algorithm for distributed reinforcement learning,”IFAC-PapersOnLine, vol. 53, no. 2, pp. 1549-1554, 2020

    2020

  15. [22]

    Policy gra- dient methods for reinforcement learning with function approximation,

    R. S. Sutton, D. A. McAllester, S. P. Singh, and Y . Mansour, “Policy gra- dient methods for reinforcement learning with function approximation,” inProc. Adv. Neural Inf. Process. Syst., vol. 12, 2000

    2000

  16. [24]

    Natural actor-critic,

    J. Peters and S. Schaal, “Natural actor-critic,”Neurocomputing, vol. 71, pp. 1180-1190, 2008

    2008

  17. [25]

    Optimistic policy iteration and natural actor-critic: A unify- ing view and a non-optimality result,

    P. Wagner, “Optimistic policy iteration and natural actor-critic: A unify- ing view and a non-optimality result,” inProc. Adv. Neural Inf. Process. Syst., vol 26, 2013

    2013

  18. [26]

    Improving sample complexity bounds for (natural) actor-critic algorithms,

    T. Xu, Z. Wang, and Y . Liang, “Improving sample complexity bounds for (natural) actor-critic algorithms,” inProc. Adv. Neural Inf. Process. Syst., vol 33, 2020

    2020

  19. [27]

    Constrained consensus and optimization in multi-agent networks,

    A. Nedi ´c, A. Ozdaglar and A. P. Parrilo, “Constrained consensus and optimization in multi-agent networks,”IEEE Trans. Autom. Control, vol. 55, no. 4, pp. 922-938, Apr. 2010

    2010

  20. [28]

    Nesterov,Introductory Lectures on Convex Optimization, Berlin, Germany: Springer, 2018

    Y . Nesterov,Introductory Lectures on Convex Optimization, Berlin, Germany: Springer, 2018

    2018

  21. [29]

    Convergence rates for localized actor-critic in networked markov potential games,

    Z. Zhou, Z. Chen, Y . Lin, and A. Wierman, “Convergence rates for localized actor-critic in networked markov potential games,” inProc. Uncertainty Artif. Intell. Conf., pp. 2563-2573, 2023

    2023

  22. [30]

    On the global convergence rates of softmax policy gradient methods,

    J. Mei, C. Xiao, C. Szepesv ´ari, and D. Schuurmans, “On the global convergence rates of softmax policy gradient methods,” inProc. Int. Conf. Mach. Learn., pp. 6820-6829, 2020

    2020

  23. [31]

    Network topology and communication-computation tradeoffs in decentralized optimization,

    A. Nedi ´c, A. Olshevsky, and M. G. Rabbat, “Network topology and communication-computation tradeoffs in decentralized optimization,” Proc. IEEE, vol. 106, no. 5, pp. 953-976, May 2018

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.