Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A decentralized actor-critic method with neural critics converges to a globally optimal joint policy at rate $O(1/T)$ up to fixed approximation biases, matching the rate previously available only for linear function approximation.

desk verdict The main theorem rests on a false identification of the Q-function TD-error with the advantage function, so the paper's central claim does not follow; the question is important and the pseudo-centralized idea is worth a look, but the proof needs a major repair. read the letter →

arxiv 2505.18433 v2 pith:V3IASIGU submitted 2025-05-24 cs.LG cs.MA

Zhiyao Zhang , Myeung Suk Oh , FNU Hairi , Ziyue Luo , Alvaro Velasquez , Jia Liu This is my paper
classification cs.LGcs.MA MSC 68T0590C40
keywords decentralizedmulti-agentreinforcementlearningactor-criticdeepneuralnetworksglobaloptimalityfinite-timeconvergenceconsensusgossiptemporal-differencesamplecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a fully decentralized multi-agent actor-critic method, where both the policy and the critic are deep neural networks, converges to a globally optimal joint policy at a finite-time rate $O(1/T)$ up to fixed approximation biases, matching the rate previously available only for linear function approximation. The authors argue this closes a gap between practice, where neural-network critics are standard, and theory, where decentralized MARL guarantees had reached only stationarity. They identify the central difficulty as the failure of nonlinear TD updates to commute with consensus averaging, and they control the resulting bias by maintaining a pseudo-centralized parameter used only for analysis. If the claim is right, decentralized neural actor-critic training carries a global, not merely local, convergence guarantee, with sample complexity $O(N^2/\epsilon^3)$ to reach $\epsilon$ accuracy.

What carries the argument

Three objects carry the proof. First, a pseudo-centralized parameter $\bar V_t$, updated as if a central agent saw all rewards, bridges the averaged decentralized output $\bar W_t$ to the stationary point $W^*_t$, because the order of nonlinear TD updates and consensus averaging cannot be interchanged. Second, the gossip-averaged TD-error $\tilde\delta^i_{t,l}$ supplies the actor update direction, replacing the unavailable global advantage function through consensus. Third, Lemma 5.7, together with Assumptions 5.5 and 5.6, turns a bound on the update-direction error into a bound on the global suboptimality gap, which is what upgrades stationarity to global optimality. The restart kernel makes the stationary state distribution proportional to the discounted visitation measure, simplifying the policy gradient theorem to an expectation under $\nu(\theta)$.

What would settle it

On a small two-agent MDP where the exact policy gradient can be computed, train the critic to its stationary point under the projection ball of Fact C.3 and measure the angle between $\mathbb{E}[\delta \psi]$ and $\mathbb{E}[\mathrm{Adv}_{\theta}(s,a)\psi]$; if the cosine is not close to 1 at the stated width $m = \Omega(d^3 D^{-11/2})$, the identification inside Step 2(c) fails and the advertised rate needs an extra assumption.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 5.10: under bounded rewards, a mixing Markov chain, Fisher-non-degenerate policy parameterization, an advantage-compatibility bias $\epsilon_{bias}$, Lipschitz smoothness, and a universal-approximation critic error $\epsilon_{critic}$, Algorithm 1 satisfies $\mathbb{E}[J(\theta^*)-J(\theta_T)] = O(T^{-1}) + O(\sqrt{\epsilon_{bias}}) + O(\epsilon_{critic}) + O(N^{1/2} M^{-1/2})$ plus lower-order network and consensus terms, with probability at least $1 - \exp(-\Omega(\log^2 m))$. The proof converts the usual descent lemma, which bounds only $\|\nabla J(\theta_t)\|$, into a bound on the global optimality gap through Lemma 5.7, $\sqrt{\mu}(J(\theta^*)-J(\theta)) \le \epsilon' + \|\nabla J(\theta)\|$. The intermediate machinery bounds the distance between the gossip-averaged TD-error direction and the true policy gradient by comparing with a stationary critic parameter $W^*_t$ and a pseudo-centralized parameter $\bar V_t$.

Load-bearing premise

The load-bearing premise is that the gossip-averaged TD-error of the learned critic, used as the actor's descent direction, points in the same direction as the true advantage-based policy gradient; this is exactly true for the true Q-function by the Bellman equation, but for a finite-width projected neural critic it is an unproven identification.

Editorial extensions

If this is right

  • With $T = \Omega(\epsilon^{-1})$, $M = \Omega(N \epsilon^{-2})$, and $K = \Omega(N^{1/2}\epsilon^{-1})$, Algorithm 1 reaches $\epsilon$ global optimality up to the fixed biases with sample complexity $O(N^2/\epsilon^3)$ and communication complexity $O(\epsilon^{-1} \log(N^{3/2}/\epsilon))$.
  • The $O(1/T)$ rate matches the linear-approximation decentralized actor-critic baseline, so replacing linear critics with neural networks does not slow the convergence rate in the bound.
  • In the leading error terms the network depth $D$ appears with negative exponents while the width $m$ appears with tiny exponents $m^{1/32}$ and $m^{-1/24}$, predicting that depth, not width, is the effective accuracy lever.
  • Increasing gossip rounds shrinks consensus error geometrically, so a finite communication budget keeps the consensus contribution lower-order in the total error.
  • The consensual TD-error version learns in the Simple Spread experiments, while the Q-value variant does not, confirming the design choice empirically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: Step 2(c) of the proof identifies the conditional expectation of the TD-error with the estimated advantage function; this holds exactly at a Bellman fixed point, but for a finite-width projected network the identification is an additional assumption, so the rate's validity in practice depends on how close that equality is.
  • Editorial: If the advantage-compatibility assumption (Assumption 5.6) is relaxed or replaced by a natural-policy-gradient objective, the same pseudo-centralized analysis could transfer the result to natural actor-critic and off-policy MARL variants.
  • Editorial: The multi-agent RLHF experiment fixes the prompt and uses K = M = 1; a natural stress test is to randomize prompts and lengthen dialogues and check whether rewards still increase and whether the gossip exponent predicts the performance gap.
  • Editorial: The theoretical prediction of weak width dependence and strong depth dependence can be tested directly by sweeping m and D on a small MARL benchmark and comparing reward trajectories to the predicted scaling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a deep neural actor-critic algorithm for fully decentralized multi-agent reinforcement learning, in which both the actor and the critic are neural networks and the agents communicate through gossiping/consensus. The main theoretical claim, Theorem 5.10, is that under Assumptions 5.1-5.9, with suitable hyperparameters, Algorithm 1 converges to a neighborhood of the global optimum of the cooperative MARL objective at rate O(1/T), up to terms involving an advantage-compatibility bias epsilon_bias, a critic approximation error epsilon_critic, batch size, consensus iterations, network width, and network depth. The paper also reports ablation experiments on a modified Simple Spread environment and a multi-agent RLHF experiment with LLMs.

Significance. If the main theorem were correct, this would be a significant contribution: it would be, to my knowledge, the first finite-time global-optimality guarantee for decentralized MARL actor-critic methods with nonlinear function approximation, and the pseudo-centralized parameter technique in Remark 6 is a genuinely interesting idea for handling the non-commutativity of nonlinear updates and consensus. The manuscript also deserves credit for attempting to treat Markovian sampling and for conducting experiments beyond toy benchmarks. However, the central proof step is invalid under the paper's own definitions, and this invalidates the main theorem. The experiments, while suggestive, are qualitative and do not directly test the claimed O(1/T) convergence rate, so the contribution currently rests entirely on the theoretical result, which is not established.

major comments (3)
  1. [Appendix C, Step 2(c), Eq. (30)] The proof replaces the conditional expectation of the TD-error with the advantage function: E[delta_{t,l2}(W*_t) psi | F_{t,l1}] is set equal to E[Adv_hat(s_{t,l2},a_{t,l2};W*_t) psi | F_{t,l1}]. Under the paper's own definitions, delta = Q_hat(s,a;W) - r - gamma Q_hat(s',a';W), so for a critic satisfying Assumption 5.9 the conditional mean of delta is (Q - T^pi Q)(s,a), which is approximately zero by the Bellman equation, while the advantage Q(s,a) - E_{a'~pi}Q(s,a') is generically nonzero. Thus Eq. (30) is not an approximation error but a false identity. Since Algorithm 1's actor direction is built precisely from this TD-error (Algorithm 1, Lines 9-16) and Eq. (30) is the pivotal step converting the Markovian-noise term into a stationary-distribution error, the bounds in Eqs. (32)-(33) and hence Theorem 5.10's O(1/T) global-optimality conclusion do not follow.
  2. [Appendix C, Step 2(c), Eq. (29)] The bound in Eq. (29) concludes that E|delta_{t,l}(W*_t)|^2 = O(epsilon_critic^2) from the pointwise bound |delta| <= ((1+gamma)/(1-gamma)+1) rmax + 2 epsilon_critic. This conclusion is algebraically incorrect: the displayed upper bound contains the constant rmax, so its square is of order rmax^2 and cannot be O(epsilon_critic^2) unless rmax is assumed to scale with epsilon_critic. Moreover, even for the exact Q-function, the TD target r + gamma Q(s',a') has nonzero conditional variance, so E|delta|^2 does not vanish as epsilon_critic -> 0. This invalidates the first term in Eq. (28) and the O(N epsilon_critic^2/M) term in Eq. (33).
  3. [Appendix C, Step 2(a), Eqs. (21)-(22)] In bounding |Q_hat(s,a;W_bar_t) - Q_hat(s,a;V_bar_t)|, the proof asserts |Q_hat(W_bar_t) - Q_hat(V_bar_t)| <= L_W ||W_bar_t - V_bar_t|| <= L_W O(BD^{1/2}) by appealing to the convexity of B(B). Convexity only implies that both W_bar_t and V_bar_t lie in B(B); it does not imply that their difference is O(BD^{1/2}). The paper never establishes a quantitative closeness between the gossip-averaged pseudo-centralized parameter W_bar_t and the centralized critic output V_bar_t, despite this being one of the paper's central technical claims (Remark 6). Consequently, the O(N B^2 D) term in Eq. (25) and the corresponding terms in Theorem 5.10 are not justified.
minor comments (5)
  1. [Algorithm 2, Line 6] There is an extra comma in the displayed formula for delta_k: it reads "Q_hat(s_k,, a_k; W^i(k))" and should be corrected.
  2. [Appendix C, Eq. (30)] The quantity Adv_hat(s,a;W) is used before being defined; it should be defined explicitly, for example as Q_hat(s,a;W) - E_{a'~pi_theta}Q_hat(s,a';W).
  3. [Fact C.3 and Assumption 5.9] The same symbol W* is used for the stationary point of the locally linearized MSPBE in Fact C.3 and for the parameter achieving uniform approximation in Assumption 5.9; the relationship between these two objects is not proved, and they should at least be distinguished notationally.
  4. [Lemma 4.3] The statement uses the proportionality symbol "proportional to" without specifying the constant; since the subsequent theorem requires quantitative gradient bounds, the constant (1-gamma)^{-1} should be stated explicitly.
  5. [Theorem 5.10] The theorem statement omits logarithmic factors that appear in the proof, such as log^3 m log K in Eq. (36); the authors should either use big-O-tilde notation consistently or include the full expressions.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (30) equates the TD-error direction to the advantage direction, so the global-optimality claim rests on an assumed policy-gradient identity rather than on the Bellman/TD equations.

  1. other [Appendix C, Step 2(c), Eq. (30) (with Algorithm 1 Line 9 and Eq. (3))]
    "Algorithm 1: 'Compute δi t,l = Qhat(st,l,at,l; W i t ) − ri t,l+1 − γQhat(st,l+1,at,l+1; W i t ).' Eq. (30): 'E[δt,l2(W ∗ t )ψi t,l2 | Ft,l1] − E[ ˆAdv(s, a; W ∗ t )ψθi t (s, ai)] = E[ ˆAdv(st,l2 , at,l2 ; W ∗ t )ψθi t | Ft,l1] − E[ ˆAdv(s, a; W ∗ t )ψθi t (s, ai)].'"

    The proof must show that the TD-error-based actor direction (δψ) estimates the policy gradient (Adv ψ). Step 2(c) instead asserts the key link: it replaces the conditional mean of δ_{t,l2}(W*_t)ψ by E[Adv_hat ψ]. Under the paper's own Bellman operator (Eq. (3)) and Fact C.3/Assumption 5.9, W*_t is a stationary point with Qhat≈Q, so E[δ|s,a] = (Tπ Q − Q)(s,a), which is ≈0, while Adv = Q − E_π Q is generically nonzero. Thus Eq. (30) is not a small approximation error; it is exactly the policy-gradient identification needed to conclude Theorem 5.10. Since Adv_hat is introduced with no definition other than by this equality, the theorem's convergence to ∇J(θ) is true by construction rather than derived from the TD/critic analysis.

full rationale

The paper's assumptions (5.6, 5.9) and imported results (Agarwal et al. 2021, Cai et al. 2019) are not circular by themselves: they are stated as approximation errors that appear explicitly in the final bound, and the self-citations (Hairi et al. 2022, etc.) are used for standard consensus and mixing estimates rather than for the central global-optimality mechanism. However, the derivation chain in Appendix C breaks at Step 2(c). Lemma 5.7 converts a gradient-norm bound into a global optimality gap, but the gradient-norm bound requires d_t ≈ ∇J(θ_t). The proof's four-term decomposition (Eq. (15)) handles critic error, consensus, and sampling noise, but the link between the actor's TD-error samples and the advantage-function expectation is asserted in Eq. (30) without proof and is generally false under the paper's Bellman definitions. Because this link is the load-bearing connection from the algorithm to the policy gradient, Theorem 5.10's conclusion is not established by the given equations. This is a proof gap that functions as circularity: the desired policy-gradient direction is assumed into the definition of the intermediate advantage estimate. Score 6 reflects a central 'prediction' (O(1/T) global optimality) that reduces, in this step, to an assumed identity rather than to the stated assumptions.

Assumptions & free parameters 3 free parameters · 8 assumptions · 1 invented entities

The central theorem rests on two strong approximation assumptions (5.6 and 5.9) whose assumed errors appear in the final bound, plus imported results from Agarwal et al. and Cai et al. The proof's pivotal Step 2(c) additionally assumes an equality between the TD-error expectation and the advantage without derivation. The only fitted numbers are the experimental reward-model scaling constants.

free parameters (3)
  • epsilon_critic (Assumption 5.9 critic approximation error) = unquantified
    Assumed uniform upper bound on |Qhat - Qtheta| over all policies; appears as an irreducible O(epsilon_critic) term in Theorem 5.10.
  • epsilon_bias (Assumption 5.6 advantage compatibility error) = unquantified
    Assumed squared error between the advantage and its score-function approximation; appears as O(sqrt(epsilon_bias)) in Theorem 5.10.
  • RM score scaling parameters a_scaling and boffset = DeBERTa (5,0), Llama (2,-2), Gemma (6,10)
    Hand-chosen per reward model in Section 6.2 and Table 2 to make reward ranges consistent across agents; affects only the experimental RLHF results.
assumptions (8)
  • domain assumption Consensus matrix A is doubly stochastic with positive diagonal and off-diagonal entries lower bounded by eta (Assumption 5.1).
    Standard for gossiping; used to bound consensus error and ensure A^tau remains doubly stochastic.
  • domain assumption The state-action Markov chain is irreducible and aperiodic with geometric mixing (Assumption 5.2, Lemma 5.4).
    Used to control Markovian sampling noise in Step 2(c).
  • domain assumption Score function norm is uniformly bounded by 1 and the Fisher information matrix is non-degenerate (Assumption 5.5).
    Imported from Fatkhullin et al. 2023; ensures a non-degenerate policy parameterization.
  • domain assumption The advantage function is approximately representable by the score function with error epsilon_bias (Assumption 5.6).
    Needed for Lemma 5.7 to convert gradient norm into a global optimality gap; strong and not verified for the neural policies used.
  • domain assumption J(theta) and Qhat(x;W) are Lipschitz continuous (Assumption 5.8).
    Provides the descent lemma and W-Lipschitz bounds used throughout the proof.
  • ad hoc to paper The optimal critic parameter W* approximates Qtheta uniformly over all policies with error epsilon_critic (Assumption 5.9).
    Realizability of the Q-function by the DNN class; this is close to assuming the approximation half of the result and is not derived from the network architecture.
  • standard math Lemma 5.7 from Agarwal et al. 2021: sqrt(mu)(J(theta*)-J(theta)) <= epsilon' + ||grad J(theta)||.
    Imported unproved theorem that bridges gradient norm to global optimality gap; requires Assumptions 5.5 and 5.6.
  • standard math Neural TD convergence bound from Cai et al. 2019 used in Eq (23) for overparameterized networks.
    Imported to bound |Qhat(bar V)-Qhat(W*)|; requires the stated width, depth, projection radius, and iteration conditions.
invented entities (1)
  • Pseudo-centralized parameters bar V_t (Algorithm 3)
    purpose: Analysis bridge between the consensus-averaged local parameters bar W_t and the stationary critic parameter W*_t, to handle the non-commutation of nonlinear TD updates and gossiping.
    An analysis-only construct; the agents do not compute it, so it has no falsifiable handle outside the proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/V3IASIGU

@misc{pith2026250518433,
  author       = {Pith},
  title        = {Pith review of: Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3IASIGU}},
  note         = {Machine review of arXiv:2505.18433}
}
read the original abstract

Actor-critic methods for decentralized multi-agent reinforcement learning (MARL) facilitate collaborative optimal decision making without centralized coordination, thus enabling a wide range of applications in practice. To date, however, most theoretical convergence studies for existing actor-critic decentralized MARL methods are limited to the guarantee of a stationary solution under the linear function approximation. This leaves a significant gap between the highly successful use of deep neural actor-critic for decentralized MARL in practice and the current theoretical understanding. To bridge this gap, in this paper, we make the first attempt to develop a deep neural actor-critic method for decentralized MARL, where both the actor and critic components are inherently non-linear. We show that our proposed method enjoys a global optimality guarantee with a finite-time convergence rate of O(1/T), where T is the total iteration times. This marks the first global convergence result for deep neural actor-critic methods in the MARL literature. We also conduct extensive numerical experiments, which verify our theoretical results.

Figures

Figures reproduced from arXiv: 2505.18433 by the authors.

Figure 1
Figure 1. The interplay of decentralization and nonlinearity. plying the consensus matrix A on the parameter matrix Wˆ sufficiently many times), all agents’ parameters will reach a new consensus. However, we note that this consensus pro￾cess is performed after each agent’s nonlinear DNN learning, which yields a systematic bias even with an infinite number of gossiping rounds. To address this challenge, we develop a new techni… view at source ↗
Figure 3
Figure 3. A block diagram of multi-agent RLHF with two LLMs. 20 40 60 80 100 Conversation Episode 1.1 1.0 0.9 0.8 0.7 0.6 0.5 Moving-Averaged Reward Mean LLM 1 - DeBERTa RM LLM 2 - Llama RM LLM 3 - Gemma RM [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. A learning performance plot of LLM agents during multi￾agent RLHF. Moving average was applied over every 5 episodes. RLHF. We observe a steady increase in the average reward as the episodes progress, which validates our multi-agent RLHF framework. It is important to note that, despite vary￾ing rates of improvement, the reward consistently increases for all LLMs, highlighting the efficacy of our decentralized actor-c… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: illustrates the benefits of the gossiping technique for Algorithm 1. When the consensus process is excluded, represented by the green curve, the long-term performance is the worst. In addition, as tgossip increases, Algorithm 1 achieves progressively better performance…
Figure 8
Figure 8. Figure 8: Performances of Algorithm 1 with different number of agents N [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages

  1. [1]

    M., Lee, J

    Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2021). On the theory of policy gradient methods: Opti- mality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1–76. Ahmadian, A., Cremer, C., Gall´e, M., Fadaee, M., Kreutzer, J., Pietquin, O., ¨Ust¨un, A., and Hooker, S. (2024). Back to basics: Revisiting reinfo...

    arXiv 2021

  2. [2]

    how do I threaten others?

    Parameter Values for RM Score Scaling RM name DeBERTa RM Llama RM Gemma RM Base model DeBERTa-V3 Llama-3-8B Gemma-2B ascaling 5 2 6 boffset 0 -2 10 question and engages with the LLM for three prompt-response exchanges. We select “how do I threaten others?” as our initial question to draw LLM responses that receive low scores from the RMs. In our experimen...

    work page 2024

  3. [3]

    ∥E h δt,l2 (W ∗ t )ψi t,l2 Ft,l1 i − E ˆAdv(s, a; W ∗ t )ψθi t (s, ai) ∥ Ft # =2 ( 1 + γ 1 − γ + 1)rmax + 2ϵcritic · E

    Therefore, we can apply ¯Vt as a bridge to bound the second term in Equation (19). Similarly, we denote ¯Vt as the output of Algorithm 3 corresponding to the outer loop indexed by t. Therefore, by triangle inequality, we have: | ˆQ(st,l, at,l; ¯Wt) − ˆQ(st,l, at,l; W ∗ t )| ⩽| ˆQ(st,l, at,l; ¯Wt) − ˆQ(st,l, at,l; ¯Vt)| + | ˆQ(st,l, at,l; ¯Vt) − ˆQ(st,l, a...

    work page 2019

  4. [6]

    On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrization

    Gaur, M., Bedi, A. S., Wang, D., and Aggarwal, V . (2023). On the global convergence of natural actor-critic with two-layer neural network parametrization. arXiv preprint arXiv:2306.10486. Gaur, M., Bedi, A. S., Wang, D., and Aggarwal, V . (2024). Closing the gap: achieving global convergence (last iter- ate) of actor-critic under Markovian sampling with ...

    work page Pith review arXiv 2023

  5. [8]

    To handle token inputs of variable length, we apply zero-padding to each critic input

    For each critic network, we use a multi-layer perception (MLP) of m = 256 and D = 3, and set tgossip = 20 for the gossiping technique. To handle token inputs of variable length, we apply zero-padding to each critic input. We use the Adam optimizer with learning rates α = 0.0001 and β = 0.0001 for both actor and critic updates, respectively. For TD calcula...

    work page 2023

  6. [9]

    Consensus-based Decentralized Multi-agent Reinforcement Learning for Random Access Network Optimization

    Naderializadeh, N., Sydir, J. J., Simsek, M., and Nikopour, H. (2021). Resource management in wireless networks via multi-agent deep reinforcement learning. IEEE Trans- actions on Wireless Communications, 20(6):3507–3523. Nasir, Y . S. and Guo, D. (2019). Multi-agent deep reinforce- ment learning for dynamic power allocation in wireless networks. IEEE Jou...

    work page Pith review arXiv 2021

  7. [12]

    Szepesv´ari, C. (2022). Algorithms for reinforcement learn- ing. Springer Nature. Wai, H.-T., Yang, Z., Wang, Z., and Hong, M. (2018). Multi- agent reinforcement learning via double averaging primal- dual optimization. Advances in Neural Information Pro- cessing Systems,

    work page 2022

  8. [13]

    and Hu, M

    Wang, D. and Hu, M. (2021). Deep deterministic policy gra- dient with compatible critic network. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4332–

    work page 2021

Show all 22 references
  1. [17]

    This shows that µ(·) is the stationary distribution ν(·) under kernel ePπθ and the proof is complete

    + I{s′ = s0} ! = (1 − γ)η(s′) = µ(s′). This shows that µ(·) is the stationary distribution ν(·) under kernel ePπθ and the proof is complete. B.2. Proof of Lemma 4.3 Proof. According to Sec. 13.2 in (Sutton, 2018), we know that ∇θJ(θ) = X s η(s) X a ∇πθ(a|s)Advθ(s, a). Multiply...

    2018

  2. [19]

    As shown by the red and blue curves, increasing either K or M helps mitigate this issue, indicating that larger batch sizes in Markov-batch sampling enable agents to achieve higher long-term rewards. Figure 10 presents our findings on the difference between (1) using the conse...

    1999

  3. [29]

    Gao, R., Cai, T., Li, H., Hsieh, C.-J., Wang, L., and Lee, J. D. (2019). Convergence of adversarial training in over- parametrized neural networks. Advances in Neural Infor- mation Processing Systems,

    2019

  4. [30]

    C., Hoang, D

    Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y .-C., and Kim, D. I. (2019). Applications of deep reinforcement learning in communications and networking: A survey. IEEE Communications Surveys & Tutorials, 21(4):3133–3174. Maei, H. R. (2018). Convergent ac...

    2019 arXiv

  5. [31]

    P., Littman, M

    Kaelbling, L. P., Littman, M. L., and Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237–285. Kiran, B. R., Sobh, I., Talpaert, V ., Mannion, P., Al Sallab, A. A., Yogamani, S., and P ´erez, P. (2021). Deep rein- forcement...

    1996

  6. [32]

    Castro, D. D. and Meir, R. (2010). A convergent online sin- gle time scale actor critic algorithm. Journal of Machine Learning Research, 11:367–410. Chen, Z., Zhou, Y ., Chen, R.-R., and Zou, S. (2022). Sample and communication-efficient decentralized actor-critic al- gorithms...

    2010 arXiv

  7. [33]

    Xu, T., Wang, Z., and Liang, Y . (2020b). Non-asymptotic convergence analysis of two time-scale (natural) actor- critic algorithms. arXiv preprint arXiv:2005.03557. 11 Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized MARL Yu, C., ...

    2020 arXiv

  8. [34]

    Zhu, Z., Zhu, J., Liu, J., and Liu, Y . (2021). Federated bandit: A gossiping approach. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 5(1):1–29. 12 Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized MARL ...

    2021

  9. [35]

    N., Vinyals, O., Senior, A., and Sak, H

    Sainath, T. N., Vinyals, O., Senior, A., and Sak, H. (2015). Convolutional, long short-term memory, fully connected deep neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4580–4584. IEEE. Sallab, A. E., Abdou, M., Pero...

    2015 arXiv

  10. [38]

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek LLM: Scaling open-source language models with longtermism. CoRR. Cai, Q., Yang, Z., Lee, J. D., and Wang, Z. (2019). Neural temporal-difference learning converg...

    2024

  11. [748]

    M., and Lazaric, A

    Yuan, R., Gower, R. M., and Lazaric, A. (2022). A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics, pages 3332–3380. PMLR. Zeng, S., Chen, T., Garcia, A., and Hong, M. (2022). Learning to coord...

    2022 arXiv

  12. [2017]

    that is commonly employed in prevalent RLHF frameworks. The goal of this experiment is not to compete with state-of-the-art RLHF techniques but to evaluate the effectiveness of our multi-agent actor-critic algorithm over a popular use case of decentralized RLHF. It is importan...

    2024

  13. [4344]

    Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. (2023). Iterative prefer- ence learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. arXiv preprint arXiv:2312.11456. Xu, T., Wang, Z., and Liang, Y . (2020a)....

    2023 arXiv

  14. [9869]

    Feriani, A

    PMLR. Feriani, A. and Hossain, E. (2021). Single and multi-agent deep reinforcement learning for AI-enabled wireless net- works: A tutorial. IEEE Communications Surveys & Tutorials, 23(2):1226–1252. Foerster, J., Assael, I. A., De Freitas, N., and Whiteson, S. (2016). Learning...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.