Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that using a health-weighted minimum-health counterfactual baseline in multi-agent PPO — actual joint value minus the value of the same state with the agent's health set to zero — significantly improves learning in…

desk verdict A genuine, simple health-based counterfactual baseline for MARL, but the 'significant improvement' claim outruns the evidence: four seeds, no significance tests, and one environment showing parity. read the letter →

arxiv 1908.01022 v4 pith:XZD7AJ4E submitted 2019-08-02 cs.LG cs.AIcs.MAstat.ML

Ross E. Allen , Jayesh K. Gupta , Jaime Pena , Yutai Zhou , Javona White Bear , Mykel J. Kochenderfer This is my paper · ORCID
classification cs.LGcs.AIcs.MAstat.ML
keywords Multi-AgentReinforcementLearning(MARL)systemhealthcreditassignmentproximalpolicyoptimizationcontinuouscontrolcounterfactualbaselinemulti-robotsystemsrisk-taking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the degree of degradation of each agent, its system health, can be turned into a credit-assignment signal for cooperative multi-agent reinforcement learning. The proposed gradient replaces the usual advantage with a counterfactual baseline: for each agent, compare the observed joint return against the value of the same state with that agent's health set to its minimum, and scale by the agent's current health. Because this baseline only needs a state value function, it works in continuous action spaces where action-enumerating methods like COMA or QMIX fail. The empirical claim is that in hazardous navigation, hazardous communication, and multiwalker environments with joint rewards and attrition, health-informed multi-agent PPO learns faster and with less variance than local-critic and central-critic MAPPO, MADDPG, and QMIX, and the advantage grows with the number of agents.

What carries the argument

The load-bearing object is the minimum-health counterfactual baseline, built on top of the paper's formal definition of system health. Health $h$ is a subvector of the state whose components lie in $[0,1]$, and the definition imposes four monotone-constriction properties: a zero-health agent cannot recover; lower health shrinks the reachable state set, the agent's available action set, and its observation set. The baseline uses $s^{\lnot i}_t$, the true joint state with agent $i$'s health component replaced by its minimum value, and the state value $V^{w_{\mathrm{old}}}(s^{\lnot i}_t)$ as the counterfactual return; the difference $V^{\mathrm{targ}}_t - V^{w_{\mathrm{old}}}(s^{\lnot i}_t)$ credits agent $i$ for value that only exists because that agent is alive. The multiplier $h_{i,t}$ attenuates the gradient for damaged agents, which the paper motivates by the constriction properties: when health limits the action set, the action chosen by the policy is less likely to match the action the agent can actually execute, so low-health experience should influence learning more slowly. In the multi-agent PPO variant, this $\Psi_{i,t}$ replaces the advantage in PPO's clipped surrogate objective, with the centralized critic trained by generalized advantage estimation.

What would settle it

Run health-informed MAPPO against central-critic MAPPO on two cooperative tasks that are identical except that in one task a zero-health agent's action set is frozen as the paper's properties require, while in the other a zero-health agent retains its full action set and health is only a label; if the minimum-health baseline still improves learning in the second task, the improvement does not depend on the constriction mechanism the paper invokes. A second check is to record the policy-gradient contribution of dead agents in a binary-health experiment: the proof of Lemma 2 requires that contribution to be exactly zero, so a nonzero measured contribution would contradict the stated convergence argument.

Watch

Extended reading notes

Core claim

The central discovery is a form of difference-reward credit assignment that uses agent death as the counterfactual. In its PPO form, the per-agent advantage-like term is $\Psi_{i,t} = h_{i,t}(V^{\mathrm{targ}}_t - V^{w_{\mathrm{old}}}(s^{\lnot i}_t))$, where $h_{i,t}\in[0,1]$ is agent $i$'s current health and $s^{\lnot i}_t$ is the joint state with agent $i$'s health replaced by its minimum value. The paper proves that the baseline part of this term contributes zero gradient, so it does not bias the policy gradient, and that for binary health states the scaled gradient reduces to the REINFORCE gradient of the survivors, which has known convergence properties. The key practical property is that this baseline does not enumerate actions or factorize the value function, so it applies to continuous-control and partially observable settings. The authors report that, across their three environment families, MAPPO with this crediting term outperforms non-crediting policy gradients and action-enumerating baselines, with the gap widest at larger agent counts.

Load-bearing premise

The whole method rests on the assumption that each agent's health is a known, observed part of the state and that lower health strictly shrinks the actions, observations, and reachable states available to that agent; if real damage does not behave that way, the health weighting can distort the gradient rather than assign credit.

Editorial extensions

If this is right

  • Health-informed MAPPO achieves higher total reward and lower cross-trial variance than central-critic MAPPO, local-critic MAPPO, MADDPG, and QMIX in the hazardous navigation, hazardous communication, and multiwalker environments tested.
  • The performance gap between health-informed crediting and the non-crediting alternatives widens as the number of agents grows, consistent with the credit-assignment problem becoming harder with more agents.
  • The baseline's action-space agnosticism means the same credit-assignment term can be dropped into any policy-gradient method that uses a state value function, including TRPO and PPO, for continuous control.
  • In environments that do not encode health or risk, the minimum-health credit term is not expected to help and can increase variance, as shown by the paper's control experiment on the original cooperative navigation task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The health semantics are replaceable: any per-agent quantity that monotonically constricts actions and observations, such as fuel, battery, sensor aperture, or communication range, could power the same baseline, so the method's scope may be broader than 'health' and worth testing under resource-depletion models.
  • The convergence argument covers binary health states only; for graded damage the paper gives no proof, so a natural stress test is whether the health multiplier should be $h_{i,t}$ or a step function $\mathbf{1}_{h_{i,t}>0}$, which may behave differently in partially degraded regimes.
  • The choice of minimum health as the counterfactual assumes death is the most informative comparison for credit; comparing against intermediate health levels would test whether the minimum is optimal or merely convenient.
  • A testable implication for downstream work is that the method should degrade gracefully when health is noisy or only partially observed, since the four properties require health to be known; this is not addressed in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a formal definition of system health for multi-agent systems and uses it to construct a counterfactual credit-assignment baseline for policy-gradient reinforcement learning. The proposed baseline, Ψ_i,t = h_i,t (G_t − V(s with agent i at minimum health)), is embedded into a multi-agent variant of PPO (health-informed MAPPO). The authors evaluate the method on two custom Multi-Agent Particle Environments (hazardous navigation and hazardous communication) and on the PettingZoo multiwalker environment, comparing against MADDPG, QMIX, local-critic MAPPO, and central-critic MAPPO. They report that health-informed crediting tends to outperform the non-crediting baselines and that the performance gap grows with the number of agents, and they claim 'significant improvement in learning performance' over policy-gradient methods without multi-agent credit assignment.

Significance. If the empirical claims are robust, the paper offers a simple, continuous-action-compatible credit-assignment technique for cooperative MARL problems with agent attrition. The minimum-health counterfactual baseline is a natural and novel idea, and the paper includes several strengths: the algorithm is clearly specified (Eqs. 8 and 12, Algorithm 1), the code and environments are publicly available, and the experiments include a centralized non-crediting baseline that shares the same critic architecture as the proposed method, which is the correct control for isolating the health-crediting effect. The principal weaknesses are statistical: the headline claim rests on only four seeds per condition with no significance testing, and the authors themselves document one environment (5-agent multiwalker) where health-informed crediting performs on par with the local-critic baseline, which is in tension with the abstract. The theoretical convergence lemma (Lemma 2) is also not a rigorous convergence proof. With additional statistical rigor and a tempered theoretical claim, the contribution would be a useful addition to the multi-agent RL literature.

major comments (4)
  1. [§5.1 / Abstract] The central empirical claim of 'significant improvement' is not supported by the reported statistics. Each learning curve in Figure 2 is an average over four independent runs with min-max shading; no significance tests, confidence intervals, or per-seed results are reported. With n=4, such aggregates are highly sensitive to outliers. The authors themselves note in §5.1 that in the 5-agent multiwalker experiment the local-critic MAPPO aggregate is 'heavily influenced by a trial that seemed to discover an exploit' and that min-health crediting performs 'on par' with the local critic in that environment. This is a documented counterexample to the universal claim in the abstract. Please add statistical tests (e.g., bootstrap or paired comparisons on final performance or learning-curve area), report all seeds, and either soften the abstract's claim or provide evidence that is consistent with it.
  2. [§4.1, Lemma 2] Lemma 2 does not prove convergence in the sense stated. Eq. (9) asserts lim ||∇θJ|| = 0, but the proof only shows that under binary health and deterministic dead-agent actions, gΨ equals the single-agent REINFORCE gradient. That equality does not establish convergence: standard REINFORCE convergence results require specific step-size schedules, appropriate reward discounting assumptions, and do not directly extend to the clipped PPO objective used in the paper. Furthermore, a gradient-norm limit of zero is not equivalent to convergence of the iterates to a locally optimal policy. I recommend either removing Lemma 2, restating it as an unbiased-estimator claim (the estimator equals the REINFORCE gradient in the binary-health case), or proving a proper convergence theorem with all required assumptions made explicit.
  3. [§4.1, Eq. (8)] The health multiplier h_i,t is not a standard baseline term; it scales the entire advantage (G_t − V(s^{-i})), so the resulting estimator is not an unbiased estimate of the original policy gradient unless h_i,t is constant with respect to the sampled action. The paper motivates this multiplier through the action-mismatch argument based on Properties 2–3, but that discussion is intuitive rather than formal. In the binary-health experiments the multiplier simply zeroes out terminated agents, which is benign; however, the paper claims applicability to general health values in [0,1]. Please clarify that for non-binary health the health multiplier is a heuristic that biases the gradient, and describe the conditions under which this bias is acceptable, or provide a derivation showing unbiasedness under a clearly stated assumption. As written, the combination of Lemma 1 and Eq. (8) can be misread as claiming the overall estimator is unbiased, which is not established.
  4. [§5.1 (architecture confound)] The comparison between centralized-critic methods and local-critic MAPPO is confounded by critic architecture: the centralized critic is an 8-layer, 64-unit ELU MLP ('developed empirically'), while the local-critic value network is a 2-layer, 64-unit tanh MLP matching the policy network. Thus performance differences between min-health crediting and local-critic MAPPO could be due to critic capacity rather than to the health-informed baseline. The comparison between min-health crediting and central-critic MAPPO is architecture-controlled and is the key evidence for the health baseline, but the abstract's claim refers to 'policy gradient methods that do not perform multi-agent credit assignment' broadly. Please either match the local critic's capacity to the central critic, restrict the paper's claim to the central-critic comparison, or explicitly discuss how the architecture difference affects interpretation of the local-critic results.
minor comments (5)
  1. [Throughout] There are several typos and grammatical slips: 'perofmance' (§5.1), 'orginally' (Appendix B), 'underpeforms' (Appendix B), and 'it’s health is set two zero' (Section 5, multiwalker description).
  2. [Figure 2 caption] The caption labels the multiwalker panels as (2e, 2f), but the text refers specifically to the '5-Agent multiwalker experiment (2f)' without stating the agent count for panel (2e). Please add agent counts to the subfigure labels or caption.
  3. [§4.2, Eq. (12)] The substitution of G_t by the value target V_targ (Eq. 10) in the baseline is not fully explained. Since V_targ = A_GAE + V_wold, the expression Ψ_i,t = h_i,t (V_targ − V_wold(s^{-i})) becomes a difference of value estimates; please clarify how this relates to GAE and why the counterfactual baseline remains valid in this form.
  4. [§5.1 (hyperparameters)] The centralized critic architecture (8-layer, 64-unit, ELU) is described as 'developed empirically.' Please report the tuning procedure, the set of architectures tried, and the selection criterion, so that the results can be reproduced and the sensitivity to this choice can be assessed.
  5. [§4.1 (related-work claim)] The sentence describing Eq. (8) as 'completely agnostic to the action space, a property not seen in prior work' is too strong: several policy-gradient methods for continuous action spaces exist (e.g., MADDPG, multi-agent TRPO), even if they do not use counterfactual credit assignment. Please rephrase to 'agnostic to action-space enumerability' or otherwise narrow the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the min-health baseline is a counterfactual critic term, not a fitted or self-referential prediction.

full rationale

The derivation chain is self-contained. Equation (8) defines Ψ_i,t = h_i,t (G_t − V^π(s^{¬h_i,t}, h_i,min)) as a standard action-independent counterfactual baseline; the learned V is a critic trained by Eq. (11), not a parameter fitted to reproduce the experimental improvement. Lemma 1 shows the baseline contribution has zero expected gradient (the usual action-independent baseline property), and Lemma 2 reduces the binary-health case to the single-agent REINFORCE gradient, citing external convergence results [32, 38]. The health multiplier h_i,t is introduced explicitly as a modeling choice motivated by Properties 2–3, with no fitted constant encoding the target outcome. The empirical comparisons against MADDPG, QMIX, and local/central-critic MAPPO are external benchmarks; even if the 'significant improvement' claim is statistically weak with four seeds and no significance tests, that is an evidence-quality concern rather than circularity. The paper itself discloses in Section 5.1 that local-critic MAPPO's aggregate in the multiwalker environment is 'heavily influenced by a trial that seemed to discover an exploit', and Appendix B admits min-health crediting is not superior on a non-health environment; these are disclosed limitations, not circular steps. Self-citations such as Gupta et al. [9] are used for parameter sharing and the multiwalker environment, not as the load-bearing justification for the health-informed baseline, so they do not create a self-citation circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim is an algorithmic improvement with no fitted constants at the mathematical core. The health construct is a domain assumption that defines the class of problems addressed. Empirical tuning of the critic architecture and entropy coefficient are the only free parameters. The questionable zero-gradient assumption in Lemma 2 is a notable axiomatic gap.

free parameters (2)
  • centralized critic architecture (8-layer, 64-unit, ELU) = 8 layers, 64 units, ELU activation
    Reported as developed empirically for these environments; the method's performance advantage may depend on this tuning.
  • entropy coefficient c = 0.01 for particle environments, 0.0 for multiwalker
    Set by hand to stabilize exploration, as stated in Section 5.1; standard PPO hyperparameter but chosen per environment.
assumptions (4)
  • domain assumption Health is a subvector of state h in [0,1]^n and obeys Properties 1-4 (non-recoverable minimum health, monotone constriction of reachable states, action sets, observation sets).
    Introduced in Section 3.1; the whole method is defined for this environment class.
  • domain assumption All agents share policy parameters and execute identical copies of the policy (parameter sharing).
    Assumed in Section 4 and Algorithm 1; enables the centralized-critic, decentralized-execution architecture.
  • standard math The policy gradient theorem and the log-derivative trick apply to the factored joint policy.
    Used in Section 4.1 and Appendix A, adapted from Sutton and Barto.
  • domain assumption For an agent with binary health 0, the action space contains exactly one action, which is deterministically executed, so its policy gradient contribution is zero.
    Stated in Lemma 2 proof; questionable when the policy distribution is defined over a fixed continuous action space and the agent is forced to execute a zero action.
invented entities (1)
  • System health vector h
    purpose: Provides the credit assignment signal and gradient attenuation in Eq. (8) and (12).
    Defined by the paper via Properties 1-4; no falsifiable prediction outside the paper's own constructed environments is offered. It is a modeling construct, not a measured physical quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/XZD7AJ4E

@misc{pith2026190801022,
  author       = {Pith},
  title        = {Pith review of: Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XZD7AJ4E}},
  note         = {Machine review of arXiv:1908.01022}
}
read the original abstract

This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is then extended to a multi-agent variant of the proximal policy optimization algorithm and demonstrated on particle and multiwalker robot environments that have characteristics such as system health, risk-taking, semi-expendable agents, continuous action spaces, and partial observability. We show significant improvement in learning performance compared to policy gradient methods that do not perform multi-agent credit assignment.

Figures

Figures reproduced from arXiv: 1908.01022 by the authors.

Figure 1
Figure 1. The 8-agent hazard navigation (1a), 16-agent hazardous communication (1b), and 5-agent multiwalker (1c) environ [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Learning curves for the hazardous navigation (2a, 2b), hazardous communication (2c, 2d), and multiwalker (2e, 2f) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Learning curves for the cooperative navigation [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 29 canonical work pages

  1. [1]

    Johnson, and Mykel J

    Edward Balaban, Stephen B. Johnson, and Mykel J. Kochenderfer. 2019. Unifying System Health Management and Automated Decision Making.Journal of Artificial Intelligence Research 65 (Aug. 2019), 487–518. https://doi.org/10.1613/jair.1.11366

    work page doi:10.1613/jair.1.11366 2019

  2. [2]

    Edward Balaban, Sriram Narasimhan, Matthew Daigle, Indranil Roychoudhury, Adam Sweet, Christopher Bond, and G Gorospe. 2013. Development of a mobile robot test platform and methods for validation of prognostics-enabled decision making algorithms. International Journal of Prognostics and Health Management 4, 1 (2013), 87

    work page 2013

  3. [3]

    Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. 2002. The complexity of decentralized control of Markov decision processes. Mathe- matics of Operations Research 27, 4 (2002), 819–840

    work page 2002

  4. [4]

    Abdeslam Boularias and Brahim Chaib-draa. 2008. Exact dynamic programming for decentralized POMDPs with lossless policy compression. In International Conference on Automated Planning and Scheduling (ICAPS) . AAAI Press, 20–27

    work page 2008

  5. [5]

    Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (ELUs).arXiv preprint arXiv:1511.07289 (2015)

    arXiv 2015

  6. [6]

    Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. 2016. Benchmarking deep reinforcement learning for continuous control. In Interna- tional Conference on Machine Learning (ICML) . 1329–1338

    work page 2016

  7. [7]

    Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI)

    work page 2018

  8. [8]

    Yasuhiro Fujita and Shin-ichi Maeda. 2018. Clipped action policy gradient. arXiv preprint arXiv:1802.07564 (2018)

    arXiv 2018

Show all 43 references
  1. [9]

    Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. 2017. Cooperative multi- agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) . Springer, 66–83

    2017

  2. [10]

    Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. 2004. Dynamic programming for partially observable stochastic games. In AAAI Conference on Artificial Intelligence (AAAI), Vol. 4. 709–715

    2004

  3. [11]

    Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. 2017. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv:1707.09183 (2017)

    2017 arXiv

  4. [12]

    Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. 2019. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems 33, 6 (2019), 750–797

    2019

  5. [13]

    Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. 2018. Local communication protocols for learning complex swarm behaviors with deep rein- forcement learning. In International Conference on Swarm Intelligence . Springer, 71–83

    2018

  6. [14]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  7. [15]

    Mykel J Kochenderfer. 2015. Decision making under uncertainty: Theory and application. MIT Press

    2015

  8. [16]

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Gold- berg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning. 3053–3062

    2018

  9. [17]

    Ryan Lowe. 2018. Multi-agent particle environment. https://github.com/openai/ multiagent-particle-envs

    2018

  10. [18]

    Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch

  11. [19]

    Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. 2019. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems. 7613–7624

    2019

  12. [20]

    Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. 2018. Credit assign- ment for collective multiagent RL with global rewards. In Advances in Neural Information Processing Systems (NIPS) . 8102–8113

    2018

  13. [21]

    Frans A Oliehoek, Julian FP Kooij, and Nikos Vlassis. 2008. The cross-entropy method for policy search in decentralized POMDPs. Informatica 32, 4 (2008), 341–357

    2008

  14. [22]

    Frans A Oliehoek, Shimon Whiteson, and Matthijs TJ Spaan. 2013. Approximate solutions for factored Dec-POMDPs with many agents. InInternational Conference on Autonomous Agents and Multiagent Systems (AAMAS) . 563–570

    2013

  15. [23]

    Shayegan Omidshafiei, Ali-akbar Agha-mohammadi, Christopher Amato, Shih- Yuan Liu, Jonathan P How, and John L Vian. 2016. Health-aware multi-UAV planning using decentralized partially observable semi-Markov decision pro- cesses. In AIAA Infotech@ Aerospace. 1407

    2016

  16. [24]

    OpenAI. 2019. OpenAI Five. https://openai.com/five/#how-openai-five-works

    2019

  17. [25]

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Far- quhar, Jakob Foerster, and Shimon Whiteson. 2018. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning.arXiv preprint arXiv:1803.11485 (2018)

    2018 arXiv

  18. [26]

    Abhinav Saxena, Jose Celaya, Edward Balaban, Kai Goebel, Bhaskar Saha, Sankalita Saha, and Mark Schwabacher. 2008. Metrics for evaluating perfor- mance of prognostic techniques. In IEEE International Conference on Prognostics and Health Management. 1–17

    2008

  19. [27]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

  20. [28]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel

  21. [29]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  22. [30]

    Matthijs TJ Spaan, Frans A Oliehoek, and Christopher Amato. 2011. Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion. In Interna- tional Joint Conference on Artificial Intelligence (IJCAI)

    2011

  23. [31]

    Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An intro- duction. MIT Press, Chapter 13, 321–338

    2018

  24. [32]

    Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems . 1057–1063

    2000

  25. [33]

    arXiv preprint arXiv:1707.06347 (2017)

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  26. [34]

    Justin K Terry, Benjamin Black, Ananth Hari, Luis Santos, Clemens Dieffendahl, Niall L Williams, Yashas Lokesh, Caroline Horsch, and Praveen Ravi. 2020. PettingZoo: Gym for Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2009.14471 (2020)

    2020 arXiv

  27. [35]

    Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, Benjamin Black, and Dinesh Manocha. 2020. Parameter Sharing is Surprisingly Useful for Multi- Agent Deep Reinforcement Learning. arXiv preprint arXiv:2005.13625 (2020)

    2020 arXiv

  28. [36]

    Kagan Tumer, Adrian K Agogino, and David H Wolpert. 2002. Learning sequences of actions in collectives of autonomous agents. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) . 378–385

    2002

  29. [37]

    Daniel Szer and François Charpillet. 2005. An optimal best-first search algorithm for solving infinite horizon Dec-POMDPs. In European Conference on Machine Learning (ECML). Springer, 389–399

    2005

  30. [38]

    Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 3-4 (1992), 229–256

    1992

  31. [39]

    ∑︁ u π(u| s𝑡, θ)· 𝑛∑︁ 𝑖=1 𝑞π(s𝑡, u)∇𝜃𝑖 log𝜋𝑖(𝑎𝑖|𝜏𝑖,𝑡,𝜃𝑖) # = Eπ

    David H Wolpert and Kagan Tumer. 2002. Optimal payoff functions for members of collectives. In Modeling Complexity in Economic and Social Systems . World Scientific, 355–369. A EXTENDED PROOFS FOR MULTI-AGENT POLICY GRADIENTS Here elaborate on the equations presented in Sectio...

    2002

  32. [41]

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (2019), 350–354

    2019

  33. [2015]

    In International Conference on Machine Learning (ICML)

    Trust region policy optimization. In International Conference on Machine Learning (ICML). 1889–1897

  34. [2016]

    In International Conference on Learning Representations

    High-dimensional continuous control using generalized advantage estima- tion. In International Conference on Learning Representations

  35. [2017]

    In Advances in Neural Information Processing Systems (NIPS)

    Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems (NIPS) . 6379–6390

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.