Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Iterative Update and Unified Representation for Multi-Agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One shared network trains all agents and beats MADDPG

desk verdict A useful parameter-sharing speedup, but the paper's iterative-update mechanism is contradicted by its own algorithm and the headline claim is not supported. read the letter →

arxiv 1908.06758 v1 pith:YUTF5NHR submitted 2019-08-16 cs.LG cs.AI

Jiancheng Long , Hongming Zhang , Tianyang Yu , Bo Xu This is my paper · ORCID
classification cs.LGcs.AI
keywords multi-agentreinforcementlearningnonstationarityiterativeupdateunifiedrepresentationvaluefixingMADDPGbatchcomputationfirst-personperspective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that multi-agent reinforcement learning can be made both stable and cheap by training one agent at a time while keeping the others fixed, and by representing every agent with a single network. It argues that freezing the waiting agents' policies turns the learning problem into a single-agent one, alleviating the nonstationarity that grows as the number of agents increases. A shared network with a first-person perspective per agent changes policy computation from sequential to batch, so interaction time and memory no longer grow linearly with agent count. On cooperative and mixed cooperative-competitive tasks, the method reports better performance than MADDPG in most comparisons and large wall-clock savings, especially as the number of agents grows. The paper is candid that the value-fixing step only makes the waiting agents' policy gradients small rather than zero.

What carries the argument

The key machinery is the value-fixing target for iterative update inside a unified network. For the current learning agent $i$, the critic target is $y_i = r_i + \gamma Q'(s, o_i, a'_1, \dots, a'_N)$ with $a'_j = \mu'(o_j)$; for waiting agents $-i$, the target is simply $y_{-i} = Q'(s, o_{-i}, a'_1, \dots, a'_N)$, so no Bellman update is applied to their part of the Q function. This is meant to keep the gradient of the waiting agents' policy objective near zero, fixing their strategies while the current agent improves. The unified representation adds each agent's own observation $o_i$ as a first-person perspective, so one network can produce distinct policies for all agents and enable batch computation.

What would settle it

Run IUUR on Spread_10 and record the action distributions of waiting agents immediately before and after the current agent's update. If the KL divergence between those distributions is large while the stated L2-norm bound on their policy gradient is small, then the value-fixing step does not actually keep waiting agents fixed, and the method's stated mechanism is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper claims that a single policy network can represent all agents in a multi-agent system, and that this representation can be trained stably by updating one agent at a time. The central discovery is a value-fixing rule based on the Bellman equation: for the agent currently being trained, the critic target uses the reward plus the discounted target-Q value, while for all other agents the target is taken directly from the target critic network without a Bellman expansion. The authors argue that this makes the policy gradient for the waiting agents small in L2 norm, so their policies stay approximately fixed even though they share the same network weights. This iterative update, combined with the unified one-network representation, is claimed to mitigate nonstationarity and to achieve state-of-the-art performance compared with MADDPG while saving significant wall-clock time, with the advantage growing as the number of agents increases.

Load-bearing premise

The load-bearing premise is that making the waiting agents' policy-gradient norm small is enough to keep their policies fixed, even though all agents share one network and any parameter update necessarily changes the network's outputs for every agent's observations.

Editorial extensions

If this is right

  • In fully cooperative tasks with many agents, iterative update reduces nonstationarity and outperforms the MADDPG baseline, with the gap growing as the number of agents increases.
  • In mixed cooperative-competitive tasks, the unified representation matches or beats MADDPG in most comparisons, with the paper noting a few cases where shared-network Q-value error hurts performance.
  • Wall-clock training time and environment interaction time drop substantially under IUUR, and interaction time no longer grows linearly with the number of agents.
  • Because policies are still computed from each agent's own observation, the method preserves decentralized execution and distributed deployment.
  • The paper states that the approach is compatible with other multi-agent algorithms such as IQL, VDN, and QMIX.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the authors leave implicit is to log the L2 norm of the waiting agents' policy gradients during IUUR training; if the norm is not actually smaller than under a Bellman target, the value-fixing premise fails in practice even if results are good.
  • The learning-frequency hyperparameter $K$ controls the balance between stability and the learning speed of waiting agents; the paper tunes it by experience, suggesting that annealing $K$ or adapting it per environment could widen the performance gains.
  • The single shared network may act as an implicit regularizer, forcing agents to share features; this could explain the fast convergence in Spread_3 and could be tested by ablating the first-person perspective encoding.
  • If the value-fixing target is sound, it should port to value-based MARL methods such as VDN and QMIX, where the critic is central and Bellman targets are already well defined.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two interlocking ideas for multi-agent reinforcement learning: iterative update (IU), in which only one agent is trained at a time while the policies of the other agents are held fixed, and unified representation (IUUR), in which all agents share a single policy network conditioned on first-person observations. The authors claim that IUUR mitigates nonstationarity, improves performance over MADDPG, reduces memory usage, and decreases wall-clock time, especially as the number of agents grows. The method is evaluated on cooperative (Spread) and mixed cooperative-competitive (Predator-Prey) continuous-control tasks, and the source code is made available online.

Significance. If the central mechanism worked as stated, the paper would make a useful practical contribution: parameter sharing via first-person perspective is a sensible way to reduce memory and computation in MARL, and fixing waiting agents' policies is a principled way to reduce nonstationarity. The availability of source code and the use of an external baseline (MADDPG) rather than fitted constants are strengths. However, the paper's own text and Algorithm 1 are inconsistent about which agents are updated, the value-fixing guarantee is asserted rather than proved or measured, and the headline performance claim is contradicted by one of the reported environments. As it stands, the experiments do not isolate the mechanism the paper claims to test.

major comments (4)
  1. [Algorithm 1] The actor update in Algorithm 1 sums over all agents: ∇θJ(μ|oi) ≈ (1/NB) Σ_i Σ_j (∇θμ(oi)∇_ai Q(...)), with no mask selecting the current learning agent l. This directly contradicts Section 3's statement that 'only the current particular agent is trained.' As written, every shared parameter update changes all agents' policies, so the training procedure degenerates to a shared-network version of MADDPG and the reported experiments would not test the iterative-update mechanism. Please correct the pseudocode to update only the current learning agent (or otherwise explicitly suppress waiting-agent gradients), and state clearly which version of the algorithm produced the reported results. If the implementation followed the printed pseudocode, the nonstationarity-related performance claims need to be rerun with the intended masking.
  2. [Section 3] The claimed guarantee in the paragraph beginning 'Luckily we can usually guarantee a smaller L2 norm of its gradient' is not established. Because all agents share the same parameters θ, any gradient step changes μ(o_-i) for waiting agents; a smaller L2 norm of the policy gradient does not imply that the waiting agents' policies are fixed, and no bound on the resulting policy change is given. The inequality is asserted without proof, and Section 5 later concedes that the equation 'cannot strictly hold.' Please provide a derivation under explicit assumptions, or empirically measure waiting-agent policy drift (for example, mean action distance or KL divergence between target policies before and after updates).
  3. [Section 3] The value-fixing target for waiting agents, y_-i = Q'(s, o_-i, a'_1, ..., a'_N), omits the immediate reward r_-i. Since Q is meant to represent a discounted return, regressing Q toward this target does not correspond to a Bellman fixed point for agent -i unless an additional argument is supplied. The paper should define the exact loss for waiting agents and explain why this target is valid; otherwise the value-fixing mechanism is underspecified and the reported training objective is ambiguous.
  4. [Section 4.2.1] In the Predator_6-Prey_2 predator-replacement condition, the paper reports that IUUR performs worse than MADDPG (Figure 6(a) and the accompanying text), attributing this to 'update error introduced by unified representation.' This is a direct counterexample to the abstract's unqualified statement that the algorithm 'achieves state-of-the-art performance.' Please qualify the performance claim to the environments and settings where the improvement is observed, and discuss whether the negative result is consistent with the value-fixing approximation or indicates a more fundamental limitation.
minor comments (5)
  1. [Section 2] The text 'For each agent's policy ri : S×A1×...×AN → R' reuses the reward notation for a policy; this should be πi.
  2. [Section 3 and Algorithm 1] The target notation y_-i = Q'(s, o_-i, ...) is confusing because Q is defined as Q(s, o_i, a_1, ..., a_N); please clarify which observation argument is used in the critic and targets.
  3. [Section 4.2.1] The caption of Figure 6 states 'In (a), IU outperforms MADDPG a lot and IU's performance is worse than that of MADDPG,' which is internally inconsistent; the text indicates that IUUR, not IU, is the worse-performing method in that condition.
  4. [Section 4.2.2] The wall-clock time savings are reported as single numbers without variance, number of runs, or details about batching/GPU utilization; adding this information would make the computational-efficiency claim more robust.
  5. [Section 4.2] Only one value of the new hyperparameter K is used (K=5000), with no sensitivity analysis; a short ablation or at least a discussion of how K affects the performance/nonstationarity trade-off would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IUUR is benchmarked against external MADDPG, and the value-fixing target is a heuristic rather than a fitted prediction.

full rationale

The derivation chain is not circular. The central performance claim is tested against MADDPG, an external baseline, and no parameter of IUUR is fitted to reproduce that baseline; the hyperparameter K is hand-tuned, which is a standard hyperparameter rather than a fitted constant that encodes the result. The value-fixing step in Section 3 defines the waiting-agent target as y_-i = Q'(s,o_-i,a') without a reward term, which is a heuristic bootstrap target, not a prediction equivalent by construction to the claimed outcome. The paper explicitly concedes the approximation: Section 3 states 'Though we can't hold the equation accurately, this method can substantially fix the other agents' polices which mitigate the nonstationarity effectively in practice,' and Section 5 states 'The value fixing method based on Bellman Equation can only guarantee a smaller L2 norm of its gradients and cannot strictly hold the equation.' These are admitted limitations, not disguised inputs. The asserted smaller L2 norm is not derived rigorously, but an unsupported inequality is a correctness risk, not circularity. The Algorithm 1 actor update sums the policy gradient over all agents with no mask selecting the current learning agent, which appears inconsistent with the paper's 'only the current particular agent is trained' claim; again, this is an implementation/correctness issue rather than a circular reduction. No load-bearing self-citation is present: the cited works on MADDPG, GANs, and continual learning are external and not authored by the current paper's authors. Since the reported results are externally benchmarked and the proposed mechanism is not a renamed fit of its own outputs, the circularity burden is not met.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the approximate value-fixing rule for a shared network, which the paper itself describes as unable to strictly hold. The remaining assumptions are standard for actor-critic MARL and are inherited from MADDPG.

free parameters (1)
  • K = 5000
    Controls how often the learning agent switches; the paper says it "plays a key role on the performance" and is "simply controlled through experience" in Section 4.2 and Section 5.
assumptions (3)
  • domain assumption Value-fixing approximation: making the L2 norm of waiting agents' policy gradients smaller than the Bellman-based alternative is sufficient to approximately fix their policies under shared parameters.
    Entered in Section 3, paragraph after Algorithm 1's value-fixing discussion. The paper itself says the equation cannot be held strictly.
  • domain assumption A single network with first-person perspective information can represent distinct, valid policies for all agents without per-agent parameters.
    Assumed in Section 3 when introducing the unified representation; no evidence or ablation is provided to verify that role information is sufficient.
  • domain assumption Fixing other agents' policies reduces the multi-agent problem to a single-agent MDP for the learning agent.
    Standard in game-theoretic best-response reasoning and stated in Section 3; used as the basis for the iterative update method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Iterative Update and Unified Representation for Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/YUTF5NHR

@misc{pith2026190806758,
  author       = {Pith},
  title        = {Pith review of: Iterative Update and Unified Representation for Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YUTF5NHR}},
  note         = {Machine review of arXiv:1908.06758}
}
read the original abstract

Multi-agent systems have a wide range of applications in cooperative and competitive tasks. As the number of agents increases, nonstationarity gets more serious in multi-agent reinforcement learning (MARL), which brings great difficulties to the learning process. Besides, current mainstream algorithms configure each agent an independent network,so that the memory usage increases linearly with the number of agents which greatly slows down the interaction with the environment. Inspired by Generative Adversarial Networks (GAN), this paper proposes an iterative update method (IU) to stabilize the nonstationary environment. Further, we add first-person perspective and represent all agents by only one network which can change agents' policies from sequential compute to batch compute. Similar to continual lifelong learning, we realize the iterative update method in this unified representative network (IUUR). In this method, iterative update can greatly alleviate the nonstationarity of the environment, unified representation can speed up the interaction with environment and avoid the linear growth of memory usage. Besides, this method does not bother decentralized execution and distributed deployment. Experiments show that compared with MADDPG, our algorithm achieves state-of-the-art performance and saves wall-clock time by a large margin especially with more agents.

Figures

Figures reproduced from arXiv: 1908.06758 by the authors.

Figure 1
Figure 1. (a) common method: all the agents are updated simultaneously (b) IU: the agents are updated iteratively (c) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Spread environments. Agents should reach di er [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Predator-Prey environments. Any of the preda [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Performance comparison in Predator_6-Prey_2. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages

  1. [1]

    A concise introduction to multiagent systems and distributed arti/f_icial intelligence.Synthesis Lectures on Arti/f_icial Intelligence and Machine Learning, 1(1):1–71, 2007

    Nikos Vlassis. A concise introduction to multiagent systems and distributed arti/f_icial intelligence.Synthesis Lectures on Arti/f_icial Intelligence and Machine Learning, 1(1):1–71, 2007

    work page 2007

  2. [2]

    Multiagent systems: A survey from a machine learning perspective

    Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. Autonomous Robots, 8(3):345–383, 2000

    work page 2000

  3. [3]

    Multiagent systems: a modern approach to distributed arti/f_icial intelligence

    Gerhard Weiss. Multiagent systems: a modern approach to distributed arti/f_icial intelligence. MIT press, 1999

    work page 1999

  4. [4]

    Re- inforcement learning for cooperating and communicating reactive agents in electrical power grids

    Martin Riedmiller, Andrew Moore, and Jeff Schneider. Re- inforcement learning for cooperating and communicating reactive agents in electrical power grids. In Workshop on Balancing Reactivity and Social Deliberation in Multi-Agent Systems, pages 137–149. Springer, 2000

    work page 2000

  5. [5]

    Game theory and multi-agent reinforcement learning

    Ann Nowé, Peter Vrancx, and Yann-Michaël De Hauwere. Game theory and multi-agent reinforcement learning. In Reinforcement Learning, pages 441–470. Springer, 2012

    work page 2012

  6. [6]

    Multi-agent reinforcement learning: An overview

    Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. In In- novations in multi-agent systems and applications-1 , pages 183–221. Springer, 2010

    work page 2010

  7. [7]

    Reinforcement learn- ing: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learn- ing: An introduction. MIT press, 2018

    2018

  8. [8]

    Markov games as a framework for multi- agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi- agent reinforcement learning. In Machine learning proceed- ings 1994, pages 157–163. Elsevier, 1994

    work page 1994

Show all 29 references
  1. [9]

    Nash q-learning for general-sum stochastic games

    Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. Journal of machine learning research, 4(Nov):1039–1069, 2003

    2003

  2. [10]

    Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games

    Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017

    2017 arXiv

  3. [11]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672–2680, 2014

    2014

  4. [12]

    Continual lifelong learning with neural networks: A review

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019

    2019

  5. [13]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , pages 6379–6390, 2017

    2017

  6. [14]

    Partially observable markov decision processes for spoken dialog systems

    Jason D Williams and Steve Young. Partially observable markov decision processes for spoken dialog systems. Com- puter Speech & Language , 21(2):393–422, 2007

    2007

  7. [15]

    Multi-agent reinforcement learning: Independent vs

    Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth interna- tional conference on machine learning , pages 330–337, 1993

    1993

  8. [16]

    Q-learning

    Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992

    1992

  9. [17]

    Op- timal and approximate q-value functions for decentralized pomdps

    Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Op- timal and approximate q-value functions for decentralized pomdps. Journal of Arti/f_icial Intelligence Research, 32:289– 353, 2008

    2008

  10. [18]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  11. [19]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Mar- ian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017

    2017 arXiv

  12. [20]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485, 2018

    2018 arXiv

  13. [21]

    Counterfactual multi-agent policy gradients

    Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Con- ference on Arti/f_icial Intelligence, 2018

    2018

  14. [22]

    Contin- ual match based training in pommerman: Technical report

    Peng Peng, Liang Pang, Yufeng Yuan, and Chao Gao. Contin- ual match based training in pommerman: Technical report. arXiv preprint arXiv:1812.07297, 2018

    2018 arXiv

  15. [23]

    Human-level performance in /f_irst-person multiplayer games with population-based deep reinforcement learning

    Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beat- tie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in /f_irst-person multiplayer games with population-based deep reinforcemen...

    2018 arXiv

  16. [24]

    Popu- lation based training of neural networks

    Max Jaderberg, Valentin Dalibard, Simon Osindero, Woj- ciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Popu- lation based training of neural networks. arXiv preprint arXiv:1711.09846, 2017

    2017 arXiv

  17. [25]

    A generalized dynamic programming princi- ple and hamilton-jacobi-bellman equation

    Shige Peng. A generalized dynamic programming princi- ple and hamilton-jacobi-bellman equation. Stochastics: An International Journal of Probability and Stochastic Processes , 38(2):119–134, 1992

    1992

  18. [26]

    Convex optimiza- tion

    Stephen Boyd and Lieven Vandenberghe. Convex optimiza- tion. Cambridge university press, 2004

    2004

  19. [27]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. InAdvances in neural information processing systems, pages 1057–1063, 2000

    2000

  20. [28]

    Wasserstein gan

    Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017

    2017 arXiv

  21. [29]

    Improved training of wasserstein gans

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Process- ing Systems, pages 5767–5777, 2017. 7

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.