Pith. sign in

REVIEW 2 major objections 5 minor 31 references

ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ToMacVF claims that asynchronous multi-agent teams can train macro-action policies with per-timestep temporal credit assignment via Mac-SJERT and a temporal IGM condition, keeping decentralized greedy execution consistent with joint…

desk verdict Useful buffer design and consistent experiments, but the headline generalization theorem is false and its proof is invalid. read the letter →

arxiv 2507.10251 v1 pith:YMSR7WKI submitted 2025-07-14 cs.MA

Wenjing Zhang , Wei Zhang This is my paper · ORCID
classification cs.MA
keywords multi-agentreinforcementlearningmacro-actionsvaluefactorizationcreditassignmentMacDec-POMDPasynchronousdecision-makingIGMcentralizedtrainingdecentralizedexecution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that asynchronous multi-agent reinforcement learning, where agents execute macro-actions of different durations, does not have to throw away the intermediate steps of those macro-actions. Existing asynchronous methods sample only the endpoints of macro-actions into a joint replay buffer and then apply synchronous value-factorization algorithms, which gives coarse and often wrong credit assignment. The authors propose ToMacVF, built on a per-agent segmented buffer (Mac-SJERT) that records the full execution of each macro-action, and a consistency condition (To-Mac-IGM) that extends the standard IGM principle to the temporal dimension. They also introduce an attention-based parameter generator (ATPG) so the mixing network can use temporal inter-agent dependence. If the claims hold, asynchronous MARL can achieve the same kind of principled factorization as synchronous MARL while still allowing each agent to act greedily from its own local history.

What carries the argument

The central object is To-Mac-IGM, a consistency condition between the joint macro-action value function and per-agent macro-action utilities that adds a per-timestep execution coordinate: only agents whose macro-action has terminated choose a new macro-action, and their greedy choices under the joint and individual value functions must coincide. The argument is carried by two supporting mechanisms: Mac-SJERT, a buffer that stores complete per-timestep joint transitions for each macro-action segment with sinusoidal positional encodings, and the ATPG module, which uses temporal and agent-oriented self-attention to generate non-negative weights for the mixing network, preserving monotonicity and enabling sufficiency. Together they let the algorithm train with a micro-TD loss at every timestep and a macro-TD loss at macro-action termination.

What would settle it

A concrete way to test the hierarchy claim is to enumerate all joint and individual value functions for a small two-agent MacDec-POMDP with, say, two primitive actions, two macro-actions, and a three-step execution horizon, and check whether every function satisfying IGM on the primitive domain admits a Mac-IGM factorization and every Mac-IGM function admits a To-Mac-IGM factorization. A single finite instance where an IGM-factorable function has no Mac-IGM factorization would refute Theorem 3. Similarly, the algorithmic claim could be tested by running ToMacVF on a domain where endpoint-only rewards are known to be misleading and measuring whether value estimates track the true per-step return.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that asynchronous macro-action MARL can be given the same principled value-factorization treatment as synchronous MARL by making time itself part of the factorization. The authors introduce To-Mac-IGM, a consistency requirement in which the joint greedy selection over macro-actions must match each terminating agent's individual greedy selection, while non-terminating agents keep their ongoing macro-actions. They prove, as Theorems 2 and 3, that the class of value functions satisfying To-Mac-IGM contains the classes satisfying IGM and Mac-IGM, and they claim the hierarchy $F^{\mathrm{IGM}} \subset F^{\mathrm{Mac\text{-}IGM}} \subset F^{\mathrm{To\text{-}Mac\text{-}IGM}}$ holds. The accompanying Mac-SJERT buffer and the ATPG module are the mechanisms that make the condition realizable in a deep RL training pipeline, and the reported experiments on BoxPushing, Warehouse, and Overcooked show ToMacVF matching or exceeding asynchronous baselines.

Load-bearing premise

The load-bearing premise is that adding macro-actions and execution timesteps to the value-function domain only makes the factorization condition easier to satisfy, so the classes of factorable functions are automatically nested; if that premise fails, the hierarchy theorem does not follow.

Editorial extensions

If this is right

  • Under To-Mac-IGM, decentralized greedy execution remains consistent with joint greedy selection even when agents have macro-actions of different durations, so prior synchronous factorization guarantees carry over to asynchronous settings.
  • Mac-SJERT removes duplicated macro-action representations, observation staleness, and reward-reset errors, yielding more accurate value estimates than endpoint-only buffers.
  • The claimed hierarchy $F^{\mathrm{IGM}} \subset F^{\mathrm{Mac\text{-}IGM}} \subset F^{\mathrm{To\text{-}Mac\text{-}IGM}}$ means every synchronous-factorizable task is also factorizable in the macro-action temporal setting, so asynchronous methods do not lose representational power.
  • The ToMacVF architecture can be instantiated with existing mixing networks like QMIX, Qtran, and QPLEX, making temporal macro-action factorization a drop-in extension of prior methods.
  • If the reported experiments are representative, ToMacVF matches or outperforms asynchronous baselines and is robust across BoxPushing, Warehouse, and Overcooked variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The nesting claim in Theorem 3 is a representational statement, not a statement about learnability; even if the class inclusion holds, optimizing over the larger temporal class may be harder in practice.
  • An immediate testable extension is to feed Mac-SJERT-style per-step data into policy-gradient or actor-critic asynchronous methods, not just Q-learning factorization, since the buffer fixes observation and reward misassignment independently of the value-update rule.
  • The claimed class hierarchy depends on a cardinality argument; a direct constructive proof that exhibits a To-Mac-IGM factor on the extended domain for every IGM factor would be stronger and could be checked on small finite MDPs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes ToMacVF, a value-factorization framework for asynchronous multi-agent reinforcement learning with macro-actions. The main components are a per-agent replay buffer (Mac-SJERT) that records timestep-level joint transitions during macro-action execution, a consistency condition called To-Mac-IGM that generalizes IGM-style greedy-selection consistency to asynchronous settings, and a mixing architecture (ATPG) that generates non-negative parameters for the mixer. Training uses micro-TD and macro-TD losses. The paper claims to prove that To-Mac-IGM generalizes IGM and Mac-IGM, and reports experiments on BoxPushing, Warehouse, and Overcooked with ablations.

Significance. The buffer design addresses real shortcomings of Mac-JERTs, and the experimental study is broad: three benchmarks, multiple difficulty levels, twenty runs, and ablations on the replay buffer and the attention module. If the theoretical claim were sound, the framework would be a meaningful step toward fine-grained credit assignment for asynchronous macro-action policies. However, the central theorem is not proved and, as stated, is false; the TD targets also appear to use an unconstrained joint maximization that is inconsistent with the asynchronous decision structure. The empirical results therefore cannot establish the paper's main theoretical and algorithmic claims.

major comments (2)
  1. [Appendix A.4, Theorem 3 (Eqs. 19–23)] The proof of Theorem 3 derives F_IGM ⊂ F_Mac-IGM ⊂ F_To-Mac-IGM solely from cardinality inequalities on the domains of the value functions. This inference is invalid: enlarging a domain does not enlarge a class of functions defined by an argmax-consistency condition, because the additional arguments also introduce new consistency constraints that can exclude previously valid pairs. The second inclusion is actually false. Consider two agents with M1 = M2 = {0, 1}, individual utilities Q1(0)=Q2(0)=0, Q1(1)=Q2(1)=1, and joint values Qtotal(1,1)=10, Qtotal(0,1)=Qtotal(1,0)=0, Qtotal(0,0)=5, extended constantly across histories. This pair satisfies Mac-IGM because the joint argmax (1,1) equals the product of the individual argmaxes. If agent 2 is ongoing with macro-action m−=0, Eq. (7) requires agent 1's conditional greedy choice to be argmax_{m1} Q1(m1,0)=1, whereas the conditional joint argmax under m2=0 selects m1=0 (since Qtotal(0,0)=5 > Qtotal(1,0)=0). Hence the pair is in F_Mac-IGM but not in F_To-Mac-IGM, contradicting Eq. (19). Because Theorem 3 is the only justification offered for the claimed generalization, and Theorem 5 explicitly builds on it, the theoretical core of the paper is unsupported.
  2. [Section 3.3, Eq. (13)] The micro-TD loss uses the target rt + γ max_{m_{t+1}} Qbar_total(ĥ_{t+1}, m_{t+1}, tm_{t+1}) with the maximum taken over the full joint macro-action space. In the asynchronous setting formalized by Eq. (7), only agents with terminated macro-actions may choose new macro-actions; agents with ongoing macro-actions keep their current choices. The unconstrained joint max therefore overestimates the value of states in which some macro-actions are still executing, and it is inconsistent with the conditional argmax in the paper's own To-Mac-IGM definition. The same problem occurs in the macro-TD target in Eq. (14), where the max should be constrained by the set of ongoing macro-actions m− at the next decision point. Since these two losses are the training objectives of the algorithm, the implementation does not realize the consistency condition that the paper claims to enforce.
minor comments (5)
  1. [Section 3.2 / Appendix A.4] The main text presents the function-class inclusion as Theorem 2 (Eq. 8), while Appendix A.4 restates it as Theorem 3 and uses Theorem 2 for the definition of To-MacAdv-IGM; renumber the theorems so each statement has a unique identifier.
  2. [Section 3.3, first paragraph] The sentence 'our ToMacVF framework incorporates Mac-JERTs' appears to be a typo for Mac-SJERT, since the paper's contribution and the rest of the text state that ToMacVF uses Mac-SJERT; this contradiction should be corrected.
  3. [Algorithm 1, lines 30–32] The pseudocode attributes the micro-TD loss to Eq. (12) and the macro-TD loss to Eq. (13), but in Section 3.3 the micro-TD loss is Eq. (13) and the macro-TD loss is Eq. (14); the references should be swapped.
  4. [Section 4 / Figures 4–8] The text reports averages over 20 independent runs and cites statistical comparison guidance [5], but no confidence intervals or error bars are visible in the figures; adding shaded intervals would support the stated robustness claims.
  5. [Reference [12]] Reference [12] is listed as 'Journal Placeholder'; providing the actual venue and year, or marking it as a preprint, is necessary for verification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical pipeline is standard RL with no fitted prediction, and the Theorem 3 proof gap is a correctness issue, not a circular reduction.

full rationale

The paper's derivation chain contains no step in which a target quantity is fitted and then reported as a prediction, and no load-bearing self-citation. To-Mac-IGM (Eq. 7) is introduced as a new definition; the observation that IGM is its single-timestep special case is a legitimate definitional specialization rather than a result used to verify itself. Experimental claims are benchmark comparisons (Figures 4-6) against external baselines, with no parameter fitted to those outcomes and then presented as a prediction. Theorem 3's inclusion claim is justified in Appendix A.4 by a cardinality argument (Eqs. 20-23); that argument is logically insufficient and, as the counterexample in the skeptic note shows, the second inclusion is false. However, an invalid proof is a correctness/rigor defect, not circularity: the paper does not assume the inclusion in order to prove it, nor does it rely on a self-citation chain (citations [1], [12], and [21] are external prior work, not the present authors' own unverified claims). Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the MacDec-POMDP model from prior work, a dueling decomposition invoked with an unresolved citation, and the monotonicity assumption for non-negative mixing parameters. The only explicit hand-tuned scientific parameter introduced ad hoc is the attention threshold delta. Training hyperparameters in Tables 2 to 4 are standard and are not counted as scientific free parameters.

free parameters (1)
  • delta (attention interdependence threshold)
    Introduced in Eq. 10 as a hand-set threshold below which inter-agent attention weights are zeroed; no value is reported and no sensitivity analysis is given.
assumptions (4)
  • domain assumption MacDec-POMDP macro-action sets include primitive actions (U^i subset M^i)
    Invoked in Appendix A.4 to show |M| > |U|; sourced to [1] and used to infer F IGM subset F Mac-IGM.
  • standard math Dueling decomposition Q = V + A holds for macro-action value functions
    Used in Appendix A.4 Theorem 4 to show To-MacAdv-IGM is equivalent to To-Mac-IGM; depends on an unresolved citation '[ ? ]'.
  • domain assumption IGM and Adv-IGM equivalence from [21], and Mac-IGM and MacAdv-IGM equivalence from [12]
    Theorem 5 combines these background results to extend the inclusion chain to advantage-based classes; [12] has a placeholder venue.
  • ad hoc to paper Monotonicity of the mixing network is sufficient for To-Mac-IGM
    Stated in Section 3.3 but not proven for the conditional argmax over terminated agents; relies on non-negative generated weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/YMSR7WKI

@misc{pith2026250710251,
  author       = {Pith},
  title        = {Pith review of: ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMSR7WKI}},
  note         = {Machine review of arXiv:2507.10251}
}
read the original abstract

Existing asynchronous MARL methods based on MacDec-POMDP typically construct training trajectory buffers by simply sampling limited and biased data at the endpoints of macro-actions, and directly apply conventional MARL methods on the buffers. As a result, these methods lead to an incomplete and inaccurate representation of the macro-action execution process, along with unsuitable credit assignments. To solve these problems, the Temporal Macro-action Value Factorization (ToMacVF) is proposed to achieve fine-grained temporal credit assignment for macro-action contributions. A centralized training buffer, called Macro-action Segmented Joint Experience Replay Trajectory (Mac-SJERT), is designed to incorporate with ToMacVF to collect accurate and complete macro-action execution information, supporting a more comprehensive and precise representation of the macro-action process. To ensure principled and fine-grained asynchronous value factorization, the consistency requirement between joint and individual macro-action selection called Temporal Macro-action based IGM (To-Mac-IGM) is formalized, proving that it generalizes the synchronous cases. Based on To-Mac-IGM, a modularized ToMacVF architecture, which satisfies CTDE principle, is designed to conveniently integrate previous value factorization methods. Next, the ToMacVF algorithm is devised as an implementation of the ToMacVF architecture. Experimental results demonstrate that, compared to asynchronous baselines, our ToMacVF algorithm not only achieves optimal performance but also exhibits strong adaptability and robustness across various asynchronous multi-agent experimental scenarios.

Figures

Figures reproduced from arXiv: 2507.10251 by the authors.

Figure 1
Figure 1. Example of two training buffers in MacDec-POMDP. To solve the issue of⃝1 duplicated macro-actions, we utilize ⃝1 sinusoidal positional encoding to incorporate the temporal information into the original macro-actions. To address ⃝2 the inconsistency between macro-observations in Mac-JERTs and the real ones, we ⃝2 sample real macro-observations at each timesteps. To eliminate ⃝3 the incorrect cumulative rewards(where … view at source ↗
Figure 2
Figure 2. Macro-action representation with temporal information. The horizontal axis represents time, while the vertical axis represents different agents. The cells are colored to distinguish between different macro-actions, and another feature introduced by temporal positional encoding is the shading intensity of each cell, which reflects the execution progress of the corresponding macro-action. Micro-transitions and Macro-t… view at source ↗
Figure 3
Figure 3. The network architecture of the ToMacVF. result, To-Mac-IGM represents a more comprehensive class of func￾tions than the original IGM and Mac-IGM: Theorem 2. Denoting with F IGM ={(Q IGM :H×U→R |U |,<QIGM i :Hi×U i→R |U i |>i∈I )} F Mac-IGM = {(Q Mac-IGM : Hˆ × M → R |M| , < QMac-IGM i : Hˆ i × Mi → R |Mi | >i∈I )} F To-Mac-IGM = {(Q To-Mac-IGM : Hˆ × M × T n → R |M|×|T n| , < QTo-Mac-IGM i : Hˆ i × Mi × T → R |Mi |… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The comparison of performance between ToMacVF and other baselines in BoxPushing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The comparison of performance between ToMacVF and other baselines in Warehouse. attention weight α ij t , which captures the correlation between agents i and j at timestep t, is computed as: α ij t = softmax( qi,t · kj,t T √ d ) (10) where α ii t = 0. In addition, if α…
Figure 6
Figure 6. Figure 6: The comparison of performance between ToMacVF and other baselines in Overcooked. ToMacVF algorithm performance The experimental results are presented in Figures 4-6. Although Q-learning-based methods such as Mac-Dec-MADDRQN and Mac-Cen-DDRQN achieve strong [PITH_FULL_…
Figure 7
Figure 7. Figure 7: Ablations on Mac-SJERT and ATPG modules on two challenging maps. Ablation studies on Mac-SJERT and ATPG To further validate the contribution of each core component, we conduct ablation stud￾ies on the Mac-SJERT buffer and the ATPG module in two of the most challenging …
Figure 8
Figure 8. Figure 8: Ablations on integrating ToMacVF with various value factorization methods. Ablation studies on integrating ToMacVF with various value fac￾torization methods We also investigate the impact of integrating ToMacVF with different value factorization methods in the Over￾coo…
Figure 9
Figure 9. Figure 9: The collection of Overcooked Environments. The Overcooked environment, adapted from GymCooking, involves three agents collaboratively preparing and delivering salads as quickly as possible. To complete the task, agents must chop vegetables (such as tomatoes and onions)…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [12]

    Marchesini, Y

    E. Marchesini, Y . Xiao, and C. Amato. Value factorization for asyn- chronous multi-agent reinforcement learning. Journal Placeholder , 2024

    work page 2024

  2. [1]

    Amato, G

    C. Amato, G. Konidaris, L. P. Kaelbling, and J. P. How. Modeling and planning with macro-actions in decentralized pomdps. Journal of Arti- ficial Intelligence Research, 64:817–859, 2019

    work page 2019

  3. [2]

    O. S. A.-M. A. Amato and C. L. S. H. J. Vian. J decentralized control of multi-robot partially observable markov decision processes using belief space macro-actions. The International Journal of Robotics Research , 36(2):231, 2017

    work page 2017

  4. [3]

    Christopher, D

    A. Christopher, D. K. George, and P. K. Leslie. Planning with macro- actions in decentralized pomdps. In International Conference on Au- tonomous Agents and Multiagent Systems, 2014

    work page 2014

  5. [4]

    T. Chu, J. Wang, L. Codecà, and Z. Li. Multi-agent deep reinforce- ment learning for large-scale traffic signal control. IEEE transactions on intelligent transportation systems, 21(3):1086–1095, 2019

    work page 2019

  6. [5]

    Colas, O

    C. Colas, O. Sigaud, and P.-Y . Oudeyer. A hitchhiker’s guide to statis- tical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.06979, 2019

    arXiv 1904

  7. [6]

    Y . Gao, B. Shi, X. Du, L. Wang, G. Chen, Z. Lian, F. Qiu, G. Han, W. Wang, D. Ye, et al. Learning diverse policies in moba games via macro-goals. Advances in Neural Information Processing Systems, 34: 16171–16182, 2021

    work page 2021

  8. [7]

    S. Gu, J. G. Kuba, Y . Chen, Y . Du, L. Yang, A. Knoll, and Y . Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:103905, 2023

    work page 2023

Show all 31 references
  1. [8]

    Gupta, A

    T. Gupta, A. Mahajan, B. Peng, W. Böhmer, and S. Whiteson. Un- even: Universal value exploration for multi-agent reinforcement learn- ing. In International Conference on Machine Learning , pages 3930–

  2. [9]

    R. A. Horn. The hadamard product. In Proc. Symp. Appl. Math , vol- ume 40, pages 87–169, 1990

    1990

  3. [10]

    Liu and J

    Y . Liu and J. Li. Runtime verification-based safe marl for optimized safety policy generation for multi-robot systems. Big Data and Cogni- tive Computing, 8(5):49, 2024

    2024

  4. [11]

    Mahajan, T

    A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi- agent variational exploration. Advances in neural information process- ing systems, 32, 2019

    2019

  5. [13]

    F. A. Oliehoek, C. Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016

    2016

  6. [14]

    Omidshafiei, A.-A

    S. Omidshafiei, A.-A. Agha-Mohammadi, C. Amato, S.-Y . Liu, J. P. How, and J. Vian. Decentralized control of multi-robot partially ob- servable markov decision processes using belief space macro-actions. The International Journal of Robotics Research, 36(2):231–258, 2017

    2017

  7. [15]

    Rashid, G

    T. Rashid, G. Farquhar, B. Peng, and S. Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing sys- tems, 33:10199–10210, 2020

    2020

  8. [16]

    Rashid, M

    T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Monotonic value function factorisation for deep multi- agent reinforcement learning. Journal of Machine Learning Research, 21(178):1–51, 2020

    2020

  9. [17]

    Seuken and S

    S. Seuken and S. Zilberstein. Improved memory-bounded dynamic pro- gramming for decentralized pomdps. arXiv preprint arXiv:1206.5295, 2012

    2012 arXiv

  10. [18]

    K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y . Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi-agent rein- forcement learning. In International conference on machine learning , pages 5887–5896. PMLR, 2019

    2019

  11. [19]

    Tuyls and G

    K. Tuyls and G. Weiss. Multiagent learning: Basics, challenges, and prospects. Ai Magazine, 33(3):41–41, 2012

    2012

  12. [20]

    L. Wan, Z. Liu, X. Chen, H. Wang, and X. Lan. Greedy-based value rep- resentation for optimal coordination in multi-agent reinforcement learn- ing. arXiv preprint arXiv:2112.04454, 2021

    2021

  13. [21]

    J. Wang, Z. Ren, T. Liu, Y . Yu, and C. Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020

    2008 arXiv

  14. [22]

    X. Wang, L. Ke, Z. Qiao, and X. Chai. Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE transactions on cybernetics, 51(1):174–187, 2020

    2020

  15. [23]

    S. A. Wu, R. E. Wang, J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman-Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. In Proceedings of the annual meeting of the cognitive science society, volume 42, 2020

    2020

  16. [24]

    Y . Xiao. Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability . PhD thesis, Northeastern Uni- versity, 2022

    2022

  17. [25]

    Y . Xiao, J. Hoffman, and C. Amato. Macro-action-based deep multi- agent reinforcement learning. In Conference on Robot Learning, pages 1146–1161. PMLR, 2020

    2020

  18. [26]

    Y . Xiao, J. Hoffman, T. Xia, and C. Amato. Learning multi-robot de- centralized macro-action-based policies via a centralized q-net. In 2020 IEEE International conference on robotics and automation (ICRA) , pages 10695–10701. IEEE, 2020

    2020

  19. [27]

    Y . Xiao, W. Tan, and C. Amato. Asynchronous actor-critic for multi- agent reinforcement learning. Advances in Neural Information Process- ing Systems, 35:4385–4400, 2022

    2022

  20. [28]

    Y . Yang, J. Hao, B. Liao, K. Shao, G. Chen, W. Liu, and H. Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020

    2002 arXiv

  21. [29]

    X. Yao, C. Wen, Y . Wang, and X. Tan. Smix ( λ): Enhancing central- ized value functions for cooperative multiagent reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 34(1): 52–63, 2021

    2021

  22. [30]

    Zhang, H

    Z. Zhang, H. Li, L. Zhang, T. Zheng, T. Zhang, X. Hao, X. Chen, M. Chen, F. Xiao, and W. Zhou. Hierarchical reinforcement learning for multi-agent moba game. arXiv preprint arXiv:1901.08004, 2019

    1901 arXiv

  23. [31]

    go to tomato

    M. Zhou, Z. Liu, P. Sui, Y . Li, and Y . Y . Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems , 33:11853–11864, 2020. A Appendix A.1 Pseudocode Algorithm 1 Temporal Macro-action Val...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.