Pith. sign in

REVIEW 5 major objections 4 minor 56 references

Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Novelty-guided replay lifts cooperative MARL win rates.

desk verdict A modest but genuine MARL trick: RND-guided per-agent extra updates with a decomposed critic, backed by a useful ablation, but the paper has presentation sloppiness and an unverified proxy assumption. read the letter →

arxiv 2412.15517 v1 pith:4YCTXJTA submitted 2024-12-20 cs.LG

Yangkun Chen , Kai Yang , Jian Tao , Jiafei Lyu This is my paper · ORCID
classification cs.LG
keywords multi-agentreinforcementlearningsampleefficiencyrandomnetworkdistillationnovelty-guideddatareuseagentdiversityQMIXStarCraftChallengeGoogleResearchFootball
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a multi-agent reinforcement learning system can be made both more sample-efficient and more behaviorally diverse by giving each agent extra Q-learning updates in proportion to the novelty of its observations, with novelty measured by random network distillation (RND). The proposed method, MANGER, builds on QMIX and reuses samples in two ways: every sample gets a base update, and agents whose observations are rare get additional updates. The paper presents results on StarCraft II (SMAC and SMAC-V2) and Google Research Football in which MANGER reaches higher win rates than QMIX, QPLEX, and Qatten within the same number of environment steps, and it shows through Q-value similarity that agents start to specialize. If correct, this would mean that targeted sample reuse based on a cheap curiosity signal is a practical way to improve cooperative MARL without adding new rewards or architectural complexity. The load-bearing empirical assertion is the win-rate superiority shown in Figures 3, 4, and 7.

What carries the argument

The central mechanism is an RND novelty scorer: a fixed random target network plus a learned predictor whose mean-squared error on an observation is that observation's novelty. This novelty score gates per-agent extra Q-learning updates, so the same transition can be replayed different numbers of times for different agents. A second piece is the split critic: each agent's Q-value is $Q_{\text{com}}(o_i, a_i) + \lambda Q^i_{\text{sep}}(o_i, a_i)$, and additional updates touch only $Q^i_{\text{sep}}$, keeping the added reuse from interfering across agents. The formula $T_i = \text{int}(\alpha (N_i - \bar{N}_{\text{total}}) / \text{Var}(N_{\text{total}}))$ converts normalized batch-level novelty into an integer number of extra updates per agent, with $\alpha = 2$ and a mean extra update count of about 0.5 that the paper argues adds little training time.

What would settle it

Measure, on a fixed SMAC task, the within-batch correlation between an agent's RND novelty $N(o_i)$ and its per-state TD-error magnitude $|Q_{\text{tot}} - y|$; if the correlation is not positive, or if shuffling the novelty scores among agents in a batch (keeping the same number of extra updates) does not decrease win rate, then MANGER's gains are not caused by targeting inaccurate Q-value estimates.

Watch

Extended reading notes

Core claim

The paper's central claim is that the frequency with which an agent has seen an observation is a usable signal for how much that agent's Q-network still needs to learn from the data. MANGER computes a novelty score $N(o_i) = \|f_{\text{target}}(o_i) - f_{\text{predictor}}(o_i)\|_2$ from a fixed random target network and a trained predictor, normalizes these scores within a batch, and assigns each agent $i$ an extra update count $T_i = \text{int}(\alpha (N_i - \bar{N}_{\text{total}}) / \text{Var}(N_{\text{total}}))$. Agents with $T_i > 0$ perform additional Q-learning steps on the same transition, updating only their separate critic layer so that extra updates do not bleed into other agents. The paper reports that this yields faster and higher final win rates than QMIX, QPLEX, and Qatten on super-hard SMAC maps, three GRF scenarios, and SMAC-V2 tasks, and it shows lower cosine similarity among agents' Q-values under the same observation, which it reads as evidence of role division and specialization.

Load-bearing premise

The method assumes that an observation visited rarely, as measured by RND prediction error, is one where the agent's Q-value estimate is inaccurate, so that spending extra updates there improves learning more than spending them elsewhere.

Editorial extensions

If this is right

  • If MANGER's win-rate gains replicate, cooperative MARL systems can improve sample efficiency by selectively reusing existing transitions rather than collecting more data or adding intrinsic rewards.
  • The diversity results imply that update-frequency asymmetry alone can induce role specialization, so division of labor can arise without explicit role assignment or mutual-information objectives.
  • The method is modular on top of value-decomposition baselines: it applies to QMIX-style critics with shared and separate layers, and the SMAC-V2 results suggest it transfers to newer benchmarks.
  • The extra update cost is small, so the reported performance gains do not come at large compute overhead.
  • If the correlation between RND novelty and Q-value inaccuracy is weak, the targeted-update mechanism could be replaced by a cheaper fixed-rate replay strategy, so the paper's ablation against Qmix-update2 and Qmix-update4 is the key comparison that separates targeted reuse from mere extra reuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • MANGER is effectively a form of prioritized experience replay in which the priority signal is observation novelty rather than TD error; directly comparing against TD-error prioritization on the same benchmarks would clarify which signal drives the gains.
  • The RND novelty signal is shared across agents, which implicitly discourages an agent from treating states mostly visited by other agents as novel and therefore discourages imitation; this diversity-preserving property could be studied formally in terms of state-visitation coverage.
  • The assumption that visit frequency proxies estimation error is untested directly; a probe of the within-batch correlation between RND MSE and per-agent TD error would tell whether the mechanism genuinely targets inaccurate estimates or simply amplifies exploration.
  • The discrete-action QMIX backbone would need replacement for continuous-control MARL, but the same novelty-guided update-count scheme could apply to actor-critic methods if the per-agent critic decomposition is retained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes MANGER, a value-based cooperative MARL method built on QMIX. It uses RND prediction error (Eq. 1) to score each agent's observation novelty, then converts normalized novelty into per-agent extra Q-learning updates via Eq. (2). The agent critic is split into a shared layer and per-agent separate layers to localize the extra updates. Experiments on SMAC, GRF, and SMAC-V2 claim higher win rates than QMIX, QPLEX, and Qatten with similar or lower training time, and Q-value cosine similarity and screenshots are used to claim that agents specialize into roles. The paper is primarily an empirical heuristic with no theoretical analysis.

Significance. If the reported gains are robust, MANGER would be an inexpensive drop-in addition to QMIX-style algorithms, with broad applicability to cooperative MARL benchmarks. The code link, the use of three benchmark suites, and the inclusion of an ablation study are strengths. However, the central mechanism rests on an unverified proxy assumption, and the diversity analysis lacks the disambiguating control, so the significance as demonstrated is moderate rather than high.

major comments (5)
  1. [Using Data Efficiently, Eq. (2), Tables 1-2] The text states alpha=2, while both hyperparameter tables list alpha=1. The denominator of Eq. (2) is labeled Var(N_total), but the prose calls it the standard deviation; these differ by a factor that changes the value of T_i. Tables 1 and 2 also list beta ("max number additional updates"), but beta never appears in Eq. (2), Algorithm 1, or the text. As written, the update rule is not uniquely specified, and the reported training-time and win-rate results cannot be exactly reproduced.
  2. [Using Data Efficiently] The method assumes that high RND novelty implies high Q-value estimation error, stated as "For data that has already appeared repeatedly, the network's estimates are highly accurate, and thus further updating is unnecessary." This correlation is never measured. In cooperative MARL, target Q-values depend on teammate policies, so frequently visited observations can carry stale values and rare observations can be accurate through bootstrapping. The ablation in Fig. 6 compares MANGER to uniform extra updates and to Qmix_sep, but it does not compare against random selection of the same number of extra updates, so it cannot confirm that the RND-selected subset is responsible for the gains.
  3. [Analysis, Fig. 5] The Q-value cosine-similarity plots compare MANGER only with QMIX. Since MANGER differs from QMIX in two respects (the separated critic and the novelty-guided update schedule), this comparison cannot isolate the diversity contribution of the latter. The Qmix_sep control in Fig. 6 should be included in Fig. 5 to show that the observed reduction in Q-value similarity is not simply an artifact of per-agent Q_sep layers.
  4. [Appendix A, Algorithm 1] The pseudocode updates target networks in line 31 at every training step, contradicting the stated M_target=200 in Tables 1 and 2. The pseudocode also performs extra updates on both theta_sep and phi in lines 27-30, whereas the text says additional training only updates the independent layer. In addition, Eq. (1) writes the norm as ||...||_2 while line 23 of the algorithm writes the same expression as a squared error; please align the notation.
  5. [Appendix D, Figure 7] The SMAC-V2 results are claimed in Appendix D and the caption for Figure 7 is present, but the figure itself is not included in the text provided, so these results cannot be checked. The central empirical claim extends to SMAC-V2, so the figure must be present and legible in the final version.
minor comments (4)
  1. [Result (GRF)] The sentence "the experimental results are shown in Figure 2" in the GRF paragraph should refer to Figure 4, which is where the GRF curves appear.
  2. [Update of the MANGER Agents, Eq. (6)] Equation (6) uses alpha as the learning rate, while Eq. (2) uses alpha as the additional-update coefficient; these are unrelated constants sharing a symbol and should be renamed.
  3. [Using Data Efficiently / Ablation Study] The claim that "the mean number of extra updates is less than 0.5" is not derived from any distribution described in the text; since T_i is an integer, a mean below 0.5 imposes a specific shape on the novelty distribution that should be stated or demonstrated empirically.
  4. [Experimental Settings, Tables 1-2] The rows labeled "T D lambda ratio to control Q-value update" and "TDlambda" are unclear; please define the quantity unambiguously and state its role in the QMIX update.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: MANGER's gains are evaluated against external benchmarks, and the RND novelty signal is trained independently of the reported win-rate targets.

full rationale

The derivation chain is self-contained. MANGER computes each agent's extra update count Ti from RND prediction error N(oi) via Eq. (2), and gates updates through h(Ti) in Eq. (9). The RND predictor is trained by its own MSE loss (Algorithm 1, lines 22-25) on the agents' observations, with no supervision from win rates or task rewards. The paper's central empirical claims are win rates on SMAC, GRF, and SMAC-V2, which are external benchmarks rather than quantities fitted by the method. The assumption that visit frequency (RND novelty) correlates with Q-value inaccuracy is an unvalidated heuristic, stated in 'Using Data Efficiently', but it is not derived from the equations and any weakness there is a correctness/validation concern rather than circularity. The separation of the critic into shared and independent layers (Eq. 3) makes per-agent Q-values differ by construction, but the paper does not use that architectural fact alone to claim the headline performance gains; the ablations compare against QMIX-with-separate-critic and fixed-multiple-update variants. Self-citations (DRND, Duan et al. 2024) appear only as related work and are not load-bearing: the novelty measure itself cites the external RND paper (Burda et al. 2018). No load-bearing step reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on two heuristic assumptions about novelty and update allocation, plus three hand-tuned hyperparameters. No new entities are introduced.

free parameters (3)
  • alpha = 1 (tables) or 2 (text)
    Controls the rate of extra updates in Eq. (2); the paper contradicts itself on its value.
  • beta = 3 (tables, unused)
    Listed as max number of additional updates, but never appears in equations, pseudocode, or analysis.
  • lambda = 0.5
    Scales the independent critic component in Eq. (3); hand-set, informed by CDS.
assumptions (3)
  • domain assumption RND prediction error measures observation novelty
    Imported from exploration literature; used as a metric rather than an intrinsic reward.
  • domain assumption Frequently visited states have accurate Q-value estimates
    Stated in 'Using Data Efficiently'; if false, the update allocation is misdirected.
  • domain assumption Updating only the independent critic layer during extra updates avoids interfering with other agents
    Assumed in 'Diversify the Agents by Separating the Critic Network', based on CDS without fresh validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/4YCTXJTA

@misc{pith2026241215517,
  author       = {Pith},
  title        = {Pith review of: Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YCTXJTA}},
  note         = {Machine review of arXiv:2412.15517}
}
read the original abstract

Recently, deep Multi-Agent Reinforcement Learning (MARL) has demonstrated its potential to tackle complex cooperative tasks, pushing the boundaries of AI in collaborative environments. However, the efficiency of these systems is often compromised by inadequate sample utilization and a lack of diversity in learning strategies. To enhance MARL performance, we introduce a novel sample reuse approach that dynamically adjusts policy updates based on observation novelty. Specifically, we employ a Random Network Distillation (RND) network to gauge the novelty of each agent's current state, assigning additional sample update opportunities based on the uniqueness of the data. We name our method Multi-Agent Novelty-GuidEd sample Reuse (MANGER). This method increases sample efficiency and promotes exploration and diverse agent behaviors. Our evaluations confirm substantial improvements in MARL effectiveness in complex cooperative scenarios such as Google Research Football and super-hard StarCraft II micromanagement tasks.

Figures

Figures reproduced from arXiv: 2412.15517 by the authors.

Figure 1
Figure 1. Visualization of the environment. In the SMAC image, the green box represents the tank role that actively absorbs damage and sacrifices itself to create an output environment for teammates. The red box represents the damage dealer role that activates attacks against enemies. The blue box represents the roaming role, similar to a guerrilla fighter, that can attract enemy aggro based on its own movement and lead some … view at source ↗
Figure 2
Figure 2. Overview of the MANGER framework. We employ the RND network to assess the novelty of each agent’s ob￾servations, thereby enabling differentiated updates among agents. Furthermore, we ensure that each additional update does not interfere with the agents by decomposing the network. regularized approach with lower TD error to address overfit￾ting problems associated with sample reuse. However, the Sample Multiple Reuse… view at source ↗
Figure 3
Figure 3. Experimental results on SMAC. All curves are averaged over 5 independent runs. 0 1 2 3 4 T(mil) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 winrate academy_counterattack_hard MANGER qmix qatten qplex 0 1 2 3 4 T(mil) 0.0 0.2 0.4 0.6 academy_3_vs_1_with_keeper 0 1 2 3 4 T(mil) 0.00 0.05 0.10 0.15 0.20 0.25 academy_single_goal_versus_lazy [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Experiments on GRF environments. All curves are averaged over 5 independent runs. Here, λ controls the scale of separate Q values. The shared layer facilitates parameter sharing among all agents, aiding them in extracting overall features from the environment, which is…
Figure 5
Figure 5. Figure 5: Graphical illustration of agent diversity. (a) shows how the agent within the red box should operate, with Q-values for actions. We calculated the Q-values of different agents and displayed three representative ones. For the same observation, Agent 1 moves southward to…
Figure 6
Figure 6. Figure 6: Left: Training time comparison between the MANGER and Qmix methods in the 6h vs 8z environment. Right: Ablation study of MANGER in the 6h vs 8z environment. It can be observed that our method does not significantly increase training time while achieving performance imp…
Figure 7
Figure 7. Figure 7: Experiments on SMAC-V2 environments. All curves are averaged over 5 independent runs. 1 https://github.com/tjuHaoXiaotian/pymarl3 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 33 canonical work pages

  1. [1]

    Baker, B.; Kanitscheider, I.; Markov, T.; Wu, Y.; Powell, G.; McGrew, B.; and Mordatch, I. 2019. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528

    arXiv 2019

  2. [2]

    Bettini, M.; Kortvelesy, R.; and Prorok, A. 2024. Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning. ArXiv, abs/2405.15054

    arXiv 2024

  3. [3]

    Burda, Y.; Edwards, H.; Storkey, A.; and Klimov, O. 2018. Exploration by random network distillation. arXiv preprint arXiv:1810.12894

    arXiv 2018

  4. [4]

    C.; Nunzio, L

    Canese, L.; Cardarilli, G. C.; Nunzio, L. D.; Fazzolari, R.; Giardino, D.; Re, M.; and Span \`o , S. 2021. Multi-Agent Reinforcement Learning: A Review of Challenges and Applications. Applied Sciences, 11: 4948

    work page 2021

  5. [5]

    Chen, M.; Hessel, M.; and van Hasselt, H. 2021. Randomized Ensemble Double Q-learning: Learning Fast Without a Model. Advances in Neural Information Processing Systems, 34: 14527--14539

    work page 2021

  6. [6]

    L.; Hernandez-Leal, P.; Kartal, B.; and Taylor, M

    Da Silva, F. L.; Hernandez-Leal, P.; Kartal, B.; and Taylor, M. E. 2020. Uncertainty-aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 5792--5799

    work page 2020

  7. [7]

    Duan, K.; Yang, K.; Liu, H.; and Wang, X. 2024. Novelty-based Sample Reuse for Continuous Robotics Control. arXiv preprint arXiv:2410.13490

    work page Pith review arXiv 2024

  8. [8]

    Ellis, B.; Cook, J.; Moalla, S.; Samvelyan, M.; Sun, M.; Mahajan, A.; Foerster, J.; and Whiteson, S. 2024. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36

    work page 2024

Show all 56 references
  1. [9]

    Eysenbach, B.; Gupta, A.; Ibarz, J.; and Levine, S. 2018. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070

    2018 arXiv

  2. [10]

    Gogineni, K.; Mei, Y.; Lan, T.; Wei, P.; and Venkataramani, G. 2023. Accmer: Accelerating multi-agent experience replay with cache locality-aware prioritization. In 2023 IEEE 34th International Conference on Application-specific Systems, Architectures and Processors (ASAP), 20...

    2023

  3. [11]

    Guestrin, C.; Koller, D.; and Parr, R. 2001. Multiagent planning with factored MDPs. Advances in neural information processing systems, 14

    2001

  4. [12]

    S.; Campbell, J.; Stepputtis, S.; Li, R.; Hughes, D.; Fang, F.; and Sycara, K

    Guo, Y. S.; Campbell, J.; Stepputtis, S.; Li, R.; Hughes, D.; Fang, F.; and Sycara, K. P. 2022. Explainable Action Advising for Multi-Agent Reinforcement Learning. 2023 IEEE International Conference on Robotics and Automation (ICRA), 5515--5521

    2022

  5. [13]

    F.; and Yamins, D

    Haber, N.; Mrowca, D.; Wang, S.; Fei-Fei, L. F.; and Yamins, D. L. 2018. Learning to play with intrinsically-motivated, self-aware agents. Advances in neural information processing systems, 31

    2018

  6. [14]

    Han, S.; Wang, H.; Su, S.; Shi, Y.; and Miao, F. 2022. Stable and Efficient Shapley Value-Based Reward Reallocation for Multi-Agent Reinforcement Learning of Autonomous Vehicles. 2022 International Conference on Robotics and Automation (ICRA), 8765--8771

    2022

  7. [15]

    A.; Wu, H.; and wei Liao, S

    Hu, J.; Jiang, S.; Harding, S. A.; Wu, H.; and wei Liao, S. 2021. Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

    2021

  8. [16]

    Hu, S.; Xie, C.; Liang, X.; and Chang, X. 2022. Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL. ArXiv, abs/2207.05683

    2022 arXiv

  9. [17]

    Jia, H.; Hu, Y.; Chen, Y.; Ren, C.; Lv, T.; Fan, C.; and Zhang, C. 2020. Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning. ArXiv, abs/2012.03204

    2020 arXiv

  10. [18]

    Kim, W.; and Sung, Y. 2023. An adaptive entropy-regularization framework for multi-agent reinforcement learning. In International Conference on Machine Learning, 16829--16852. PMLR

    2023

  11. [19]

    Kurach, K.; Raichuk, A.; Stańczyk, P.; Zajac, M.; Bachem, O.; Espeholt, L.; Riquelme, C.; Vincent, D.; Michalski, M.; Bousquet, O.; et al. 2020. Google Research Football: A Novel Reinforcement Learning Environment. In Proceedings of the AAAI Conference on Artificial Intelligen...

    2020

  12. [20]

    Lazaridou, A.; and Baroni, M. 2020. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419

    2020 arXiv

  13. [21]

    Li, C.; Wang, T.; Wu, C.; Zhao, Q.; Yang, J.; and Zhang, C. 2021. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 3991--4002

    2021

  14. [22]

    Li, X.; Qiao, Z.; Gong, A.; Lyu, J.; Yu, C.; Yan, J.; and Li, X. 2022. Prag: Periodic regularized action gradient for efficient continuous control. In Pacific Rim International Conference on Artificial Intelligence, 106--119. Springer

    2022

  15. [23]

    Liu, Y.-C.; Tian, J.; Glaser, N.; and Kira, Z. 2020. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 4106--4115

    2020

  16. [24]

    Lobel, S.; Bagaria, A.; and Konidaris, G. 2023. Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning. arXiv preprint arXiv:2306.03186

    2023 arXiv

  17. [25]

    I.; Tamar, A.; Harb, J.; Pieter Abbeel, O.; and Mordatch, I

    Lowe, R.; Wu, Y. I.; Tamar, A.; Harb, J.; Pieter Abbeel, O.; and Mordatch, I. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30

    2017

  18. [26]

    Lyu, J.; Bai, C.; Yang, J.; Lu, Z.; and Li, X. 2024 a . Cross-domain policy adaptation by capturing representation mismatch. arXiv preprint arXiv:2405.15369

    2024 arXiv

  19. [27]

    Lyu, J.; Ma, X.; Yan, J.; and Li, X. 2022. Efficient continuous control with double actors and regularized critics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 7655--7663

    2022

  20. [28]

    Lyu, J.; Wan, L.; Li, X.; and Lu, Z. 2024 b . Off-policy RL algorithms can be sample-efficient for continuous control via sample multiple reuse. Information Sciences, 666: 120371

    2024

  21. [29]

    Lyu, J.; Yang, Y.; Yan, J.; and Li, X. 2023. Value activation for bias alleviation: Generalized-activated deep double deterministic policy gradients. Neurocomputing, 518: 70--81

    2023

  22. [30]

    Mahajan, A.; Rashid, T.; Samvelyan, M.; and Whiteson, S. 2019. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32

    2019

  23. [31]

    Oh, J.; Hessel, M.; and van Hasselt, H. 2021. Adaptive Value-Targeted Learning. Proceedings of the 38th International Conference on Machine Learning (ICML)

    2021

  24. [32]

    A.; Amato, C.; et al

    Oliehoek, F. A.; Amato, C.; et al. 2016. A concise introduction to decentralized POMDPs, volume 1. Springer

    2016

  25. [33]

    A.; and Darrell, T

    Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, 2778--2787. PMLR

    2017

  26. [34]

    M.; Liu, C.; and Zhou, B

    Peng, Z.; Li, Q.; Hui, K. M.; Liu, C.; and Zhou, B. 2021. Learning to simulate self-driven particles system with coordinated policy optimization. Advances in Neural Information Processing Systems, 34: 10784--10797

    2021

  27. [35]

    S.; Farquhar, G.; Foerster, J.; and Whiteson, S

    Rashid, T.; Samvelyan, M.; De Witt, C. S.; Farquhar, G.; Foerster, J.; and Whiteson, S. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1): 7234--7284

    2020

  28. [36]

    S.; Farquhar, G.; Nardelli, N.; Rudner, T

    Samvelyan, M.; Rashid, T.; De Witt, C. S.; Farquhar, G.; Nardelli, N.; Rudner, T. G.; Hung, C.-M.; Torr, P. H.; Foerster, J.; and Whiteson, S. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043

    2019 arXiv

  29. [37]

    J.; Hostallero, D

    Son, K.; Kim, D.; Kang, W. J.; Hostallero, D. E.; and Yi, Y. 2019. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, 5887--5896. PMLR

    2019

  30. [38]

    M.; Zambaldi, V.; Jaderberg, M.; Lanctot, M.; Sonnerat, N.; Leibo, J

    Sunehag, P.; Lever, G.; Gruslys, A.; Czarnecki, W. M.; Zambaldi, V.; Jaderberg, M.; Lanctot, M.; Sonnerat, N.; Leibo, J. Z.; Tuyls, K.; et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296

    2017 arXiv

  31. [39]

    S.; Barto, A

    Sutton, R. S.; Barto, A. G.; et al. 1998. Introduction to reinforcement learning

    1998

  32. [40]

    Tao, J.; Chen, Y.; Zhang, Y.; Yang, K.; and Li, X. 2024. Multi-agent Exploration with Sub-state Entropy Estimation. In 2024 International Joint Conference on Neural Networks (IJCNN), 1--9. IEEE

    2024

  33. [41]

    M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D.; Powell, R.; Ewalds, T.; Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.; Huang, A.; Sifre, L.; Cai, T.; Agapiou, J

    Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D.; Powell, R.; Ewalds, T.; Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.; Huang, A.; Sifre, L.; Cai, T.; Agapiou, J. P.; Jaderberg, M.; Vezhnevets, A. S.; Leblond, R.; Pohle...

    2019

  34. [42]

    Wang, J.; Ren, Z.; Liu, T.; Yu, Y.; and Zhang, C. 2020 a . Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062

    2020 arXiv

  35. [43]

    Wang, T.; Dong, H.; Lesser, V.; and Zhang, C. 2020 b . Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039

    2020 arXiv

  36. [44]

    Wang, T.; Gupta, T.; Mahajan, A.; Peng, B.; Whiteson, S.; and Zhang, C. 2020 c . Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523

    2020 arXiv

  37. [45]

    Wang, W.; Mao, L.; Wang, R.; and Min, B.-C. 2023. Multi-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning. 2024 IEEE International Conference on Robotics and Automation (ICRA), 12353--12360

    2023

  38. [46]

    Wang, W.; Yang, T.; Liu, Y.; Hao, J.; Hao, X.; Hu, Y.; Chen, Y.; Fan, C.; and Gao, Y. 2019. Action semantics network: Considering the effects of actions in multiagent systems. arXiv preprint arXiv:1907.11461

    2019 arXiv

  39. [47]

    Yan, M.; Lyu, J.; and Li, X. 2024. Enhancing visual reinforcement learning with State--Action Representation. Knowledge-Based Systems, 304: 112487

    2024

  40. [48]

    Yang, K.; Fang, Z.; Li, X.; and Tao, J. 2024 a . Cmbe: Curiosity-driven model-based exploration for multi-agent reinforcement learning in sparse reward settings. In 2024 International Joint Conference on Neural Networks (IJCNN), 1--8. IEEE

    2024

  41. [49]

    Yang, K.; Gong, A.; Tao, J.; Zhang, Y.; and Li, X. 2023. GTLMA: Generalizable Hierarchical Learning for Tasks with Variable Entities. In 2023 International Conference on Frontiers of Robotics and Software Engineering (FRSE), 97--103. IEEE

    2023

  42. [50]

    Yang, K.; Tao, J.; Lyu, J.; and Li, X. 2024 b . Exploration and Anti-Exploration with Distributional Random Network Distillation. arXiv preprint arXiv:2401.09750

    2024 arXiv

  43. [51]

    Yang, Y.; Hao, J.; Liao, B.; Shao, K.; Chen, G.; Liu, W.; and Tang, H. 2020. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939

    2020 arXiv

  44. [52]

    Yu, C.; Velu, A.; Vinitsky, E.; Gao, J.; Wang, Y.; Bayen, A.; and Wu, Y. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 24611--24624

    2022

  45. [53]

    Zhang, Z.; Han, S.; Wang, J.; and Miao, F. 2022. Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios. 2023 IEEE International Conference on Robotics and Automation (ICRA), 5574--5580

    2022

  46. [54]

    Zhang, Z.; Li, H.; Zhang, L.; Zheng, T.; Zhang, T.; Hao, X.; Chen, X.; Chen, M.; Xiao, F.; and Zhou, W. 2019. Hierarchical Reinforcement Learning for Multi-agent MOBA Game. ArXiv, abs/1901.08004

    2019 arXiv

  47. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.