REVIEW 4 major objections 6 minor 28 references
Towards Heterogeneous Multi-Agent Reinforcement Learning with Graph Neural Networks
T0 review · 4 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash
Pith's one-line read This paper argues that giving each pair of agent and entity classes its own communication channel improves policies in heterogeneous multi-agent reinforcement learning, and reports higher rewards on a StarCraft battle map than…
desk verdict A modest, honest architecture paper for heterogeneous MARL; the class-pair relational convolution idea is genuinely new, but the experiments don't isolate it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is HMAGQ-Net, a three-module network: class-specific encoders $\varphi_c$ that map heterogeneous state vectors to a common size, a stack of relational graph convolution layers with relation set $\mathcal{R} = \mathcal{Z} \times \mathcal{C}$ (every agent class paired with every node class), and class-specific Q-functions $Q_c$ for action selection. The RGCN update gives each relation its own transformation matrix $W_r$, with basis matrices shared across relations to control parameter growth; this is the mechanism that creates specialized communication channels between entity classes.
What would settle it
Run a controlled experiment on the same 2s3z map: keep the HMAGQ-Net backbone and all hyperparameters fixed, but replace the per-class-pair relation set with a single shared relation for all communication, across at least five random seeds. If the single-channel variant matches or exceeds the class-specialized variant's mean reward, the paper's central claim is not supported.
Extended reading notes
Core claim
The paper's central claim is that in a multi-agent setting with heterogeneous entities, specializing the communication channels between entity classes yields better policies than using a single shared communication mechanism. Concretely, HMAGQ-Net encodes each node class with its own MLP, passes messages through relational graph convolution layers with one relation per directed agent-class/entity-class pair, and outputs action values through a distinct Q-network for each agent class. On the StarCraft II 2s3z map, the best RGCN configuration earned about 4.69 mean episode reward, the best GAT configuration about 3.98, and random play about 2.22; the paper reads this as evidence that class-specialized relational communication is a promising direction.
Load-bearing premise
The load-bearing premise is that the higher reward of the RGCN models on the 2s3z map is caused by the class-specialized communication channels, rather than by any other difference between RGCN and GAT message passing or by run-to-run noise.
Editorial extensions
If this is right
- On SMAC's 2s3z map, relational graph convolution configurations achieved higher mean episode reward than graph attention configurations, with the best RGCN model at roughly 4.69 versus 3.98 for the best GAT model.
- Adding full agent communication arcs improved reward in both communication families, while full receptive field and temporal relation regularization did not consistently help on this heterogeneous map.
- The architecture supports entity classes with different state-vector lengths and action spaces end-to-end, sharing parameters through basis decomposition so the model size grows with the number of classes, not the number of agents.
- The same training protocol (target network, prioritized replay, per-class DQN loss) works for this graph-based heterogeneous architecture, making it a straightforward structure for DQN-style multi-agent training.
Reading between the lines
- A direct ablation the paper does not run is to keep the RGCN backbone but collapse all relations into one shared channel; the size of that reward drop would quantify how much the class specialization itself contributes.
- Because no repeated-seed statistics are reported, the observed roughly 0.7-point reward gap could be within run-to-run variance; a multi-seed replication with confidence intervals would make the effect size interpretable.
- If the mechanism transfers, a natural design rule for heterogeneous robot teams is to let the communication graph's edge types mirror the roles and sensing capabilities of each agent class, rather than forcing a single peer-to-peer channel.
- Combining class-specialized relational communication with policy-gradient action modules, which the paper lists as future work, could test whether the same advantage appears in continuous-action and larger-scale heterogeneous settings.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HMAGQ-Net, a graph-neural-network architecture for heterogeneous multi-agent reinforcement learning. States are represented as directed labeled graphs whose nodes are agents or environment entities; class-specific encoders normalize features of different sizes, relational graph convolution (RGCN) layers implement communication with one relation per agent-class/entity-class pair, and class-specific Q-networks output action values. The model is trained with DQN-style updates, a target network, and prioritized experience replay. Experiments on the SMAC 2s3z map compare RGCN communication with graph attention (GAT) communication under ablations of full receptive field (FRF), full agent communication (FAC), and temporal relation regularization (TRR). The best RGCN variant achieves a mean reward of 4.69, versus 3.98 for the best GAT variant and 2.22 for a random policy, and the paper concludes that specializing communication channels between entity classes is a promising step.
Significance. The problem is well motivated and the architecture is a reasonable first step toward heterogeneous MARL. The paper deserves credit for explicitly handling heterogeneous feature dimensions and action spaces, for testing on the standard SMAC benchmark, and for being honest in framing the result as 'promising' rather than conclusive. However, the empirical basis is too thin to establish the central claim: the comparison rests on a single map, a single run per configuration, no ablation that isolates class-specialized relations, and no standard MARL baselines. With additional controlled experiments, this could become a useful contribution to the MARL+graph literature.
major comments (4)
- [§5, Table 2] Each configuration is reported as a single training run with no random-seed repetition, error bars, or significance test. The headline comparison (RGCN FRF FAC mean reward 4.69 versus GAT best mean reward 3.98) is a difference of about 0.7 on one map, and Figure 3 shows visibly noisy training curves. Without repeated-seed statistics, the reported gap could plausibly be run-to-run variance, so the claim that specialized channels 'achieve higher performance' is not statistically supported by the current experiments.
- [§4.1.2, Table 2] The RGCN-versus-GAT comparison is not a controlled test of class-specialized communication. The RGCN models use relational graph convolutions with |R| = |Z|·|C| relation-specific parameter matrices, while the GAT models use attention-weighted averaging; these differ in aggregation mechanism, parameter count, and inductive bias. There is no RGCN variant with a single shared relation channel, and no GAT variant with relation-specific channels, so Table 2 cannot isolate whether class-pair-specific relational convolutions are what drive the reward improvement. An RGCN ablation with one relation (or no relation labels) is needed to support the abstract's central claim.
- [§5, §6] The experimental evaluation covers only the 2s3z map of SMAC, and the only comparators are a random policy and the authors' own GAT variants. No established MARL algorithms (for example QMIX, COMA, or IQL) are compared, and no second map or domain is used to test whether the observed improvement generalizes. Consequently the statement that 'results have shown' higher performance for specialized channels rests on a single map against weak baselines, which is disproportionate to the generality of the conclusion.
- [§7] The conclusion lists as future work the isolation of the contribution of per-class policies, but the contribution of class-specialized communication channels is likewise never isolated in Sections 4–6. As argued above, the present architecture and baselines conflate relation specialization with the choice of message-passing mechanism. The manuscript therefore does not yet provide the experiment that would justify its main claim, even though the claim is appropriately hedged as 'promising'.
minor comments (6)
- [§2.1] The generic message-passing equation indexes the edge feature as e_{(j,i)} while the surrounding text defines the edge as e(i,j); the notation should be made consistent.
- [Table 2] The two GAT rows that each show two checkmarks are visually ambiguous because the column positions are not identified by row labels; the exact configuration of FRF, FAC, and TRR for each row should be made explicit.
- [§5, Table 2] TRR (temporal relation regularization) is included as a configuration option but is never defined in the text or given an equation or a self-contained explanation; readers cannot tell what regularization is being applied.
- [References] The citation for prioritized experience replay is listed as [Schaul et al. 2015] with the title 'Universal Value Function Approximators'; the correct title is 'Prioritized Experience Replay'.
- [§3, References] The text attributes NerveNet to [Wang et al. 2018a], but the bibliography lists NerveNet as [Wang et al. 2018b], while [Wang et al. 2018a] appears to be an unrelated paper; these citations should be corrected.
- [§4.2] The loss function uses r_i without defining it; in SMAC the reward is a team scalar, so the paper should state that r_i is the shared team reward for all agents or clarify the per-agent reward setup.
Circularity Check
No circularity: the paper reports measured RL outcomes from an architecture comparison against an external benchmark, with no derivation that reduces to its own inputs.
full rationale
The paper's central claim is empirical: that class-specialized relational graph convolutions yield higher reward on the SMAC 2s3z map than attention-based alternatives. Nothing in the paper defines a result in terms of the quantity it purports to predict. The rewards in Table 2 are recorded from trained policies, not constructed from fitted parameters, and the random baseline provides an external point of comparison. HMAGQ-Net is specified openly through class-wise encoding functions, RGCN layers, and per-class Q-networks, with a standard DQN-style loss; no equation equates an output with an input by definition. The RGCN versus GAT comparison is not a clean ablation of relation specialization, and the paper itself defers isolating the contribution of per-class policies to future work, but a control-group weakness is an experimental validity concern, not circularity. The paper contains no self-citations used as load-bearing evidence, no imported uniqueness theorems, and no renamed known result presented as a derivation. Its building blocks come from external published work, and the reported numbers are measured outcomes on an external benchmark. Thus the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Shared training and architecture hyperparameters =
K=4 communication layers, encoding size=64, hidden size=128, learning rate=2.5e-4, discount=0.99, PER alpha=0.6…
assumptions (5)
- domain assumption SMAC 2s3z can be modeled as a stochastic game with discrete joint actions and a shared reward.
- domain assumption A directed labeled graph whose nodes are units/entities and whose arcs encode observability and communication fully represents the state needed for Q-learning.
- domain assumption Arcs are drawn according to agents' observation and communication capabilities, so graph convolution over in-neighbors implements true local observability in SMAC.
- ad hoc to paper The relation set R = Z x C, one channel per agent-class to entity-class pair, is the right granularity for specializing communication.
- standard math RGCN basis decomposition provides sufficient expressiveness for the communication task.
invented entities (1)
-
HMAGQ-Net components (per-class encoders, class-pair relational convolutions, per-class Q-networks)
Cite this review
Pith. "Pith review of Towards Heterogeneous Multi-Agent Reinforcement Learning with Graph Neural Networks." pith.science (2026). https://pith.science/paper/FHH5Q35P
@misc{pith2026200913161,
author = {Pith},
title = {Pith review of: Towards Heterogeneous Multi-Agent Reinforcement Learning with Graph Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/FHH5Q35P}},
note = {Machine review of arXiv:2009.13161}
}
read the original abstract
This work proposes a neural network architecture that learns policies for multiple agent classes in a heterogeneous multi-agent reinforcement setting. The proposed network uses directed labeled graph representations for states, encodes feature vectors of different sizes for different entity classes, uses relational graph convolution layers to model different communication channels between entity types and learns distinct policies for different agent classes, sharing parameters wherever possible. Results have shown that specializing the communication channels between entity classes is a promising step to achieve higher performance in environments composed of heterogeneous entities.
Figures
Reference graph
Works this paper leans on
-
[1]
Agarwal, A. (2019). Learning Transferable Cooperative Behavior in Multi-Agent Teams . Master's Thesis , Carnegie Mellon University, Pittsburg, USA
work page 2019
-
[2]
Agarwal, A., Kumar, S., and Sycara, K. (2019). Learning Transferable Cooperative Behavior in Multi - Agent Teams . In ICML 2019 Workshop on Learning and Reasoning with Graph - Structured Representations
work page 2019
-
[3]
Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez , A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer, J., Dahl, G., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D., Kohli, P., Botvinick, M., Vinyals, O., Li, Y., and Pascanu, R. ...
work page 2018
-
[4]
Bondy, J. A. and Murty, U. S. R. (2008). Graph Theory . Springer London
work page 2008
-
[5]
Bowling, M. and Veloso, M. (2000). An Analysis of Stochastic Game Theory for Multiagent Reinforcement Learning . Resreport, School of Computer Science, Carnegie Mellon University , Pittsburgh, PA
work page 2000
-
[6]
Busoniu, L., Babuska, R., and Schutter, B. D. (2008). A Comprehensive Survey of Multiagent Reinforcement Learning . IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 38(2):156--172
work page 2008
-
[7]
da Silva , F. L., Glatt, R., and Costa, A. H. R. (2019). MOO - MDP : An Object - Oriented Representation for Cooperative Multiagent Reinforcement Learning . IEEE Transactions on Cybernetics , 49
work page 2019
-
[8]
Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rabbat, M., and Pineau, J. (2019). TarMAC : Targeted Multi - Agent Communication . Proceedings of the 36th International Conference on Machine Learning , 97:1538--1546
work page 2019
Show all 28 references
-
[9]
Duvenaud, D., Maclaurin, D., Aguilera-Iparraguirre , J., G \'o mez-Bombarelli , R., Hirzel, T., Aspuru-Guzik , A., and Adams, R. P. (2015). Convolutional Networks on Graphs for Learning Molecular Fingerprints
2015
-
[10]
S., Riley, P
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural Message Passing for Quantum Chemistry . arXiv:1704.01212 [cs]
2017 arXiv
-
[11]
Gori, M., Monfardini, G., and Scarselli, F. (2005). A new model for learning in graph domains. In Proceedings of the International Joint Conference on Neural Networks , volume 2, pages 729--734. IEEE
2005
-
[12]
Guestrin, C., Koller, D., Gearhart, C., and Kanodia, N. (2003). Generalizing Plans to New Environments in Relational MDPs . In Proceedings of the 18th International Joint Conference on Artificial Intelligence , IJCAI '03, pages 1003--1010, San Francisco, CA, USA . Morgan Kaufm...
2003
-
[13]
Jiang, J., Dun, C., Huang, T., and Lu, Z. (2020). Graph Convolutional Reinforcement Learning . In International Conference on Learning Representations
2020
-
[14]
and Lu, Z
Jiang, J. and Lu, Z. (2018). Learning Attentional Communication for Multi - Agent Cooperation . In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi , N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31 , pages 7254--7264. Curran...
2018
-
[15]
Kipf, T. N. and Welling, M. (2017). Semi- Supervised Classification with Graph Convolutional Networks . In 5th International Conference on Learning Representations , ICLR 2017 - Conference Track Proceedings . International Conference on Learning Representations, ICLR
2017
-
[16]
Littman, M. L. (1994). Markov Games as a Framework for Multi - Agent Reinforcement Learning . In Proceedings of the Eleventh International Conference on Machine Learning , volume 157, pages 157--163
1994
-
[17]
Malysheva, A., Kudenko, D., and Shpilman, A. (2019). MAGNet : Multi -agent Graph Network for Deep Multi -agent Reinforcement Learning . In Adaptive and Learning Agents Workshop at AAMAS ( ALA 2019) , Montreal, Canada
2019
-
[18]
Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., and Wang, J. (2017). Multiagent Bidirectionally - Coordinated Nets : Emergence of Human - Level Coordination in Learning to Play StarCraft Combat Games
2017
-
[19]
S., Farquhar, G., Nardelli, N., Rudner, T
Samvelyan, M., Rashid, T., de Witt , C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., and Whiteson, S. (2019). The StarCraft Multi - Agent Challenge . arXiv:1902.04043 [cs, stat]
2019 arXiv
-
[20]
C., Hagenbuchner, M., and Monfardini, G
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009a). Computational capabilities of graph neural networks. IEEE Transactions on Neural Networks , 20(1):81--102
2009
-
[21]
C., Hagenbuchner, M., and Monfardini, G
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009b). The Graph Neural Network Model . IEEE Transactions on Neural Networks , 20(1):61--80
2009
-
[22]
Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015). Universal Value Function Approximators . In International Conference on Machine Learning , pages 1312--1320
2015
-
[23]
N., Bloem, P., van den Berg, R., Titov, I., and Welling, M
Schlichtkrull, M., Kipf, T. N., Bloem, P., van den Berg, R., Titov, I., and Welling, M. (2018). Modeling relational data with graph convolutional networks. In European Semantic Web Conference , pages 593--607. Springer
2018
-
[24]
Sukhbaatar, S., Szlam, A., and Fergus, R. (2016). Learning Multiagent Communication with Backpropagation . In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems 29 , pages 2244--2252. Curran Associates, Inc
2016
-
[25]
Veli c kovi \'c , P., Casanova, A., Li \`o , P., Cucurull, G., Romero, A., and Bengio, Y. (2018). Graph attention networks. In 6th International Conference on Learning Representations , ICLR 2018 - Conference Track Proceedings . International Conference on Learning Representat...
2018
-
[26]
Wang, D., Duan, Y., and Weng, J. (2018a). Motivated Optimal Developmental Learning for Sequential Tasks Without Using Rigid Time - Discounts . IEEE Transactions on Neural Networks and Learning Systems , 29
2018
-
[27]
Wang, T., Liao, R., Ba, J., and Fidler, S. (2018b). Nervenet: Learning structured policy with graph neural networks. In 6th International Conference on Learning Representations , ICLR 2018 - Conference Track Proceedings . International Conference on Learning Representations, ICLR
2018
-
[28]
Wasser, C. G. D., Cohen, A., and Littman, M. L. (2008). An Object - Oriented Representation for Efficient Reinforcement Learning . In Proceedings of the 25th International Conference on Machine Learning , pages 240--247. ACM
2008
Reviewed August 27, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.