REVIEW 3 major objections 4 minor 33 references
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Training agents on small battles first, then scaling up, beats from-scratch multiagent RL
desk verdict The DyAN architecture is the real contribution; the curriculum idea is plausible, but the SOTA claim and MAgent numbers need a tighter budget and baseline story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is Dynamic Agent-number Network (DyAN), a network that splits each agent's observation into a fixed-size part for its own features and environment information and a variable-size set of observations of other agents, embeds each other-agent observation, then aggregates the embeddings with a permutation-invariant operator (sum, max, or mean). This aggregation makes the network indifferent to the number of agents in view, which is what allows curriculum stages of different battle sizes to share one architecture and enables Model Reload between stages. The paper treats this GNN aggregation as an approximate implementation of the Semantic Mapping Function from Definition 1, and reports that sum aggregation produces the clearest semantic clusters in t-SNE embeddings and the best final performance.
What would settle it
A decisive test is to extract observations from 5-vs-5, 10-vs-10, and 15-vs-15 battles that share identical local semantics, such as exactly one teammate in view, and check whether DyAN's sum-aggregated embeddings form one cluster while different-semantics observations form another; if same-semantics points do not cluster, the semantic-mapping property that Model Reload depends on fails. A controlled ablation that removes only Model Reload while keeping the same curriculum would then reveal whether the reported gains come from cross-size transfer or from some other curriculum effect.
Extended reading notes
Core claim
The central claim is that a dynamic multiagent curriculum, in which agents learn on a small scenario and progressively face more agents, greatly improves final performance on large-scale multiagent tasks, and that the improvement depends on a network whose input size can vary. The paper formalizes a semantic-mapping assumption: observations from environments with different agent counts can be projected into a common latent space in which semantically similar observations sit close together, and it builds DyAN to realize that mapping with a graph-neural-network aggregation over per-agent observations. With Model Reload, the trained network from one curriculum stage initializes the next stage directly. Empirically, on a 15-immortals-versus-15-immortals StarCraft II battle and on 50-versus-50 battles in MAgent, curriculum-trained agents reach nonzero win rates and higher survival and kill counts, whereas from-scratch baselines lag substantially, in some cases never leaving zero win rate.
Load-bearing premise
The approach rests on the idea that what an agent sees in a small battle is semantically the same as what it sees in a large battle, so a network trained on the small setting can be reused in the larger one.
Editorial extensions
If this is right
- Curriculum-trained IQL and VDN reach nonzero win rates on the 15-vs-15 StarCraft II task in the reported training budget, while from-scratch IQL stays at zero win rate.
- Model Reload beats both Buffer Reuse and Curriculum Distillation across the tested tasks, so reusing the trained network rather than reusing data or distilling behavior carries the largest part of the curriculum benefit.
- Sum aggregation in DyAN outperforms max and mean on both benchmarks, which the paper links to sum producing better semantic separation of same-meaning states across different-sized scenarios.
- The curriculum benefit is not specific to one algorithm family: IQL, VDN, PPO, A2C, and ACER all improve on MAgent when trained with DyMA-CL rather than from scratch.
- Both StarCraft II and MAgent results support the paper's claim that increasing agent count during training is an effective route into large-scale multiagent learning problems.
Reading between the lines
- If the semantic-mapping assumption holds generally, DyAN could serve as a backbone for any multiagent task where roster size changes mid-episode, such as allies dying or reinforcements arriving, not only for curricula with fixed growing stages.
- The reported superiority of sum aggregation suggests that for these combat tasks, counting and total-mass features over nearby units matter more than picking out specific individuals; a direct test would compare sum against attention-based aggregation on tasks where a particular enemy's identity is crucial.
- Because the paper fixes total training steps rather than reporting steps-to-threshold, whether curriculum training also reduces wall-clock compute for reaching a given win rate is an open question that a time-to-threshold comparison would settle.
- The curriculum schedule is manually designed; if the benefit scales with schedule quality, automatic curriculum generation over agent counts could yield further gains on even larger arenas.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DyMA-CL, a curriculum learning framework for large-scale multiagent reinforcement learning that starts from small agent-count scenarios and progressively increases the number of agents. Three transfer mechanisms are introduced: Buffer Reuse, Curriculum Distillation, and Model Reload. To support Model Reload across changing input dimensions, the authors design DyAN, a GNN-based network that aggregates per-agent observations. Experiments on StarCraft II (IQL, VDN) and MAgent (IQL, PPO, A2C, ACER) report that curriculum training with DyAN improves final performance over learning-from-scratch baselines, with Model Reload performing best. The paper also analyzes aggregation operators (SUM/MEAN/MAX) via t-SNE visualizations.
Significance. If the empirical claims hold, the paper offers a practical recipe for scaling multiagent RL via agent-count curricula and a network design that makes parameter transfer feasible across tasks with varying input sizes. The consistent improvement across five algorithms and two environments is a useful existence proof, and the comparison of transfer mechanisms gives actionable insight. The manuscript is clearly presented and the experimental scope is substantial. The central limitation is that the headline comparison may conflate curriculum benefit with extra environment interactions in MAgent (and possibly VDN in StarCraft II); the paper also overstates 'state-of-the-art' comparisons. These are addressable by additional experimental controls.
major comments (3)
- [MAgent simulation and supplementary materials] The total environment steps for DyMA-CL on MAgent are 7,500 + 4,500 + 1,500 + 750 + 10,000 = 24,250 steps, whereas the paper only reports that the target task is trained 'for 1e+4 steps' and never states the budget of the 'w/o CL' baselines. If the baselines receive only the 10,000 target-task steps, the Table 1 gains (e.g., IQL Max survivors 20.35±4.87 vs 0.54±2.65) could be due to the 2.4× larger environment budget rather than curriculum transfer. The authors should either match total steps for the baselines or explicitly report the baseline budget and re-run at matched budget.
- [Abstract and Introduction] The claim that DyMA-CL 'greatly improves ... compared with state-of-the-art deep reinforcement learning approaches' is not supported by the baselines, which are IQL, VDN, PPO, A2C, and ACER trained from scratch. These are standard baselines, but not state-of-the-art for large-scale multiagent learning; no comparison is made with QMIX, mean-field MARL, or other curriculum/transfer methods. The claim should be qualified to 'from-scratch deep RL baselines' or the comparison set should be expanded.
- [Dynamic Number Agent Network, Definition 1] The semantic mapping function Φ(·) is formalized by an inequality, but the paper does not provide a quantitative test of this property on held-out states. The only evidence is the qualitative t-SNE visualization in Figure 7. Since Model Reload transfers parameters through the same GNN embeddings, the authors should validate the semantic mapping with a numerical measure (e.g., nearest-neighbor distances between semantic classes across scenarios) or explicitly state that the mapping is an assumption that is not independently verified.
minor comments (4)
- [Abstract] The phrase 'state dimension varies across curricula,, and existing' contains a double comma and a grammatical break; please fix the typo.
- [Figures 5 and 6] The StarCraft II results are reported without error bars or confidence intervals; please state the number of random seeds and include variance information in the plots.
- [Equation (2)] The displayed formula for the distillation loss is ambiguous in its typesetting (the division between the softmax terms is not clearly expressed); please rewrite the KL expression in a clearer form.
- [References] The reference list contains duplicate entries for Narvekar et al. 2017a and 2017b, which appear to be the same paper; please check and merge.
Circularity Check
No significant circularity: DyMA-CL is an empirical curriculum-training method whose reported gains are measured against external baselines, and no central claim reduces to a fitted input or a self-citation chain.
full rationale
The paper's central claim is an empirical one: curriculum training with DyMA-CL and the DyAN architecture improves large-scale multiagent learning relative to learning from scratch. The curriculum schedule, transfer mechanisms, and aggregation choices are design decisions validated by experiments, not quantities fitted to the target result. No equation in the paper defines a prediction in terms of the outcome it is said to predict, and no fitted parameter is renamed as a prediction. The formal Definition 1 (Semantic Mapping Function) is an assumption about the existence of a useful latent mapping; it is used to motivate the DyAN architecture, but the performance of that architecture is tested against external baselines rather than derived from the definition. The t-SNE analysis in Figure 7 is a post-hoc visualization of learned embeddings, not a circular justification. There are no load-bearing self-citations: the references are to standard external works on curriculum learning, multiagent RL, GNNs, and the benchmark environments, and no uniqueness theorem or architecture constraint is imported from the authors' own prior work. The paper explicitly acknowledges that the curriculum is manually designed and that automatic curriculum generation is future work; this is a limitation of autonomy, not circularity. The MAgent comparison may raise an experimental-budget question, because the text does not state whether the from-scratch baselines received the same total number of environment steps as the curriculum condition, but that is a potential confound in the empirical evaluation, not a circularity in the derivation. Overall, the derivation chain is self-contained against external empirical benchmarks, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- curriculum schedule (training steps per task) =
1.5e6 steps per task (SC2); 7500/4500/1500/750/1e4 steps (MAgent)
- distillation temperature omega =
not specified
- GNN aggregation operator =
SUM selected as best
assumptions (3)
- domain assumption Property 3: State Semanticity
- domain assumption Property 1: Partial Observability
- standard math Standard RL/POSG formulation
Cite this review
Pith. "Pith review of From Few to More: Large-scale Dynamic Multiagent Curriculum Learning." pith.science (2026). https://pith.science/paper/QNLPPW2F
@misc{pith2026190902790,
author = {Pith},
title = {Pith review of: From Few to More: Large-scale Dynamic Multiagent Curriculum Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/QNLPPW2F}},
note = {Machine review of arXiv:1909.02790}
}
read the original abstract
A lot of efforts have been devoted to investigating how agents can learn effectively and achieve coordination in multiagent systems. However, it is still challenging in large-scale multiagent settings due to the complex dynamics between the environment and agents and the explosion of state-action space. In this paper, we design a novel Dynamic Multiagent Curriculum Learning (DyMA-CL) to solve large-scale problems by starting from learning on a multiagent scenario with a small size and progressively increasing the number of agents. We propose three transfer mechanisms across curricula to accelerate the learning process. Moreover, due to the fact that the state dimension varies across curricula,, and existing network structures cannot be applied in such a transfer setting since their network input sizes are fixed. Therefore, we design a novel network structure called Dynamic Agent-number Network (DyAN) to handle the dynamic size of the network input. Experimental results show that DyMA-CL using DyAN greatly improves the performance of large-scale multiagent learning compared with state-of-the-art deep reinforcement learning approaches. We also investigate the influence of three transfer mechanisms across curricula through extensive simulations.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Agarwal, A.; Kumar, S.; and Sycara, K. P. 2019. Learning transferable cooperative behavior in multi-agent teams. CoRR abs/1906.01202
arXiv 2019
-
[2]
Andreas, J.; Klein, D.; and Levine, S. 2017. Modular multitask reinforcement learning with policy sketches. In Proceedings of ICML , 166--175
work page 2017
-
[3]
Arnekvist, I.; Kragic, D.; and Stork, J. A. 2019. VPE: variational policy embedding for transfer reinforcement learning. In Proceedings of ICRA , 36--42
work page 2019
-
[4]
Bengio, Y.; Louradour, J.; Collobert, R.; and Weston, J. 2009. Curriculum learning. In Proceedings of ICML , 41--48
work page 2009
-
[5]
Busoniu, L.; Babuska, R.; and Schutter, B. D. 2008. A comprehensive survey of multiagent reinforcement learning. IEEE T SYST MAN CY C 38(2):156--172
work page 2008
-
[6]
Chen, Y.; Zhou, M.; Wen, Y.; Yang, Y.; Su, Y.; Zhang, W.; Zhang, D.; Wang, J.; and Liu, H. 2018. Factorized q-learning for large-scale multi-agent systems. arXiv preprint arXiv:1809.03738
work page Pith review arXiv 2018
-
[7]
Claus, C., and Boutilier, C. 1998. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI 1998:746--752
work page 1998
-
[8]
Hansen, E. A.; Bernstein, D. S.; and Zilberstein, S. 2004. Dynamic programming for partially observable stochastic games. In Proceedings of AAAI , 709--715
work page 2004
Show all 33 references
-
[9]
Z.; and Gruslys, A
Hester, T.; Vecer \' k, M.; Pietquin, O.; Lanctot, M.; Schaul, T.; Piot, B.; Horgan, D.; Quan, J.; Sendonaris, A.; Osband, I.; Dulac - Arnold, G.; Agapiou, J.; Leibo, J. Z.; and Gruslys, A. 2018. Deep q-learning from demonstrations. In Proceedings of AAAI , 3223--3230
2018
-
[10]
A.; Matthey, L.; Burgess, C.; Pritzel, A.; Botvinick, M.; Blundell, C.; and Lerchner, A
Higgins, I.; Pal, A.; Rusu, A. A.; Matthey, L.; Burgess, C.; Pritzel, A.; Botvinick, M.; Blundell, C.; and Lerchner, A. 2017. DARLA: improving zero-shot transfer in reinforcement learning. In Proceedings of ICML , 1480--1490
2017
-
[11]
Jiang, J., and Lu, Z. 2018. Learning attentional communication for multi-agent cooperation. In Proceedings of NeurIPS , 7254--7264
2018
-
[12]
Littman, M. L. 1994. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of ICML , 157--163
1994
-
[13]
A.; Veness, J.; Bellemare, M
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529
2015
-
[14]
P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K
Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of ICML , 1928--1937
2016
-
[15]
Narvekar, S., and Stone, P. 2019. Learning curriculum policies for reinforcement learning. In Proceedings of AAMAS , 25--33
2019
-
[16]
Narvekar, S.; Sinapov, J.; Leonetti, M.; and Stone, P. 2016. Source task creation for curriculum learning. In Proceedings of AAMAS , 566--574
2016
-
[17]
Narvekar, S.; Sinapov, J.; and Stone, P. 2017a. Autonomous task sequencing for customized curriculum design in reinforcement learning. In Proceedings of IJCAI , 2536--2542
-
[18]
Narvekar, S.; Sinapov, J.; and Stone, P. 2017b. Autonomous task sequencing for customized curriculum design in reinforcement learning. In Proceedings of IJCAI , 2536--2542
-
[19]
S.; Farquhar, G.; Foerster, J.; and Whiteson, S
Rashid, T.; Samvelyan, M.; Witt, C. S.; Farquhar, G.; Foerster, J.; and Whiteson, S. 2018. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of ICML , 4292--4301
2018
-
[20]
A.; Colmenarejo, S
Rusu, A. A.; Colmenarejo, S. G.; G \" u l c ehre, C .; Desjardins, G.; Kirkpatrick, J.; Pascanu, R.; Mnih, V.; Kavukcuoglu, K.; and Hadsell, R. 2016. Policy distillation. In Proceedings of ICLR
2016
-
[21]
S.; Farquhar, G.; Nardelli, N.; Rudner, T
Samvelyan, M.; Rashid, T.; de Witt, C. S.; Farquhar, G.; Nardelli, N.; Rudner, T. G.; Hung, C.-M.; Torr, P. H.; Foerster, J.; and Whiteson, S. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043
2019 arXiv
-
[22]
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
2017 arXiv
-
[23]
Singh, A.; Jain, T.; and Sukhbaatar, S. 2019. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. In Proceedings of ICLR
2019
-
[24]
M.; Zambaldi, V
Sunehag, P.; Lever, G.; Gruslys, A.; Czarnecki, W. M.; Zambaldi, V. F.; Jaderberg, M.; Lanctot, M.; Sonnerat, N.; Leibo, J. Z.; Tuyls, K.; and Graepel, T. 2018. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of AAMAS , 2085--2087
2018
-
[25]
S., and Barto, A
Sutton, R. S., and Barto, A. G. 2018. Reinforcement learning: An introduction . MIT press
2018
-
[26]
Tampuu, A.; Matiisen, T.; Kodelja, D.; Kuzovkin, I.; Korjus, K.; Aru, J.; Aru, J.; and Vicente, R. 2017. Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE 12:1--15
2017
-
[27]
Wang, Z.; Bapst, V.; Heess, N.; Mnih, V.; Munos, R.; Kavukcuoglu, K.; and de Freitas, N. 2017. Sample efficient actor-critic with experience replay. In Proceedings of ICLR
2017
-
[28]
Wattenberg, M.; Vi \'e gas, F.; and Johnson, I. 2016. How to use t-sne effectively. Distill 1(10):e2
2016
-
[29]
Wu, Y., and Tian, Y. 2017. Training agent for first-person shooter game with actor-critic curriculum learning. In Proceedings of ICLR
2017
-
[30]
Xu, K.; Hu, W.; Leskovec, J.; and Jegelka, S. 2019. How powerful are graph neural networks? In Proceedings of ICLR
2019
-
[31]
Yang, Y.; Luo, R.; Li, M.; Zhou, M.; Zhang, W.; and Wang, J. 2018a. Mean field multi-agent reinforcement learning. In Proceedings of ICML , volume 80, 5571--5580
-
[32]
Yang, Y.; Yu, L.; Bai, Y.; Wen, Y.; Zhang, W.; and Wang, J. 2018b. A study of AI population dynamics with million-agent reinforcement learning. In Proceedings of AAMAS , 2133--2135
-
[33]
Zheng, L.; Yang, J.; Cai, H.; Zhou, M.; Zhang, W.; Wang, J.; and Yu, Y. 2018. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Proceedings of AAAI
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.