{"id":"6868375c-059b-487f-a9a3-8067c43fa4c4","arxiv_id":"1909.02790","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A curriculum that grows the number of agents over training, combined with a graph neural network that accepts variable agent counts, improves large-scale multiagent RL performance in StarCraft II and MAgent.","lead":"This paper proposes a curriculum training method for multiagent reinforcement learning that starts with few agents and gradually adds more, together with a graph-based neural network that handles changing input sizes. The method improves win rates in StarCraft II and MAgent battles compared with training from scratch, though comparisons to strong baselines are missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MAgent from-scratch baselines may not be matched on total training steps, so the headline curriculum gains could reflect extra environment interactions.","rationale":"The reader identified Property 3 / State Semanticity as the weakest assumption. That is a real gap: Definition 1 postulates a semantic mapping function, and the only evidence is a t-SNE visualization, with no quantitative or formal verification. However, I do not think it is the most load-bearing issue for the central empirical claim. Model reload can provide a speedup through feature reuse even without a rigorously established semantic alignment; the permutation-invariant aggregation in DyAN already makes transfer possible at the architectural level. The more direct threat is that the MAgent comparisons may not be controlled for total environment interactions. The curriculum schedule gives 14,250 steps on easier tasks before the target, and the target itself is trained for 1e4 steps. If the from-scratch baseline is trained only for 1e4 steps on the target, then the w/ CL method receives more than twice as much environment interaction, so the large improvements in Table 1 could be an artifact of training budget rather than curriculum transfer. The authors explicitly state that the StarCraft II comparisons use the same total training time, but no such statement is made for MAgent, making the omission conspicuous. This is a concrete, testable concern: re-running the baselines with matched total steps would settle it. Since the issue is addressable by additional experiments, the existing CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":14162,"tokens_out":8715,"duration_ms":90404,"concrete_test":"Re-run each 'w/o CL' row in Table 1 on the 50v50 task with 24,250 total environment steps (or otherwise exactly match DyMA-CL's total step budget), preserving all other hyperparameters and random seeds. If survivors and kill counts remain at the reported low levels, the concern is resolved; if they rise to near the w/ CL values, the curriculum's reported benefit is not from knowledge transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is empirical, and the MAgent table is the most direct comparison of DyMA-CL with DyAN against from-scratch learning with the same network architecture. The reported curriculum uses 7500 + 4500 + 1500 + 750 = 14,250 environment steps on tasks 10v10 through 40v40 before the 50v50 target, plus 1e4 steps on the target (supplementary). The text states only that the target task runs 'for 1e+4 steps'; it never states that the 'w/o CL' baselines were trained for the same total number of steps (24,250) or the same wall-clock time. If the baselines were trained only on 50v50 for 1e4 steps, then w/ CL has roughly 2.4x more environment interactions, so the large gaps in Table 1 (e.g., IQL Max: 20.35±4.87 survivors vs 0.54±2.65) could be a training-budget effect, not a curriculum or transfer effect. For StarCraft II the authors explicitly match totals ('same amount of total training time (7.5e+6 steps)'), but no equivalent statement appears for MAgent. This is a load-bearing confound because the abstract's claim that DyMA-CL 'greatly improves' performance rests on these numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DyMA-CL, a curriculum learning framework for large-scale multiagent reinforcement learning that starts from small agent-count scenarios and progressively increases the number of agents. Three transfer mechanisms are introduced: Buffer Reuse, Curriculum Distillation, and Model Reload. To support Model Reload across changing input dimensions, the authors design DyAN, a GNN-based network that aggregates per-agent observations. Experiments on StarCraft II (IQL, VDN) and MAgent (IQL, PPO, A2C, ACER) report that curriculum training with DyAN improves final performance over learning-from-scratch baselines, with Model Reload performing best. The paper also analyzes aggregation operators (SUM/MEAN/MAX) via t-SNE visualizations.","tokens_in":14394,"tokens_out":4543,"duration_ms":43851,"significance":"If the empirical claims hold, the paper offers a practical recipe for scaling multiagent RL via agent-count curricula and a network design that makes parameter transfer feasible across tasks with varying input sizes. The consistent improvement across five algorithms and two environments is a useful existence proof, and the comparison of transfer mechanisms gives actionable insight. The manuscript is clearly presented and the experimental scope is substantial. The central limitation is that the headline comparison may conflate curriculum benefit with extra environment interactions in MAgent (and possibly VDN in StarCraft II); the paper also overstates 'state-of-the-art' comparisons. These are addressable by additional experimental controls.","major_comments":[{"comment":"The total environment steps for DyMA-CL on MAgent are 7,500 + 4,500 + 1,500 + 750 + 10,000 = 24,250 steps, whereas the paper only reports that the target task is trained 'for 1e+4 steps' and never states the budget of the 'w/o CL' baselines. If the baselines receive only the 10,000 target-task steps, the Table 1 gains (e.g., IQL Max survivors 20.35±4.87 vs 0.54±2.65) could be due to the 2.4× larger environment budget rather than curriculum transfer. The authors should either match total steps for the baselines or explicitly report the baseline budget and re-run at matched budget.","section":"MAgent simulation and supplementary materials"},{"comment":"The claim that DyMA-CL 'greatly improves ... compared with state-of-the-art deep reinforcement learning approaches' is not supported by the baselines, which are IQL, VDN, PPO, A2C, and ACER trained from scratch. These are standard baselines, but not state-of-the-art for large-scale multiagent learning; no comparison is made with QMIX, mean-field MARL, or other curriculum/transfer methods. The claim should be qualified to 'from-scratch deep RL baselines' or the comparison set should be expanded.","section":"Abstract and Introduction"},{"comment":"The semantic mapping function Φ(·) is formalized by an inequality, but the paper does not provide a quantitative test of this property on held-out states. The only evidence is the qualitative t-SNE visualization in Figure 7. Since Model Reload transfers parameters through the same GNN embeddings, the authors should validate the semantic mapping with a numerical measure (e.g., nearest-neighbor distances between semantic classes across scenarios) or explicitly state that the mapping is an assumption that is not independently verified.","section":"Dynamic Number Agent Network, Definition 1"}],"minor_comments":[{"comment":"The phrase 'state dimension varies across curricula,, and existing' contains a double comma and a grammatical break; please fix the typo.","section":"Abstract"},{"comment":"The StarCraft II results are reported without error bars or confidence intervals; please state the number of random seeds and include variance information in the plots.","section":"Figures 5 and 6"},{"comment":"The displayed formula for the distillation loss is ambiguous in its typesetting (the division between the softmax terms is not clearly expressed); please rewrite the KL expression in a clearer form.","section":"Equation (2)"},{"comment":"The reference list contains duplicate entries for Narvekar et al. 2017a and 2017b, which appear to be the same paper; please check and merge.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The step-budget confound in MAgent is the key technical issue; if the authors can show matched-budget results still favor DyMA-CL, the paper would be acceptable. I also note the 'state-of-the-art' phrasing is stronger than the experiments support, and the semantic-mapping evidence is thin. These are fixable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the DyAN architecture: a GNN aggregation layer that lets a network accept a variable number of agent observations, which makes parameter transfer across curricula with different agent counts feasible. That is a genuine, useful idea, and combining it with a curriculum that scales up the number of agents is a natural and sensible direction. The StarCraft II experiments are the cleanest part of the paper: they explicitly state that all methods get the same total training steps, and the win-rate curves show the curriculum with model reload beating learning from scratch. The t-SNE analysis of the GNN embeddings is a nice attempt to justify the semantic-mapping claim, though it is illustrative rather than conclusive.\n\nThe soft spots are real but manageable. First, the headline claim \"compared with state-of-the-art deep reinforcement learning approaches\" is not supported by the baselines. IQL, PPO, A2C, and ACER are standard single-agent or independent learners, not SOTA multiagent algorithms; the paper never compares against QMIX, COMA, or mean-field methods, which are the obvious references for large-scale MARL. Second, the MAgent comparison has a potential confound that the stress-test note flags correctly: the curriculum schedule totals 24,250 steps, while the text only says the target task runs for 1e4 steps. It never states that the from-scratch baselines were trained for the same total step budget. If they were trained only on the 50v50 task for 1e4 steps, then the large gaps in Table 1 partly reflect extra environment interactions, not the curriculum or transfer mechanism. That is a load-bearing ambiguity for the MAgent results, and it needs an explicit sentence or a matching-budget experiment. Third, Property 3 / Definition 1 is an assumption, not a result. The paper asserts that a learned Phi maps semantically similar states close together, but it never proves or independently validates that the single GNN aggregation layer actually realizes this in a way that generalizes beyond the tested scenarios.\n\nOn the citation pattern and reproducibility: no meaningful self-citation, references are standard, and the authors are honest in the discussion that the curriculum schedule is manually designed and critical. They do not ship code or data, which is a limitation but not disqualifying for a 2019 preprint.\n\nOverall: the central concept is sound and the StarCraft II evidence is decent. The MAgent numbers, as reported, are not airtight. This paper deserves a serious referee who can ask for tighter baselines and a budget-matched MAgent comparison; it is not a desk reject, and it is not a clear accept yet either. I would bring it to a reading group focused on MARL or transfer learning, and I would cite the DyAN architecture in my own work.","headline":"The DyAN architecture is the real contribution; the curriculum idea is plausible, but the SOTA claim and MAgent numbers need a tighter budget and baseline story.","tokens_in":14945,"tokens_out":1971,"would_cite":true,"duration_ms":23831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training agents on small battles first, then scaling up, beats from-scratch multiagent RL","keywords":["curriculum learning","multiagent reinforcement learning","large-scale multiagent systems","knowledge transfer","graph neural network","dynamic agent-number network","StarCraft II","MAgent"],"falsifier":"A decisive test is to extract observations from 5-vs-5, 10-vs-10, and 15-vs-15 battles that share identical local semantics, such as exactly one teammate in view, and check whether DyAN's sum-aggregated embeddings form one cluster while different-semantics observations form another; if same-semantics points do not cluster, the semantic-mapping property that Model Reload depends on fails. A controlled ablation that removes only Model Reload while keeping the same curriculum would then reveal whether the reported gains come from cross-size transfer or from some other curriculum effect.","tokens_in":13951,"feed_emoji":"🤖","tokens_out":10315,"duration_ms":97772,"temperature":0.7,"pith_summary":"This paper argues that large-scale multiagent reinforcement learning can be made tractable by curriculum training: agents first learn in a battle with few combatants, then the roster is grown until the full-scale task is reached. The authors propose three ways to carry knowledge across stages: reusing replay samples from earlier tasks, distilling earlier policies or Q-values into the current learner, and reloading the previous network as initialization for the next. Because the observation size changes when the number of agents changes, they design a Dynamic Agent-number Network (DyAN) that embeds each other-agent observation and aggregates them, so one network accepts any roster size. In StarCraft II and MAgent, curriculum-trained versions of IQL, VDN, PPO, A2C, and ACER outperform their from-scratch counterparts, with model reload and sum aggregation delivering the largest gains.","feed_headline":"Small-first curriculum beats from-scratch in large-scale agent games","feed_subtitle":"Growing a fighting force step by step, with a size-flexible network, beats training on the full battle directly.","key_machinery":"The load-bearing mechanism is Dynamic Agent-number Network (DyAN), a network that splits each agent's observation into a fixed-size part for its own features and environment information and a variable-size set of observations of other agents, embeds each other-agent observation, then aggregates the embeddings with a permutation-invariant operator (sum, max, or mean). This aggregation makes the network indifferent to the number of agents in view, which is what allows curriculum stages of different battle sizes to share one architecture and enables Model Reload between stages. The paper treats this GNN aggregation as an approximate implementation of the Semantic Mapping Function from Definition 1, and reports that sum aggregation produces the clearest semantic clusters in t-SNE embeddings and the best final performance.","core_discovery":"The central claim is that a dynamic multiagent curriculum, in which agents learn on a small scenario and progressively face more agents, greatly improves final performance on large-scale multiagent tasks, and that the improvement depends on a network whose input size can vary. The paper formalizes a semantic-mapping assumption: observations from environments with different agent counts can be projected into a common latent space in which semantically similar observations sit close together, and it builds DyAN to realize that mapping with a graph-neural-network aggregation over per-agent observations. With Model Reload, the trained network from one curriculum stage initializes the next stage directly. Empirically, on a 15-immortals-versus-15-immortals StarCraft II battle and on 50-versus-50 battles in MAgent, curriculum-trained agents reach nonzero win rates and higher survival and kill counts, whereas from-scratch baselines lag substantially, in some cases never leaving zero win rate.","pith_inferences":["If the semantic-mapping assumption holds generally, DyAN could serve as a backbone for any multiagent task where roster size changes mid-episode, such as allies dying or reinforcements arriving, not only for curricula with fixed growing stages.","The reported superiority of sum aggregation suggests that for these combat tasks, counting and total-mass features over nearby units matter more than picking out specific individuals; a direct test would compare sum against attention-based aggregation on tasks where a particular enemy's identity is crucial.","Because the paper fixes total training steps rather than reporting steps-to-threshold, whether curriculum training also reduces wall-clock compute for reaching a given win rate is an open question that a time-to-threshold comparison would settle.","The curriculum schedule is manually designed; if the benefit scales with schedule quality, automatic curriculum generation over agent counts could yield further gains on even larger arenas."],"forward_implications":["Curriculum-trained IQL and VDN reach nonzero win rates on the 15-vs-15 StarCraft II task in the reported training budget, while from-scratch IQL stays at zero win rate.","Model Reload beats both Buffer Reuse and Curriculum Distillation across the tested tasks, so reusing the trained network rather than reusing data or distilling behavior carries the largest part of the curriculum benefit.","Sum aggregation in DyAN outperforms max and mean on both benchmarks, which the paper links to sum producing better semantic separation of same-meaning states across different-sized scenarios.","The curriculum benefit is not specific to one algorithm family: IQL, VDN, PPO, A2C, and ACER all improve on MAgent when trained with DyMA-CL rather than from scratch.","Both StarCraft II and MAgent results support the paper's claim that increasing agent count during training is an effective route into large-scale multiagent learning problems."],"supporting_citations":[{"why":"Supplies the StarCraft II multiagent challenge that defines the combat benchmark, win-rate measurement, and the from-scratch difficulty that motivates curriculum learning.","marker":"Samvelyan et al. 2019"},{"why":"Supplies the MAgent platform, reward settings, and large-scale battle scenarios used in the second set of experiments.","marker":"Zheng et al. 2018"},{"why":"Provides DQN, the base off-policy learning algorithm whose replay buffer is extended by the Buffer Reuse transfer mechanism.","marker":"Mnih et al. 2015"},{"why":"Motivates Buffer Reuse by showing that deep Q-learning can incorporate expert demonstrations as extra supervision.","marker":"Hester et al. 2018"},{"why":"Provides the policy-distillation technique on which the Curriculum Distillation transfer mechanism via KL divergence is based.","marker":"Rusu et al. 2016"},{"why":"Supplies the graph-neural-network formulation and aggregation operators that DyAN adapts to handle a variable number of agents.","marker":"Xu et al. 2019"},{"why":"Provides value-decomposition multiagent RL context and the StarCraft II settings that VDN and the paper's combat benchmarks build on.","marker":"Rashid et al. 2018"},{"why":"Gives the t-SNE embedding method used to visualize whether the learned semantic mapping clusters same-meaning states across scenarios.","marker":"Wattenberg, Viégas, and Johnson 2016"}],"fun_headline_variants":["Dynamic curriculum plus flexible network wins large-scale agent battles","Growing agent count stepwise beats full-scale training","Size-adaptive network enables curriculum for many agents","Small-to-large agent curriculum outperforms from-scratch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach rests on the idea that what an agent sees in a small battle is semantically the same as what it sees in a large battle, so a network trained on the small setting can be reused in the larger one.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic curriculum plus flexible network wins large-scale agent battles","Growing agent count stepwise beats full-scale training","Size-adaptive network enables curriculum for many agents","Small-to-large agent curriculum outperforms from-scratch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1542,"prompt_tokens":925,"completion_tokens":617,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":555}},"tokens_in":541,"tokens_out":617,"duration_ms":6268,"temperature":1.0,"reasoning_tokens":555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:39:02.247640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to extract observations from 5-vs-5, 10-vs-10, and 15-vs-15 battles that share identical local semantics, such as exactly one teammate in view, and check whether DyAN's sum-aggregated embeddings form one cluster while different-semantics observations form another; if same-semantics points do not cluster, the semantic-mapping property that Model Reload depends on fails. A controlled ablation that removes only Model Reload while keeping the same curriculum would then reveal whether the reported gains come from cross-size transfer or from some other curriculum effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MAgent platform, reward settings, and large-scale battle scenarios used in the second set of experiments."},{"cited_title":"A.; Veness, J.; Bellemare, M","cited_arxiv_id":null,"evidence_quote":"Provides DQN, the base off-policy learning algorithm whose replay buffer is extended by the Buffer Reuse transfer mechanism."},{"cited_title":"Z.; and Gruslys, A","cited_arxiv_id":null,"evidence_quote":"Motivates Buffer Reuse by showing that deep Q-learning can incorporate expert demonstrations as extra supervision."},{"cited_title":"A.; Colmenarejo, S","cited_arxiv_id":null,"evidence_quote":"Provides the policy-distillation technique on which the Curriculum Distillation transfer mechanism via KL divergence is based."},{"cited_title":"S.; Farquhar, G.; Foerster, J.; and Whiteson, S","cited_arxiv_id":null,"evidence_quote":"Provides value-decomposition multiagent RL context and the StarCraft II settings that VDN and the paper's combat benchmarks build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the t-SNE embedding method used to visualize whether the learned semantic mapping clusters same-meaning states across scenarios."}],"review_version":1}