{"id":"74166671-d425-48bc-abe3-e82cc42787f2","arxiv_id":"2501.10367","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GTDE dynamically groups agents during training using Gumbel-Sigmoid sampling, enabling decentralized execution that outperforms CTDE and DTDE on large-scale multi-agent benchmarks.","lead":"GTDE proposes a new training paradigm for multi-agent reinforcement learning where agents form dynamic groups during training and act independently during execution. It reports large performance gains on large-scale benchmarks (up to 495 agents) compared with centralized and decentralized baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that GTDE outperforms CTDE in large-scale tasks is not yet established because the only CTDE baselines in Battle/Gather are an impractical monolithic MAAC variant; a strong CTDE baseline comparison is needed before accepting the headline.","rationale":"The reader's weakest assumption targets the adaptive grouping module's ability to infer useful links from observation history alone. That is a legitimate concern, but the paper's own ablations show that even GTDE-U (uniform random grouping) and GTDE-F (fixed grouping) achieve strong results in Battle and Gather (e.g., 91.8 and 97.2 win rates in Battle, and only 10-18% lower total reward in Gather), so the adaptive grouping mechanism is not the sole driver of the headline performance. The more load-bearing weakness is the comparison protocol: the claimed superiority over CTDE at scale rests on a CTDE baseline that is intentionally or accidentally handicapped by an impractical monolithic critic. Since the paper explicitly argues that CTDE becomes untrainable or ineffective as agent count increases, the fairness of the CTDE baseline is central to that argument. If a reasonable CTDE baseline performs competitively, the paper's main contribution shifts from a new paradigm that beats CTDE to a sparse-attention variant of existing methods. This is an empirical question that can be settled by a specific baseline rerun, and the paper currently provides no theory that would protect against this outcome. I therefore agree with the reader's conditional verdict but locate the controlling concern in baseline fairness rather than in the grouping module alone. No significant internal mathematical contradiction was found in the Gumbel-Sigmoid derivation or the actor-critic update; the concern is about whether the empirical comparison supports the central claim.","tokens_in":13845,"tokens_out":7417,"duration_ms":85507,"concrete_test":"Reproduce Battle and Gather with the released code, replacing the MAAC baseline with at least one strong CTDE method: (1) MAPPO with a centralized state-based critic using global features, or (2) MAAC with its original attention mechanism over per-entity observations, or (3) QMIX with a factorized value function. Use the same training budget, seeds, and evaluation protocol as in the paper. If GTDE still achieves a decisive win-rate/total-reward advantage in Battle and Gather, the central claim survives; if a CTDE baseline matches or exceeds GTDE, the claim that GTDE outperforms CTDE in large-scale MARL is falsified. In parallel, recompute the Battle win rates from the raw evaluation logs to resolve the apparent discrepancy between Table 1 and the text's '100% win rate' statement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's contribution is framed as a paradigm that overcomes limitations of both DTDE and CTDE, with the abstract and conclusions claiming superiority as the number of agents grows. In §5, however, the large-scale Battle (64 agents) and Gather (495 agents) experiments compare GTDE only against IAC (DTDE) and MAAC (CTDE), and the MAAC baseline is implemented with a fully connected critic over the raw joint observation. The authors state that MAAC's input dimension of (13×13×5+32)×64 in Battle causes it to perform worse than IAC when using the same network size. This is an implementation choice, not an inherent property of CTDE: modern CTDE methods such as MAPPO with a state-based critic, MAAC with its original entity-wise attention, or value-factorization methods (QMIX) do not consume a flattened joint observation and would not exhibit this failure mode. On SMACv2, where the CTDE baseline is MAPPO, GTDE's advantage over CTDE is small and often within one standard deviation (Zerg: 53.4 vs 50.6; Terran: 48.4 vs 45.3; Protoss: 38.8 vs 26.6). Thus the evidence that GTDE beats CTDE at scale is weak. The Gather 382% improvement is measured only against IAC, a DTDE baseline, not against any CTDE method. Additionally, the Battle column of Table 1 is difficult to reconcile with the text's claim of a 100% win rate against MAAC and IAC, further undermining the empirical basis. The load-bearing question is whether the paradigm advantage survives a fair CTDE baseline; the paper currently answers it only with a strawman.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GTDE (Grouped Training with Decentralized Execution), a training-time grouping paradigm for multi-agent actor-critic methods. The key idea is to let each agent, during training, condition its value function on a dynamically selected subset of other agents' observation histories (a 'group') rather than on individual or joint observations. Grouping is learned end-to-end via a Gumbel-Sigmoid reparameterization of an adjacency matrix, with group information aggregated by matrix multiplication or by graph attention. The authors evaluate GTDE against MAPPO/IPPO on SMACv2 20v20, against MAAC/IAC on a 64-agent Battle environment, and against IAC on a 495-agent Gather environment, reporting improvements and a 100% win rate against the two baselines in Battle.","tokens_in":14161,"tokens_out":6340,"duration_ms":56920,"significance":"If fully supported, the GTDE idea would be a useful contribution to the large-scale MARL literature, offering a way to retain some inter-agent coordination during training without the costly joint observation. The paper provides code, ablations (GTDE-F, GTDE-U, GTDE-A), and reports the average number of agents used in the critic, which indicates a substantial reduction in input information. However, the headline claim that GTDE outperforms CTDE as the number of agents grows is not convincingly established by the current experiments, because the CTDE baselines in the large-scale scenarios are either absent or implemented in an unfavorable way, and the SMACv2 advantage over MAPPO is within one standard deviation on two of three maps. The contribution is therefore promising but needs additional empirical support.","major_comments":[{"comment":"The claim that GTDE outperforms CTDE at scale is not supported by the experiments on Battle and Gather. The only CTDE baseline in these scenarios is MAAC with a flattened joint observation, and the paper itself states that MAAC's input dimension of (13×13×5+32)×64 makes it perform worse than IAC. This is an implementation artifact of a particular CTDE design rather than a fundamental property of CTDE. A proper CTDE baseline such as MAPPO with a state-based critic, MAAC with entity-wise attention, or a value-factorization method like QMIX would not suffer from this failure mode. Consequently, the 100% win rate against MAAC in Battle cannot be interpreted as a win against CTDE in general.","section":"Section 5, Table 1, Fig. 4"},{"comment":"The reported 382% average total-reward improvement is computed against IAC, a DTDE baseline, and no CTDE baseline is trained in this scenario. The statement in the conclusion that 'CTDE becomes untrainable due to the high input dimensionality' is a conclusion about the particular MAAC implementation, not an inherent limitation of CTDE. The paper should either run a state-based MAPPO or another scalable CTDE method in Gather, or explicitly qualify the claim as a comparison against DTDE only.","section":"Section 5, Gather, Fig. 6"},{"comment":"The table is difficult to interpret and appears to contradict the text. The text states that 'GTDE defeated them with a 100% win rate' against IAC and MAAC, while Table 1 lists GTDE's Battle value as 52.3(2.8) and CTDE/DTDE as 100.0(0.0). If the entries in the Battle column are the win rates of GTDE against the algorithm in the corresponding row, the caption must say so explicitly; if they are the win rates of the algorithms themselves, the 52.3 value for GTDE needs explanation. As it stands, the table and the text cannot both be correct, undermining the empirical basis of the Battle claim.","section":"Table 1, Battle column"},{"comment":"The paper does not provide evidence that the adaptive grouping module learns meaningful, non-trivial groupings. In Battle, GTDE-U (uniform/random grouping) still achieves a 91.8% win rate, and in SMACv2 the gap between GTDE and GTDE-U is modest (53.4 vs 46.4, 48.4 vs 43.5, 38.8 vs 32.8). This suggests that a large part of the observed benefit may come from the aggregation of a subset of observations rather than from the specific learned grouping. The authors should report the learned adjacency matrices qualitatively and provide a quantitative measure of grouping quality, or temper the claim that adaptive grouping is the key component.","section":"Section 4, Adaptive Grouping, and Table 1 ablations"}],"minor_comments":[{"comment":"The term 'group' is used for the set of agents that a given agent is linked to, but this does not define a partition of the agent set; two agents can have different groups and the 'number of groups' is therefore not well-defined. Suggest clarifying this terminology.","section":"Section 4, Eq. (3)"},{"comment":"The derivation of Gumbel-Sigmoid uses two independent Gumbel noises (ϵ1 and ϵ2) but the text just says ϵ ∼ Gumbel(0,1); please specify that the two noises are independent.","section":"Appendix A"},{"comment":"The paper does not report sensitivity to the mask dropout probability (0.1) or the Gumbel temperature; a short sensitivity analysis or a note on their choice would strengthen the paper.","section":"Section 5, Hyperparameter"},{"comment":"The partial links in the Battle scenario are described in the text but the figure lacks labels or arrows that make it easy to see which agents are linked; the figure needs a clearer caption and legend.","section":"Figure 5"},{"comment":"The reference 'Ruan et al. 2023' appears in the bibliography but is not cited in the body of the paper.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the MARL community, but the experimental design does not currently support the central claim that GTDE outperforms CTDE at scale. In particular, the Battle column of Table 1 is confusing and should be clarified before any further consideration. I would be willing to review a revised version that includes a strong CTDE baseline in the large-scale scenarios."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: GTDE is a training-time grouping scheme for MARL that drops the centralized critic and learns dynamic groups from each agent's observation history. The adaptive grouping via Gumbel-Sigmoid is clever, and it scales to 495 agents in the Gather scenario. But the evidence for the headline claim that it beats CTDE at scale is weak, because the CTDE baseline in the large-scale experiments is a flatten-joint-observation MAAC, a known pathology, not a representative CTDE method.\n\nWhat's actually new: GTDE lets the number of groups and members vary per timestep, learned end-to-end from local observation histories, without a centralized clustering step or a fixed partition. That is a real difference from earlier grouping methods like VAST or DHCG. The Gumbel-Sigmoid pointwise sampling is a sensible reparameterization, though not a deep theoretical contribution.\n\nWhat it does well: The authors are honest that MAAC cannot be trained in Gather due to memory, and that on SMACv2 GTDE is only marginally ahead of MAPPO on two maps and clearly ahead on one. The ablation against fixed and uniform grouping shows that learned groups matter. They also report training times, which is more than many papers do.\n\nSoft spots, in order of severity:\n\n1. The CTDE comparison in Battle and Gather is weak. MAAC with a flattened joint observation is known to fail as n grows. A proper CTDE baseline like MAPPO with a compact state-based critic should have been run, or at least discussed. Without it, the claim that GTDE overcomes CTDE's limitations is unsupported.\n\n2. Table 1's Battle column is internally inconsistent. The text says GTDE wins 100% against IAC and MAAC, but the table lists CTDE and DTDE rows as 100.0. Either the table labels are wrong or the numbers don't match the text. A referee needs to sort that out.\n\n3. The mask dropout probability (0.1) is a free parameter with no ablation. Same for the Gumbel temperature, which isn't even mentioned in the main text.\n\n4. There's no theoretical analysis of what grouping is learned; the paper states it works on the two-buttons intuition, but provides no beyond the one figure.\n\nThat said, the core idea is interesting and the paper is a reasonable building block. It deserves a serious referee, but not acceptance as is. The authors should add a fair CTDE baseline, clarify the Battle table, and ablate the mask dropout.\n\nWho for: researchers working on scaling MARL to many agents, especially those interested in training-time grouping or avoiding centralized critics. I'd probably not cite it as strong evidence of the paradigm's superiority, but might mention it as related work on adaptive grouping.\n\nRecommendation: send to peer review with a request for major revision. A good referee can push the authors to make the empirical case rigorous.","headline":"GTDE's adaptive grouping is a plausible new idea, but the paper's CTDE baselines are too weak to support the claim that it beats centralized training at scale.","tokens_in":14715,"tokens_out":6000,"would_cite":false,"duration_ms":55097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes GTDE, a training paradigm in which each agent learns from a small, self-chosen group of other agents, and reports that it beats both the full-centralized and fully-decentralized baselines in large-scale multi-agent tasks.","keywords":["multi-agent reinforcement learning","centralized training decentralized execution","decentralized training and execution","grouped training","actor-critic","Gumbel-Sigmoid","large-scale cooperation","adaptive grouping"],"falsifier":"Run GTDE on the two-button task described in the paper, where a reward is obtained only when two agents separated by a large distance press two buttons at the same time; if the two relevant agents fail to learn a link and performance stays at DTDE levels, the grouping module's assumption of inferring links from observation history alone would be falsified.","tokens_in":13617,"feed_emoji":"🤖","tokens_out":17048,"duration_ms":146894,"temperature":0.7,"pith_summary":"The paper claims that the two standard training schemes for multi-agent reinforcement learning—centralized training with decentralized execution (CTDE), which uses all agents' information, and decentralized training and execution (DTDE), which uses only each agent's own information—both lose effectiveness as the number of agents grows. It proposes a middle path, grouped training with decentralized execution (GTDE), in which each agent selects a small set of other agents to share information with during training, and the set is chosen from its own observation history rather than from global knowledge. The authors support the proposal with experiments showing a 382% average total-reward gain over the DTDE baseline IAC in a 495-agent cooperative gathering scenario, a 100% win rate against both CTDE and DTDE representatives in a 64-agent competitive battle, and win rates on a 20-agent StarCraft benchmark that match or exceed CTDE. If correct, GTDE would make actor-critic training feasible at scales where a centralized critic is untrainable.","feed_headline":"Grouped training beats global and independent baselines at 495-agent scale","feed_subtitle":"The method learns local coordination groups from each agent's own observations, raising total reward 382%.","key_machinery":"The central object is the directed graph $G^t=(V^t,E^t)$ whose edge $\\langle v_i, v_j\\rangle$ means agent $i$ uses agent $j$'s information at step $t$; the group $g(v_i)$ of an agent is its set of out-neighbors. The argument is carried by the adaptive grouping module, a network that maps each agent's observation history $\\tau_i$ to an $n$-variate Bernoulli distribution over possible links and samples it with Gumbel-Sigmoid to preserve differentiability, and by the group information aggregation module, which merges the histories of linked agents using either matrix multiplication with the adjacency matrix or graph attention. Together they make GTDE a spectrum that contains the prior paradigms as extremes: CTDE is recovered when all agents are pairwise linked (the complete graph), and DTDE is recovered when each agent is linked only to itself (the self-loop graph). The critic is trained on the aggregated group history $\\tau'_{g(v_i)}$, while the actor uses only $\\tau_i$.","core_discovery":"The paper's central claim is that the information an agent needs for value estimation can be represented as a directed graph of links to other agents, and that training on the local subgraph—not the complete graph (CTDE) and not the self-loop graph (DTDE)—is both sufficient and more scalable. GTDE learns these links end-to-end: an adaptive grouping network outputs, for each agent and each other agent, a Bernoulli probability of linkage based on the agent's observation history; Gumbel-Sigmoid sampling converts this into a differentiable adjacency matrix. A group information aggregation module then combines the observation histories of linked agents, either by matrix multiplication or by masked graph attention, and the actor's value function is trained on the aggregated group trajectory. Execution remains fully decentralized because at action time each policy sees only its own observation. The paper reports that this single framework improves on both paradigms across three benchmarks, and that ablations with fixed or random grouping confirm the learned links, not the aggregation mechanism, drive the gains.","pith_inferences":["The paper's two-button counterexample suggests a boundary condition the authors do not test: GTDE may fail when the agents that need to coordinate are mutually invisible, so a natural next experiment is to stress-test the grouping module on long-range coordination tasks and decide whether a small amount of global hint at training time is needed.","Because the adjacency matrix has fixed size $n \\times n$ and new agents can only link to existing ones, the framework as presented is not open to population growth; a forward-looking extension would replace the matrix with a learned neighbor-querying function or a graph generative model.","The paper does not ablate the two aggregation methods (matrix multiplication versus graph attention), so a reader can infer that the large gains come from the adaptive grouping module rather than from the aggregation backbone; a direct comparison of the two aggregators under identical grouping would test whether attention weights add value."],"forward_implications":["Training a critic no longer requires input that grows with the number of agents: the value input is the aggregated group trajectory, so the framework can be extended to populations far beyond 495 agents without changing network architecture.","The framework can be combined with communication-based methods: the learned adjacency matrix gives a natural routing protocol for sharing information, an avenue the paper notes as complementary to GTDE.","In the tested scenarios, GTDE uses roughly five times less input information per agent than CTDE, which translates directly into lower communication and memory costs in deployment.","The adaptive grouping mechanism doubles as a discoverability tool: the links it learns can be interpreted as task decompositions, as in the Battle scenario where distant agents link to form a besieging group."],"supporting_citations":[{"why":"MAAC baseline; supplies the CTDE attention-based critic that GTDE is compared against and whose centralized attention motivates the grouping idea.","marker":"Iqbal and Sha 2019"},{"why":"MAPPO baseline; the centralized PPO variant used as a CTDE reference on SMACv2.","marker":"Yu et al. 2022"},{"why":"IPPO baseline; the decentralized PPO variant whose poor performance at larger scales motivates GTDE.","marker":"de Witt et al. 2020"},{"why":"SMACv2 benchmark; provides the 20-agent cooperative StarCraft tasks on which GTDE is tested.","marker":"Ellis et al. 2022"},{"why":"MAgent platform; supplies the Battle (64 agents) and Gather (495 agents) scenarios used in the main experiments.","marker":"Zheng et al. 2018"},{"why":"MAgent repository; cited as the source of the 495-agent Gather scenario in the introduction.","marker":"Terry, Black, and Jayakumar 2020"},{"why":"Graph attention networks; the aggregation method used in the SMACv2 implementations of GTDE.","marker":"Veličković et al. 2017"}],"fun_headline_variants":["Grouped training with decentralized execution scales to 495 agents, +382% reward","No central critic needed: GTDE achieves 100% win rate at 64 agents","Adaptive grouping network learns local links, boosting reward 382% at 495 agents","Training on learned subgraphs beats full-graph and independent training in MARL","GTDE: learnable grouping for scalable multi-agent RL, no central critic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an agent's own observation history is enough to infer which other agents it should coordinate with; if the relevant collaborators are far away and invisible in local observations, the learned groups become arbitrary and the reported gains would disappear, and the paper provides only empirical evidence rather than a theoretical guarantee for this step.","fun_headline_variants_meta":{"raw":{"variants":["Grouped training with decentralized execution scales to 495 agents, +382% reward","No central critic needed: GTDE achieves 100% win rate at 64 agents","Adaptive grouping network learns local links, boosting reward 382% at 495 agents","Training on learned subgraphs beats full-graph and independent training in MARL","GTDE: learnable grouping for scalable multi-agent RL, no central critic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3297,"prompt_tokens":1011,"completion_tokens":2286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2190}},"tokens_in":627,"tokens_out":2286,"duration_ms":17892,"temperature":1.0,"reasoning_tokens":2190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:27:29.964124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GTDE on the two-button task described in the paper, where a reward is obtained only when two agents separated by a large distance press two buttons at the same time; if the two relevant agents fail to learn a link and performance stays at DTDE levels, the grouping module's assumption of inferring links from observation history alone would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MAAC baseline; supplies the CTDE attention-based critic that GTDE is compared against and whose centralized attention motivates the grouping idea."},{"cited_title":"N.; and Whiteson, S","cited_arxiv_id":null,"evidence_quote":"SMACv2 benchmark; provides the 20-agent cooperative StarCraft tasks on which GTDE is tested."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MAgent platform; supplies the Battle (64 agents) and Gather (495 agents) scenarios used in the main experiments."},{"cited_title":"K.; Black, B.; and Jayakumar, M","cited_arxiv_id":null,"evidence_quote":"MAgent repository; cited as the source of the 495-agent Gather scenario in the introduction."}],"review_version":1}