{"id":"87a614d4-fced-4c61-9fa6-43a2d8ade53e","arxiv_id":"2412.15517","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MANGER uses RND-computed observation novelty to give each agent a different number of extra Q-learning updates, improving sample efficiency and behavioral diversity in cooperative MARL.","lead":"A new MARL training trick, MANGER, uses a curiosity-style novelty signal to decide which agents' samples to replay more often, and separates each agent's critic into shared and private parts to encourage role specialization. The paper reports higher win rates than QMIX, QPLEX, and Qatten on StarCraft II and Google Research Football, but the description has internal inconsistencies that need fixing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RND novelty (visit frequency) is used as proxy for Q-value inaccuracy in Eq. (2), but in MARL the Bellman error is driven by teammate-policy nonstationarity, not visit count; if the proxy is weak, the targeted-reuse mechanism and claimed gains lose support.","rationale":"The reader's weakest assumption is that RND visit frequency proxies Q-value inaccuracy; I agree and sharpen it. The load-bearing issue is not just missing correlation evidence: in multi-agent settings the value function is nonstationary because other agents' policies evolve, so repeated visits do not guarantee accurate estimates. The paper's own justification for reusing rare samples is therefore theoretically fragile. However, I do not see a reason to move the verdict beyond CONDITIONAL: the empirical win-rate comparisons are the central claim, and the concern is about the explanatory mechanism. The reader already marked the paper conditional; my analysis does not change that assessment. I would still require the correlation check before accepting the mechanism as established. The internal inconsistencies (alpha=2 vs alpha=1, beta unused, target-network update typos in Algorithm 1) further support a conditional verdict, but they are secondary to the proxy assumption.","tokens_in":13567,"tokens_out":3887,"duration_ms":36552,"concrete_test":"Instrument MANGER on 3s_vs_5z and 6h_vs_8z: for every agent transition used in an update, log the RND novelty N(o_i) and the per-agent TD error magnitude |Q_i - (r + gamma * max_{a'} Q_i^-(s', a'))| at update time. Compute the Spearman rank correlation between these two series per agent over training. If the median correlation is below roughly 0.3, the premise of Eq. (2) is not supported. As a second check, run an ablation where the extra updates in Eq. (2) are allocated by measured TD error instead of RND novelty; if TD-error-guided reuse matches or beats MANGER, the claimed RND-specific targeting is not the causal mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism rests on the claim in 'Using Data Efficiently' that 'For data that has already appeared repeatedly, the network's estimates are highly accurate, and thus further updating is unnecessary.' This equates RND prediction error with Q-value estimation error. In cooperative MARL, the value of an observation depends on the joint policy of all agents. As teammates' policies change during training, an observation that has been seen many times can carry an outdated or inaccurate Q-value, while a rare observation can have an accurate value due to function approximation and bootstrapping. The paper provides no measurement of the correlation between N(o_i) (Eq. 1) and any TD/Bellman error signal. If the correlation is weak, the additional updates selected by Eq. (2) are not targeted at inaccurate estimates; the observed gains could come from generic extra updates, from the separate critic module, or from seed noise. The ablation in Fig. 6 compares against fixed extra updates, but it does not verify that the RND-selected subset is the right subset; it only verifies that some subset selection differs from uniform reuse.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MANGER, a value-based cooperative MARL method built on QMIX. It uses RND prediction error (Eq. 1) to score each agent's observation novelty, then converts normalized novelty into per-agent extra Q-learning updates via Eq. (2). The agent critic is split into a shared layer and per-agent separate layers to localize the extra updates. Experiments on SMAC, GRF, and SMAC-V2 claim higher win rates than QMIX, QPLEX, and Qatten with similar or lower training time, and Q-value cosine similarity and screenshots are used to claim that agents specialize into roles. The paper is primarily an empirical heuristic with no theoretical analysis.","tokens_in":13796,"tokens_out":5266,"duration_ms":48273,"significance":"If the reported gains are robust, MANGER would be an inexpensive drop-in addition to QMIX-style algorithms, with broad applicability to cooperative MARL benchmarks. The code link, the use of three benchmark suites, and the inclusion of an ablation study are strengths. However, the central mechanism rests on an unverified proxy assumption, and the diversity analysis lacks the disambiguating control, so the significance as demonstrated is moderate rather than high.","major_comments":[{"comment":"The text states alpha=2, while both hyperparameter tables list alpha=1. The denominator of Eq. (2) is labeled Var(N_total), but the prose calls it the standard deviation; these differ by a factor that changes the value of T_i. Tables 1 and 2 also list beta (\"max number additional updates\"), but beta never appears in Eq. (2), Algorithm 1, or the text. As written, the update rule is not uniquely specified, and the reported training-time and win-rate results cannot be exactly reproduced.","section":"Using Data Efficiently, Eq. (2), Tables 1-2"},{"comment":"The method assumes that high RND novelty implies high Q-value estimation error, stated as \"For data that has already appeared repeatedly, the network's estimates are highly accurate, and thus further updating is unnecessary.\" This correlation is never measured. In cooperative MARL, target Q-values depend on teammate policies, so frequently visited observations can carry stale values and rare observations can be accurate through bootstrapping. The ablation in Fig. 6 compares MANGER to uniform extra updates and to Qmix_sep, but it does not compare against random selection of the same number of extra updates, so it cannot confirm that the RND-selected subset is responsible for the gains.","section":"Using Data Efficiently"},{"comment":"The Q-value cosine-similarity plots compare MANGER only with QMIX. Since MANGER differs from QMIX in two respects (the separated critic and the novelty-guided update schedule), this comparison cannot isolate the diversity contribution of the latter. The Qmix_sep control in Fig. 6 should be included in Fig. 5 to show that the observed reduction in Q-value similarity is not simply an artifact of per-agent Q_sep layers.","section":"Analysis, Fig. 5"},{"comment":"The pseudocode updates target networks in line 31 at every training step, contradicting the stated M_target=200 in Tables 1 and 2. The pseudocode also performs extra updates on both theta_sep and phi in lines 27-30, whereas the text says additional training only updates the independent layer. In addition, Eq. (1) writes the norm as ||...||_2 while line 23 of the algorithm writes the same expression as a squared error; please align the notation.","section":"Appendix A, Algorithm 1"},{"comment":"The SMAC-V2 results are claimed in Appendix D and the caption for Figure 7 is present, but the figure itself is not included in the text provided, so these results cannot be checked. The central empirical claim extends to SMAC-V2, so the figure must be present and legible in the final version.","section":"Appendix D, Figure 7"}],"minor_comments":[{"comment":"The sentence \"the experimental results are shown in Figure 2\" in the GRF paragraph should refer to Figure 4, which is where the GRF curves appear.","section":"Result (GRF)"},{"comment":"Equation (6) uses alpha as the learning rate, while Eq. (2) uses alpha as the additional-update coefficient; these are unrelated constants sharing a symbol and should be renamed.","section":"Update of the MANGER Agents, Eq. (6)"},{"comment":"The claim that \"the mean number of extra updates is less than 0.5\" is not derived from any distribution described in the text; since T_i is an integer, a mean below 0.5 imposes a specific shape on the novelty distribution that should be stated or demonstrated empirically.","section":"Using Data Efficiently / Ablation Study"},{"comment":"The rows labeled \"T D lambda ratio to control Q-value update\" and \"TDlambda\" are unclear; please define the quantity unambiguously and state its role in the QMIX update.","section":"Experimental Settings, Tables 1-2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's evaluation is based on five seeds without reporting variances or significance tests, so the reader cannot assess whether the win-rate gaps are statistically meaningful. The high density of self-citations in the sample-reuse discussion (e.g., SMR, AVTD, DRND) is disproportionate, and several of those citations are peripheral to the method. The missing SMAC-V2 figure must be verified before the paper can be assessed fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: MANGER is a modest but genuine MARL trick. It uses RND novelty per agent to decide how many extra Q-learning updates each agent gets, and routes those extra updates through per-agent separate critic layers so the shared layers don't drag everyone along. The components are known—RND, CDS-style decomposition, sample reuse—but the specific packaging is new, and the empirical work is clean enough to be useful.\n\nWhat I like most is the ablation. Comparing against fixed extra updates (QMIX-update2/4) and the separate critic alone shows the RND-targeting is doing real work, not just generic sample reuse. That's the right experiment and it gives the method some teeth. Results on SMAC, SMAC-V2, and GRF with five seeds and code links are also in the paper's favor.\n\nThe soft spots are mostly presentational, but there are a few real ones. The alpha=2 in the text vs alpha=1 in the hyperparameter tables is a direct contradiction. Beta appears in the tables but never in the method. Equation (2) says \"Var\" and then calls it standard deviation. The pseudo-code in Algorithm 1 looks like it updates target networks and shared layers during the extra-update loop, which contradicts the method description. And the introduction promises \"theoretical analysis\" that never appears.\n\nThe deeper concern from the stress-test is the proxy assumption: treating RND prediction error (visit frequency) as a reliable indicator of Q-value estimation error. In cooperative MARL, teammate policy nonstationarity makes old observations stale, so a frequently visited state can still have an inaccurate value. The ablation shows RND-selected updates beat uniform updates, which is indirect evidence that the proxy captures something useful, but the paper never measures the correlation directly. That's a gap, not a fatal flaw—the method works empirically, the mechanism is just under-supported.\n\nWho is this for? People working on sample efficiency or diversity in cooperative MARL. It deserves a serious referee; a careful reviewer can ask for the missing measurements and cleanup. I'd send it to review rather than desk-rejecting.","headline":"A modest but genuine MARL trick: RND-guided per-agent extra updates with a decomposed critic, backed by a useful ablation, but the paper has presentation sloppiness and an unverified proxy assumption.","tokens_in":14317,"tokens_out":2449,"would_cite":true,"duration_ms":24258,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Novelty-guided replay lifts cooperative MARL win rates.","keywords":["multi-agent reinforcement learning","sample efficiency","random network distillation","novelty-guided data reuse","agent diversity","QMIX","StarCraft Multi-Agent Challenge","Google Research Football"],"falsifier":"Measure, on a fixed SMAC task, the within-batch correlation between an agent's RND novelty $N(o_i)$ and its per-state TD-error magnitude $|Q_{\\text{tot}} - y|$; if the correlation is not positive, or if shuffling the novelty scores among agents in a batch (keeping the same number of extra updates) does not decrease win rate, then MANGER's gains are not caused by targeting inaccurate Q-value estimates.","tokens_in":1904,"feed_emoji":"🤖","tokens_out":2661,"duration_ms":51170,"temperature":0.7,"pith_summary":"The paper claims that a multi-agent reinforcement learning system can be made both more sample-efficient and more behaviorally diverse by giving each agent extra Q-learning updates in proportion to the novelty of its observations, with novelty measured by random network distillation (RND). The proposed method, MANGER, builds on QMIX and reuses samples in two ways: every sample gets a base update, and agents whose observations are rare get additional updates. The paper presents results on StarCraft II (SMAC and SMAC-V2) and Google Research Football in which MANGER reaches higher win rates than QMIX, QPLEX, and Qatten within the same number of environment steps, and it shows through Q-value similarity that agents start to specialize. If correct, this would mean that targeted sample reuse based on a cheap curiosity signal is a practical way to improve cooperative MARL without adding new rewards or architectural complexity. The load-bearing empirical assertion is the win-rate superiority shown in Figures 3, 4, and 7.","feed_headline":"Novelty-guided replay lifts cooperative MARL win rates","feed_subtitle":"MANGER adds extra Q-updates for rare states, beating QMIX, QPLEX, and Qatten on StarCraft and football tasks.","key_machinery":"The central mechanism is an RND novelty scorer: a fixed random target network plus a learned predictor whose mean-squared error on an observation is that observation's novelty. This novelty score gates per-agent extra Q-learning updates, so the same transition can be replayed different numbers of times for different agents. A second piece is the split critic: each agent's Q-value is $Q_{\\text{com}}(o_i, a_i) + \\lambda Q^i_{\\text{sep}}(o_i, a_i)$, and additional updates touch only $Q^i_{\\text{sep}}$, keeping the added reuse from interfering across agents. The formula $T_i = \\text{int}(\\alpha (N_i - \\bar{N}_{\\text{total}}) / \\text{Var}(N_{\\text{total}}))$ converts normalized batch-level novelty into an integer number of extra updates per agent, with $\\alpha = 2$ and a mean extra update count of about 0.5 that the paper argues adds little training time.","core_discovery":"The paper's central claim is that the frequency with which an agent has seen an observation is a usable signal for how much that agent's Q-network still needs to learn from the data. MANGER computes a novelty score $N(o_i) = \\|f_{\\text{target}}(o_i) - f_{\\text{predictor}}(o_i)\\|_2$ from a fixed random target network and a trained predictor, normalizes these scores within a batch, and assigns each agent $i$ an extra update count $T_i = \\text{int}(\\alpha (N_i - \\bar{N}_{\\text{total}}) / \\text{Var}(N_{\\text{total}}))$. Agents with $T_i > 0$ perform additional Q-learning steps on the same transition, updating only their separate critic layer so that extra updates do not bleed into other agents. The paper reports that this yields faster and higher final win rates than QMIX, QPLEX, and Qatten on super-hard SMAC maps, three GRF scenarios, and SMAC-V2 tasks, and it shows lower cosine similarity among agents' Q-values under the same observation, which it reads as evidence of role division and specialization.","pith_inferences":["MANGER is effectively a form of prioritized experience replay in which the priority signal is observation novelty rather than TD error; directly comparing against TD-error prioritization on the same benchmarks would clarify which signal drives the gains.","The RND novelty signal is shared across agents, which implicitly discourages an agent from treating states mostly visited by other agents as novel and therefore discourages imitation; this diversity-preserving property could be studied formally in terms of state-visitation coverage.","The assumption that visit frequency proxies estimation error is untested directly; a probe of the within-batch correlation between RND MSE and per-agent TD error would tell whether the mechanism genuinely targets inaccurate estimates or simply amplifies exploration.","The discrete-action QMIX backbone would need replacement for continuous-control MARL, but the same novelty-guided update-count scheme could apply to actor-critic methods if the per-agent critic decomposition is retained."],"forward_implications":["If MANGER's win-rate gains replicate, cooperative MARL systems can improve sample efficiency by selectively reusing existing transitions rather than collecting more data or adding intrinsic rewards.","The diversity results imply that update-frequency asymmetry alone can induce role specialization, so division of labor can arise without explicit role assignment or mutual-information objectives.","The method is modular on top of value-decomposition baselines: it applies to QMIX-style critics with shared and separate layers, and the SMAC-V2 results suggest it transfers to newer benchmarks.","The extra update cost is small, so the reported performance gains do not come at large compute overhead.","If the correlation between RND novelty and Q-value inaccuracy is weak, the targeted-update mechanism could be replaced by a cheaper fixed-rate replay strategy, so the paper's ablation against Qmix-update2 and Qmix-update4 is the key comparison that separates targeted reuse from mere extra reuse."],"supporting_citations":[{"why":"Supplies the Random Network Distillation novelty measure that the entire update-count rule relies on.","marker":"Burda et al. 2018"},{"why":"Supplies the shared/independent critic decomposition that lets extra updates target one agent without affecting others.","marker":"Li et al. 2021"},{"why":"QMIX is the base algorithm MANGER builds on and the primary baseline for the win-rate comparisons.","marker":"Rashid et al. 2020"},{"why":"REDQ motivates the broader idea that larger update-to-data ratios improve sample efficiency, which MANGER extends with selective updates.","marker":"Chen, Hessel, and van Hasselt 2021"},{"why":"Defines the SMAC benchmark whose super-hard maps are the primary testbed for the main win-rate claims.","marker":"Samvelyan et al. 2019"},{"why":"Defines the Google Research Football benchmark used as a second testbed.","marker":"Kurach et al. 2020"},{"why":"Defines SMAC-V2, the newer benchmark used for the additional results in Appendix D.","marker":"Ellis et al. 2024"},{"why":"Provides the PyMARL2 codebase in which MANGER is implemented for the SMAC experiments.","marker":"Hu et al. 2021"}],"fun_headline_variants":["Rare-state replay boosts cooperative MARL win rates","MANGER: extra updates for unique observations beat QMIX and QPLEX","Novelty-guided replay lifts performance in StarCraft and football","Unique states get extra training steps, yielding diverse agent roles","MANGER assigns more Q-updates to rare states, improving MARL"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The method assumes that an observation visited rarely, as measured by RND prediction error, is one where the agent's Q-value estimate is inaccurate, so that spending extra updates there improves learning more than spending them elsewhere.","fun_headline_variants_meta":{"raw":{"variants":["Rare-state replay boosts cooperative MARL win rates","MANGER: extra updates for unique observations beat QMIX and QPLEX","Novelty-guided replay lifts performance in StarCraft and football","Unique states get extra training steps, yielding diverse agent roles","MANGER assigns more Q-updates to rare states, improving MARL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3142,"prompt_tokens":951,"completion_tokens":2191,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2099}},"tokens_in":567,"tokens_out":2191,"duration_ms":15432,"temperature":1.0,"reasoning_tokens":2099,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:21:15.810937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a fixed SMAC task, the within-batch correlation between an agent's RND novelty $N(o_i)$ and its per-state TD-error magnitude $|Q_{\\text{tot}} - y|$; if the correlation is not positive, or if shuffling the novelty scores among agents in a batch (keeping the same number of extra updates) does not decrease win rate, then MANGER's gains are not caused by targeting inaccurate Q-value estimates.","supporting_citations":[{"cited_title":"S.; Farquhar, G.; Foerster, J.; and Whiteson, S","cited_arxiv_id":null,"evidence_quote":"QMIX is the base algorithm MANGER builds on and the primary baseline for the win-rate comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"REDQ motivates the broader idea that larger update-to-data ratios improve sample efficiency, which MANGER extends with selective updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Google Research Football benchmark used as a second testbed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SMAC-V2, the newer benchmark used for the additional results in Appendix D."}],"review_version":1}