{"id":"3b260f87-7efb-4cbe-b2a5-3f6408f06ce1","arxiv_id":"2502.06261","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proves that communication in decentralized critics adds variance to policy gradients and introduces baseline plus KL techniques that reduce this variance and improve learning.","lead":"This paper studies how communication between agents during training affects the variance of policy gradient updates in decentralized multi-agent reinforcement learning. It proposes a message-dependent baseline and a KL regularization term that reduce training variance and improve performance on StarCraft and Traffic Junction benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's proof uses a message-conditioned policy in the critic's Bellman equation, contradicting DCCDA's decentralized actors; Theorem 1 may not apply to the proposed estimator.","rationale":"The reader's weakest assumption (idealistic communication) is about realism, which is a concern but not fatal given that the paper also provides a noisy-communication theorem. The more fundamental issue is internal consistency: the proof of Lemma 1, which underpins Theorem 1 and Corollary 1, uses a policy πi(a'i|haoi, m'−i) that depends on received messages in the critic's Bellman backup, while the DCCDA gradient estimator and the algorithms in Appendix D.2 use the decentralized policy πi(ai|hi, θi). If actors do not communicate, the backup must use πi(a'i|haoi); the fixed-point argument that identifies Em[Qi] with Qπ(h,a) is then not guaranteed. This is not a matter of whether Assumption 1 is realistic; it is a question of whether the theorem applies to the algorithm actually proposed. The numerical/tabular check would settle whether the proof can be repaired. Independent support: the paper provides detailed proofs and reproducible algorithm pseudocode, but no machine-checked proofs; the empirical variance measure (standard deviation of gradient norms across seeds) is also a different quantity from the theoretical estimator variance, so the experiments do not currently validate the central claim. Hence the verdict remains conditional pending a corrected derivation.","tokens_in":36604,"tokens_out":10815,"duration_ms":86958,"concrete_test":"Re-derive Appendix A with Eq. (5)'s next-policy replaced by the actual DCCDA policy πi(a'i|haoi). If the derivation of Qπ(h,a)=Em−i|h,a[Qi(hi,ai,m−i)] then fails—for instance, if the fixed-point argument requires the policy to condition on m'−i—Theorem 1's variance comparison does not hold for the proposed estimator. As a numerical check, implement a small tabular Dec-POMDP with Assumption 1 satisfied exactly (message = true (h−i,a−i)) and compare Var(ĝ_DCCDA) and Var(ĝ_CTDE) using (a) the paper's message-conditioned backup and (b) the true decentralized backup; only (a) should satisfy Theorem 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The DCCDA gradient estimator (Section 3.2) uses the decentralized policy πi(ai|hi,θi), and the paper states that actors do not communicate during training or execution. However, the Bellman equation for the communicating critic in Appendix A (Eq. 5) backs up with πi(a'i|haoi, m'−i), a policy that conditions on the next received messages. Line 7l of Lemma 1's proof explicitly factorizes the next-action distribution using this message-conditioned policy. If the actor policy is truly decentralized, the correct backup is πi(a'i|haoi). The proof's factorization then changes: the next-action distribution no longer depends on m'−i, and it is not demonstrated that Em−i|h,a[Qi(hi,ai,m−i)] satisfies the same Bellman equation as the centralized Qπ(h,a). Since Lemma 1 is the basis for the equal-expectation result used in Theorem 1, Theorem 1 and Corollary 1 may be proving variance properties of a different algorithm that uses communicating actors, not the DCCDA method the paper proposes and evaluates. This is an internal consistency issue, independent of the realism of Assumption 1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies variance in policy gradients for decentralized multi-agent deep reinforcement learning with communicating critics and decentralized actors (DCCDA). It claims that DCCDA sample gradients have variance at least as large as CTDE sample gradients under an idealistic perfect-communication assumption (Theorem 1) and under a binary-reward noisy communication model (Theorem 2). It then proposes a message-dependent optimal baseline (Theorem 3 and Corollary 1) plus a KL regularization term to align decentralized actors with communicating critics, and it reports improved win rates and lower gradient-norm variability on SMAC and Traffic Junction when extending GAAC and IPPO-Comm with these techniques.","tokens_in":36915,"tokens_out":8731,"duration_ms":81032,"significance":"If the theoretical results hold, the paper provides a useful first analysis of communication-induced variance in the DCCDA setting and a modular variance-reduction technique that transfers across algorithms. The empirical evaluation is reasonably broad, with ablations identifying the separate contributions of the baseline (OB) and the KL term, and the variance decomposition in Theorem 1 and the baseline derivation in Theorem 3 are explicit and non-circular. However, the central Lemma 1 proof currently uses a message-conditioned policy in the critic's Bellman backup even though DCCDA actors are decentralized, which means Theorem 1 may characterize a different algorithm than the one evaluated. The non-idealistic theorem also relies on a surrogate reward that depends on the unobserved true reward, limiting its reach. These issues are load-bearing for the paper's main claims, so the manuscript needs a substantive revision.","major_comments":[{"comment":"The Bellman equation for the communicating critic backs up with the message-conditioned policy pi_i(a'_i|hao_i,m'_{-i}), and line 7l factorizes the next-action distribution using exactly this policy. This contradicts the DCCDA definition in Section 3.2, where actors use decentralized policies pi_i(a_i|h_i) and do not communicate during training or execution. Consequently, Lemma 1, and with it Theorem 1 and Corollary 1, may only characterize a variant with communicating actors rather than the DCCDA estimator evaluated in the experiments. The authors should either re-derive the fixed-point equivalence using the decentralized backup pi_i(a'_i|hao_i) and prove that E_{m_{-i}|h,a}[Q^pi_i(hi,ai,m_{-i})] satisfies the centralized Bellman equation under that backup, or explicitly state and justify an alternative interpretation of Q^pi_i. The acknowledgement in Section 4.2 that non-communicating experience 'can mislead the training of communicating critics' does not repair the proof; it underscores the mismatch.","section":"Appendix A, Eq. (5) and line 7l"},{"comment":"The non-idealistic analysis restricts rewards to the binary set {r_+,r_-} and defines a surrogate reward hat{R}_i that depends on the true reward r_i, which is not observed by the agent once messages are noisy. The estimator hat{g}_{DCCDA-noise} built from the surrogate Q-function is therefore not the noisy DCCDA gradient that a practical agent would compute, and Theorem 2 does not establish the claimed variance ordering for the actual DCCDA estimator under noisy communication. The paper should state this limitation explicitly and, if the claim is intended to cover practical noisy communication, provide an analysis that does not require access to unobserved true rewards.","section":"Appendix B, Eq. (14) and Theorem 2"},{"comment":"The proof writes the noisy gradient with Q^pi_i(h,a,epsilon_i), whereas Lemma 3 and Eq. (16) define the surrogate object hat{Q}^pi_i(h,a,epsilon_i). The equality of expectations in (21c) is therefore not demonstrated as written, because the objects on the two sides do not match. Please correct the notation and verify that the variance comparison in (22) uses the same estimator throughout.","section":"Appendix B, Theorem 2 proof, derivations (21a) and (22f)"}],"minor_comments":[{"comment":"The reported quantity is the standard deviation of gradient norms across 8 seeds, which is a between-run statistic rather than the variance of the single-sample gradient estimators analyzed in Theorems 1--3; the text should clarify the relationship between this empirical measure and the theoretical variance, or report within-training gradient variance alongside it.","section":"Section 5.2, Table 1"},{"comment":"The theoretical gradient definitions use on-policy Q-values Q^pi_i, while the practical algorithms update critics by TD learning; the paper should state explicitly when Q_i is assumed to have converged to Q^pi_i, since Theorem 3 and Corollary 1 rely on that convergence.","section":"Section 3.2 and Algorithms 1--2"},{"comment":"COMA is listed with 0.0 win rate on several maps but with nonzero standard deviations, and e.g. '0.1 (0.2)' appears for 5m_vs_6m; please check the formatting and rounding so the table is internally consistent.","section":"Appendix D.4, Table 5"},{"comment":"The KL term is written as L_KL(theta_i) = -D_KL(...), and Eq. (3) then adds beta * grad_theta_i L_KL(theta_i) to the policy gradient. Since L_KL is the negative KL divergence, the sign is correct but non-standard; a one-sentence explanation of the sign convention would help readers avoid confusion.","section":"Equations (2)--(3)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and the empirical contribution is potentially useful, but I would not recommend acceptance before the Lemma 1 proof gap is addressed, because the main variance-ordering claim currently rests on a Bellman backup that uses communicating actors. The non-idealistic theorem's reliance on a surrogate reward that depends on the true reward should also be stated as a limitation in the main text. Given that the empirical claims are central and the main text says code will be released only upon acceptance, I suggest the editor require the code and configuration files as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the main theoretical result is likely not about the algorithm the paper evaluates. Lemma 1's proof (Appendix A) defines the communicating critic's Bellman backup with the next action drawn from πi(a'i | haoi, m'−i), a policy that conditions on next messages. But the paper's DCCDA setting (Section 3.2) explicitly keeps actors decentralized: gradients are computed against ∇ log πi(ai | hi). So the proof establishes an equality for a policy that sees messages before acting, not for the decentralized policy. Line 7l's factorization is the point: p(a'i | h, a, m−i, oi, m'−i) is set to πi(a'i | hi, ai, oi, m'−i), contradicting the paper's own actor definition. Theorem 1 and Corollary 1 therefore may be proving variance properties of a different algorithm. This is a load-bearing gap, not a minor misstatement.\n\nThat said, the paper has real merit. The question is well chosen—nobody has formally analyzed variance from communication in DCCDA before—and the proposed recipe (message-dependent baseline plus KL regularization) is clean and modular. The optimal baseline formula follows the standard derivation, and applying it to messages is a reasonable extension. The citation pattern is appropriate: it builds on Lyu et al. and Kuba et al. rather than ignoring them. The experiments are honestly executed: two base algorithms, two environments, ablations of OB and KL, and gradient-norm variance are reported. The gains on SMAC and Traffic Junction are consistent, and the ablation shows both components matter. If the theory were fixed, this would be a solid applied contribution.\n\nOther soft spots are smaller. The non-idealistic Theorem 2 is limited to binary rewards with a noise rate and uses a surrogate reward that depends on the unobserved true reward, so it is a narrow illustration rather than a general statement. The variance measurement in Table 1 is the standard deviation of gradient norms across seeds, a proxy, not the variance of the estimator itself. And the code is not provided, so the empirical claims cannot be independently checked.\n\nBottom line: this deserves a serious referee—the question and the recipe are worth engaging—but as it stands the central theoretical claim is not established for the DCCDA estimator the paper actually runs. Researchers working on communication in MARL will find the recipe useful; theorists will want to see the proof repaired. A referee should ask the authors to redo Lemma 1 with the decentralized policy's backup, or clearly restrict the theory to a communicating-actor variant.","headline":"Theorem 1's proof uses a message-conditioned next-action distribution, contradicting the paper's decentralized actors, so the main variance result may not apply to the evaluated estimator.","tokens_in":37339,"tokens_out":8758,"would_cite":false,"duration_ms":73389,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Communication between decentralized critics inflates policy-gradient variance, and a message-dependent baseline provably reduces it.","keywords":["multi-agent deep reinforcement learning","decentralized training","communication","variance reduction","policy gradient","baseline technique","KL regularization","DCCDA"],"falsifier":"Measure the variance of a single policy-gradient update in a DCCDA agent whose messages are deliberately uninformative (e.g., random noise) while the critic is fixed at its true on-policy values: the paper's inequality predicts the DCCDA gradient variance is still no smaller than the corresponding CTDE variance, so observing a strictly smaller variance at any step would contradict Theorem 1's scope.","tokens_in":36434,"feed_emoji":"📡","tokens_out":4867,"duration_ms":40577,"temperature":0.7,"pith_summary":"This paper claims that in decentralized multi-agent reinforcement learning where critics exchange messages during training (the DCCDA setting), communication itself is a source of variance in policy gradients, and that a carefully chosen baseline can remove part of that variance. It proves that the DCCDA sample gradient has variance at least as large as the centralized-training (CTDE) gradient, both when messages perfectly represent sender information and when they are corrupted by noise. The proposed cure is an optimal message-dependent baseline that leaves the expected gradient unchanged but lowers variance, plus a KL-divergence regularizer that aligns the non-communicating execution policy with the policy implied by the communicating critic. Experiments on StarCraft Multi-Agent Challenge and Traffic Junction show that the two techniques together reduce gradient variance and improve win rates.","feed_headline":"Communication inflates gradient variance; a baseline tames it","feed_subtitle":"Proven variance bounds for DCCDA training plus a message-dependent baseline that cuts noise and boosts win rates.","key_machinery":"The load-bearing objects are the decentralized communicating critic $Q_i(h_i,a_i,m_{-i})$, where $m_{-i}$ collects the messages from all other agents, and the message-dependent baseline $b_i(h_i,m_{-i})$. The critics are the only channel for communication during training; actors sample from $\\pi_i(\\cdot|h_i,\\theta_i)$ without seeing messages. The baseline is the variance-minimizing centering term derived in Theorem 3, and the KL term $D_{\\mathrm{KL}}(\\pi_i(\\cdot|h_i,\\theta_i)\\,\\|\\,\\mathrm{SoftMax}(Q_i(h_i,\\cdot,m_{-i})/\\alpha))$ anchors the actor to the communicating critic's implied policy. The analytical mechanism is a Bellman-uniqueness argument: under ideal communication the expected communicating critic equals the joint centralized critic, turning the variance comparison into a Jensen gap.","core_discovery":"On the paper's own terms, under the DCCDA setting — decentralized actors that never communicate, decentralized critics that do — the single-sample policy gradient $\\hat{g}_{\\mathrm{DCCDA}}^i$ has variance greater than or equal to the CTDE single-sample gradient $\\hat{g}_{\\mathrm{CTDE}}^i$. In the idealistic case, the equality $Q^\\pi(h,a)=\\mathbb{E}_{m_{-i}|h,a}[Q_i^\\pi(h_i,a_i,m_{-i})]$ (Lemma 1) converts the variance comparison into a Jensen gap, giving $\\mathrm{Var}(\\hat{g}_{\\mathrm{DCCDA}}^i)\\ge \\mathrm{Var}(\\hat{g}_{\\mathrm{CTDE}}^i)$. The proof extends to non-idealistic communication through a surrogate reward function, yielding $\\mathrm{Var}(\\hat{g}_{\\mathrm{DCCDA-noise}}^i)\\ge \\mathrm{Var}(\\hat{g}_{\\mathrm{CTDE}}^i)$. The constructive result is the optimal message-dependent baseline $b_i^*(h_i,m_{-i})=\\mathbb{E}_{a_i}[Q_i(h_i,a_i,m_{-i})S]/\\mathbb{E}_{a_i}[S]$, where $S=\\nabla_{\\theta_i}\\log \\pi_i(a_i|h_i,\\theta_i)^\\top \\nabla_{\\theta_i}\\log \\pi_i(a_i|h_i,\\theta_i)$; this baseline is unbiased and cuts the variance by a non-negative squared term (Corollary 1). A KL regularizer then keeps the decentralized execution policy close to the policy suggested by the communicating critic, which the paper shows empirically improves both learning speed and final performance.","pith_inferences":["If the variance inequality holds in other environments, variance reduction in communicating critics should be treated as a first-class design goal rather than a side effect of better critics.","The ideal-communication equality suggests a stronger statement may hold for any message distribution that makes the critic conditionally independent of the sender's history, a relaxation worth testing.","An adaptive temperature or scaling factor for the KL term would be a natural extension, since the paper's own ablation shows the regularizer can hurt when the critic's implied policy is unreliable.","The binary-reward noise model in Theorem 2 could likely be generalized, but continuous rewards may require a different variance decomposition because the surrogate-reward construction relies on flipping a binary reward."],"forward_implications":["The variance inequality identifies communication as a genuine cost in DCCDA training: even perfect information, once sent stochastically, makes gradient estimates noisier than a centralized critic that receives everything.","The optimal message-dependent baseline is model-agnostic, so it applies to any stochastic message function and can extend existing DCCDA methods without redesigning their communication protocols.","The KL regularizer matters empirically: removing either the baseline or the regularizer lowers win rates on most tested maps, and the two together deliver lower gradient-norm variance than the underlying DCCDA methods.","Because the baseline is computed from critic values and messages already stored in the replay buffer, it adds negligible compute overhead and preserves decentralized execution."],"supporting_citations":[{"why":"Supplies the CTDE-versus-DTDE variance analysis framework and the Bellman fixed-point argument that Lemma 1 adapts to communicating critics.","marker":"[16]"},{"why":"Defines the DCCDA setting with decentralized communicating critics and decentralized actors that this paper analyzes.","marker":"[12]"},{"why":"Provides GAAC, one of the two DCCDA algorithms that the proposed baseline and KL regularizer are tested on.","marker":"[13]"},{"why":"Supplies the optimal-baseline derivation technique and the analytical inner-product form for softmax policies used in Theorem 3.","marker":"[33]"},{"why":"Introduces the surrogate reward function that the non-idealistic communication analysis (Theorem 2) builds on.","marker":"[39]"},{"why":"Provides IPPO, the base algorithm extended with a communication architecture to form IPPO-Comm, and the MAPPO optimization strategy used in experiments.","marker":"[38]"}],"fun_headline_variants":["Communication hikes gradient variance; a baseline brings it down","Tame the variance that communication adds to policy gradients","How to slash variance from communication in decentralized MARL","A fix for variance caused by communication in multi-agent RL","Cut the gradient variance that communication brings to MADRL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument requires that received messages are perfect representations of the sender's local history and action (Assumption 1) and that learned critics converge to true on-policy values; with learned, noisy message encoders neither condition is exactly met, and the non-idealistic theorem additionally assumes a binary reward structure.","fun_headline_variants_meta":{"raw":{"variants":["Communication hikes gradient variance; a baseline brings it down","Tame the variance that communication adds to policy gradients","How to slash variance from communication in decentralized MARL","A fix for variance caused by communication in multi-agent RL","Cut the gradient variance that communication brings to MADRL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1503,"prompt_tokens":1021,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":637,"tokens_out":482,"duration_ms":4923,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:13:07.586087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the variance of a single policy-gradient update in a DCCDA agent whose messages are deliberately uninformative (e.g., random noise) while the critic is fixed at its true on-policy values: the paper's inequality predicts the DCCDA gradient variance is still no smaller than the corresponding CTDE variance, so observing a strictly smaller variance at any step would contradict Theorem 1's scope.","supporting_citations":[{"cited_title":"On centralized critics in multi-agent reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the CTDE-versus-DTDE variance analysis framework and the Bellman fixed-point argument that Lemma 1 adapts to communicating critics."},{"cited_title":"Actor-attention-critic for multi-agent reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Defines the DCCDA setting with decentralized communicating critics and decentralized actors that this paper analyzes."},{"cited_title":"Multi-agent game abstraction via graph attention neural network","cited_arxiv_id":null,"evidence_quote":"Provides GAAC, one of the two DCCDA algorithms that the proposed baseline and KL regularizer are tested on."},{"cited_title":"Settling the variance of multi-agent policy gradients","cited_arxiv_id":null,"evidence_quote":"Supplies the optimal-baseline derivation technique and the analytical inner-product form for softmax policies used in Theorem 3."},{"cited_title":"Reinforcement learning with perturbed rewards","cited_arxiv_id":null,"evidence_quote":"Introduces the surrogate reward function that the non-idealistic communication analysis (Theorem 2) builds on."},{"cited_title":"Bayen, and Yi Wu","cited_arxiv_id":null,"evidence_quote":"Provides IPPO, the base algorithm extended with a communication architecture to form IPPO-Comm, and the MAPPO optimization strategy used in experiments."}],"review_version":1}