{"id":"aca72535-2ee6-4d65-a76c-8b4db39c110c","arxiv_id":"2505.21985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MARL-CPC lets decentralized agents learn to send informative messages through a self-supervised reconstruction objective, and outperforms message-as-action baselines in non-cooperative multi-agent tasks.","lead":"This paper proposes MARL-CPC, a training method that lets independent AI agents learn useful communication without needing a shared reward or any direct incentive to help each other. The method trains each agent to compress its own observations into messages, and other agents learn to read those messages to improve their decisions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not compare against its own stated reduction to Lin et al.'s autoencoder, so the observed gains may be attributable to the generic reconstruction loss rather than to the CPC-specific joint decoder and KL term.","rationale":"The reader's verdict is CONDITIONAL and lists the autoencoder baseline as one condition, but the weakest_assumption focuses on receiver decodability in complex settings. I think the most load-bearing issue is more immediate: the paper's own Section 3.2 identifies a parameter setting (β=0, decoder restricted to own message) that reduces MARL-CPC to a previously published autoencoder method, yet no experiment isolates the contribution of the CPC-specific ingredients. The central empirical claim—that MARL-CPC establishes communication in non-cooperative settings—could hold even if the distinctive CPC machinery is inert, because the reconstruction loss alone is sufficient to make each sender's message a code for its own observation, and the receiver's RL objective can learn to decode that code. The current comparisons against no-comm and message-as-action cannot rule out this alternative explanation. This is a threat to attribution rather than to internal correctness: the math in Eqs. 5-8 checks out, and the reported improvements are plausible. But for a paper whose contribution is a new framework, demonstrating that the framework's distinctive parts matter is essential. I therefore set verdict_should_be to UNCHANGED (keeping the reader's CONDITIONAL) because the missing baseline is precisely the kind of control the reader should demand. Agreement: partial—the reader flagged the comparison as a condition but did not make it the core concern.","tokens_in":10259,"tokens_out":11862,"duration_ms":126688,"concrete_test":"Re-run the bandit and observer experiments with the reduced model defined in §3.2: decoder Pθi(xi|m) = Pθi(xi|mi) (own message only), KL weight β=0, other settings identical. Also run cpc with KL weight β=0 as an intermediate. If the reduced model reaches the same welfare and episode length as cpc, the observed advantage over message-as-action is not attributable to the CPC-specific components; the paper must reposition its claim. If cpc clearly outperforms the reduced model, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that with β=0 and Pθi(xi|m)≜Pθi(xi|mi), MARL-CPC reduces to the autoencoder method of Lin et al. [23]. The experiments (Figures 5, 7, 8) compare only against no-comm and message-as-action baselines; they never run this reduced model or any ablation that removes the KL term or restricts the decoder to the agent's own message. Because the sender's reconstruction loss alone already forces each message mi to encode its own observation xi, a receiver trained by RL can decode mi. Thus the empirical success in the two non-cooperative environments may be entirely due to this generic autoencoder pressure. If the reduced model performs as well, then the paper's central claim that the CPC formulation (joint decoder, KL term, state-estimation interpretation) is what enables reward-independent communication is not supported by the data. The missing baseline is therefore the most load-bearing gap for the paper's contribution as framed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MARL-CPC, a framework for decentralized multi-agent reinforcement learning in which agents learn to send discrete messages through a collective predictive coding (CPC) module rather than by treating messages as actions. The authors derive an ELBO for a factorized generative model, decompose it into agent-wise objectives, and combine the resulting CPC loss with either a contextual-bandit objective (Bandit-CPC) or an independent PPO objective (IPPO-CPC). Experiments in a two-agent contextual bandit and a four-by-four grid 'observer' environment compare no-communication, message-as-action, CPC, and fully shared baselines. The reported results show that CPC-based agents outperform the no-communication and message-as-action baselines in non-cooperative settings, and an ablation shows that replacing or zeroing the trained messages degrades performance.","tokens_in":10553,"tokens_out":7764,"duration_ms":82879,"significance":"If the empirical findings hold, the paper makes a useful contribution to decentralized MARL by demonstrating a mechanism for communication that does not rely on the sender receiving a reward for its message. The authors provide two complete algorithms, report IQM with bootstrapped confidence intervals, include an ablation study, and clearly describe the relationship to prior autoencoder-based communication methods. The main conceptual value is in formulating messaging as variational inference over a joint generative model and in showing that this can work in settings where message-as-action approaches fail. However, the evidence for the specific CPC mechanism is incomplete: the paper never compares against its own stated reduction to the autoencoder method of Lin et al., and the KL approximation used in the derivation is biased. These gaps prevent the current version from fully supporting the claim that the CPC formulation, rather than generic reconstruction pressure, is responsible for the observed communication.","major_comments":[{"comment":"The central novelty claim is not tested against the paper's own stated reduction. Section 3.2 says that with beta=0 and P_theta_i(x_i|m) constrained to P_theta_i(x_i|m_i) the method reduces to the autoencoder method of Lin et al., but neither this reduced model nor an ablation that separately removes the joint decoder or the KL term is run in the experiments. Because the sender's reconstruction loss alone already forces each message m_i to encode its own observation x_i, a receiver trained by RL can decode m_i, so the observed gains in Figures 5, 7, 8, and 9 could be entirely due to generic autoencoding pressure. Add the reduced-model baseline and the two component ablations in both environments; if the reduced model performs as well as MARL-CPC, the conclusion that the CPC formulation (joint decoder, KL term, state-estimation interpretation) enables reward-independent communication is not supported by the data.","section":"Section 3.2 and Figures 5, 7, 8, 9"},{"comment":"The KL approximation in Eq. (11) is not a valid estimator of the divergence it claims to approximate. Since the message m is sampled from Q_phi(m|x) in Eq. (9), the expectation of (kappa-1)-log kappa under Q_phi equals E_Q[Q_phi/P] - 1 - D_KL(Q_phi||P), which is generally not D_KL(Q_phi||P); sampling from P would estimate the reverse direction D_KL(P||Q_phi). With a flat prior and one-hot categorical Q, the bias is generally nonzero. The paper therefore optimizes a quantity different from the ELBO-derived KL penalty. Use the unbiased single-sample estimator log kappa, or sample from P and clearly state that the reverse KL is being used, and report an ablation of the KL term to show its empirical role.","section":"Equations (8)-(12)"},{"comment":"The statement that 'the message variable m ... corresponds to a state estimation of the entire environment s' is an overclaim. The variational posterior Q_phi(m|x) is a latent-variable posterior in the generative model of Eq. (1); without identifiability or other consistency conditions, there is no guarantee that m tracks the true state s. In the experiments, both observations are one-hot encodings of the two components of a factorized state, so the concatenated message can trivially encode the full state, but this does not justify the general statement. Rephrase as 'the message variable is a latent variable that can support state inference' and either add an identifiability discussion or test an environment with overlapping or ambiguous observations.","section":"Section 3.2, after Eq. (7)"}],"minor_comments":[{"comment":"The notation m(x_i) = (m_1, ..., tilde{m_i}(x_i), ..., m_K) uses K for both the vocabulary size and the number of agents; with two agents and K=5 this is inconsistent. Use N for the number of agents.","section":"Section 3.2, Eq. (10)"},{"comment":"The caption of Figure 8 says 'Overview of observer environment' but the figure displays the ablation results; the caption should be updated accordingly.","section":"Figure 8 caption"},{"comment":"Line 6 says 'for all mini-matches do'; this should be 'mini-batches'.","section":"Algorithm 1"},{"comment":"The term 'reward-independent' should be qualified: the sender's message generation is reward-independent, but the receiver's use of the message is trained through the RL objective. The current phrasing in the abstract and conclusions could be read as claiming that the whole communication channel is learned without any reward signal, which is not the case.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites Lin et al. as a baseline and even states that the method reduces to it, but no such comparison is run. Since the missing baseline directly affects the interpretation of all reported gains, I would ask for it before sending the paper back for a final decision. The KL estimator issue is also technical and should be fixed or explicitly discussed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine, well-scoped step forward — variational CPC for decentralized MARL is new, the ELBO derivation is correct, and the non-cooperative evaluation setting is a real extension. But the paper skips the baseline that would actually distinguish the CPC machinery from a plain autoencoder, and that omission is load-bearing for the central claim.\n\nWhat's new and good: the shift from MCMC-based CPC to variational inference with deep nets is real and useful. Equations (5)-(7) decompose the ELBO into per-agent terms cleanly, and the straight-through estimator for discrete messages is standard and appropriate. The two environments are simple but well chosen to show communication emerging without incentive alignment. Using IQM with bootstrapped CIs is good practice, and the ablation (random/zero messages) shows the messages are being used. That is honest evidence.\n\nSoft spots: first, the missing baseline. Section 3.2 explicitly says that with beta=0 and a decoder restricted to the sender's own message, MARL-CPC reduces to Lin et al.'s autoencoder method. The experiments never run that reduced model, nor an ablation that removes the KL term or restricts the decoder. Because the sender's reconstruction loss already forces each message to encode its own observation, a receiver trained by RL can plausibly decode it. If the reduced model performs the same, then the CPC-specific machinery — the joint decoder and KL term — is not what's doing the work, and the paper's framed contribution is overreaching. This is not a technical error but a gap in experimental support for the claim.\n\nSecond, Eq. (11) is a biased estimator of the KL. The paper cites Schulman's blog for it, but it's an approximation; this doesn't sink the method, but it should be acknowledged and ideally replaced with an unbiased estimator.\n\nThird, the claim that m corresponds to a state estimate of the environment is an over-interpretation. In the bandit, the informed agent's observation is a one-hot encoding of the true state, so the message can be a state estimate in that toy setting, but that doesn't follow from the ELBO generally.\n\nFourth, the observations are one-hot. That's fine for a proof of concept, but the reader's worry about decodability is fair: in richer observation spaces, a self-reconstruction objective can produce codes unreadable by the separately trained receiver.\n\nThe citation pattern is fine; self-citations are to the prior CPC literature that the paper extends, and the algorithms stand independently. No circular fitting.\n\nBottom line: this is a solid incremental paper that deserves referee time. The missing baseline is the one thing I'd want before believing the specific CPC-attribution claim. Code and statistical test details should also be released. If the reduced model matches, the paper becomes a much weaker claim; if it doesn't, the paper's story is supported.","headline":"Solid variational CPC framework for decentralized MARL with a clean derivation, but the missing autoencoder-reduction baseline leaves the central attribution claim unsupported until that experiment is run.","tokens_in":10983,"tokens_out":2562,"would_cite":true,"duration_ms":25332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MARL-CPC lets fully decentralized agents establish functional communication even when the sender gains nothing from sending.","keywords":["multi-agent reinforcement learning","emergent communication","collective predictive coding","variational inference","decentralized learning","non-cooperative environments","independent agents","state estimation"],"falsifier":"Train the same algorithms on the same tasks but with high-dimensional pixel observations instead of one-hot state encodings; if the receiver's performance stops improving over the no-communication baseline, the claim that the sender's self-reconstruction objective produces decodable messages is refuted. A sharper measurement is to compute, after training, the mutual information between the sender's message and the receiver's estimate of the true state, since near-zero mutual information would directly contradict the paper's claim that the message is a state estimate.","tokens_in":10065,"feed_emoji":"🗣️","tokens_out":9263,"duration_ms":79114,"temperature":0.7,"pith_summary":"This paper tries to establish that communication can arise among fully decentralized reinforcement-learning agents even when the sender receives no reward for helping and the agents do not cooperate. The proposed framework, MARL-CPC, treats a message as a latent variable in a joint generative model of all agents' observations rather than as an action chosen for a reward, and it decomposes the resulting evidence lower bound into one objective per agent. Two algorithms, Bandit-CPC and IPPO-CPC, implement this idea for contextual bandits and for sequential tasks. In non-cooperative experiments, agents using the framework outperform message-as-action baselines and approach the performance of a shared-observation upper bound. The paper concludes that messages learned this way function as estimates of the environmental state, so communication emerges independently of reward-driven signaling.","feed_headline":"Decentralized agents learn to talk even when senders gain nothing","feed_subtitle":"MARL-CPC lets independent agents talk without parameter sharing or cooperative rewards.","key_machinery":"The carrying object is the CPC module: a joint generative model over all agents' observations and discrete messages, $P_\\theta(x,m)=P(m)\\prod_{i=1}^N P_{\\theta_i}(x_i\\mid m)$, with a factorized variational distribution $Q_\\phi(m\\mid x)=\\prod_i Q_{\\phi_i}(m_i\\mid x_i)$ supplied by each agent's message encoder. The identity that makes it work is the agent-wise decomposition of the ELBO, $\\log P_\\theta(x)\\ge \\sum_i J_{\\mathrm{CPC}}(\\theta_i,\\phi_i)$, where $J_{\\mathrm{CPC}}$ contains a reconstruction term $\\log P_{\\theta_i}(x_i\\mid m)$ and a KL term $D_{\\mathrm{KL}}(Q_{\\phi_i}(m_i\\mid x_i)\\|P(m_i))$. Because the reconstruction is of the agent's own observation from the concatenated message vector, the message becomes an auxiliary variable for global state inference, and straight-through gradients make the discrete messages trainable. The RL loss, a bandit likelihood or PPO objective, is optimized in parallel with no gradient flowing through the CPC module, so communication learning is decoupled from reward.","core_discovery":"The central claim is that MARL-CPC lets independent agents learn to communicate without parameter sharing, centralized training, or a cooperative reward structure. In the CPC module, a pseudo-joint generative model $P_\\theta(x,m)=P(m)\\prod_{i=1}^N P_{\\theta_i}(x_i\\mid m)$ ties all observations to a shared message vector, and each agent maximizes its own evidence lower bound $J_{\\mathrm{CPC}}(\\theta_i,\\phi_i)=\\mathbb{E}_{Q_\\phi(m\\mid x)}[\\log P_{\\theta_i}(x_i\\mid m)]-D_{\\mathrm{KL}}(Q_{\\phi_i}(m_i\\mid x_i)\\|P(m_i))$. Optimizing this objective makes the concatenated messages $m$ an estimate of the global state $s$, even though each agent's decoder only reconstructs that agent's own observation $x_i$. The RL loss is added separately and gradients do not flow between the RL and CPC modules, so the sender has no direct incentive to encode information for the receiver, yet the receiver's policy learns to use the message as input. Experiments in a non-cooperative contextual bandit and an observer grid-world show that this yields communication that helps the group, while message-as-action baselines fail.","pith_inferences":["Editorial inference: the decodability of the sender's message is only tested with one-hot observations, so a natural stress test is to move to raw images or high-dimensional observations; the paper itself does not condition the sender's objective on receiver decoding success.","Editorial inference: because messages are interpreted as global state estimates, they could plausibly be fed into a learned world model or used for planning by the receiver, although the paper only uses them as policy and value inputs.","Editorial inference: the framework presumes agents are not in direct competition; under strictly adversarial rewards, a sender would have an incentive to distort the shared state estimate, and nothing in the predictive-coding objective prevents deceptive codes."],"forward_implications":["Fully decentralized agents with no shared parameters can establish functional communication by each optimizing only a local predictive-coding objective plus their own RL loss.","Message-as-action communication, which fails in the tested non-cooperative settings, can be replaced by treating messages as state estimates, supporting communication when the sender has no reward incentive to signal.","The framework is a strict generalization of autoencoder-based message learning: zeroing the KL term and restricting the decoder to the agent's own message recovers the earlier method.","In the observer environment, an agent that receives zero reward still learns to send messages, and the receiving agent uses them to improve task performance; ablations show that randomizing or zeroing the messages causes a clear performance drop.","The same CPC objective drives communication in both a one-step bandit and a longer sequential task via IPPO-CPC, indicating the mechanism is not restricted to stateless settings."],"supporting_citations":[{"why":"introduces collective predictive coding, the emergent-communication theory that MARL-CPC adapts to deep RL.","marker":"[40]"},{"why":"shows emergent communication via deep generative models and a Metropolis-Hastings naming game, motivating the pseudo-joint generative model.","marker":"[16]"},{"why":"the autoencoder-based multi-agent communication method whose objective MARL-CPC generalizes when the KL term is zeroed and reconstruction uses only the agent's own message.","marker":"[23]"},{"why":"the deep MARL message-as-action baseline that MARL-CPC outperforms in non-cooperative tests.","marker":"[9]"},{"why":"an earlier message-as-action emergent-communication model used as a comparison baseline.","marker":"[5]"},{"why":"defines non-cooperative multi-agent settings with unequal reward functions, framing the problem studied here.","marker":"[4]"},{"why":"supplies the PPO algorithm that IPPO-CPC extends with the CPC module.","marker":"[34]"},{"why":"the straight-through gradient estimator used to train the discrete message variable.","marker":"[3]"}],"fun_headline_variants":["Talking for free: decentralized agents learn to communicate","No shared rewards, no shared weights—yet they communicate","Communication without reward: MARL-CPC lets agents share state","Agents that talk even when it pays them nothing","Decentralized agents coordinate via messages, no reward needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a receiver can learn to extract useful information from a message even though the sender's training objective only rewards reconstructing the sender's own observation, something the experiments make easy with simple one-hot observations.","fun_headline_variants_meta":{"raw":{"variants":["Talking for free: decentralized agents learn to communicate","No shared rewards, no shared weights—yet they communicate","Communication without reward: MARL-CPC lets agents share state","Agents that talk even when it pays them nothing","Decentralized agents coordinate via messages, no reward needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1423,"prompt_tokens":940,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":556,"tokens_out":483,"duration_ms":5623,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:19:49.706870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same algorithms on the same tasks but with high-dimensional pixel observations instead of one-hot state encodings; if the receiver's performance stops improving over the no-communication baseline, the claim that the sender's self-reconstruction objective produces decodable messages is refuted. A sharper measurement is to compute, after training, the mutual information between the sender's message and the receiver's estimate of the true state, since near-zero mutual information would directly contradict the paper's claim that the message is a state estimate.","supporting_citations":[{"cited_title":"Frontiers in Robotics and AI11, 1353870 (2024)","cited_arxiv_id":null,"evidence_quote":"introduces collective predictive coding, the emergent-communication theory that MARL-CPC adapts to deep RL."},{"cited_title":"Frontiers in Robotics and AI10, 1290604 (2024)","cited_arxiv_id":null,"evidence_quote":"shows emergent communication via deep generative models and a Metropolis-Hastings naming game, motivating the pseudo-joint generative model."},{"cited_title":"Advances in Neural Information Processing Systems 34, 15230–15242 (2021)","cited_arxiv_id":null,"evidence_quote":"the autoencoder-based multi-agent communication method whose objective MARL-CPC generalizes when the KL term is zeroed and reconstruction uses only the agent's own message."},{"cited_title":"Advances in neural information pro- cessing systems 29 (2016)","cited_arxiv_id":null,"evidence_quote":"the deep MARL message-as-action baseline that MARL-CPC outperforms in non-cooperative tests."},{"cited_title":"Connection Science10(2), 83–97 (1998)","cited_arxiv_id":null,"evidence_quote":"an earlier message-as-action emergent-communication model used as a comparison baseline."},{"cited_title":"IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)38(2), 156–172 (2008)","cited_arxiv_id":null,"evidence_quote":"defines non-cooperative multi-agent settings with unequal reward functions, framing the problem studied here."}],"review_version":1}