Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning

Naoto Yoshida; Tadahiro Taniguchi

REVIEW 3 major objections 4 minor 1 cited by

Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MARL-CPC lets fully decentralized agents establish functional communication even when the sender gains nothing from sending.

desk verdict Solid variational CPC framework for decentralized MARL with a clean derivation, but the missing autoencoder-reduction baseline leaves the central attribution claim unsupported until that experiment is run. read the letter →

arxiv 2505.21985 v1 pith:LRCX72Z3 submitted 2025-05-28 cs.MA cs.AIcs.LG

Naoto Yoshida , Tadahiro Taniguchi This is my paper

classification cs.MAcs.AIcs.LG

keywords multi-agentreinforcementlearningemergentcommunicationcollectivepredictivecodingvariationalinferencedecentralizednon-cooperativeenvironmentsindependentagentsstateestimation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that communication can arise among fully decentralized reinforcement-learning agents even when the sender receives no reward for helping and the agents do not cooperate. The proposed framework, MARL-CPC, treats a message as a latent variable in a joint generative model of all agents' observations rather than as an action chosen for a reward, and it decomposes the resulting evidence lower bound into one objective per agent. Two algorithms, Bandit-CPC and IPPO-CPC, implement this idea for contextual bandits and for sequential tasks. In non-cooperative experiments, agents using the framework outperform message-as-action baselines and approach the performance of a shared-observation upper bound. The paper concludes that messages learned this way function as estimates of the environmental state, so communication emerges independently of reward-driven signaling.

What carries the argument

The carrying object is the CPC module: a joint generative model over all agents' observations and discrete messages, $P_\theta(x,m)=P(m)\prod_{i=1}^N P_{\theta_i}(x_i\mid m)$, with a factorized variational distribution $Q_\phi(m\mid x)=\prod_i Q_{\phi_i}(m_i\mid x_i)$ supplied by each agent's message encoder. The identity that makes it work is the agent-wise decomposition of the ELBO, $\log P_\theta(x)\ge \sum_i J_{\mathrm{CPC}}(\theta_i,\phi_i)$, where $J_{\mathrm{CPC}}$ contains a reconstruction term $\log P_{\theta_i}(x_i\mid m)$ and a KL term $D_{\mathrm{KL}}(Q_{\phi_i}(m_i\mid x_i)\|P(m_i))$. Because the reconstruction is of the agent's own observation from the concatenated message vector, the message becomes an auxiliary variable for global state inference, and straight-through gradients make the discrete messages trainable. The RL loss, a bandit likelihood or PPO objective, is optimized in parallel with no gradient flowing through the CPC module, so communication learning is decoupled from reward.

What would settle it

Train the same algorithms on the same tasks but with high-dimensional pixel observations instead of one-hot state encodings; if the receiver's performance stops improving over the no-communication baseline, the claim that the sender's self-reconstruction objective produces decodable messages is refuted. A sharper measurement is to compute, after training, the mutual information between the sender's message and the receiver's estimate of the true state, since near-zero mutual information would directly contradict the paper's claim that the message is a state estimate.

Watch

Extended reading notes

Core claim

The central claim is that MARL-CPC lets independent agents learn to communicate without parameter sharing, centralized training, or a cooperative reward structure. In the CPC module, a pseudo-joint generative model $P_\theta(x,m)=P(m)\prod_{i=1}^N P_{\theta_i}(x_i\mid m)$ ties all observations to a shared message vector, and each agent maximizes its own evidence lower bound $J_{\mathrm{CPC}}(\theta_i,\phi_i)=\mathbb{E}_{Q_\phi(m\mid x)}[\log P_{\theta_i}(x_i\mid m)]-D_{\mathrm{KL}}(Q_{\phi_i}(m_i\mid x_i)\|P(m_i))$. Optimizing this objective makes the concatenated messages $m$ an estimate of the global state $s$, even though each agent's decoder only reconstructs that agent's own observation $x_i$. The RL loss is added separately and gradients do not flow between the RL and CPC modules, so the sender has no direct incentive to encode information for the receiver, yet the receiver's policy learns to use the message as input. Experiments in a non-cooperative contextual bandit and an observer grid-world show that this yields communication that helps the group, while message-as-action baselines fail.

Load-bearing premise

The load-bearing premise is that a receiver can learn to extract useful information from a message even though the sender's training objective only rewards reconstructing the sender's own observation, something the experiments make easy with simple one-hot observations.

Editorial extensions

If this is right

Fully decentralized agents with no shared parameters can establish functional communication by each optimizing only a local predictive-coding objective plus their own RL loss.
Message-as-action communication, which fails in the tested non-cooperative settings, can be replaced by treating messages as state estimates, supporting communication when the sender has no reward incentive to signal.
The framework is a strict generalization of autoencoder-based message learning: zeroing the KL term and restricting the decoder to the agent's own message recovers the earlier method.
In the observer environment, an agent that receives zero reward still learns to send messages, and the receiving agent uses them to improve task performance; ablations show that randomizing or zeroing the messages causes a clear performance drop.
The same CPC objective drives communication in both a one-step bandit and a longer sequential task via IPPO-CPC, indicating the mechanism is not restricted to stateless settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the decodability of the sender's message is only tested with one-hot observations, so a natural stress test is to move to raw images or high-dimensional observations; the paper itself does not condition the sender's objective on receiver decoding success.
Editorial inference: because messages are interpreted as global state estimates, they could plausibly be fed into a learned world model or used for planning by the receiver, although the paper only uses them as policy and value inputs.
Editorial inference: the framework presumes agents are not in direct competition; under strictly adversarial rewards, a sender would have an incentive to distort the shared state estimate, and nothing in the predictive-coding objective prevents deceptive codes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuine, well-scoped step forward — variational CPC for decentralized MARL is new, the ELBO derivation is correct, and the non-cooperative evaluation setting is a real extension. But the paper skips the baseline that would actually distinguish the CPC machinery from a plain autoencoder, and that omission is load-bearing for the central claim.

What's new and good: the shift from MCMC-based CPC to variational inference with deep nets is real and useful. Equations (5)-(7) decompose the ELBO into per-agent terms cleanly, and the straight-through estimator for discrete messages is standard and appropriate. The two environments are simple but well chosen to show communication emerging without incentive alignment. Using IQM with bootstrapped CIs is good practice, and the ablation (random/zero messages) shows the messages are being used. That is honest evidence.

Soft spots: first, the missing baseline. Section 3.2 explicitly says that with beta=0 and a decoder restricted to the sender's own message, MARL-CPC reduces to Lin et al.'s autoencoder method. The experiments never run that reduced model, nor an ablation that removes the KL term or restricts the decoder. Because the sender's reconstruction loss already forces each message to encode its own observation, a receiver trained by RL can plausibly decode it. If the reduced model performs the same, then the CPC-specific machinery — the joint decoder and KL term — is not what's doing the work, and the paper's framed contribution is overreaching. This is not a technical error but a gap in experimental support for the claim.

Second, Eq. (11) is a biased estimator of the KL. The paper cites Schulman's blog for it, but it's an approximation; this doesn't sink the method, but it should be acknowledged and ideally replaced with an unbiased estimator.

Third, the claim that m corresponds to a state estimate of the environment is an over-interpretation. In the bandit, the informed agent's observation is a one-hot encoding of the true state, so the message can be a state estimate in that toy setting, but that doesn't follow from the ELBO generally.

Fourth, the observations are one-hot. That's fine for a proof of concept, but the reader's worry about decodability is fair: in richer observation spaces, a self-reconstruction objective can produce codes unreadable by the separately trained receiver.

The citation pattern is fine; self-citations are to the prior CPC literature that the paper extends, and the algorithms stand independently. No circular fitting.

Bottom line: this is a solid incremental paper that deserves referee time. The missing baseline is the one thing I'd want before believing the specific CPC-attribution claim. Code and statistical test details should also be released. If the reduced model matches, the paper becomes a much weaker claim; if it doesn't, the paper's story is supported.

Referee Report

3 major / 4 minor

Summary. The paper proposes MARL-CPC, a framework for decentralized multi-agent reinforcement learning in which agents learn to send discrete messages through a collective predictive coding (CPC) module rather than by treating messages as actions. The authors derive an ELBO for a factorized generative model, decompose it into agent-wise objectives, and combine the resulting CPC loss with either a contextual-bandit objective (Bandit-CPC) or an independent PPO objective (IPPO-CPC). Experiments in a two-agent contextual bandit and a four-by-four grid 'observer' environment compare no-communication, message-as-action, CPC, and fully shared baselines. The reported results show that CPC-based agents outperform the no-communication and message-as-action baselines in non-cooperative settings, and an ablation shows that replacing or zeroing the trained messages degrades performance.

Significance. If the empirical findings hold, the paper makes a useful contribution to decentralized MARL by demonstrating a mechanism for communication that does not rely on the sender receiving a reward for its message. The authors provide two complete algorithms, report IQM with bootstrapped confidence intervals, include an ablation study, and clearly describe the relationship to prior autoencoder-based communication methods. The main conceptual value is in formulating messaging as variational inference over a joint generative model and in showing that this can work in settings where message-as-action approaches fail. However, the evidence for the specific CPC mechanism is incomplete: the paper never compares against its own stated reduction to the autoencoder method of Lin et al., and the KL approximation used in the derivation is biased. These gaps prevent the current version from fully supporting the claim that the CPC formulation, rather than generic reconstruction pressure, is responsible for the observed communication.

major comments (3)

[Section 3.2 and Figures 5, 7, 8, 9] The central novelty claim is not tested against the paper's own stated reduction. Section 3.2 says that with beta=0 and P_theta_i(x_i|m) constrained to P_theta_i(x_i|m_i) the method reduces to the autoencoder method of Lin et al., but neither this reduced model nor an ablation that separately removes the joint decoder or the KL term is run in the experiments. Because the sender's reconstruction loss alone already forces each message m_i to encode its own observation x_i, a receiver trained by RL can decode m_i, so the observed gains in Figures 5, 7, 8, and 9 could be entirely due to generic autoencoding pressure. Add the reduced-model baseline and the two component ablations in both environments; if the reduced model performs as well as MARL-CPC, the conclusion that the CPC formulation (joint decoder, KL term, state-estimation interpretation) enables reward-independent communication is not supported by the data.
[Equations (8)-(12)] The KL approximation in Eq. (11) is not a valid estimator of the divergence it claims to approximate. Since the message m is sampled from Q_phi(m|x) in Eq. (9), the expectation of (kappa-1)-log kappa under Q_phi equals E_Q[Q_phi/P] - 1 - D_KL(Q_phi||P), which is generally not D_KL(Q_phi||P); sampling from P would estimate the reverse direction D_KL(P||Q_phi). With a flat prior and one-hot categorical Q, the bias is generally nonzero. The paper therefore optimizes a quantity different from the ELBO-derived KL penalty. Use the unbiased single-sample estimator log kappa, or sample from P and clearly state that the reverse KL is being used, and report an ablation of the KL term to show its empirical role.
[Section 3.2, after Eq. (7)] The statement that 'the message variable m ... corresponds to a state estimation of the entire environment s' is an overclaim. The variational posterior Q_phi(m|x) is a latent-variable posterior in the generative model of Eq. (1); without identifiability or other consistency conditions, there is no guarantee that m tracks the true state s. In the experiments, both observations are one-hot encodings of the two components of a factorized state, so the concatenated message can trivially encode the full state, but this does not justify the general statement. Rephrase as 'the message variable is a latent variable that can support state inference' and either add an identifiability discussion or test an environment with overlapping or ambiguous observations.

minor comments (4)

[Section 3.2, Eq. (10)] The notation m(x_i) = (m_1, ..., tilde{m_i}(x_i), ..., m_K) uses K for both the vocabulary size and the number of agents; with two agents and K=5 this is inconsistent. Use N for the number of agents.
[Figure 8 caption] The caption of Figure 8 says 'Overview of observer environment' but the figure displays the ablation results; the caption should be updated accordingly.
[Algorithm 1] Line 6 says 'for all mini-matches do'; this should be 'mini-batches'.
[Throughout] The term 'reward-independent' should be qualified: the sender's message generation is reward-independent, but the receiver's use of the message is trained through the RL objective. The current phrasing in the abstract and conclusions could be read as claiming that the whole communication channel is learned without any reward signal, which is not the case.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the variational derivation is self-contained and the empirical claims are tested against external baselines; self-citations appear in framing but are not load-bearing.

full rationale

The paper's derivation chain starts from a generative model over agents' observations and a message variable (Eq. 1), applies a standard variational ELBO (Eq. 2), and obtains per-agent objectives through a factorized variational distribution and prior (Eqs. 3-8). The claim that the message variable estimates the global state is a direct reading of the latent-variable model being introduced, not a quantity fitted to reproduce a target result. Each agent maximizes its own reconstruction-KL objective, and the RL module uses the resulting messages as inputs; no parameter is fitted to a predicted outcome. The experiments compare against no-comm, message-as-action, and shared-observation baselines, and the ablation study (random/no message) tests message utility empirically. The paper explicitly notes that with beta=0 and a decoder restricted to the agent's own message, the framework reduces to Lin et al.'s autoencoder, but it does not run that reduced model as a baseline; that omission is a completeness or correctness risk, not a circularity, because the reduced model is not used in defining, fitting, or deriving the reported CPC results. Several citations are to prior work by the same research group, especially the CPC hypothesis and MCMC-based predecessor models, but the present variational formulation and the empirical evaluations do not depend on an unverified uniqueness theorem or on a fitted input being relabeled as a prediction. The central derivation is therefore self-contained; minor self-citation in the background and related-work framing does not rise to load-bearing circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a mean-field variational model and a self-reconstruction objective. The most consequential free parameters are the message vocabulary size K and the implicit relative weighting of the RL and CPC losses, neither of which is tuned or analyzed. The load-bearing unproven assumption is that a message trained only to reconstruct its sender's observation remains decodable by a different agent's policy. The invented entities are the latent message and the conceptual 'pseudo-joint agent'; neither has externally falsifiable predictions outside this paper.

free parameters (3)

Message vocabulary size K = K=5 (bandit), K=20 (observer)
Chosen by hand for each environment; no sensitivity analysis or tuning reported. The capacity of the channel directly affects whether the sender's observation can be encoded.
Implicit RL/CPC loss balance = 1 (implicit)
The total objective is J = JRL + JCPC without an explicit weighting hyperparameter (Eq. 13). The relative scale of the two losses is set implicitly by the network outputs and optimizer, which serves as an unstated free parameter.
KL regularization strength = 1 (not weighted)
The KL term in Eq. (6) is added with weight 1. No beta or coefficient is discussed, so the regularization strength is fixed implicitly and may affect code compactness and message informativeness.

assumptions (5)

domain assumption State space is factorized as S = S1 x ... x SN and each agent observes only its own component (Xi = Si).
Section 3.1 introduces this to make private observations and communication meaningful; it is a task-level assumption, not derived from first principles.
ad hoc to paper Joint distribution Ptheta(x,m) factorizes as P(m) prod_i Ptheta_i(xi|m), meaning observations are conditionally independent given the messages.
This defines the CPC generative model. It is a modeling choice specific to this paper and is not justified by the POMG formulation.
domain assumption Variational distribution factorizes as Qphi(m|x) = prod_i Qphi_i(mi|xi) (mean-field).
Standard variational inference assumption used in Eq. (3). It limits the family of posteriors but is widely used.
ad hoc to paper The KL divergence DKL(Qphi_i(mi|xi)||P(mi)) is approximated by (kappa-1)-log kappa with kappa = Q/P, and this biased estimator is treated as the optimization objective.
Eq. (11). This is a biased estimator whose expectation does not equal the KL divergence; the paper does not discuss the bias or alternative estimators.
domain assumption Each agent can obtain the full joint message m = (m1,...,mN) during training and execution despite observing only its own xi.
The algorithms require the concatenated global message as input to policies and decoders. In the experiments this is available; the paper does not address how a fully decentralized physical system would supply all messages to every agent without a shared message board.

invented entities (2)

Discrete message variable mi
purpose: Auxiliary variable in the generative model meant to encode agent i's observation and support state estimation by other agents
The messages are internal latent variables; the only evidence of their informativeness comes from the ablation in this paper. There is no externally falsifiable prediction about these messages outside the paper.
Pseudo-joint agent
purpose: Conceptual entity formed by the CPC module that aggregates messages to perform joint state inference
Figure 2 describes a 'pseudo-joint agent' as the central processing unit; this is an architectural metaphor rather than a physical or external entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/LRCX72Z3

@misc{pith2026250521985,
  author       = {Pith},
  title        = {Pith review of: Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRCX72Z3}},
  note         = {Machine review of arXiv:2505.21985}
}

read the original abstract

In multi-agent reinforcement learning (MARL), effective communication improves agent performance, particularly under partial observability. We propose MARL-CPC, a framework that enables communication among fully decentralized, independent agents without parameter sharing. MARL-CPC incorporates a message learning model based on collective predictive coding (CPC) from emergent communication research. Unlike conventional methods that treat messages as part of the action space and assume cooperation, MARL-CPC links messages to state inference, supporting communication in non-cooperative, reward-independent settings. We introduce two algorithms -Bandit-CPC and IPPO-CPC- and evaluate them in non-cooperative MARL tasks. Benchmarks show that both outperform standard message-as-action approaches, establishing effective communication even when messages offer no direct benefit to the sender. These results highlight MARL-CPC's potential for enabling coordination in complex, decentralized environments.

Figures

Figures reproduced from arXiv: 2505.21985 by the authors.

**Figure 1.** Graphical model of the CPC module (2 agents). This study formulates CPC using variational inference within a deep generative modeling framework. In this context, a joint generative model is constructed by aggregating the observations xi of individual agents (i = 1, 2, . . . , N). This joint model is then decomposed to derive an objective function for the communication modules of individual agents. An overview of … view at source ↗

**Figure 2.** Overview of the MARL-CPC architecture. The figure is a model with two agents. The components of each agent are represented by filled regions—white and gray, respectively. The central panel corresponds to the CPC module, which forms a pseudojoint agent and enables message generation and exchange. Based on the messages m and the hidden states z acquired through the CPC module, the agent performs action selection and … view at source ↗

**Figure 3.** Agent architectures compared in this experiments. A) Independent agents without communication [6]. B) Message agents, where communication is defined as an extension of action [5, 9]. C) CPC-based agents in which messages function as auxiliary variables for the state inference process (ours). D) Agents whose observations are shared in advance (performance upper bound). 4.1 Contextual Bandit with Information Sharing… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Multi-agent conditional bandit environment. communication can increase the total group reward. CPC has the potential to establish such communication autonomously. We evaluated Bandit-CPC under the cpc condition and compared it with three baselines: independent, message…

**Figure 5.** Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Overview of observer environment. This environment evaluates communication in a non-cooperative setting with asymmetric information access ( [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Results in observer environment. A) Episode length. B) Group welfare. 5 Additional analysis 5.1 Ablation Study Contexual Bandit Observer Normal Random No Normal Random No Welfare [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Overview of observer environment. The effectiveness of communication in MARL remains under debate [24, 30]. To assess message utility in MARLCPC, we conducted an ablation study to evaluate the impact of disrupting trained agents’ messages. Two conditions were tested:…

**Figure 9.** Figure 9: Results in cooperative environment. We examined communication learning under cooperative conditions by modifying the reward structure in the contextual bandit environment. Both agents now receive a reward of +1 only if they select the correct answer simultaneously; o…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

AI Mother Tongue: Self-Emergent Communication in MARL via Endogenous Symbol Systems
cs.AI 2025-07 reject novelty 4.0 of 10

Agents with a shared discrete-symbol codebook reportedly converge on cooperative communication in a Prisoner's Dilemma variant, but the paper provides no reproducible data to support the claim.

Reference graph

Works this paper leans on

47 extracted references · 37 canonical work pages · cited by 1 Pith paper

[1]

Advances in Neural Information Processing Systems34 (2021)

Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A.C., Bellemare, M.: Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems34 (2021)

work page 2021
[2]

MIT Press (2024)

Albrecht, S.V., Christianos, F., Schäfer, L.: Multi-agent reinforcement learning: Foundations and modern approaches. MIT Press (2024)

work page 2024
[3]

arXiv preprint arXiv:1308.3432 (2013)

Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

arXiv 2013
[4]

IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)38(2), 156–172 (2008)

Busoniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)38(2), 156–172 (2008)

work page 2008
[5]

Connection Science10(2), 83–97 (1998)

Cangelosi, A., Parisi, D.: The emergence of a’language’in an evolving population of neural networks. Connection Science10(2), 83–97 (1998)

work page 1998
[6]

De Witt, C.S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P.H., Sun, M., Whiteson, S.: Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 (2020)

arXiv 2020
[7]

In: 2023 15th international congress on advanced applied informatics winter (IIAI- AAI-Winter)

Ebara, H., Nakamura, T., Taniguchi, A., Taniguchi, T.: Multi-agent reinforcement learning with emergent communication using discrete and indifferentiable message. In: 2023 15th international congress on advanced applied informatics winter (IIAI- AAI-Winter). pp. 366–371. IEEE (2023) 14 N. Yoshida and T. Taniguchi

work page 2023
[8]

Farrell,J.,Rabin,M.:Cheaptalk.JournalofEconomicperspectives 10(3),103–118 (1996)

work page 1996

Show all 47 references

[9]

Advances in neural information pro- cessing systems 29 (2016)

Foerster, J., Assael, I.A., De Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information pro- cessing systems 29 (2016)

2016
[10]

Journal of physiology-Paris 100(1-3), 70–87 (2006)

Friston, K., Kilner, J., Harrison, L.: A free energy principle for the brain. Journal of physiology-Paris 100(1-3), 70–87 (2006)

2006
[11]

Ar- tificial Intelligence Review55(2), 895–943 (2022)

Gronauer, S., Diepold, K.: Multi-agent deep reinforcement learning: a survey. Ar- tificial Intelligence Review55(2), 895–943 (2022)

2022
[12]

Frontiers in Robotics and AI6, 134 (2019)

Hagiwara, Y., Kobayashi, H., Taniguchi, A., Taniguchi, T.: Symbol emergence as an interpersonal multimodal categorization. Frontiers in Robotics and AI6, 134 (2019)

2019
[13]

In: AAAI

Hansen, E.A., Bernstein, D.S., Zilberstein, S.: Dynamic programming for partially observable stochastic games. In: AAAI. vol. 4, pp. 709–715 (2004)

2004
[14]

arXiv preprint arXiv:1606.08415 (2016)

Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

2016 arXiv
[15]

In: International conference on learning representations (2017)

Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A.: beta-vae: Learning basic visual concepts with a constrained vari- ational framework. In: International conference on learning representations (2017)

2017
[16]

Frontiers in Robotics and AI10, 1290604 (2024)

Hoang, N.L., Taniguchi, T., Hagiwara, Y., Taniguchi, A.: Emergent communication of multimodal deep generative models based on metropolis-hastings naming game. Frontiers in Robotics and AI10, 1290604 (2024)

2024
[17]

arXiv preprint arXiv:1412.6980 (2014)

Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

2014 arXiv
[18]

arXiv preprint arXiv:1312.6114 (2013)

Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

2013 arXiv
[19]

arXiv preprint arXiv:2006.02419 (2020)

Lazaridou, A., Baroni, M.: Emergent multi-agent communication in the deep learn- ing era. arXiv preprint arXiv:2006.02419 (2020)

2020 arXiv
[20]

In: International Conference on Learning Representations (2018)

Lazaridou, A., Hermann, K.M., Tuyls, K., Clark, S.: Emergence of linguistic com- munication from referential games with symbolic and pixel input. In: International Conference on Learning Representations (2018)

2018
[21]

In: International Conference on Learning Repre- sentations (2017)

Lazaridou, A., Peysakhovich, A., Baroni, M.: Multi-agent cooperation and the emergence of (natural) language. In: International Conference on Learning Repre- sentations (2017)

2017
[22]

Harvard University Press (1969)

Lewis, D.: Convention: A philosophical study. Harvard University Press (1969)

1969
[23]

Advances in Neural Information Processing Systems 34, 15230–15242 (2021)

Lin, T., Huh, J., Stauffer, C., Lim, S.N., Isola, P.: Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Systems 34, 15230–15242 (2021)

2021
[24]

In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems

Lowe, R., Foerster, J., Boureau, Y.L., Pineau, J., Dauphin, Y.: On the pitfalls of measuring emergent communication. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. pp. 693–701 (2019)

2019
[25]

Advances in neural information processing systems30 (2017)

Lowe, R., Wu, Y.I., Tamar, A., Harb, J., Pieter Abbeel, O., Mordatch, I.: Multi- agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems30 (2017)

2017
[26]

Mirolli, M., Parisi, D.: Producer Biases and Kin Selection in the Evolution of Com- munication, pp. 135–159. Springer Berlin Heidelberg, Berlin, Heidelberg (2010)

2010
[27]

arXiv preprint arXiv:2307.05004 (2023) Title Suppressed Due to Excessive Length 15

Nakamura,T.,Taniguchi,A.,Taniguchi,T.:Controlasprobabilisticinferenceasan emergent communication mechanism in multi-agent reinforcement learning. arXiv preprint arXiv:2307.05004 (2023) Title Suppressed Due to Excessive Length 15

2023 arXiv
[28]

arXiv preprint arXiv:2504.03353 (2025)

Nomura,K.,Aoki,T.,Taniguchi,T.,Horii,T.:Decentralizedcollectiveworldmodel for emergent communication and coordination. arXiv preprint arXiv:2504.03353 (2025)

2025 arXiv
[29]

Proceedings of the Na- tional Academy of Sciences96(14), 8028–8033 (1999)

Nowak, M.A., Krakauer, D.C.: The evolution of language. Proceedings of the Na- tional Academy of Sciences96(14), 8028–8033 (1999)

1999
[30]

arXiv preprint arXiv:2409.02645 (2024)

Peters, J., de Puiseau, C.W., Tercan, H., Gopikrishnan, A., De Carvalho, G.A.L., Bitter, C., Meisen, T.: A survey on emergent language. arXiv preprint arXiv:2409.02645 (2024)

2024 arXiv
[31]

arXiv preprint arXiv:2401.15059 (2024)

Pina, R., De Silva, V., Artaud, C., Liu, X.: Fully independent communication in multi-agent reinforcement learning. arXiv preprint arXiv:2401.15059 (2024)

2024 arXiv
[32]

Nature neuroscience2(1), 79–87 (1999)

Rao, R.P., Ballard, D.H.: Predictive coding in the visual cortex: a functional inter- pretation of some extra-classical receptive-field effects. Nature neuroscience2(1), 79–87 (1999)

1999
[33]

In: International Con- ference on Learning Representations (ICLR) (2016)

Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. In: International Con- ference on Learning Representations (ICLR) (2016)

2016
[34]

arXiv preprint arXiv:1707.06347 (2017)

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[35]

http://joschu.net/blog/kl- approx.html (3 2020)

Schulman, J.D.: Approximating kl divergence. http://joschu.net/blog/kl- approx.html (3 2020)

2020
[36]

Sci- ence 210(4471), 801–803 (1980)

Seyfarth, R.M., Cheney, D.L., Marler, P.: Monkey responses to three different alarm calls: evidence of predator classification and semantic communication. Sci- ence 210(4471), 801–803 (1980)

1980
[37]

Skyrms, B.: Signals: Evolution, learning, and information (2010)

2010
[38]

Advances in neural information processing systems29 (2016)

Sukhbaatar, S., Fergus, R., et al.: Learning multiagent communication with back- propagation. Advances in neural information processing systems29 (2016)

2016
[39]

In: Proceed- ings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., et al.: Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceed- ings of the 17th International Con...

2018
[40]

Frontiers in Robotics and AI11, 1353870 (2024)

Taniguchi, T.: Collective predictive coding hypothesis: Symbol emergence as de- centralized bayesian inference. Frontiers in Robotics and AI11, 1353870 (2024)

2024
[41]

Advanced Robotics37(19), 1266–1282 (2023)

Taniguchi, T., Yoshida, Y., Matsui, Y., Le Hoang, N., Taniguchi, A., Hagiwara, Y.: Emergent communication through metropolis-hastings naming game with deep generative models. Advanced Robotics37(19), 1266–1282 (2023)

2023
[42]

MIT press (2009)

Tomasello, M.: Why we cooperate. MIT press (2009)

2009
[43]

MIT press (2010)

Tomasello, M.: Origins of human communication. MIT press (2010)

2010
[44]

In: The Twelfth International Conference on Learning Rep- resentations (2024)

Ueda, R., Taniguchi, T.: Lewis’s signaling game as beta-vae for natural word lengths and segments. In: The Twelfth International Conference on Learning Rep- resentations (2024)

2024
[45]

Artificial Intelligence Review56(6), 5023–5056 (2023)

Wong, A., Bäck, T., Kononova, A.V., Plaat, A.: Deep multiagent reinforcement learning: challenges and directions. Artificial Intelligence Review56(6), 5023–5056 (2023)

2023
[46]

Advances in neural information processing systems35, 24611–24624 (2022)

Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., Wu, Y.: The sur- prising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems35, 24611–24624 (2022)

2022
[47]

Autonomous Agents and Multi-Agent Systems38(1), 4 (2024)

Zhu, C., Dastani, M., Wang, S.: A survey of multi-agent deep reinforcement learn- ing with communication. Autonomous Agents and Multi-Agent Systems38(1), 4 (2024)

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Advances in Neural Information Processing Systems34 (2021)

Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A.C., Bellemare, M.: Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems34 (2021)

work page 2021

[2] [2]

MIT Press (2024)

Albrecht, S.V., Christianos, F., Schäfer, L.: Multi-agent reinforcement learning: Foundations and modern approaches. MIT Press (2024)

work page 2024

[3] [3]

arXiv preprint arXiv:1308.3432 (2013)

Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

arXiv 2013

[4] [4]

IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)38(2), 156–172 (2008)

Busoniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)38(2), 156–172 (2008)

work page 2008

[5] [5]

Connection Science10(2), 83–97 (1998)

Cangelosi, A., Parisi, D.: The emergence of a’language’in an evolving population of neural networks. Connection Science10(2), 83–97 (1998)

work page 1998

[6] [6]

De Witt, C.S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P.H., Sun, M., Whiteson, S.: Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 (2020)

arXiv 2020

[7] [7]

In: 2023 15th international congress on advanced applied informatics winter (IIAI- AAI-Winter)

Ebara, H., Nakamura, T., Taniguchi, A., Taniguchi, T.: Multi-agent reinforcement learning with emergent communication using discrete and indifferentiable message. In: 2023 15th international congress on advanced applied informatics winter (IIAI- AAI-Winter). pp. 366–371. IEEE (2023) 14 N. Yoshida and T. Taniguchi

work page 2023

[8] [8]

Farrell,J.,Rabin,M.:Cheaptalk.JournalofEconomicperspectives 10(3),103–118 (1996)

work page 1996