REVIEW 3 major objections 4 minor 21 references
Unsupervised Translation of Emergent Communication
T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that unsupervised neural machine translation can translate emergent communication — the languages agents invent in referential games — into English without any parallel EC–English data.
desk verdict First UNMT-for-EC paper with a transparent setup, but the central evaluation may be measuring recognition of seen messages rather than translation of novel ones. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the argument is a three-stage unsupervised neural machine translation pipeline applied to an emergent communication corpus. A high-resource English corpus first pretrains the model; the model is then fine-tuned jointly on the monolingual EC corpus and an English caption corpus, which creates a shared embedding space between the two languages; finally, iterative back-translation and denoising auto-encoding align the spaces without any parallel pairs. The referential game supplies the EC: a Sender sees an image and emits a six-symbol message from a 64-symbol vocabulary, and a Receiver must identify the target among distractors, with complexity set by how semantically close the distractors are. The load-bearing identity is that both languages are grounded in the same images: English captions and emergent messages describe the same visual world, so the unsupervised model can learn a mapping through that shared ground.
What would settle it
Hold out all messages generated from images never seen during game training, or translate messages from a game seed not used in fine-tuning, and compare BLEU and ROUGE against the reported values; if scores drop toward the random-agent baseline (BLEU near 0.07), the reported translation is largely memorization rather than decoding of new emergent language.
Extended reading notes
Core claim
The central claim is that emergent communication from referential games can be translated into English without any parallel EC–English data, and that the degree of translatability is shaped by the semantic structure of the task rather than by task difficulty alone. On the paper's own terms: an unsupervised neural machine translation system, initialized on English and fine-tuned on the corpus of recorded messages plus English captions, produces coherent English for all four game types, with BLEU scores between 6.08 and 9.21, ROUGE-L around 0.34–0.37, BERTScore near 0.73, and image–text alignment scores near 0.18, all clearly above a random-agent baseline. Inter-category games, whose distractors come from different but overlapping categories, yield the most translatable EC, while Supercategory games, with semantically close distractors, produce the richest but least translatable messages, and Category games, with fine-grained distinctions, produce pragmatic, low-entropy messages that remain translatable. The authors conclude that semantic diversity in the task enhances translatability, whereas constrained semantic variability produces pragmatic EC that is harder to interpret but still suitable for translation.
Load-bearing premise
The evaluation assumes the test EC messages are genuinely new inputs the translator must decode, but Table 1 shows that fewer than 7% of test messages are novel for every game, so the scores may largely reward recognizing messages the model already saw during fine-tuning.
Editorial extensions
If this is right
- If the claim holds, emergent communication can be inspected by humans without collecting any paired EC–English data, removing a major bottleneck for interpretability of agent communication.
- Translation quality does not increase monotonically with task difficulty; it is mediated by semantic diversity, so game design choices directly determine how readable the resulting language will be.
- The near-zero correlation of translation scores across seeds on the same test set implies that each trained agent pair develops its own protocol, so there is no single 'emergent language' to translate.
- Compositionality metrics can serve as predictors of translatability: messages aligned with image concepts translate better, while strongly disentangled symbol codes produce more novel but harder-to-translate outputs.
Reading between the lines
- The shared-grounding recipe suggests a concrete transfer test: apply the same UNMT pipeline to other non-human communication systems whenever an aligned observational channel, such as video of the behavior, can serve as the common ground that captions provide here.
- The differing translatability rankings across game types imply that 'complexity' is too coarse a design knob; future emergent-communication research could use translatability itself as a selection pressure when the goal is human-readable agent communication.
- A targeted probe that removes the final words from English captions during fine-tuning could separate what an EC message genuinely encodes from what the language model fills in from caption-length priors, sharpening the notion of 'translatability'.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using unsupervised neural machine translation (UNMT) to translate emergent communication (EC) from referential games into English without parallel EC–NL data. The authors train Sender–Receiver agents in four game types (Random, Category, Supercategory, Inter-category), record the messages as a monolingual EC corpus, and fine-tune an XLM-based UNMT with back-translation and denoising on this corpus together with English MSCOCO captions. They report BLEU, ROUGE-L, METEOR, BERTScore, CLIP score, Jaro similarity, and TTR, finding that trained EC messages translate above a random-agent baseline, with Inter-category achieving the highest BLEU (9.21). The paper also analyzes correlations between EC compositionality metrics and translatability, and includes a Chinese-to-English UNMT sanity check. The central claim is that UNMT can decode emergent agent-invented languages using only monolingual corpora.
Significance. If the evaluation truly reflects generalization to unseen emergent messages, this is a novel and useful demonstration of unsupervised translation of emergent communication, with practical value for interpreting agent-invented languages. The paper contributes a fairly comprehensive set of evaluation metrics and a systematic comparison across game complexities, and it includes a concrete sanity check against Chinese-to-English UNMT and a random-agent baseline. However, the significance is conditional on the evaluation not being confounded by memorization: the paper's own Table 1 shows that test EC messages have near-zero novelty (0.84%–6.85%), which directly threatens the interpretation of the translation results as generalization to new emergent language.
major comments (3)
- [Section 6, Table 1 vs. Sections 4–5] The test EC messages have near-zero novelty (0.84%–6.85% across game types, Table 1), meaning they largely appear in the agents' training messages. Since the UNMT fine-tuning corpus is compiled from the same games (Section 4) and the paper does not state that held-out game test messages are excluded, the default reading is that the test messages used for evaluation in Table 2 are present in the UNMT fine-tuning corpus. Under that reading, the reported BLEU/ROUGE/CLIP scores may measure recognition of memorized source messages rather than translation of unseen EC. This is load-bearing for the central claim of 'translation.' I request an evaluation on held-out EC messages that were not seen during UNMT fine-tuning, or at minimum a breakdown of translation performance by whether the test message occurred in the fine-tuning corpus.
- [Section 4, 'UNMT Architecture' and Section 5, 'Implementation Details'] The paper does not specify the train/test split of the EC corpus used for UNMT fine-tuning. Please clarify whether the fine-tuning corpus includes the messages used as the test set in Table 2, and report the number of unique messages and the total corpus size per game type. Without this information, the reader cannot determine whether the translation metrics reflect generalization to novel messages or retrieval of memorized message–caption associations.
- [Table 2, Baseline column] The random-agent baseline does not adequately control for the memorization concern. As shown in Table 1, random-agent messages also have zero novelty, so their messages would also appear in a corresponding fine-tuning corpus; the increased BLEU for trained EC could stem from a structured message-to-image mapping even under a memorization-only account. A stronger baseline would include an untrained but symbol-sampling agent, or a held-out evaluation where both trained and random EC messages are excluded from fine-tuning.
minor comments (4)
- [Appendix E] The heading '7. Bag-of-Symbols Disentanglement (BosDis):' appears twice, and the equations are not numbered, which makes referencing specific definitions difficult.
- [Table 2] The 'Novelty (%)' row should state whether it refers to the source EC messages or the translated English text, and the Baseline column should be described in the main text (e.g., whether it uses the same random-agent EC corpus as Table 1).
- [Section 5 vs. Appendix D] The main text says each message is composed of 6 symbols plus an EOS symbol, while Appendix D states a message length of 7 including EOS; please reconcile the notation.
- [Section 6, 'Game complexity vs. Translatability'] The claim that Inter-category achieves the highest BLEU (9.21) is based on only 3 seeds; reporting per-seed results would help assess the stability of this comparison.
Circularity Check
No significant circularity: the UNMT translation claim is empirical and self-contained; self-citations are background, and the low message-novelty issue is an evaluation-leakage risk rather than a circular derivation.
full rationale
The paper's claimed derivation chain is empirical rather than equation-driven: EC games produce messages; the messages form a monolingual EC corpus; a pretrained XLM is fine-tuned on that corpus and on MSCOCO captions with back-translation and denoising; translation quality is then measured against the MSCOCO captions and against a random-agent baseline. No step defines the predicted translation as the fitted input by construction. The key advertised property is that UNMT is trained without EC-English parallel data, so the translation is not statistically forced by paired supervision. The self-citations (Carmeli et al. 2023, 2024; Goldwasser et al. 2024) are used as background or design choices—quantized communication, concept-mapping compositionality, and a theoretical UMT motivation—and are not load-bearing for the central translation result. The strongest concern is evaluative, not circular: Table 1 reports very low message novelty (0.84–6.85%), indicating that agents largely reuse messages from game training, and the paper does not state that game-test messages were excluded from the UNMT fine-tuning corpus. Under the default reading, the reported BLEU and ROUGE scores may overstate generalization to genuinely unseen emergent messages. However, seeing a source EC message in the fine-tuning corpus does not determine its English translation in the absence of parallel data, so this is a data-splitting and evaluation-validity risk rather than a circular derivation. The use of the same MSCOCO captions for both UNMT fine-tuning and evaluation is likewise an evaluation-leakage risk, not a definitional equivalence. The central claim retains independent empirical content, supported by the random-agent baseline and the Chinese-to-English calibration, so the circularity score is low.
Assumptions & free parameters
free parameters (4)
- Communication channel vocabulary size =
64 symbols
- Message length =
6 symbols + EOS
- Number of distractors =
9
- Number of game seeds used for UNMT translation =
3
assumptions (4)
- domain assumption The infoNCE training objective guides agents to produce messages that encode task-relevant image information.
- domain assumption MSCOCO captions are an appropriate target-language prior for the images used in the games.
- domain assumption UNMT with back-translation and denoising can create a shared embedding space between EC and English.
- domain assumption Automatic metrics (BLEU, METEOR, CLIP) computed against captions are valid proxies for translation quality.
Cite this review
Pith. "Pith review of Unsupervised Translation of Emergent Communication." pith.science (2026). https://pith.science/paper/I7QQPD6L
@misc{pith2026250207552,
author = {Pith},
title = {Pith review of: Unsupervised Translation of Emergent Communication},
year = {2026},
howpublished = {\url{https://pith.science/paper/I7QQPD6L}},
note = {Machine review of arXiv:2502.07552}
}
read the original abstract
Emergent Communication (EC) provides a unique window into the language systems that emerge autonomously when agents are trained to jointly achieve shared goals. However, it is difficult to interpret EC and evaluate its relationship with natural languages (NL). This study employs unsupervised neural machine translation (UNMT) techniques to decipher ECs formed during referential games with varying task complexities, influenced by the semantic diversity of the environment. Our findings demonstrate UNMT's potential to translate EC, illustrating that task complexity characterized by semantic diversity enhances EC translatability, while higher task complexity with constrained semantic variability exhibits pragmatic EC, which, although challenging to interpret, remains suitable for translation. This research marks the first attempt, to our knowledge, to translate EC without the aid of parallel data.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Game Accuracy (ACC): ACC i = Number of Correct Predictions Total Number of Predictions Game accuracy measures how many correct identifica- tions the agents make, and i represents how many candi- dates they must choose from
-
[2]
Vocabulary Usage (VU): VU = |{si | count(si) > 0}| |S| where |{si | count(si) > 0}| is the number of symbols used in messages and |S| is the total vocabulary size
-
[3]
Message Entropy (ME): ME = − X m∈M P (m) logP (m) where M is the set of messages and P (m) is the proba- bility of message m
-
[4]
Message Novelty (MN): MN = |{m ∈ Mtest | m /∈ Mtrain}| |Mtest| where Mtest is the set of messages in the test set andMtrain is the set of messages in the training set
-
[5]
TopSim captures the alignment of distances in input and message spaces
Topographic Similarity (TopSim): TopSim = ρ(dinput, dmessage) where ρ is the Spearman correlation between dinput and dmessage, with dinput representing the Euclidean distance in the input space and dmessage representing the edit distance between corresponding messages. TopSim captures the alignment of distances in input and message spaces
-
[6]
Bag-of-Symbols Disentanglement (BosDis):
-
[7]
Bag-of-Symbols Disentanglement (BosDis): BosDis = |V |X j=1 M I sj; a − M I sj; b H(sj) where sj is the j-th symbol in the vocabularyV , M I(·; ·) denotes mutual information, and H(·) denotes entropy. We compare a per-symbol measure of how much more strongly that symbol encodes one attribute a over the other b
-
[8]
Positional Disentanglement (PosDis) : PosDis follows the same logic as BosDis, but instead of looking at the MI of each symbol in the vocabulary, it focuses on each specific position in the message
Show all 21 references
-
[9]
AMI evaluates how well messages align with un- derlying image concepts, while adjusting for random- ness
Adjusted Mutual Information (AMI): AMI(U, V) = MI(U, V) − E [MI(U, V)] max(H(U ), H(V )) − E [MI(U, V)] where I(U, V) is the mutual information between message clusters U and ground-truth labels V , and E[MI(U, V)] is the expected MI under random assign- ment. AMI evaluates ho...
-
[10]
Metrics for UNMT
Multi-Concept AMI: mAMI = 1 |C′| X c∈C′ AMIc(U, V) where c is concept and C′ is the set of valid concepts. Metrics for UNMT
-
[14]
BERTScore: BERTScore(R, C) = 1 |R| X r∈R max c∈C cos(er, ec) where R is the set of reference embeddings, C is the set of candidate embeddings, and cos (er, ec) is the cosine similarity between embeddings er and ec
-
[15]
BLEU Score: BLEU = exp min 1 − |r| |c| , 0 + NX n=1 wn log pn ! where |r| is the length of the reference, |c| is the length of the candidate translation, wn are the weights, and pn is the precision of n-grams
-
[16]
, wi−1) ! where N is the total number of words, and P (wi | w1,
Perplexity (PPL): PPL = exp − 1 N NX i=1 log P (wi | w1, . . . , wi−1) ! where N is the total number of words, and P (wi | w1, . . . , wi−1) is the probability of word wi given the previous words
-
[17]
Each matched word is classified into one of these categories, and the total number of matches m is adjusted accordingly
METEOR: METEOR = Fmean · (1 − Penalty) METEOR considers stemming and synonymy by using WordNet to account for matches beyond exact match, in- cluding stem and synonym matches. Each matched word is classified into one of these categories, and the total number of matches m is ad...
-
[18]
All embeddings are obtained from the CLIP multi-modal model
CLIP Score: CLIP Score(I, T) = 1 |T | X t∈T cos(et, ei) where I is the set of image embeddings, T is the set of translation candidate embeddings, et is the embedding of the translation candidate, and ei is the embedding of the corresponding image. All embeddings are obtained f...
-
[19]
This metric evaluates the proportion of novel n-grams in the translations that do not appear in the training data
Novelty Score: Novelty = 1 |T | X t∈T I(t /∈ D) where T is the set of n-grams in the translated texts, D is the set of n-grams in the training corpus, and I is an in- dicator function that is 1 if t is not in D and 0 otherwise. This metric evaluates the proportion of novel n-g...
-
[20]
Jaro Similarity: Jaro(s1, s2) = 1 3 m |s1| + m |s2| + m − t m where s1 and s2 are the input strings, m is the number of matching characters, and t is the transpositions number
-
[21]
F Chinese UNMT Before translating EC, we calibrated our expectations by evaluating how UNMT performs on a clear, interpretable and more complex language than EC
Text-Type Ratio (TTR): TTR = Number of Unique Words Total Number of Words measures the lexical diversity within the translated texts. F Chinese UNMT Before translating EC, we calibrated our expectations by evaluating how UNMT performs on a clear, interpretable and more complex...
2014
-
[2018]
ArXiv, abs/1804.03984
Emergence of Linguistic Communication from Ref- erential Games with Symbolic and Pixel Input. ArXiv, abs/1804.03984. Lazaridou, A.; Peysakhovich, A.; and Baroni, M. 2016. Multi-agent cooperation and the emergence of (natural) lan- guage. arXiv preprint arXiv:1612.07182. Lee, J...
2024 arXiv
-
[2023]
In Proceedings of the 3rd Workshop on Multi-lingual Repre- sentation Learning (MRL)
Learning to translate by learning to communicate. In Proceedings of the 3rd Workshop on Multi-lingual Repre- sentation Learning (MRL). Giles, C. L.; and Jim, K.-C. 2003. Learning communica- tion for multi-agent systems. In Innovative Concepts for Agent-Based Systems: First Int...
2003
-
[2024]
Advances in Neu- ral Information Processing Systems, 36
A Theory of Unsupervised Translation Motivated by Understanding Animal Communication. Advances in Neu- ral Information Processing Systems, 36. Graesser, A. C.; McNamara, D. S.; Louwerse, M. M.; and Cai, Z. 2003. Automated Evaluation of Discourse Coher- ence Quality in Essays. ...
2003 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.