{"id":"acb8552c-3681-4e5a-8370-65e1f6f5774c","arxiv_id":"2502.07552","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Unsupervised neural machine translation can decode emergent agent languages from referential games into English without parallel data, with translation quality varying by game complexity.","lead":"Researchers trained AI agents to communicate in invented languages during image-guessing games, then used unsupervised translation to convert those languages into English without any paired examples. The method works modestly well, and it shows that games with more varied image categories produce more translatable agent languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test EC messages largely appear in the UNMT fine-tuning corpus (Table 1 novelty ≤6.85%), so the reported BLEU/ROUGE/CLIP scores may measure memorization rather than generalization to unseen emergent language.","rationale":"I agree with the reader's assessment. The low novelty is acknowledged in the paper itself, and it creates a direct threat to the validity of the translation evaluation. The UNMT is fine-tuned on the game EC corpus, and the test messages are almost certainly part of that corpus. This is a logical gap between the claim 'UNMT can translate EC' and the evidence 'UNMT can map memorized EC strings to plausible captions.' The random-agent baseline cannot completely control for this, because the baseline's messages are also memorized (novelty 0%) but lack the structured mapping to images that trained EC has. If the held-out test suggested above shows a large drop, the central claim would need to be weakened to 'UNMT can translate known EC messages.' The paper's other weaknesses (small seed count, overlapping error bars, lenient max-over-captions evaluation) are secondary to this one. The verdict should remain CONDITIONAL: the concern is addressable by a clean held-out evaluation, and the paper's other contributions (benchmark, complexity analysis) still stand conditionally.","tokens_in":18223,"tokens_out":10017,"duration_ms":95616,"concrete_test":"Re-run the UNMT fine-tuning after deleting from the EC corpus every message that also appears in the held-out game test set, leaving only messages from the game training split. Recompute Table 2 on the same test set. If BLEU, ROUGE-L, and CLIP Score drop to the random-agent baseline levels (e.g., BLEU ~0.07), the original scores reflect memorized test messages rather than translation of unseen EC.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that UNMT can translate EC without parallel data. In Table 1, message novelty for the game test sets is 0.84–6.85% for all four game types, meaning nearly every test EC message was already seen during the game training phase. The authors themselves state in Section 6 that 'agents largely rely on memorized messages generated during training.' The UNMT fine-tuning stage uses an EC corpus compiled from the same games (Section 4), and the paper does not state that held-out game test messages are excluded from this corpus. Under the default reading, the translation metrics in Table 2 may be produced by recognizing exact source messages that the model has seen in fine-tuning, rather than by translating new EC. The comparison to a random-agent baseline (novelty 0.0%) does not resolve this, because both conditions can exhibit memorization; the trained EC still benefits from a structured message-to-image mapping. To support the advertised generalization, the authors need a held-out evaluation where the source EC messages are truly unseen by the UNMT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using unsupervised neural machine translation (UNMT) to translate emergent communication (EC) from referential games into English without parallel EC–NL data. The authors train Sender–Receiver agents in four game types (Random, Category, Supercategory, Inter-category), record the messages as a monolingual EC corpus, and fine-tune an XLM-based UNMT with back-translation and denoising on this corpus together with English MSCOCO captions. They report BLEU, ROUGE-L, METEOR, BERTScore, CLIP score, Jaro similarity, and TTR, finding that trained EC messages translate above a random-agent baseline, with Inter-category achieving the highest BLEU (9.21). The paper also analyzes correlations between EC compositionality metrics and translatability, and includes a Chinese-to-English UNMT sanity check. The central claim is that UNMT can decode emergent agent-invented languages using only monolingual corpora.","tokens_in":18433,"tokens_out":6002,"duration_ms":56371,"significance":"If the evaluation truly reflects generalization to unseen emergent messages, this is a novel and useful demonstration of unsupervised translation of emergent communication, with practical value for interpreting agent-invented languages. The paper contributes a fairly comprehensive set of evaluation metrics and a systematic comparison across game complexities, and it includes a concrete sanity check against Chinese-to-English UNMT and a random-agent baseline. However, the significance is conditional on the evaluation not being confounded by memorization: the paper's own Table 1 shows that test EC messages have near-zero novelty (0.84%–6.85%), which directly threatens the interpretation of the translation results as generalization to new emergent language.","major_comments":[{"comment":"The test EC messages have near-zero novelty (0.84%–6.85% across game types, Table 1), meaning they largely appear in the agents' training messages. Since the UNMT fine-tuning corpus is compiled from the same games (Section 4) and the paper does not state that held-out game test messages are excluded, the default reading is that the test messages used for evaluation in Table 2 are present in the UNMT fine-tuning corpus. Under that reading, the reported BLEU/ROUGE/CLIP scores may measure recognition of memorized source messages rather than translation of unseen EC. This is load-bearing for the central claim of 'translation.' I request an evaluation on held-out EC messages that were not seen during UNMT fine-tuning, or at minimum a breakdown of translation performance by whether the test message occurred in the fine-tuning corpus.","section":"Section 6, Table 1 vs. Sections 4–5"},{"comment":"The paper does not specify the train/test split of the EC corpus used for UNMT fine-tuning. Please clarify whether the fine-tuning corpus includes the messages used as the test set in Table 2, and report the number of unique messages and the total corpus size per game type. Without this information, the reader cannot determine whether the translation metrics reflect generalization to novel messages or retrieval of memorized message–caption associations.","section":"Section 4, 'UNMT Architecture' and Section 5, 'Implementation Details'"},{"comment":"The random-agent baseline does not adequately control for the memorization concern. As shown in Table 1, random-agent messages also have zero novelty, so their messages would also appear in a corresponding fine-tuning corpus; the increased BLEU for trained EC could stem from a structured message-to-image mapping even under a memorization-only account. A stronger baseline would include an untrained but symbol-sampling agent, or a held-out evaluation where both trained and random EC messages are excluded from fine-tuning.","section":"Table 2, Baseline column"}],"minor_comments":[{"comment":"The heading '7. Bag-of-Symbols Disentanglement (BosDis):' appears twice, and the equations are not numbered, which makes referencing specific definitions difficult.","section":"Appendix E"},{"comment":"The 'Novelty (%)' row should state whether it refers to the source EC messages or the translated English text, and the Baseline column should be described in the main text (e.g., whether it uses the same random-agent EC corpus as Table 1).","section":"Table 2"},{"comment":"The main text says each message is composed of 6 symbols plus an EOS symbol, while Appendix D states a message length of 7 including EOS; please reconcile the notation.","section":"Section 5 vs. Appendix D"},{"comment":"The claim that Inter-category achieves the highest BLEU (9.21) is based on only 3 seeds; reporting per-seed results would help assess the stability of this comparison.","section":"Section 6, 'Game complexity vs. Translatability'"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern in the review is well-founded: the near-zero message novelty in Table 1 combined with the ambiguous composition of the UNMT fine-tuning corpus raises a serious possibility that the reported translation scores are inflated by memorization. This is the central empirical claim, so the authors should be required to either provide a proper held-out evaluation or substantially downscale the claim. The manuscript otherwise shows a thorough experimental structure, but the evaluation protocol needs to be made explicit and rigorous before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first paper I know of to apply unsupervised neural machine translation to emergent communication without parallel data, and the gap is real. It also ships a much more complete evaluation than most EC papers, with a random-agent baseline, a Chinese calibration, and a whole set of metrics. But the headline result has a leak. The test EC messages are largely the same messages the agents produced during game training (novelty 0.84–6.85%), and the UNMT fine-tunes on the EC corpus compiled from those games. The paper never says the held-out game test messages are excluded from that corpus. So the BLEU and ROUGE scores in Table 2 may reflect the model recognizing sequences it saw in fine-tuning, not translating new emergent language. The authors themselves note that agents rely on memorized messages, and the random-agent baseline doesn't help because both conditions can memorize.\n\nI should be clear that UNMT is trained without parallel pairs, so this isn't classic memorization of a source-target mapping. But cycle-consistency can still latch onto frequent source messages and generate plausible translations for them. Without an evaluation on messages that are genuinely unseen, we can't tell whether the model is translating or pattern-matching. That's a central flaw for the paper's core claim.\n\nCredit where it's due: the experimental setup is transparent, the limitations section is honest, and the observation that translation performance is uncorrelated across seeds is a genuinely interesting finding. The message-to-concept analysis in Appendix D gives some evidence that the emergent protocols have structure aligned with natural-language concepts, which is worth building on.\n\nThe weaker spots beyond the leak: the complexity comparisons rely on small BLEU/ROUGE differences with no significance testing; only three seeds were used for the UNMT runs while five were run for the games; and the absolute scores are low (BLEU 7–9), close to the Chinese calibration (5.65), so the practical interpretability is modest. The 'pragmatic EC' explanation is plausible but speculative.\n\nBottom line: the idea deserves a serious referee. The empirical support for it does not yet justify the abstract's claim. I'd send it to review with a major-revision request, specifically asking for a held-out set of novel EC messages and a breakdown of performance on those versus seen messages. If that holds up, this becomes a solid contribution to EC interpretability. For now, I wouldn't rely on the numbers.","headline":"First UNMT-for-EC paper with a transparent setup, but the central evaluation may be measuring recognition of seen messages rather than translation of novel ones.","tokens_in":18967,"tokens_out":5806,"would_cite":false,"duration_ms":53704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that unsupervised neural machine translation can translate emergent communication — the languages agents invent in referential games — into English without any parallel EC–English data.","keywords":["emergent communication","unsupervised neural machine translation","referential games","multi-agent communication","image captioning","compositionality","translatability","semantic diversity"],"falsifier":"Hold out all messages generated from images never seen during game training, or translate messages from a game seed not used in fine-tuning, and compare BLEU and ROUGE against the reported values; if scores drop toward the random-agent baseline (BLEU near 0.07), the reported translation is largely memorization rather than decoding of new emergent language.","tokens_in":18031,"feed_emoji":"💬","tokens_out":8650,"duration_ms":75902,"temperature":0.7,"pith_summary":"The paper tries to establish that agent-invented languages are not opaque: unsupervised neural machine translation (UNMT), trained only on monolingual corpora, can turn emergent communication into English captions without paired examples. The authors run referential games at four levels of contextual difficulty, record the messages agents exchange, and feed those messages together with an English image-caption corpus into a pretrained language model that is fine-tuned and refined by back-translation and denoising. Translation scores beat a random-communication baseline and roughly match an unsupervised Chinese-to-English run on the same pipeline, which the authors read as evidence that the emergent languages carry genuine, decodable structure. The practical payoff would be a way to inspect and interpret agent communication, and potentially other non-human communication, using only monolingual text and a shared visual world.","feed_headline":"Agent-invented languages can be decoded without parallel data","feed_subtitle":"Unsupervised machine translation reads emergent communication from referential games using only monolingual English captions.","key_machinery":"The machinery that carries the argument is a three-stage unsupervised neural machine translation pipeline applied to an emergent communication corpus. A high-resource English corpus first pretrains the model; the model is then fine-tuned jointly on the monolingual EC corpus and an English caption corpus, which creates a shared embedding space between the two languages; finally, iterative back-translation and denoising auto-encoding align the spaces without any parallel pairs. The referential game supplies the EC: a Sender sees an image and emits a six-symbol message from a 64-symbol vocabulary, and a Receiver must identify the target among distractors, with complexity set by how semantically close the distractors are. The load-bearing identity is that both languages are grounded in the same images: English captions and emergent messages describe the same visual world, so the unsupervised model can learn a mapping through that shared ground.","core_discovery":"The central claim is that emergent communication from referential games can be translated into English without any parallel EC–English data, and that the degree of translatability is shaped by the semantic structure of the task rather than by task difficulty alone. On the paper's own terms: an unsupervised neural machine translation system, initialized on English and fine-tuned on the corpus of recorded messages plus English captions, produces coherent English for all four game types, with BLEU scores between 6.08 and 9.21, ROUGE-L around 0.34–0.37, BERTScore near 0.73, and image–text alignment scores near 0.18, all clearly above a random-agent baseline. Inter-category games, whose distractors come from different but overlapping categories, yield the most translatable EC, while Supercategory games, with semantically close distractors, produce the richest but least translatable messages, and Category games, with fine-grained distinctions, produce pragmatic, low-entropy messages that remain translatable. The authors conclude that semantic diversity in the task enhances translatability, whereas constrained semantic variability produces pragmatic EC that is harder to interpret but still suitable for translation.","pith_inferences":["The shared-grounding recipe suggests a concrete transfer test: apply the same UNMT pipeline to other non-human communication systems whenever an aligned observational channel, such as video of the behavior, can serve as the common ground that captions provide here.","The differing translatability rankings across game types imply that 'complexity' is too coarse a design knob; future emergent-communication research could use translatability itself as a selection pressure when the goal is human-readable agent communication.","A targeted probe that removes the final words from English captions during fine-tuning could separate what an EC message genuinely encodes from what the language model fills in from caption-length priors, sharpening the notion of 'translatability'."],"forward_implications":["If the claim holds, emergent communication can be inspected by humans without collecting any paired EC–English data, removing a major bottleneck for interpretability of agent communication.","Translation quality does not increase monotonically with task difficulty; it is mediated by semantic diversity, so game design choices directly determine how readable the resulting language will be.","The near-zero correlation of translation scores across seeds on the same test set implies that each trained agent pair develops its own protocol, so there is no single 'emergent language' to translate.","Compositionality metrics can serve as predictors of translatability: messages aligned with image concepts translate better, while strongly disentangled symbol codes produce more novel but harder-to-translate outputs."],"supporting_citations":[{"why":"This paper supplies the three-stage UNMT system (pretraining, fine-tuning, and back-translation plus denoising) that the authors adapt to translate emergent communication.","marker":"Chronopoulou, Stojanovski, and Fraser 2020"},{"why":"This work establishes the feasibility of neural unsupervised machine translation without parallel data, which is the methodological foundation the paper builds on.","marker":"Lample et al. 2018a"},{"why":"This work defines the referential game framework in which agents develop emergent communication, the source of the EC corpora translated here.","marker":"Lazaridou, Peysakhovich, and Baroni 2016"},{"why":"This earlier work translates emergent communication using parallel EC–natural-language pairs, serving as the contrast that motivates the paper's no-parallel-data contribution.","marker":"Andreas, Dragan, and Klein 2017"},{"why":"This paper provides the image–text alignment score used to evaluate whether translations reflect the image that prompted the original message.","marker":"Hessel et al. 2021"},{"why":"This dataset supplies the English image captions used as the monolingual target corpus and as the reference captions for evaluating translation quality.","marker":"Lin et al. 2014"}],"fun_headline_variants":["Agent languages translated without any parallel data","Semantic diversity boosts translation of emergent communication","Unsupervised translation decodes emergent communication","No parallel data needed to decode agent languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the test EC messages are genuinely new inputs the translator must decode, but Table 1 shows that fewer than 7% of test messages are novel for every game, so the scores may largely reward recognizing messages the model already saw during fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Agent languages translated without any parallel data","Semantic diversity boosts translation of emergent communication","Unsupervised translation decodes emergent communication","No parallel data needed to decode agent languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3267,"prompt_tokens":898,"completion_tokens":2369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2315}},"tokens_in":514,"tokens_out":2369,"duration_ms":14271,"temperature":1.0,"reasoning_tokens":2315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:20:30.328086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out all messages generated from images never seen during game training, or translate messages from a game seed not used in fine-tuning, and compare BLEU and ROUGE against the reported values; if scores drop toward the random-agent baseline (BLEU near 0.07), the reported translation is largely memorization rather than decoding of new emergent language.","supporting_citations":[],"review_version":1}