{"id":"95ffece6-17d5-4915-b194-747e65817f33","arxiv_id":"2607.03752","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"EmCom-Diffusion measures visual reflection of emergent languages by reconstructing source images from messages with a fine-tuned diffusion model and scoring perceptual similarity to the originals.","lead":"The paper introduces EmCom-Diffusion, a way to measure how much visual information sits inside an emergent language by fine-tuning a text-to-image diffusion model on message-image pairs and scoring how similar the reconstructed image is to the original. It matters because existing EmCom metrics only check proxies (concepts, captions, distances, game accuracy) and can miss or invent visual content.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The generative bottleneck claim is under-tested: SD prior + fine-tuning can invent content the message never encoded, so perceptual similarity may overstate visual reflection.","rationale":"The Reader correctly flags the SD prior + perceptual s as the weakest assumption and still assigns CONDITIONAL. That diagnosis is load-bearing: if the generative bottleneck does not hold, the claim that EmCom-Diffusion uniquely recovers visual content missed by the four baselines is no longer secure, even though the ranking Random/Fixed < EL < SD-NL and the triplet experiments remain informative. The paper already acknowledges the prior and the single-game/domain scope; the concrete ablation above would settle whether the residual concern is fatal or merely a quantified bias. No stronger internal inconsistency appears, so the verdict stays CONDITIONAL rather than moving to REJECT or ACCEPT.","tokens_in":12557,"tokens_out":570,"duration_ms":7598,"concrete_test":"On a held-out set of 500 images, ablate message capacity (e.g., K=2 vs K=8, or randomly mask half the tokens) and measure both EmCom-Diffusion (DINOv2/CLIP-img) and a controlled attribute probe: fraction of original object categories / spatial relations that appear in ˆx (via an off-the-shelf detector or VQA). If similarity remains high while attribute recovery collapses, the score is inflated by prior invention and the bottleneck claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that EmCom-Diffusion 'directly' measures visual reflection rests on the generative-bottleneck argument in §2.2: because G_θ* must produce a full image from m alone, content absent from m should be absent from ˆx and lower s(ˆx, x). Limitations and Table 2 already show that random/fixed tokens still yield plausible images (CLIP-img ~0.49, non-zero Vendi/Recall), so the pretrained prior supplies substantial visual content independent of m. Fine-tuning on (x, m) pairs can further teach the model to map sparse tokens onto full MS-COCO-like scenes by exploiting dataset regularities rather than recovering only what m encodes. Consequently Eq. (2) can credit invented content (or suppress content the message actually carries but the prior/s cannot express), undermining the claim that the score is a faithful external decoder independent of speaker–listener and free of the proxy failures of CBM/Translation/TopSim/R@1. The multi-seed analysis (Table 3) and random/fixed baselines control only for input-independent plausibility, not for message-conditioned hallucination of unencoded attributes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper defines visual reflection as the extent to which emergent-language messages preserve recoverable information about their source images without further appeal to the original speaker–listener pair. It proposes EmCom-Diffusion: fine-tune a pretrained text-to-image diffusion model on (image, emergent-message) pairs (Eq. 1) and score reflection as the average perceptual similarity s between the generated reconstruction and the original image (Eq. 2). Instantiated with a Referential Game on MS-COCO, Experiment 1 shows the expected ranking Random/Fixed < EL < SD-NL across CLIP-img, DINOv2 and SigLIP, plus multi-seed controls (Tables 2–3, Fig. 2). Experiment 2 constructs targeted triplets that expose known limitations of CBM, supervised translation, TopSim and R@1 and shows EmCom-Diffusion retains more signal (Tables 4–6). The authors conclude that the generative comparison captures visual content the four baselines miss or spuriously credit.","tokens_in":12844,"tokens_out":1190,"duration_ms":9016,"significance":"If the generative score is a faithful external decoder of visual content, EmCom-Diffusion would supply a missing, annotation-free instrument for a central open problem in emergent communication—what the language actually encodes about its visual inputs—beyond formal properties such as compositionality or task success. The work is concrete: it ships a clear formal definition (Eqs. 1–2), a reproducible pipeline (code link, Appendix A–C), multi-encoder and multi-seed controls, and carefully constructed contrastive tests against four established metrics. Even if the absolute score is partly prior-dependent, the relative ranking and the triplet results already give the community a usable generative probe that existing proxies lack.","major_comments":[{"comment":"§2.2 generative-bottleneck argument and Eq. (2): the claim that content absent from m is necessarily absent from ˆx (and therefore lowers s) is under-supported. Table 2 already shows that random/fixed tokens produce non-degenerate images (CLIP-img ≈0.49, non-zero Vendi/Recall), so the Stable Diffusion prior supplies substantial visual content independent of m. Fine-tuning on (x,m) pairs can further teach the model to map sparse tokens onto full MS-COCO-like scenes by exploiting dataset regularities rather than recovering only what m encodes. The multi-seed analysis (Table 3) and random/fixed baselines control only for input-independent plausibility, not for message-conditioned hallucination of unencoded attributes. Without an ablation that isolates prior contribution (e.g., frozen vs. fine-tuned generator, or controlled attribute-masking of m), the assertion that EmCom-Diffusion is a fai","section":null},{"comment":"§3.1 / Appendix A and §3.3: the speaker uses a frozen DINOv2 ViT-B/14 backbone, and DINOv2 is also one of the three readout encoders for the EmCom-Diffusion score and the primary readout in the triplet tests of Experiment 2. Although CLIP and SigLIP are also reported in Table 2, the strongest discrimination claims (Tables 4–5) rest on the DINOv2 cosine. This shared backbone introduces a mild circularity risk: the metric may preferentially recover features already privileged by the speaker’s visual encoder rather than arbitrary visual content. A control that freezes a different backbone for the speaker (or reports the full triplet tables under CLIP/SigLIP readouts) is needed to confirm that the advantage over baselines is not backbone-specific.","section":null}],"minor_comments":[{"comment":"Limitations section: the dependence on the generator prior and the single-game/single-domain scope are acknowledged, but the text still asserts that the generative bottleneck “fundamentally restricts information hiding.” Soften that language to match the empirical caveats already present in Table 2.","section":null},{"comment":"Table 2 / Fig. 2: qualitative examples show individual EL samples occasionally outscoring SD-NL; a short note that these are sample-level fluctuations (as the caption already hints) would prevent over-reading.","section":null},{"comment":"§3.3: the visual-similarity threshold τ=0.7 is justified as the 99th percentile, but sensitivity of the triplet accuracies to τ (or to an alternative object-category match) is not reported; a one-paragraph robustness check would strengthen the claim.","section":null},{"comment":"Notation: L_diff in Eq. (1) is never expanded; a brief reminder that it is the standard latent-diffusion denoising loss would help readers outside the diffusion literature.","section":null},{"comment":"Appendix C: CBM inventory is restricted to the 80 MS-COCO categories via dominant-instance count; this design choice is reasonable but should be flagged when claiming that CBM “cannot” capture content outside the inventory, since a richer inventory might change the gap.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the experimental design is careful; the main risk is over-claiming “direct” measurement given the strong SD prior. If the authors add the requested prior-isolation and backbone-control ablations (or substantially qualify the generative-bottleneck language), the paper would be a solid contribution. Scope fits a CV/ML venue that already publishes evaluation-metric and multi-agent communication work."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is a concrete, annotation-free pipeline for probing what visual content sits in emergent messages: fine-tune a diffusion model on (image, message) pairs and score reconstruction similarity. That is new for EmCom and cleaner than the usual proxies.\n\nWhat they do well is the experimental design. Experiment 1 gives the expected ranking Random/Fixed < EL < SD-NL across CLIP, DINOv2, and SigLIP, plus a multi-seed check that the signal is input-specific. Experiment 2 builds targeted triplets that isolate the known failure modes of translation (caption-similarity control), TopSim/CBM (matched edit distance), and R@1 (random vs hard distractors). EmCom-Diffusion keeps signal where the others collapse or become setup-dependent. Bootstrap CIs are reported, code is promised, and the limitations section is honest about single-game/single-domain scope. The formal definition (Eq. 2) and the comparison table are clear.\n\nThe soft spot is real but not fatal. The generative-bottleneck argument in §2.2 claims that content absent from m must be absent from the reconstruction. Table 2 already shows random/fixed tokens still produce plausible images (CLIP-img ~0.49), so the SD prior supplies a lot of visual content. Fine-tuning can teach the model to fill in MS-COCO regularities rather than recover only what the tokens encode. The random/fixed baselines and multi-seed analysis control input-independent plausibility, not message-conditioned hallucination. So the score is not as “direct” or speaker-listener-independent as advertised; it is still a useful relative probe once you treat the prior as part of the instrument. Mild circularity risk from reusing DINOv2 in the speaker and as a readout is secondary.\n\nMath and citations look standard for the area; no invented formalisms beyond the named score. This is for people who evaluate multi-agent communication or want better diagnostics of discrete codes. It deserves a serious referee. I would engage with it, re-run the code, and test other generators/games before treating the absolute numbers as ground truth.","headline":"Solid new EmCom evaluation method with careful triplet tests; the generative-bottleneck claim is overstated but the baselines and experiments still make the metric useful.","tokens_in":13471,"tokens_out":522,"would_cite":true,"duration_ms":4457,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"EmCom-Diffusion measures how much visual content emergent languages encode by regenerating each source image from its message and scoring the match.","keywords":["emergent communication","visual reflection","text-to-image diffusion","evaluation metrics","referential game","image reconstruction","MS-COCO"],"falsifier":"If reconstructions from random or fixed tokens scored as high as those from trained emergent messages on held-out images, or if EmCom-Diffusion failed to prefer the visually similar candidate in the caption-controlled and edit-distance-matched triplets while the other metrics succeeded, the claim that it measures recoverable visual content would be falsified.","tokens_in":13428,"feed_emoji":"🖼️","tokens_out":970,"duration_ms":15163,"temperature":0.7,"pith_summary":"When agents invent a private language to talk about images, how much of the actual visual scene ends up in the messages? Existing checks only use proxies—fixed concept lists, natural-language captions, distance correlations, or success at a referential game—and can miss content that is present or credit content that is not. This paper introduces EmCom-Diffusion: fine-tune a pretrained text-to-image diffusion model on pairs of images and emergent messages, then generate images from the messages alone and compare them to the originals with perceptual similarity. On MS-COCO with a referential game, the method ranks real emergent languages above random and fixed tokens across three visual encoders, and it still separates visually similar images in settings where translation, concept matching, topographic similarity, and game accuracy fail. A reader cares because it offers a direct, annotation-free way to ask how much of the physical world is reflected in a language that formed from scratch.","feed_headline":"Agent languages scored by how well they regenerate images","feed_subtitle":"A diffusion model fine-tuned on message–image pairs recovers visual content that concept and caption metrics miss.","key_machinery":"EmCom-Diffusion: fine-tune a text-conditional diffusion model so it reconstructs each image from its emergent message alone, then score average perceptual similarity between reconstructions and originals on held-out pairs. The generative bottleneck is the instrument: content absent from the message should be absent from the generated image.","core_discovery":"Visual reflection—the recoverability of an image’s visual content from its emergent message without using the original speaker–listener pair—can be measured directly by fine-tuning a pretrained text-to-image diffusion model on (image, message) pairs and scoring the perceptual similarity between the generated reconstruction and the original image. On MS-COCO with a Referential Game, EmCom-Diffusion correctly ranks Random/Fixed < Emergent Language < natural-language captions across CLIP, DINOv2, and SigLIP, and in targeted tests it captures visual distinctions that CBM, supervised translation, TopSim, and R@1 miss or spuriously credit.","pith_inferences":["If the method holds beyond referential games and MS-COCO, it could become a standard check whenever someone claims an emergent protocol is grounded rather than merely task-successful.","Tracking the score over training epochs could show whether visual reflection appears early then drifts, or accumulates—something formal compositionality metrics do not capture.","The larger EL-versus-random gap on a vision-only encoder hints that emergent codes may lock onto coarse object structure more readily than fine layout; token-level attribution would test that.","The same generative probe could be applied to other discrete visual codes to measure preserved content without human labels."],"forward_implications":["Visual reflection can be scored without human concept inventories or paired captions.","Caption translation, concept matching, distance-rank correlation, and distractor-dependent game accuracy systematically mis-estimate what emergent messages encode.","Generative reconstruction yields a stable information ranking (random/fixed < emergent language < natural-language captions) across multiple vision encoders.","The same pipeline can be run across training to track how world-reflecting structure forms in a language grown from scratch.","Regularities that never map to English nouns remain measurable because the comparison target is the image itself."],"fun_headline_variants":["Emergent languages scored by diffusion image reconstructions","Visual reflection measured via message-to-image regeneration","EmCom-Diffusion recovers images from agent language codes","Agent messages ranked by source-image reconstruction fidelity","Diffusion probes how well emergent messages retain visual content"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes a fine-tuned image generator plus a perceptual similarity score can recover what the message encodes without inventing missing details from its own prior or discarding content the message actually carried.","fun_headline_variants_meta":{"raw":{"variants":["Emergent languages scored by diffusion image reconstructions","Visual reflection measured via message-to-image regeneration","EmCom-Diffusion recovers images from agent language codes","Agent messages ranked by source-image reconstruction fidelity","Diffusion probes how well emergent messages retain visual content"]},"model":"grok-4.5","effort":"low","cost_usd":0.00658,"raw_usage":{"total_tokens":1724,"prompt_tokens":845,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":65800000,"prompt_tokens_details":{"text_tokens":845,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":805,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":845,"tokens_out":74,"duration_ms":6076,"temperature":1.0,"reasoning_tokens":805,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T00:11:22.798896+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If reconstructions from random or fixed tokens scored as high as those from trained emergent messages on held-out images, or if EmCom-Diffusion failed to prefer the visually similar candidate in the caption-controlled and edit-distance-matched triplets while the other metrics succeeded, the claim that it measures recoverable visual content would be falsified.","supporting_citations":[],"review_version":1}