{"id":"5928fccd-8ca3-4d04-bb92-256df91dcaf9","arxiv_id":"2505.18218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoMet couples a hypothesis-testing metaphor reasoner with a self-improving metaphor generator, and the resulting LLM agents win more often in metaphor-heavy language games.","lead":"CoMet is a prompting framework that helps LLM-based agents generate and interpret metaphors in two multi-agent language games, Undercover and Adversarial Taboo. The paper reports higher win rates and better concealment metrics for CoMet agents, but the evaluation lacks error bars and statistical tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoMet's experience pool stores exact word-pair identities and is populated on the same 200 word pairs later used for evaluation, so self-improvement gains may be pair-specific memorization rather than generalizable metaphor skill.","rationale":"I considered the reader's concern about LLM-judged metrics and absence of significance tests; both are real. However, the most load-bearing issue for the central claim is the training/evaluation overlap in the experience pool. The paper makes the self-improving metaphor generator a core contribution (Section 3.5), and Figure 6(b) explicitly credits more experiences for higher success. But because the same 200 word pairs are reused for all evaluation episodes and the pool stores the exact pair, later episodes can exploit pair-specific memories. This would not require any metric-validation problem or statistical subtlety to invalidate the generalization claim. I do not see evidence of intentional misconduct; the issue is an experimental design gap. A held-out split or first-episode-vs-later comparison would settle it. My final recommendation stays CONDITIONAL, matching the reader: the paper's evidence is suggestive but not yet sufficient, and this leakage test should be required before acceptance.","tokens_in":25883,"tokens_out":7135,"duration_ms":61575,"concrete_test":"Re-run the Undercover evaluation with a strict split: train/populate the experience pool only on one half of the word pairs, evaluate on the other half, and compare CoMet to CoT and CoMet w/o Met on those unseen pairs. Additionally, within the current setup, compare episodes 1 vs 10 on the same word pair: if later episodes win more only when the same pair is in the pool, that is memorization. If CoMet's advantage survives on held-out pairs, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that CoMet improves general metaphor-based communication, not just performance on a fixed word list. Section 4.1 says 200 word pairs are each tested across 10 episodes. Appendix C describes an experience pool whose entries record the exact word pair (Figure 7: 'words': ['snake','lizard']) and that persists across games, with retrieval of 'relevant experiences' and replacement after use. Therefore, after the first episode with a given pair, the metaphor generator can retrieve a metaphor already tuned for that exact pair from earlier self-play on the same pair. The 0/50/100-experience curve in Figure 6(b) is thus confounded: more experiences also means more repeated exposures to the same evaluation pairs. Table 1's CoMet win rates may reflect pair-specific memorization (including knowledge of the other word from stored entries) rather than improved metaphor reasoning/generation. The paper reports no held-out word-pair evaluation and no separation between experience-pool training pairs and evaluation pairs. This is load-bearing because the self-improving generator is a core claimed contribution, and if the effect evaporates on unseen pairs the abstract's 'significantly enhances' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CoMet is a framework that augments LLM-based agents with a hypothesis-based metaphor reasoner and a self-improving metaphor generator, and evaluates it in two multi-agent language games: Undercover and Adversarial Taboo. The paper reports that CoMet raises attackers' win rates by 47% and defenders' by 30% in Adversarial Taboo compared with baselines, and that in Undercover the undercover win rate rises from 0.20 with CoT to 0.35 with CoMet (Table 1). Additional metrics (FER, OIAA, SIAA, PPC, IISC) computed from game logs are used to argue that CoMet improves feature extraction, identity reasoning, privacy protection, and deceptive capability. The central claim is that metaphor-driven reasoning and generation significantly improve covert and deceptive communication in these language games.","tokens_in":26014,"tokens_out":4180,"duration_ms":32645,"significance":"The paper opens a genuinely underexplored direction: using metaphors as a communication strategy in interactive multi-agent language games rather than on static metaphor-detection or generation benchmarks. Its concrete assets include an implemented framework, released code, a curated dataset of 200 word pairs, experiments across five different LLMs, and an ablation study of the main components. If the empirical claims hold, the work would be a useful demonstration of LLM agents exploiting metaphorical expression for concealment, misdirection, and privacy-aware communication. However, the current evaluation does not establish that the improvements generalize beyond the exact word pairs used for experience accumulation, and the reported magnitudes are not supported by error bars, confidence intervals, or human-validated metrics. The significance is therefore conditional on resolving these evaluation threats.","major_comments":[{"comment":"The experience pool stores exact word-pair identities, as shown in Figure 7, which records the entry \"words\": [\"snake\",\"lizard\"], and Section 4.1 states that the 200 word pairs used for evaluation are each tested across 10 episodes. Because the metaphor generator retrieves and reuses experiences across games, including self-play on the same evaluation pairs, the improvements reported in Figure 6(b) and Table 1 may reflect pair-specific memorization rather than a generalizable metaphor-generation capability. The paper provides no held-out word-pair split and no comparison on unseen pairs, so the \"self-improving\" claim is not yet established. Please add a held-out evaluation in which the experience pool is trained on one set of word pairs and evaluated on a disjoint set, or otherwise demonstrate that gains transfer to pairs never encountered during experience accumulation.","section":"Appendix C, Section 4.1, Figure 6(b)"},{"comment":"The metrics FER, OIAA, SIAA, PPC, and IISC (defined in Table 4 and Appendix D) are computed by LLM-based analysis of game logs, without independent human annotation or grounding in a held-out benchmark. Since these metrics support the non-win-rate claims in Table 1 and Figure 5, their reliability is load-bearing for the paper's assertions about feature extraction, identity reasoning, and privacy protection. The authors should provide human validation on a sample of logs, report inter-annotator agreement, or otherwise demonstrate that the LLM judgments correspond to verifiable ground truth.","section":"Section 4.1, Table 4, Appendix D"},{"comment":"No variability measure or statistical test is reported for any of the main results. Figure 4 states attackers' win rates increased by 47% and defenders' by 30%, and Table 1 reports differences such as 0.35 vs 0.20, but there are no confidence intervals, per-pair distributions, or significance tests. With only 200 word pairs and 10 episodes per pair, the reader cannot assess whether these differences are robust or within the noise of the games. Please report per-pair win-rate distributions, bootstrap confidence intervals, or appropriate significance tests for the key comparisons.","section":"Section 4.2, Table 1, Figure 4"}],"minor_comments":[{"comment":"The column header \"Win Rate↓\" appears to use the wrong arrow, because the full CoMet row has the highest value (0.70) and the paper presents it as the best configuration; presumably ↑ was intended.","section":"Table 2"},{"comment":"The axis labels in Figure 4(b) are difficult to parse; please clarify whether each entry denotes attacker-method versus defender-method, and specify which role is fixed as CoT when reading the curves.","section":"Figure 4(b)"},{"comment":"Adversarial Taboo appears to be run with a simplified prompt that does not instantiate the full CoMet pipeline shown in Figure 2 (Feature Extractor, Belief Mapper, Self-Monitor, Strategy Planner). Please clarify which modules are active in that game and whether the reported CoMet results use the same framework as Undercover.","section":"Appendix F, Figure 19"},{"comment":"The Balanced Metrics formula M'_i = avg(M_i^Civ, M_i^Und) - Var(M_i^Civ, M_i^Und) is nonstandard and can produce values outside the range of the original metrics; please provide a justification for subtracting the variance or use a more conventional correction.","section":"Section 4.1, Balanced Metrics"},{"comment":"The assertion that \"it is infeasible to use our method to make LLMs output risky content\" is unsupported by the experiments, which only test daily-vocabulary game words; please soften or remove this categorical claim.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The experience-pool contamination is the main risk to the paper's central claim, and it is fixable in principle by a held-out word-pair evaluation. The LLM-computed metrics also need human validation. The work is within scope for the journal and has useful released artifacts, but the current evidence is not yet sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoMet is the first thing I've seen that treats metaphor generation as a communication strategy for LLM agents in interactive games, and the framework is described clearly enough to reproduce. The hypothesis-based reasoner (binary yes/no) is a neat simplification, and the generator's experience pool is a plausible mechanism for self-improvement. The ablations are thorough, they test several LLMs, and they release code and word lists. Credit where due: the paper makes a genuine new move, not just a new benchmark.\n\nBut the evidence as reported does not establish the abstract's 'significantly enhances.' The win rates in Figure 4 come with no error bars, confidence intervals, or significance tests. With 200 pairs × 10 episodes, the effect could be real, but the paper doesn't show it. The auxiliary metrics (FER, PPC, IISC, etc.) are computed by LLM judgment on the same logs, with no human validation. That's a weakness, not a fatal one.\n\nThe load-bearing problem is experience-pool contamination. Appendix C shows entries storing the exact word pair (e.g., 'snake'/'lizard') and says the pool persists across games. The evaluation uses the same 200 word pairs for all episodes. So after the first episode on a pair, CoMet can retrieve a metaphor already tuned on that exact pair from earlier self-play. Figure 6(b) shows a 29% improvement with 100 experiences, but that curve is exactly what you'd see from repeated exposure to the test items. There is no held-out word-pair evaluation, no separation between pool-training pairs and evaluation pairs. This means the self-improvement claim may be pair-specific memorization, not generalizable metaphor skill. The Limitations section talks about cultural transfer of idiomatic metaphors but never mentions this overlap, which is more serious.\n\nAlso minor: the reasoner depends on threshold T and weights wf/wm that are never specified, so reproducibility suffers.\n\nWho is this for? Researchers working on LLM agents for language games or on non-literal communication in interactive settings. It's a worthwhile design probe, but I would not build on it until the contamination is resolved.\n\nRecommendation: send to peer review. The idea is novel and the framework is well-specified, and the main flaw is empirically fixable. Referees should demand held-out pair evaluation, significance testing, and human-validated metrics. If the effect survives, this is a solid contribution.","headline":"Good idea, shaky evidence: the self-improvement claim is undercut by experience-pool contamination on the same word pairs used for evaluation.","tokens_in":26624,"tokens_out":5600,"would_cite":false,"duration_ms":42391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoMet gives LLM agents metaphor reasoning and self-improving generation, raising win rates in two covert-communication games.","keywords":["metaphor reasoning","covert communication","multi-agent language games","large language models","Undercover","Adversarial Taboo","self-improving metaphor generation","semantic evasion"],"falsifier":"Run the same two games with human annotators labeling each speech for whether it leaked the secret word and each vote for whether it identified the correct teammate, over enough episodes to compute confidence intervals; CoMet's claimed advantage would be refuted if its win-rate gap over baselines disappears or the human labels contradict the LLM-judged metrics.","tokens_in":25568,"feed_emoji":"🎭","tokens_out":6935,"duration_ms":52592,"temperature":0.7,"pith_summary":"The paper sets out to show that metaphors can be a working communication strategy for LLM agents, not just a literary decoration. It introduces CoMet, a framework that adds a hypothesis-testing metaphor reasoner and a self-improving metaphor generator to a standard LLM game agent, and evaluates it in two multiplayer language games where concealment and misdirection decide the outcome. The central empirical claim is that CoMet materially improves covert communication: in Adversarial Taboo, attacker win rate rises 47% and defender win rate 30% over baselines, and in Undercover, undercover agents' win rate increases from 0.20 with Chain-of-Thought to 0.35 with CoMet. If true, the result matters because covert communication—saying something that allies decode but opponents cannot—is a general social capability, and language games stress the exact skills an agent would need for it.","feed_headline":"Metaphor reasoning lifts LLM win rates by up to 47 percent","feed_subtitle":"A reasoner-plus-generator loop helps agents hide meaning and deceive opponents in two language games.","key_machinery":"The carrying object is a five-stage agent pipeline with two metaphor-specific components. The Hypothesis-Based Metaphor Reasoner receives any utterance that does not literally fit the agent's secret word, frames two competing hypotheses (speaker means my word, or speaker means the other word), expands the utterance into ontological, structural, and spatial metaphor aspects, and scores the match between those aspects and the word's features on a six-level scale before accepting or rejecting the hypothesis. The Metaphor Generator maintains a self-play experience pool: each generated metaphor is stored with its explanation, evaluator feedback, usage statistics, and success score, and after every five games low-performing, over-referenced entries are pruned while new high-quality ones replace them, so the agent's metaphorical speech improves with accumulated gameplay.","core_discovery":"CoMet's central claim is that LLM agents fail at covert communication mainly because they interpret and produce language literally, and that a scaffolded metaphor process fixes this. The paper argues that in Undercover, civilians can use metaphors as 'concept camouflage': a clue like 'it wears a crown' lets teammates identify 'pineapple' while the undercover team cannot pin down the word. In Adversarial Taboo, the same machinery lets an attacker steer a defender toward a secret word or a defender avoid it, by substituting metaphorical descriptions for direct ones. The authors report that the framework generalizes across six LLMs and that each component contributes, with the Self-Monitor module being especially load-bearing: removing it drops the Undercover win rate to 0.05, below the Chain-of-Thought baseline.","pith_inferences":["Because the reasoner deliberately reduces metaphor interpretation to a binary match against a secret word, the same design could be lifted out of games into any setting where two parties share private context and want to exclude eavesdroppers, such as identity verification or protected coordination.","A natural stress test is to push the experience pool beyond its 100-entry cap and beyond the food and animal word domains; the paper's own statistics suggest metaphor categories differ in quality, so domain transfer is not guaranteed.","The quantitative metrics are computed by LLM judgment on game logs with no reported human annotation or significance testing, so a human-labeled replication is the most direct way to see whether the covert-communication advantage is real or partly an artifact of the evaluator model."],"forward_implications":["In Adversarial Taboo, CoMet outperforms baseline agents as both attacker and defender, with reported win-rate gains of 47% and 30% respectively.","In Undercover, CoMet raises the undercover win rate from 0.20 (CoT) to 0.35 and the civilian win rate to 0.85 when facing the CoT baseline.","The win-rate advantage generalizes across the models tested, and CoMet reduces failure rates below 15% for every LLM in the Adversarial Taboo generalization experiment.","The ablation results imply that identity self-monitoring, not metaphor alone, is necessary for the gains: without it, CoMet's win rate drops to 0.05."],"supporting_citations":[{"why":"Defines the Undercover game and the multi-agent reasoning baseline that CoMet is evaluated against.","marker":"Xu et al., 2024"},{"why":"Provides the Adversarial Taboo implementation and word-selection protocol used in the experiments.","marker":"Cheng et al., 2024"},{"why":"Original Adversarial Taboo game that defines the attacker-defender task CoMet is tested on.","marker":"Yao et al., 2021"},{"why":"Chain-of-Thought prompting, the main reasoning baseline CoMet is compared with.","marker":"Wei et al., 2022"},{"why":"Replace-based metaphor reasoning, the alternative reasoning method CoMet's hypothesis-based reasoner is measured against.","marker":"Tong et al., 2024"},{"why":"Conceptual metaphor theory whose ontological, structural, and spatial categories are injected as knowledge into both the reasoner and the generator.","marker":"Lakoff and Johnson, 2008"}],"fun_headline_variants":["Metaphor reasoning lifts LLM win rates by up to 47%","CoMet teaches LLMs covert metaphor communication","LLMs use metaphor camouflage to win language games","Self-monitoring key to LLM metaphor game strategy","Undercover agents win more with metaphor-driven clues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the win rates and the LLM-computed metrics of privacy protection, feature extraction, identity assessment, and deceptive statements faithfully measure real covert-communication ability, despite having no human annotation or significance testing.","fun_headline_variants_meta":{"raw":{"variants":["Metaphor reasoning lifts LLM win rates by up to 47%","CoMet teaches LLMs covert metaphor communication","LLMs use metaphor camouflage to win language games","Self-monitoring key to LLM metaphor game strategy","Undercover agents win more with metaphor-driven clues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1375,"prompt_tokens":863,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":435}},"tokens_in":479,"tokens_out":512,"duration_ms":4538,"temperature":1.0,"reasoning_tokens":435,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:43:09.674663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two games with human annotators labeling each speech for whether it leaked the secret word and each vote for whether it identified the correct teammate, over enough episodes to compute confidence intervals; CoMet's claimed advantage would be refuted if its win-rate gap over baselines disappears or the human labels contradict the LLM-judged metrics.","supporting_citations":[],"review_version":1}