{"id":"cba7a454-4ce7-4a4a-9ca9-7dc618d97a1a","arxiv_id":"2502.06060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A listening-plus-speaking training scheme, where language model agents are rewarded for influencing teammates' beliefs about the impostor, doubles crewmate win rates in a simulated Among Us game compared to RL alone.","lead":"Researchers trained a small language model to play a simplified version of the social deduction game Among Us, teaching crewmates to talk about what they see and to use each other's statements to guess who the impostor is. The method improved crewmate win rates to about double what standard reinforcement learning achieved, without using any human game transcriptions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) rewards the speaker's change in its own belief because C_t in Eq. (5) includes the speaker, so the speaking-reward gain may reflect self-persuasion rather than influence on other agents.","rationale":"The reader's weakest assumption was that the belief-change reward attributes observed changes to the speaker's message and that the beliefs are calibrated. I agree that the causal attribution is the central risk, but I sharpen it into a concrete, internal defect: Eq. (5) sums over all living crewmates, including the speaker, so the reward in Eq. (6) mechanically includes the speaker's own pre-to-post belief change. This does not require any assumption about calibration; it follows from the equations as written. If the implementation matches the equations, the speaking reward can be optimized by self-influence, and the small but positive RL+L+S over RL+L gap in Fig. 3 becomes ambiguous. If the implementation already excludes the speaker, then the paper should say so, and the concern reduces to a documentation error. I would not reject the paper: the listening loss and the overall RL+L+S pipeline still produce win rates around 0.51-0.56 and the code release makes the check straightforward. The reader's conditional verdict remains appropriate, hence UNCHANGED.","tokens_in":18396,"tokens_out":7508,"duration_ms":73603,"concrete_test":"On held-out rollouts from the released RL+L+S checkpoint, recompute each speaking reward as published and with the speaker excluded: r_other = sum_{k in C_t \\ {i}} [p_k^post(q) - p_k^pre(q)] and r_self = p_i^post(q) - p_i^pre(q). Report the mean absolute and variance of r_self and r_other across all speaking turns. If r_self contributes materially (e.g., more than 10 percent of the total reward magnitude or variance), retrain RL+L+S from the same seeds with Eq. (5) corrected to exclude the speaker and compare base-environment win rate against RL+L. The speaking-reward claim stands only if the corrected RL+L+S still exceeds RL+L by a comparable margin.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the speaking reward, Eq. (6), r_s^t = B_t - B_t', where Eq. (5) defines B_t as the sum over all living crewmates C_t. The speaker i is a living crewmate, so B_t and B_t' both include i's own belief. The only event between the two survey timesteps is i's own message, so the change in the speaker's own belief is credited to that speaker. This is internally inconsistent with the paper's stated goal of rewarding influence on other agents (abstract and Section 5.2). Because the trained policy generates both the message and the post-message belief, the reward can be increased by messages that shift the model's own subsequent vote distribution toward the true imposter, without any requirement that other agents learned anything. The reported fabrication of evidence (Section 6.1, Appendix D) is consistent with this misspecification. The consequence is that the marginal win-rate gain of RL+L+S over RL+L, which the paper itself describes as relatively small, may not be evidence of improved communication. If the released code follows Eq. (5), the speaking-loss claim is not yet supported; if it excludes the speaker, then Eq. (5) is mis-specified and the discrepancy needs to be documented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for training language-model agents in the social deduction game Among Us without human demonstrations. The method decomposes communication into listening (supervised prediction of the true imposter after discussion messages) and speaking (a reward based on the change in teammates' beliefs in the true imposter after each message), combined with PPO and a world-modeling loss. The authors report that the full method roughly doubles crewmate win rates relative to RL-only, and that trained agents exhibit human-like behaviors such as direct accusations and evidence sharing. The paper includes code and model releases.","tokens_in":18621,"tokens_out":6872,"duration_ms":62053,"significance":"If the method works as described, it would be a useful contribution to training LLM agents to coordinate through natural language in partially observable settings, without needing task-specific human demonstrations. The idea of grounding communication in a dense prediction reward is appealing, and the Among Us-like environment is a good testbed. The paper ships code and models, which is a strength. However, the speaking reward as defined appears to credit the speaker with changes in its own belief, and the reported error bars are min-max over three seeds, which weakens the empirical support for the 'doubling' claim and for the additive benefit of the speaking term.","major_comments":[{"comment":"The speaking reward r_s^t = B_t - B_t' uses B_t = sum_{k in C_t} pi_k(q|tau_k^t) over all living crewmates C_t, which includes the speaker i. Since the speaker's own post-message belief is part of B_t, the reward can be increased by messages that shift the speaker's own posterior toward the true imposter, without requiring any effect on other agents. This contradicts the stated goal in the abstract and Section 5.2 of rewarding 'influence on other agents.' The paper should either exclude the speaker from the sum (if that is the intended semantics) or justify why self-influence is a legitimate component of the reward. If the released code already excludes the speaker, then Eq. (5) is mis-specified and should be corrected; if not, the current experiments do not isolate the effect of communication on other agents.","section":"Section 5.2, Eq. (5)-(6)"},{"comment":"The paper states that the win-rate difference between RL+L and RL+L+S is 'relatively small' (Section 6.1), but the central claim of 'doubling the win rates' refers to RL+L+S vs RL. The reported error bars are only the minimum and maximum over three seeds, with no standard errors, confidence intervals, or number of evaluation episodes. This makes it impossible to assess whether the speaking reward provides a statistically significant improvement over RL+L, or whether the differences among the curves in Fig. 4 are meaningful. Please report the evaluation protocol (number of episodes per seed, standard errors) and, if possible, provide a significance test or at least non-overlapping confidence intervals for the key comparisons.","section":"Section 6.1, Fig. 3"},{"comment":"The speaking reward is an internal measure based on the trained agents' own belief updates, and the paper acknowledges (Appendix D, Section 7) that agents sometimes make up evidence and state whatever is most convincing. This raises the concern that the reward may incentivize persuasive falsehoods rather than truthful, informative communication, especially since the belief probabilities are produced by the same models being trained. The paper should validate the speaking reward against an external, task-level signal (e.g., correlation with eventual voting accuracy or with the factual correctness of statements) or at least discuss this risk explicitly as a limitation of the reward definition itself. The current discussion treats deception as an emergent human-like behavior, but it may also indicate a misspecified reward.","section":"Section 5.2 and Appendix D"}],"minor_comments":[{"comment":"Typo: 'animposter' should be 'an imposter'.","section":"Section 5.2"},{"comment":"The caption says 'maximum and minimum expected win rates,' but the values are observed min/max across three runs, not expected values; please rephrase.","section":"Fig. 3 caption"},{"comment":"The hyperparameter lambda_BC is listed but never defined in the main text or appendix; please either define it or remove it.","section":"Appendix E, Table 2"},{"comment":"Please state the number of evaluation episodes used per seed for the win-rate estimates in Figs. 3-5; without this information the reader cannot gauge the precision of the reported differences.","section":"Section 6.1"},{"comment":"The introductory sentence 'All samples below are from the perspective of the imposter' is confusing because some samples appear to be from the perspective of a crewmate; please clarify the perspective in each sample.","section":"Appendix C"},{"comment":"The design choice of freezing one crewmate to the listening-only policy pi_L is central to the training and evaluation, but the paper does not analyze how sensitive the results are to this choice (e.g., varying the number of frozen agents or the frozen policy type); a brief ablation or discussion would strengthen the robustness claims.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the definition of the speaking reward, which appears to credit the speaker with its own belief change. I recommend the authors be asked to either correct Eq. (5) to exclude the speaker or provide a justification for including it, and to re-run the key experiments if necessary. The empirical claims also need proper statistics (standard errors, evaluation episodes). The paper's core idea is promising and the code release is a plus, so I do not recommend rejection, but the paper should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. The core idea—decomposing communication into a supervised listening loss (predict the imposter from the discussion) and an RL speaking reward (belief change after a message)—is clean, and training without human demonstrations in a social deduction game is genuinely new. The self-play robustness analysis is a nice addition, and the promised release of code and models is the right call.\n\nWhat the paper does well: the listening loss is a sensible grounded signal, and the empirical gap between base models and RL+L is large and consistent across environment variations. The qualitative examples do show emergent accusing and evidence-sharing, which is plausible behavior for a trained policy.\n\nThe soft spots are real but not fatal. First, the stress-test note holds up. Eq. (5) defines B_t as the sum over all living crewmates C_t, which includes the speaker. Eq. (6) then rewards the speaker for the change in that sum between surveys. Since the only event between the two surveys is the speaker's own message, part of the reward is the speaker's change in its own belief. That is not \"influence on other agents\" as promised in Section 5.2. If the released code excludes the speaker from C_t, then Eq. (5) is mis-specified relative to the implementation; if it includes the speaker, the speaking reward is partly self-persuasion. Either way, the paper needs to clarify this, and the marginal contribution of the speaking term (RL+L+S vs RL+L) is small enough that the main win is the listening loss. The paper's own report that agents \"make up evidence\" (Section 6.1, Appendix D) is consistent with the reward rewarding persuasiveness rather than truthfulness, a limitation the authors do flag.\n\nSecond, the statistical reporting is thin: min-max over three seeds, no standard errors, and no number of evaluation episodes. That makes the doubled win-rate claim less precise than it should be, though the qualitative trend is clear.\n\nNone of this sinks the paper. The listening loss is a legitimate contribution, the environment is reasonable, and the limitations are acknowledged honestly. But the speaking reward needs either a corrected formula or a clear implementation note before the influence claim is supported.\n\nThis paper is for researchers working on LLM agents, emergent communication, and social deduction. It deserves a serious referee: I would send it out, with the specific request to check the code for how C_t is computed and to validate the speaking reward against a speaker-excluded variant. The authors can fix this cleanly. I would not desk-reject it.","headline":"A solid training recipe whose headline win rate is carried by the supervision signal, not the speaking reward; the speaking reward has a real specification bug worth checking in the code.","tokens_in":19178,"tokens_out":1732,"would_cite":true,"duration_ms":18977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that natural-language communication among agents can be learned without human demonstrations by training language models to listen and speak through dense belief-based rewards, roughly doubling crewmate win rates in a…","keywords":["multi-agent reinforcement learning","language models","social deduction games","emergent communication","listening loss","speaking reward","Among Us","self-play"],"falsifier":"The claim would be falsified by a controlled experiment in which the speaking reward is replaced by a reward for truthfulness—say, matching the message's content to the speaker's actual observations—and win rates do not drop; or by showing that a speaker trained only to maximize $r_s^t$ systematically makes false accusations and still wins, which would demonstrate the reward exploits uncalibrated listener beliefs rather than genuine information transfer.","tokens_in":1724,"feed_emoji":"🕵️","tokens_out":3982,"duration_ms":75704,"temperature":0.7,"pith_summary":"The paper claims that natural-language communication among agents can be learned from scratch in a partially observable social deduction game, without any human demonstrations. The authors decompose communication into listening—training each agent to predict the true imposter from the discussion so far—and speaking—using multi-agent reinforcement learning to reward messages that raise the other crewmates' summed probability assigned to the true imposter. In a simplified Among Us environment, the combined training roughly doubles crewmate win rates relative to standard reinforcement learning alone and produces qualitative behaviors familiar from human play, such as directly accusing suspects and citing observations as evidence. The point is that dense, task-relevant belief signals can replace human dialogue data as the learning signal for multi-agent language communication.","feed_headline":"Belief-shifting rewards double LLM win rates in Among Us","feed_subtitle":"LLM crewmates trained without human demos learn to accuse suspects and cite evidence, beating much larger base models.","key_machinery":"The load-bearing objects are two auxiliary training signals added to the sparse win/loss game reward. The listening loss $\\mathcal{L}_L(\\pi,\\tau_t^i) = -\\log\\pi(q|\\tau_t^i)$ trains each crewmate to assign high probability to the true imposter $q$ given its action-observation history at belief-query points. The speaking reward $r_s^t = B_t - B_{t'}$ with $B_t = \\sum_{k\\in C_t}\\pi_k(q|\\tau_t^k)$ rewards a message by the increase in the other living crewmates' summed belief in the true imposter after the message. These are combined with a sparse RL loss, a KL penalty that keeps generations in natural language, and a world-modeling loss that preserves the model's language abilities; one crewmate is frozen to the listening-only policy to prevent degenerate conventions. The mechanism turns the open-ended problem of saying something useful into a dense signal tied to the game's actual deduction question.","core_discovery":"The central discovery is that the goal of predicting a hidden world state (who is the imposter) supplies a dense reward that makes language-based communication learnable without demonstrations. Listening is trained as supervised prediction of the true imposter at belief-query timesteps; speaking is trained by the reward $r_s^t = B_t - B_{t'}$, the change in the sum of living crewmates' beliefs in the true imposter after a message. Trained crewmates reach win rates of roughly $0.51$--$0.56$ against adversarially trained imposters on the base environment, about twice the RL-only baseline, while a much larger untrained base model stays below $0.2$. The paper argues that this shows self-improving multi-agent discussion does not require task-specific human data.","pith_inferences":["The speaking reward only measures belief shifts, not truthfulness, so the observed fabrication of evidence may be a direct consequence of the reward design; a natural test is to add a truthfulness penalty and check whether win rates and message honesty both improve.","The reported gain from adding the speaking reward over listening-plus-RL is small, which suggests a cheaper recipe for other domains might omit the RL speaking term and rely on pretrained language priors for generation.","The reward is computed from the same policy being trained, so it is a self-referential signal; generalizing to humans would require tracking human belief updates, which the paper does not test.","If the method scales to larger models, it offers a path to training language agents for coordination tasks where human demonstrations are scarce or impossible, an application the paper mentions but does not pursue."],"forward_implications":["If correct, training an LLM crewmate with the listening loss and speaking reward yields win rates around twice those of RL-only training and well above much larger untrained models.","The learned discussion strategies carry across environment shapes, task counts, and player counts, including settings outside the training distribution.","Crewmate strategies keep their effectiveness against imposters that are adversarially trained in self-play, with win rates converging to between roughly 0.51 and 0.56.","The approach produces interpretable, human-like discussion behavior—accusations and evidence-giving—without any human dialogue data.","The same listen-and-speak decomposition could apply to other social deduction games and partially observable tasks that have a well-defined hidden question."],"supporting_citations":[{"why":"Supplies the pretrained recurrent language model used as the policy initialization and as the KL anchor for all trained agents.","marker":"[30]"},{"why":"Supplies the RLHF-style soft KL constraint that keeps trained policies from drifting out of natural language.","marker":"[28]"},{"why":"Supplies the proximal policy optimization algorithm used as the inner optimization loop for all RL losses.","marker":"[32]"},{"why":"Supplies the PPO implementation used in the training code, including the shared network and clipping details.","marker":"[16]"},{"why":"Provides the prior state of the art that requires large human demonstration data, which this method is explicitly designed to avoid.","marker":"[8]"},{"why":"Supplies the optimizer used in training, which stabilizes the recurrent-model updates.","marker":"[6]"},{"why":"Supplies the ad hoc teamwork setting of freezing one crewmate to prevent degenerate conventions from being shared by all agents.","marker":"[37]"},{"why":"Supplies evidence of human Among Us discussion behaviors such as accusing and providing evidence, used as the comparison for emergent behavior.","marker":"[22]"}],"fun_headline_variants":["No demos needed: LLMs learn to talk in Among Us","Belief-shift rewards teach LLMs to sniff out imposters","AI crewmates learn to accuse and cite evidence, no human data","Self-taught LLM discussion wins twice as often in Among Us"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The whole scheme rests on the assumption that a message is good exactly when it increases the listeners' stated probability of the true imposter, so if those belief updates can be manipulated by confident lies, the reward will select persuasive rather than truthful communication.","fun_headline_variants_meta":{"raw":{"variants":["No demos needed: LLMs learn to talk in Among Us","Belief-shift rewards teach LLMs to sniff out imposters","AI crewmates learn to accuse and cite evidence, no human data","Self-taught LLM discussion wins twice as often in Among Us"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1407,"prompt_tokens":942,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":558,"tokens_out":465,"duration_ms":4422,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:52:06.810397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The claim would be falsified by a controlled experiment in which the speaking reward is replaced by a reward for truthfulness—say, matching the message's content to the speaker's actual observations—and win rates do not drop; or by showing that a speaker trained only to maximize $r_s^t$ systematically makes false accusations and still wins, which would demonstrate the reward exploits uncalibrated listener beliefs rather than genuine information transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PPO implementation used in the training code, including the shared network and clipping details."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the optimizer used in training, which stabilizes the recurrent-model updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ad hoc teamwork setting of freezing one crewmate to prevent degenerate conventions from being shared by all agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies evidence of human Among Us discussion behaviors such as accusing and providing evidence, used as the comparison for emergent behavior."}],"review_version":1}