{"id":"de6e5333-d5c6-4eb9-a050-179f0f952829","arxiv_id":"2502.04686","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LSPO iteratively clusters an LLM's utterances into latent strategies, solves the abstracted game with CFR, and fine-tunes the LLM with DPO, improving Werewolf win rates over iterations and against prior agents.","lead":"Researchers built an AI agent that learns to play the hidden-role game Werewolf by repeatedly turning players' free-form statements into a compact set of 'latent strategies', solving the simplified game with game theory, and fine-tuning a language model to follow the winning strategy. The agent achieved the highest win rates among four prior Werewolf agents in simulated matches, suggesting a general recipe for training strategic language agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The strongest risk is that §3.1's k-means latent abstraction merges strategically different utterances, so the CFR equilibrium and DPO regret labels in the abstracted game do not transfer to the original language game, and the paper's qualitative/toy validation does not close this gap.","rationale":"The central claim rests on the iterative loop: cluster utterances -> solve abstracted game with CFR -> DPO-align LLM -> expand latent space. The only step that connects the abstracted game to the original language game is the clustering. If the clustering does not group strategically equivalent utterances, then all downstream components optimize the wrong objective. The paper does not provide a quantitative test of this equivalence: Figure 3 is qualitative; the RPSLS experiment has no language clustering; Table 4 shows robustness to k but not fidelity. A secondary concern is the statistical weakness of Table 2, where LSPO and SLA overlap within error bars on both sides, but that would weaken the empirical claim rather than the method's foundation. The proposed test (intra-cluster strategic agreement and regret-vs-outcome correlation) would directly determine whether the abstraction is faithful. If it fails, the central claim collapses; if it passes, the mechanism is plausible. The reader already identifies this as the weakest assumption and assigns CONDITIONAL. I agree: the paper should be accepted only if the abstraction's fidelity is demonstrated, so the verdict remains CONDITIONAL (UNCHANGED).","tokens_in":24257,"tokens_out":16969,"duration_ms":150928,"concrete_test":"Collect 200 game states from LSPO self-play and have the LSPO LLM generate 10 diverse utterance candidates per state (2,000 utterances). Cluster these with the paper's k-means protocol. Then (a) ask independent annotators (or a strong LLM) to label each utterance's strategic intent and whether it would increase the speaker's side's win probability in that state, and measure intra-cluster label agreement; if same-cluster utterances diverge on intent/effect in >15% of pairs, the abstraction is lossy. (b) For each cluster, compute the correlation between the CFR regret label and the empirical win rate of the sampled utterances when actually played in the original game across matched states; a Spearman correlation below ~0.3 would indicate the DPO labels are not aligned with true game outcomes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs the latent strategy space by k-means clustering of sentence embeddings of utterances, without conditioning on the game state. This assumes embedding similarity implies strategic equivalence. Two utterances with similar wording but different game contexts (e.g., 'I am the Seer' in round 1 vs. late game, or when the real Seer is already dead) will fall in the same cluster yet have very different effects on opponents' beliefs and the win probability. The abstracted game then treats them as the same action, so CFR's regret values for clusters are computed under a coarsened game that does not reflect within-cluster strategic variation. DPO fine-tuning uses these regret values as preference labels, so if the abstraction is lossy, the LLM is trained toward an objective that can diverge from true game utility. The paper's validation of the abstraction is qualitative (Figure 3 shows clusters but no measure of strategic purity) and a toy RPSLS experiment that involves no clustering at all. Table 4 shows final win rates are insensitive to k (0.24 vs 0.25), which suggests clustering granularity is not the active ingredient, but does not validate equivalence. Without a quantitative check that utterances in the same cluster are strategically interchangeable, the core mechanism of LSPO is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LSPO, an iterative framework for building LLM agents in free-form strategic language games such as Werewolf. The method maps LLM-generated utterances into a finite latent strategy space via k-means clustering of sentence embeddings, solves the resulting abstracted extensive-form game with Deep CFR, and fine-tunes the LLM with Direct Preference Optimization (DPO) using CFR regret values as preferences. Iterating this loop expands the latent strategy space. Experiments in a seven-player Werewolf game report improved prediction accuracy and win rates over iterations, and the final agent is claimed to outperform ReAct, ReCon, Cicero-like, and SLA baselines on both the Werewolf and Village sides.","tokens_in":24591,"tokens_out":9318,"duration_ms":84289,"significance":"If the results hold, LSPO is a novel and potentially influential combination of game-theoretic abstraction and LLM alignment, directly addressing two known weaknesses of prompt-based language agents: intrinsic action bias and insufficient exploration of the text action space. The paper is clearly written, includes detailed appendices with prompts, game logs, and hyperparameters, and demonstrates a concrete iterative pipeline. However, the empirical support for the headline claims is weakened by an asymmetric evaluation design, a lack of statistical significance testing, and an unvalidated clustering assumption that is central to the method. These issues are fixable in a revision, and the core idea is worth pursuing.","major_comments":[{"comment":"The head-to-head comparison is asymmetric. In the first setup, LSPO always plays the Werewolf side and each of the five agents (including LSPO itself) plays the Village side; in the second setup, LSPO always plays the Village side and each of the five agents plays the Werewolf side. Consequently, LSPO's reported win rates are aggregated over matches against all five opponents, while each baseline's win rates are computed only from matches against LSPO. This makes the 'highest win rate' claim unsupported because the opponent pools differ across agents. Please re-run a symmetric round-robin evaluation (or report per-pair win rates for every agent in both roles) and provide significance tests for the pairwise differences.","section":"Section 4.4, Table 2"},{"comment":"The k-means latent abstraction is the core mechanism, but the paper provides no quantitative evidence that utterances in the same cluster are strategically interchangeable. The validation is qualitative (Figure 3), and the RPSLS proof-of-concept in Section 4.1 does not involve clustering because the action space is already discrete. Table 4 shows that final win rates are similar for k=1,2,3 in a four-player game, which suggests that clustering granularity is not the active ingredient, but it does not validate cluster equivalence. Because DPO preference labels are derived from CFR regrets in the abstracted game, a lossy clustering can train the LLM toward an objective that diverges from the original game's utility. Please add a direct validation of the abstraction, such as measuring intra-cluster outcome variance, comparing to an oracle clustering conditioned on game state, or showing that within-cluster utterances have statistically indistinguishable win rates when played in the original game.","section":"Section 3.1"},{"comment":"The ablations do not isolate the latent-space clustering component. The paper removes fine-tuning and policy learning, but there is no ablation without clustering (e.g., treating each generated utterance as a separate action, or using raw LLM action probabilities). The cluster-size sensitivity analysis (Table 4) is also run in a four-player game, which may not capture the strategic complexity of seven-player Werewolf. Without an ablation that varies the abstraction itself, the paper's claim that clustering is what enables policy transfer to the original language game is not established.","section":"Section 4.5"},{"comment":"The reported win-rate differences between LSPO and the strongest baseline (SLA) are within one standard deviation (0.73 ± 0.11 vs 0.69 ± 0.12 as Werewolf; 0.27 ± 0.11 vs 0.25 ± 0.08 as Village). The number of games per cell is not reported, and no significance tests are provided. Please report confidence intervals, p-values, or effect sizes for the pairwise comparisons to support the claim that LSPO outperforms all baselines.","section":"Section 4.4, Table 2"}],"minor_comments":[{"comment":"The theoretical convergence argument via a finite vocabulary N_v^L is trivial and does not explain the empirical convergence in five iterations; please replace it with a more substantive discussion of when the cluster expansion is sufficient to cover the strategic space.","section":"Appendix B.4"},{"comment":"Please clarify whether the 'additional prediction phase' before each voting phase is purely internal (not affecting the game state) or whether it changes the agents' observations or the game flow.","section":"Section 4.3"},{"comment":"Please specify how the mean and standard deviation for LSPO's row are computed: over 100 games per opponent or over the aggregated 500 games.","section":"Table 2"},{"comment":"The regret update equation omits the player index, information-set index, and iteration superscripts used in standard CFR notation; please align it with the notation in Appendix B.2.","section":"Section 3.2, Eq. (1)"},{"comment":"The qualitative description of clusters would be strengthened by reporting quantitative cluster-quality metrics (e.g., silhouette score) to support the claim that clusters become clearer over iterations.","section":"Figure 3"},{"comment":"The RPSLS proof-of-concept uses discrete actions and does not exercise the clustering step; please clarify how the latent-space construction applies to this setting or explain why the analogy is valid.","section":"Section 4.1"},{"comment":"There are several typos, including 'human-AI interation' in Section 1, 'out-fo-distribution' in Section 2.2, and 'startegy' in Appendix B.2; these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written paper with a promising framework, but the experimental validation of the headline claims needs substantial work. If the authors provide a symmetric evaluation, significance tests, and a direct validation of the clustering abstraction, the paper could become acceptable. The lack of code release is also a concern for reproducibility, though the appendix contains sufficient detail. I recommend major_revision rather than reject because the flaws are fixable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me get straight to it. The contribution is the loop itself: cluster free-form utterance embeddings into a finite latent strategy space, solve the abstracted extensive-form game with Deep CFR, fine-tune the LLM with DPO using cluster-level regret as preference labels, then expand the cluster set and repeat. That specific combination is not in SLA, PSRO, or the Cicero line, and it is a plausible answer to the exploration problem in free-form language games. The paper also does a few things right: clear exposition, a proof-of-concept in RPSLS, ablations showing both CFR and fine-tuning matter, and an honest look at five-iteration convergence. The qualitative examples of emergent bluffing and misdirection are consistent with the story.\n\nThe soft spots are real but not fatal. First, Table 2 is not a like-for-like comparison. The LSPO rows include self-mirror games, while each baseline only faces LSPO; the differences are inside the confidence intervals (Werewolf 0.73±0.11 vs SLA 0.69±0.12, Village 0.27±0.11 vs 0.25±0.08). I'd want a round-robin with all pairs and significance tests before believing 'highest win rate.' Second, the latent abstraction is the load-bearing assumption and it is validated only qualitatively. Clustering embeddings without conditioning on game state can merge utterances that are strategically different in context, and the paper gives no quantitative check that same-cluster utterances are interchangeable; the RPSLS toy does not exercise clustering at all. The k-insensitivity in Table 4 is nice robustness evidence, but it does not close the gap. Third, the convergence argument in Appendix B.4 is a tautology: the bound is finite but astronomically large, so it guarantees nothing about the observed five-iteration convergence. And there is no code or data.\n\nNone of this kills the core idea. The empirical improvements over iterations and the ablations suggest the loop does something real. I'd want the comparison redone, artifacts released, and at least one direct measure of cluster purity (e.g., within-cluster outcome variance under a fixed policy) before I'd take the headline literally.\n\nThis is a paper for people building LLM agents for social deduction or other free-form language games, and for anyone thinking about how to marry game-theoretic policies with LLM alignment. It deserves a serious referee; I'd engage with it, not desk-reject it.","headline":"The iterative cluster–CFR–DPO loop is a real step forward; the win-rate comparison is softer than the abstract suggests.","tokens_in":25093,"tokens_out":5278,"would_cite":true,"duration_ms":52035,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LSPO claims that an iterative loop—cluster an LLM's utterances into latent strategies, solve the abstracted game with CFR, fine-tune with DPO—builds Werewolf agents whose win rates improve each iteration and beat all tested baselines.","keywords":["latent space policy optimization","Werewolf game","strategic language agents","counterfactual regret minimization","direct preference optimization","LLM fine-tuning","social deduction games","game-theoretic reinforcement learning"],"falsifier":"A direct test is to take one LSPO agent, freeze it, and replay its games while swapping each utterance with another utterance from the same cluster; if win rates change as much as when utterances are swapped across clusters, the latent abstraction is not capturing the strategic content. A second test is to shuffle cluster labels after training and check whether the win-rate gap between LSPO and its ablations persists.","tokens_in":24038,"feed_emoji":"🐺","tokens_out":5354,"duration_ms":47626,"temperature":0.7,"pith_summary":"LSPO is an iterative method for building LLM agents that play free-form strategic language games. The paper's central claim is that although natural language has an unbounded action space, the strategies behind utterances are compact; clustering utterances into a finite latent strategy space lets a game solver (Deep CFR) compute a near-optimal policy, and fine-tuning the LLM with DPO toward that policy both removes the model's intrinsic action bias and expands the strategy space. In seven-player Werewolf, the agent's win rate rises steadily across iterations—from 0.54 to 0.73 on the Werewolf side and 0.18 to 0.27 on the Village side—and the final agent outperforms ReAct, ReCon, a Cicero-like agent, and SLA on both sides. A sympathetic reader would care because it offers a path from game-theoretic equilibrium computation to open-ended language behavior without hand-labeling strategic actions.","feed_headline":"LLM agents win Werewolf by solving a latent-strategy game","feed_subtitle":"Clustering utterances into strategies lets a game solver choose the moves, then DPO teaches the model to speak them.","key_machinery":"The load-bearing object is the latent strategy space: a finite set of clusters obtained by embedding each free-form discussion utterance with a sentence embedder and applying k-means. Each cluster is treated as an atomic action in an abstracted extensive-form game, so CFR/Deep CFR can solve it despite the unbounded original text space. The second half of the loop is the DPO fine-tuning step, which converts the abstracted policy's regret values into preference labels over the LLM's utterance candidates, so the model learns to say things that the solved latent policy would choose. Iteration adds one cluster per role, so the latent space expands as the model improves.","core_discovery":"The paper establishes that an agent can learn to play a free-form language game by solving an abstracted version of the game in a learned discrete latent strategy space and then aligning the LLM's language output to the solved policy. Each iteration works as follows: the current LLM self-plays and generates discussion utterances; those utterances are embedded and k-means clustered into latent strategies; the game is reformulated as an extensive-form game whose discussion actions are the clusters; Deep CFR learns a near-equilibrium policy in that abstracted game; and DPO fine-tunes the LLM using the regret values of the clusters as preference labels. The next iteration regenerates utterances with the fine-tuned model and expands the cluster count, so the latent space grows. The paper reports that this loop raises prediction accuracy and win rates over iterations, achieves the Nash equilibrium in Rock-Paper-Scissors-Spock-Lizard by iteration 3, and reaches the highest win rates among the compared agents in seven-player Werewolf.","pith_inferences":["Editorial inference: LSPO is a general template: any game with unbounded text actions could be approached by clustering utterances into a latent strategy set, solving the abstraction, and aligning the model—so social-deduction variants like Avalon or negotiation games are natural next tests.","Editorial inference: the method's ceiling depends on clustering granularity; if the paper's premise holds, better semantic representations or hierarchical clustering could push win rates further, and cluster-quality metrics could predict how well the abstraction transfers.","Editorial inference: a direct human-play or cross-agent evaluation would test whether the equilibrium learned against LLM opponents transfers to human conversational tactics, since the paper evaluates against LLM-based agents."],"forward_implications":["Iterating the LSPO loop improves both role-prediction accuracy and win rate in seven-player Werewolf, with reported performance converging by iteration five.","A game-theoretic solver can shape an LLM's free-form language behavior without requiring hand-designed action taxonomies or human preference data.","The final LSPO agent beats prompt-based agents (ReAct, ReCon), a fixed-action Cicero-like agent, and a fixed-LLM RL agent (SLA) on both the Werewolf side and the Village side.","The framework transfers from the toy RPSLS game (where it reaches zero exploitability) to the conversational Werewolf setting, suggesting scalability to other free-form language games.","Ablations show both the CFR policy-learning component and the DPO fine-tuning component contribute, and reported performance is stable across cluster counts and DPO temperatures in a simpler four-player variant."],"supporting_citations":[{"why":"Supplies the CFR algorithm that the latent-space solver uses to minimize counterfactual regret in the abstracted game.","marker":"Zinkevich et al., 2007"},{"why":"Supplies Deep CFR, the neural variant that lets the paper scale the solver to the abstracted Werewolf game tree.","marker":"Brown et al., 2019"},{"why":"Supplies DPO, the preference-optimization method used to fine-tune the LLM toward the latent-space policy.","marker":"Rafailov et al., 2024"},{"why":"Provides the SLA baseline and documents the intrinsic-bias problem in LLM action distributions that LSPO targets.","marker":"Xu et al., 2023c"},{"why":"Provides the Cicero-like baseline with a fixed atomic action space against which LSPO compares.","marker":"Meta et al., 2022"},{"why":"Provides the ReAct baseline, a prompt-based agent that LSPO outperforms in the Werewolf evaluation.","marker":"Yao et al., 2022b"},{"why":"Provides the ReCon baseline, a prompt-based agent for social deduction that LSPO outperforms in the Werewolf evaluation.","marker":"Wang et al., 2023b"}],"fun_headline_variants":["Latent-space solver trains LLMs to beat Werewolf","LSPO maps chatter to strategies, then wins Werewolf","Compact strategy space lets LLM agents master Werewolf","Iterative latent policy optimization outperforms Werewolf agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that grouping similar-sounding utterances into strategy clusters does not destroy the strategic differences between them; if two statements that mean different things land in the same cluster, the solver's equilibrium is for the wrong game and the fine-tuning labels can reinforce the model's original biases.","fun_headline_variants_meta":{"raw":{"variants":["Latent-space solver trains LLMs to beat Werewolf","LSPO maps chatter to strategies, then wins Werewolf","Compact strategy space lets LLM agents master Werewolf","Iterative latent policy optimization outperforms Werewolf agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1486,"prompt_tokens":1002,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":418}},"tokens_in":618,"tokens_out":484,"duration_ms":4574,"temperature":1.0,"reasoning_tokens":418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:52:22.237650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take one LSPO agent, freeze it, and replay its games while swapping each utterance with another utterance from the same cluster; if win rates change as much as when utterances are swapped across clusters, the latent abstraction is not capturing the strategic content. A second test is to shuffle cluster labels after training and check whether the win-rate gap between LSPO and its ablations persists.","supporting_citations":[],"review_version":1}