{"id":"3c2553d2-2e21-4bf7-825b-f99cb4e88a23","arxiv_id":"2501.06695","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DVM uses a win rate constrained decision chain reward to make a Werewolf LLM agent's actual win rate trend toward a requested target.","lead":"This paper presents DVM, a system that combines large language models with reinforcement learning to make Werewolf agents adjust how often they win. The agent's actual win rate rises when a higher target is requested, but it does not precisely hit the requested rate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The win-rate controllability claim depends on an offline human-game decision-chain database whose coverage of the deployed policy's decision chains is unverified; without coverage statistics or released parameters, the observed trend in Fig. 2d does not establish that targets are met.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the controllability reward depends on an offline human-game database (WR_dc) rather than actual outcomes, and the deployment environment (Thinker opponents) may differ from FanLang-9 human play. My independent reading of Section II.B and Section III.A leads to the same conclusion. The paper's framework is coherent and the ablation study (Table III) supports the value of the Predictor and chain reward, but the headline claim that DVM 'successfully modulates its performance levels to meet predefined win rate targets' is not supported by the reported evidence: 30 games per cell, no error bars, and a visible gap between achieved and target win rates. The monotonic trend in Fig. 2d is encouraging but insufficient. I therefore recommend the same CONDITIONAL verdict, with the condition requiring artifact release and a stronger evaluation, possibly including a coverage analysis of the decision-chain database. This is not a rejection because the concern is empirical and testable, not an internal contradiction.","tokens_in":8128,"tokens_out":2194,"duration_ms":19449,"concrete_test":"Release the trained DVM policy, the FanLang-9 decision-chain database, and the exact values of α, s, ϵ, and k. Run 10 target win rates (e.g., 10%, 20%, ... 100%) with at least 100 games per cell against Thinker opponents. Compute: (1) the fraction of DVM-generated decision chains found in the FanLang-9 database, and (2) the correlation between database WR_dc and the actual win rate of those chains in the deployment environment. If coverage is low or correlation is weak, the controllability result is an artifact of the offline proxy; if both are high and the achieved win rates track the targets, the central claim survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section II.B, cr_ctrl is computed from WR_dc, a win rate looked up in a database built from FanLang-9 human games, not from actual game outcomes during training. The deployed environment pits DVM against Thinker agents, so the database must cover the decision chains DVM actually produces and its WR_dc values must track real win rates. The paper provides no coverage statistics, no distributional analysis of decision chains, and omits the hyperparameters α, s, ϵ, and k. With only 30 games per cell in Fig. 2d, no error bars, and a self-admitted persistent gap between achieved and target win rates, the monotonic trend is consistent with the model learning to modulate some proxy-correlated behavior rather than genuinely hitting predefined win rate targets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DVM, a framework for controllable LLM agents in the social deduction game Werewolf. DVM consists of a Predictor, a Decider, and a Discussor, and is trained in two stages: supervised fine-tuning on the FanLang-9 human-game dataset, followed by PPO reinforcement learning. The central novelty is a decision-chain reward, where the win rate of a full decision chain is looked up from a precomputed (DC, WR) database, combined with a win-rate-constrained reward that is intended to steer the agent toward a specified target win rate. The paper reports that DVM outperforms prior methods in Werewolf and shows an upward trend in achieved win rate as the win-rate constraint is increased. The evaluation is conducted in a 9-player Werewolf setup with 30 games per condition, and the text acknowledges that a gap remains between achieved and target win rates.","tokens_in":8435,"tokens_out":5741,"duration_ms":52573,"significance":"If the central claims were fully supported, the paper would make a useful contribution to controllable LLM game agents, with potential applications to dynamic difficulty scaling and to the safety/fairness analysis of LLM agents. The idea of using decision-chain-level win rates from a human-game database as a reward signal is interesting and goes beyond single-step reward shaping. The authors also provide a component ablation and a prediction study, which are appropriate steps. However, the experimental evidence is currently too weak to support the abstract's claim that DVM 'meets predefined win rate targets,' and several load-bearing details of the reward mechanism and evaluation are missing. The contribution is promising but needs substantial additional evidence and clarification.","major_comments":[{"comment":"The controllability claim is not supported at the level stated in the abstract. Each condition uses only 30 games, no error bars or statistical tests are reported, and the text concedes 'there remains a gap between the achieved win rates and the target constraints.' A monotonic upward trend is consistent with a noisy proxy effect and does not establish that the agent 'meets predefined win rate targets.' Please report the exact achieved rates, confidence intervals, and a significance test for both the trend and the deviation from target.","section":"III.A, Fig. 2"},{"comment":"The constrained reward cr_ctrl is computed from WR_dc, the win rate of a decision chain looked up in a database built from FanLang-9 human games. The deployment evaluation, however, pits DVM against Thinker-controlled roles, not against the human players who produced the database. The paper gives no coverage statistics for DVM's produced decision chains in this database and no calibration evidence that WR_dc tracks actual online win rates; without such evidence, optimizing this reward does not imply achieving the requested actual win rate. The hyperparameters α, ε, k, s and the step reward srt are also omitted, so the mechanism is not reproducible.","section":"II.B"},{"comment":"The performance and ablation comparisons are based on 30 games per cell. A difference of one game corresponds to 3.3 percentage points; for example, DVM's werewolf win rate of 66.6% versus Thinker's 63.3% is a one-game difference, and the -w/o DCR ablation (63.6% versus 63.3%) is essentially unchanged. Without confidence intervals or significance tests, the claims that DVM 'outperforms existing methods' and that each ablated component is necessary are not established.","section":"Tables II and III"},{"comment":"The prediction test set is described as 'selected 600 games from the FanLang-9 dataset and from games played by different agents.' Since the same FanLang-9 corpus is used for supervised training and for building the (DC, WR) database, the paper must specify how the 600-game test set is disjoint from these training and database sources. Without a strict split, the improvements in Table I may be inflated by overlap.","section":"III.B"},{"comment":"The equations defining cr_ctrl are dimensionally inconsistent. d is defined as (WR_cons − WR_dc)^2, but the second branch uses (d − ε)/(1 − ε) rather than (d − ε^2)/(1 − ε^2), and the first branch uses d/ε rather than d/ε^2. This is not merely a notational issue, because the sign and magnitude of the reward depend on this comparison; please clarify the intended formula and verify that the reward is continuous where intended.","section":"II.B, cr_ctrl definition"}],"minor_comments":[{"comment":"The PPO objective shows A_π(s,a) = r_t + γV(s_{t+1}) − V(s_t), which is a TD residual rather than the standard advantage function used in PPO, and no clipped surrogate objective is presented; the training objective should be corrected or clarified.","section":"II.B"},{"comment":"The Random baseline probabilities do not match the hypergeometric distribution for selecting 3 werewolves out of 8; for example, ACC@3 should be 1/56 ≈ 1.8%, not 0.8%. Please verify the Random baseline computation.","section":"Table I"},{"comment":"The actual achieved win-rate values are not listed in the text; please include the numerical values and error bars in the figure or caption.","section":"Fig. 2"},{"comment":"The paper omits many implementation details needed for replication: the Werewolf environment, prompt templates, number of PPO iterations, self-play opponent details, and whether code or data will be released.","section":"General"},{"comment":"The row labels 'Werewolf,' 'Villager,' and 'Other Roles' should be defined precisely, specifying which camp is controlled by the tested method and which roles are included in 'Other Roles.'","section":"Tables II and III"},{"comment":"The notation 'W Rcons.' and 'W Rdc' appears with inconsistent spacing; please use consistent subscripts throughout.","section":"Notation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's abstract and conclusion overstate the experimental results relative to the reported evidence. The controllability result in Section III.A is presented as a trend rather than target achievement, yet the abstract claims the agent 'successfully modulates its performance levels to meet predefined win rate targets.' I would advise the editor to require the authors to reconcile this discrepancy before publication. The authors should also be asked to specify the hyperparameters, the data split, and the coverage/calibration of the (DC, WR) database, since these are central to the paper's reproducibility and to the validity of the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: DVM is the first framework I've seen that tries to make an LLM agent hit a specified win rate in a social deduction game, and the paper shows a monotonic trend in the right direction. That is a real, useful contribution to a growing subfield. But the evidence is underpowered, and the reward mechanism rests on an offline database that may not transfer to the deployment setting.\n\nThe novelty is legitimate. Prior work maximizes win rate; this paper adds a win-rate constraint as a tunable objective. The decision chain reward, where a whole game trajectory gets a reward based on its historical win rate, is a reasonable idea and does seem to help in the ablation (removing it drops win rate from 66.6 to 63.6). The authors also give credit to baselines and are candid about the persistent gap between achieved and requested win rates, especially at the high end. That honesty is appreciated.\n\nThe soft spots are real but addressable. The main one: the (DC, WR) database is built from FanLang-9 human games, and the reward during training looks up the agent's own decision chain in that database. No coverage statistics are given, no match fallback is described, and the distribution shift between human play and Thinker opponent play is unexamined. So the reward may be rewarding a proxy that only correlates with real win rate. This is not a fatal flaw in principle, but it means the controllability result could come from a spuriously correlated signal. The paper needs at least a coverage analysis and ideally a version of the reward computed from actual game outcomes.\n\nSecond, 30 games per cell is far too few. For a binary outcome, the standard error is around 9 percentage points, so the differences between adjacent constraints are within noise. The monotonic pattern across five constraints adds some confidence, but without error bars or a test, the trend is fragile.\n\nThird, the abstract says 'successfully modulates its performance levels to meet predefined win rate targets,' but the body acknowledges a persistent gap, particularly for high constraints. That overstatement should be fixed.\n\nFinally, the hyperparameters α, ε, k, s are not reported, and no code or data are released. That makes the method hard to reproduce.\n\nNet: the central idea is sound and worth pursuing, but the current evidence is not enough to certify the claims. I'd send it to peer review with strong encouragement to revise, not desk-reject. A reader interested in controllable game agents or dynamic difficulty scaling will find it useful. I would not cite it yet in its current form, but I'd watch for a revised version.\n\nRecommendation: give it a proper referee, but push for reproducibility and more rigorous evaluation.","headline":"A genuinely new control objective for LLM agents in social deduction games, but the win-rate evidence is thinner than the abstract suggests.","tokens_in":8806,"tokens_out":3065,"would_cite":false,"duration_ms":28299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM agent can be made to play Werewolf at a requested win rate while still beating existing agents, by rewarding whole decision chains rather than single steps.","keywords":["large language models","game agents","controllable agents","reinforcement learning","social deduction games","Werewolf","decision chain reward","win-rate constraint"],"falsifier":"Retrain DVM with $cr_{ctrl}$ computed online from the agent's actual game outcomes against its real opponents, and compare the resulting dial curve to the one in the paper. If the monotone relationship between constraint and actual win rate disappears, or shifts meaningfully with more than 30 games per setting, the central claim is falsified. A second check is to take a DVM agent set to a low target (say 30%) and play it against human players from the same pool as FanLang-9: it should land near 30% if the database lookup is actually controlling performance.","tokens_in":7946,"feed_emoji":"🐺","tokens_out":7335,"duration_ms":61988,"temperature":0.7,"pith_summary":"This paper is trying to establish that controllability and competence are not in tension for LLM game agents: the same Werewolf agent can be made stronger or weaker on demand. The proposed system, DVM, splits the agent into a Predictor, a Decider, and a Discussor, and trains the Decider with a decision-chain reward that scores the whole sequence of a player's decisions, plus a win-rate-constrained term. The central claim is that DVM exceeds existing methods when unrestricted, and that its actual win rate moves in the intended direction when the win-rate constraint is changed, while prompting-only methods stay flat. That matters because game developers need NPCs with adjustable difficulty, and the same mechanism could be used to calibrate the strength of LLM agents for fair evaluation.","feed_headline":"DVM tunes Werewolf agents to preset win rates","feed_subtitle":"A decision-chain reward lets one trained agent adjust its own strength, enabling difficulty scaling for NPCs.","key_machinery":"The load-bearing object is the decision chain reward. A decision chain is the sequence of actions a player takes across a full Werewolf game; DVM builds a database of (decision chain, win rate) pairs from FanLang-9, a dataset of over 18,000 human-player Werewolf games, and after each game the agent's chain is looked up to give $cr(DC)=\\alpha(WR-0.5)$. Controllability comes from $cr_{ctrl}$, which computes $d=(WR_{cons}-WR_{dc})^2$, applies a threshold $\\epsilon$ and tanh scaling, and returns positive reward when $d$ is small and negative reward when $d$ is large, so the Decider is pushed to repeat decision chains whose database win rate is close to the requested target.","core_discovery":"On its own terms, the paper's discovery is a training recipe for a \"dialable\" agent. The Decider is optimized by PPO with a reward $r_t = sr_t + cr$, where $cr(DC)=\\alpha(WR-0.5)$ evaluates the whole decision chain using a precomputed win rate $WR$ from human games; a second term $cr_{ctrl}$ converts the squared deviation between the requested win rate and the chain's win rate into a bounded reward that is positive for small deviations and negative for large ones. In evaluation with 30 games per setting, DVM reached 66.6% win rate as werewolf, 63.3% as villager, and 53.3% for other roles, ahead of the Thinker baseline, and its actual win rate rose with the requested target when it controlled the village side, unlike ReAct, LtM, and Thinker given the same constraint in their prompts. The paper is explicit that a gap remains between target and achieved win rate, and treats the method as a way to tune an agent below its optimal ceiling.","pith_inferences":["Because $cr_{ctrl}$ is optimized against the database win rate $WR_{dc}$ rather than outcomes in the live environment, the dial's accuracy is only as good as the match between the human-games database and the real opponent policy; an obvious extension is to rebuild the database online from self-play games and compare dial curves.","Nothing in the reward design is Werewolf-specific beyond the definition of a decision chain, so the same constrained chain reward should transfer to other social deduction games with a measurable win rate; a testable variant would apply DVM to a 5-player or 7-player setup and check whether the monotone dial survives.","The paper's controllability plots use 30 games per setting; with binomial noise, a more decisive demonstration would report confidence intervals or more games, and would test whether the achieved win rate tracks the constraint beyond the village side."],"forward_implications":["If the claim holds, a single trained NPC policy can serve multiple difficulty levels: the same weights, with a different win-rate constraint at inference, produce a different skill tier.","The decision-chain reward should reduce the need for per-step reward engineering in conversation-driven games, because long-horizon quality is scored at the episode level.","For evaluation, a calibrated agent gives a controllable opponent for testing other agents; an evaluator can match strengths rather than relying on a fixed strong or weak baseline.","The method's limitation to targets below the agent's ceiling means the practical use is difficulty lowering, not superhuman boosting; the paper says so explicitly."],"supporting_citations":[{"why":"Supplies the previous RL-plus-LLM Werewolf agent that DVM extends and compares against as a baseline.","marker":"[15]"},{"why":"Introduces the Thinker baseline and the single-step reward approach that the decision chain reward is designed to replace.","marker":"[16]"},{"why":"Provides the PPO algorithm used to train the Decider.","marker":"[21]"},{"why":"Introduces the dense-network friend/foe prediction approach that the LLM-based Predictor extends.","marker":"[27]"},{"why":"Provides the base language model used for the Predictor and Discussor.","marker":"[28]"},{"why":"Provides DPO, used to train the Predictor from positive and negative predictions.","marker":"[29]"},{"why":"Serves as a baseline in the controllability experiments, with the constraint inserted into the prompt.","marker":"[30]"},{"why":"Serves as a baseline in both prediction and controllability experiments.","marker":"[31]"}],"fun_headline_variants":["DVM gives Werewolf AI a dial for win-rate control","Controllable LLM agents: set the win rate, DVM delivers","AI agents in Werewolf can now be tuned to target win rates","DVM: adjust LLM agent strength in social deduction games"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training signal for controllability is not the win rate the agent actually achieves in games; it is the win rate of a decision chain found in a database built from human games, and if that database does not behave like the live environment where the agent plays other trained agents, hitting the database target does not mean hitting the requested win rate.","fun_headline_variants_meta":{"raw":{"variants":["DVM gives Werewolf AI a dial for win-rate control","Controllable LLM agents: set the win rate, DVM delivers","AI agents in Werewolf can now be tuned to target win rates","DVM: adjust LLM agent strength in social deduction games"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1352,"prompt_tokens":990,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":287}},"tokens_in":606,"tokens_out":362,"duration_ms":4055,"temperature":1.0,"reasoning_tokens":287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:54:09.235746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain DVM with $cr_{ctrl}$ computed online from the agent's actual game outcomes against its real opponents, and compare the resulting dial curve to the one in the paper. If the monotone relationship between constraint and actual win rate disappears, or shifts meaningfully with more than 30 games per setting, the central claim is falsified. A second check is to take a DVM agent set to a low target (say 30%) and play it against human players from the same pool as FanLang-9: it should land near 30% if the database lookup is actually controlling performance.","supporting_citations":[{"cited_title":"Finding friend and foe in multi-agent games,","cited_arxiv_id":null,"evidence_quote":"Introduces the dense-network friend/foe prediction approach that the LLM-based Predictor extends."},{"cited_title":"Direct preference optimization: Your language model is secretly a reward model,","cited_arxiv_id":null,"evidence_quote":"Provides DPO, used to train the Predictor from positive and negative predictions."},{"cited_title":"Least-to-most prompting enables complex reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Serves as a baseline in both prediction and controllability experiments."}],"review_version":1}