{"id":"5f2ad43a-c178-490f-9357-57d827c8a79b","arxiv_id":"2501.05501","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reward decomposition plus a learnable mask lets a Coup-playing DQN agent suppress lying at inference time without retraining and with little loss of win rate.","lead":"This paper introduces strategy masking, which splits an agent's reward into behavior channels and lets a user turn one channel off or down after training is complete. The method is tested in the deception game Coup, where it cuts lying sharply while keeping the agent's win rate roughly unchanged, suggesting a cheap guardrail for value-based reinforcement learning agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The performance-preservation claim rests on Figure 4's unseeded, error-bar-free sweep; 'minimally impacted' is asserted, not demonstrated.","rationale":"Reading in good faith, the paper's central claim has two parts: (1) a linear mask over decomposed reward Q-values can suppress lying at inference, and (2) this does not compromise the agent's ability to win. Part (1) is mechanistically expected and the figures show a clear directional effect; the method itself is coherent, the masked TD updates are clearly specified, and Theorem II.1 gives a valid tabular convergence result for the masked scalar Q-function (though it does not extend to the DRQN function approximation used in the experiments). Part (2) is the load-bearing lynchpin, and it is not statistically established. Figure 4 is the only evidence for win-rate preservation: a sweep over lie weights with 5000 games per point, no reported variance or seeds, no baseline policy or retraining comparison, and no test of whether the observed win-rate changes are within noise. In a three-player game, random play wins 33% of the time, so a drop from, say, 40% to 37% would need error bars to interpret. The robustness run with updated PFSP priorities is better in design but still lacks error bars. There is also an internal inconsistency: Section V-A states that each agent's top two reward dimensions were the masked-in dimensions, but Table IV reports the opposite for both agents (Win-Lie: Challenge 9.09% > Lie 7.41%; Win-Challenge: Lie 9.10% > Challenge 0.79%). This does not directly refute the post-training suppression result, but it undermines the supporting claim that reward decomposition creates predictably targeted behaviors and suggests a possible error in reported statistics. No code is released, which is an additional reproducibility barrier, but the statistical gap is the primary reason to keep the verdict conditional. The reader's listed weakest assumption (unambiguous observability of the unwanted behavior) is a legitimate limitation for real-world generalization, but for the paper's stated Coup experiments the defining assumption holds by construction; the missing statistical support for win-rate preservation is more directly load-bearing. Hence partial agreement with the reader, and the verdict should remain conditional pending the concrete test above.","tokens_in":14278,"tokens_out":8517,"duration_ms":83143,"concrete_test":"Run 10 independent training runs (different seeds) of the win-only agent. For each run and for lie weights -5, -2, -1, 0, 1, 2, 5, evaluate 5000 games per weight against the same fixed league, and compute the mean and 95% CI of win rate across runs. Then perform a paired equivalence test (same run, lie=0 vs lie=-1) with a pre-specified bound (e.g., |difference| < 2 percentage points) for 'minimally impacted.' If the equivalence test passes, the central claim is supported; if the CI shows a material decline, the claim fails. Also re-run the Table IV reward collection to verify the reported percentages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the central claim's performance-preservation half ('without compromising their ability to perform effectively') rests entirely on Figure 4, which reports no error bars, no multiple seeds, no confidence intervals, and no comparison baseline. The post-training mask sweep was run with a single trained agent (or at least with no seed information) and 5000 games per mask weight; in a three-player game the baseline win rate is ~33%, so a small drop is hard to interpret without variance estimates or a paired test. The robustness run with updated PFSP priorities also lacks error bars. A secondary internal inconsistency: Section V-A asserts that each agent's top two reward dimensions matched the masked-in dimensions, but Table IV shows Win-Lie's top two collected rewards are Win (78.22%) and Challenge (9.09%), not Lie (7.41%), and Win-Challenge's top two are Win (83.72%) and Lie (9.10%), not Challenge (0.79%). This weakens the 'controllable behavior shaping' sub-claim, though not the post-training suppression direction, which is mechanistically expected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes \"strategy masking,\" a method that decomposes a value-based RL agent's reward into multiple scalar dimensions, trains the agent with access to a masked subset of those dimensions, and then adjusts the mask at inference time to suppress or penalize an unwanted behavior. The method is demonstrated in the social deception game Coup, where the unwanted behavior is lying (taking an action while not holding the matching card). The authors train DQN/DRQN agents with four reward dimensions (Win, Challenge, Lie, Bait), show that different training masks produce agents with different behavioral tendencies, and then show that varying the Lie mask weight post-training can drive lying actions to near zero while claiming that win rate is only minimally impacted. A convergence proof is provided for masked tabular Q-learning in Appendix C.","tokens_in":14503,"tokens_out":3717,"duration_ms":37091,"significance":"If the central performance-preservation claim holds, strategy masking would be a lightweight, post-training guardrail for value-based RL systems, and the Coup environment is a well-chosen testbed because lying is unambiguous and observable. The paper is clearly written, provides pseudocode, and includes a tabular convergence proof with a standard contraction argument. The strength of the contribution, however, is currently limited by the weak empirical support for the headline claim: the key figure has no statistical uncertainty quantification, and one of the training-behavior claims is contradicted by the paper's own table.","major_comments":[{"comment":"The central claim that lying can be suppressed \"without compromising agent ability to perform effectively\" rests entirely on Figure 4, which reports a single sweep over lie-dimension weights with no error bars, no confidence intervals, no multiple seeds, and no statistical test. In a three-player game the baseline win rate is about 33%, so a difference of a few percentage points is hard to interpret without variance information. The robustness rerun with updated PFSP priorities also lacks error bars. The manuscript should report means and variances over at least several independent training runs (or clearly state if only one agent was used), and ideally include a paired comparison (e.g., win rate of the same agent with lie weight 0 vs. -1 on the same set of episodes) or a statistical test to support the phrase \"minimally impacted.\" Without this, the main conclusion is not quantitatively supported.","section":"V-C, Figure 4"},{"comment":"The text in Section V-A asserts that \"for either agent their top two dimensions by reward were the dimensions we biased them towards,\" but Table IV shows the opposite. For the Win-Lie agent, the top two collected rewards are Win (78.22%) and Challenge (9.09%), with Lie (7.41%) third. For the Win-Challenge agent, the top two are Win (83.72%) and Lie (9.10%), with Challenge (0.79%) last. This internal inconsistency weakens the sub-claim that reward decomposition and strategy masking create agents with predictable, controllable behavioral priorities during training. The authors should either correct the claim or discuss why the top-two reward distributions do not match the intended mask.","section":"V-A, Table IV"},{"comment":"The convergence theorem in Section II-C and Appendix C proves convergence only for the scalarized quantity Q(s,a)\\cdot m under a restricted update rule, and only for tabular MDPs. It does not cover convergence of the individual decomposed components, nor does it cover the DQN/DRQN function-approximation setting used in all the experiments. The paper should explicitly state that the theoretical guarantee applies only to the scalarized tabular update, and should not imply that the experimental algorithm inherits these guarantees. The proof also contains small errors (e.g., the variance expression is missing a closing parenthesis, and an extra (s,a) appears in the definition of Delta_{t+1}) that should be fixed.","section":"II-C, Appendix C"},{"comment":"The authors themselves list a critical scope limitation: strategy masking as demonstrated relies on the unwanted behavior being unambiguous and directly observable, so it can be encoded as a reward dimension. The paper's abstract and introduction frame the method as a general guardrail mechanism, but the only fully demonstrated case is one where a clean ground-truth signal (holding a specific card in Coup) is available. The manuscript should more prominently temper the generalization claims and discuss what would be needed to apply the method when the target behavior (e.g., LLM hallucination) must first be detected or estimated. This is not a correctness error in the presented experiments, but it is load-bearing for the paper's stated significance.","section":"VI"}],"minor_comments":[{"comment":"There is a typo in the abstract: the quotation mark after \"unethical\" is not closed, and the following sentence begins with \"Without\" but should be a new sentence or be connected grammatically.","section":"Abstract"},{"comment":"In Algorithm 1 (Masked DQN), the loss in line 11 sums over all K dimensions but the target y is defined in Eq. (1) as a vector; it would be clearer to explicitly write y^{(j)}_k = r_k(s^{(j)}, a^{(j)}, s'^{(j)}) + gamma Q^{(k)}(s'^{(j)}, a^*_m(s'^{(j)}) | w') for non-terminal states.","section":"II-A, Algorithm 1"},{"comment":"The eight strategy masks in Table III are presented without explicit column numbers; the text refers to \"column 1,\" \"column 2,\" etc., which is understandable but could be made more reader-friendly by numbering the columns.","section":"Table III"},{"comment":"The Win-Lie agent's win rate in Table IV (30.56%) is below the 33% expected against uniform random play in a three-player game, and the Win-Challenge agent's win rate is 42.58%. The authors may want to comment on whether the Win-Lie agent's below-chance win rate indicates a weak agent or a strong league, since this affects interpretation of the subsequent post-training results.","section":"V-A, Table IV"},{"comment":"The proof of Lemma C.1 uses the notation a^{*,i}_m(s) and then switches to argmax; the argument is standard but a few steps are compressed, and the transition from the triangle inequality to the max-norm bound would be clearer with an explicit statement that max_a |Q1(s,a) - Q2(s,a)| <= ||Q1 - Q2||_inf.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a borderline case. The core idea is simple and potentially useful, and the Coup environment is a reasonable proof-of-concept. However, the main empirical claim is not backed by statistical evidence, and there is an internal inconsistency in the training-behavior results. I recommend major revision rather than rejection because the missing statistical analysis and the table-vs-text mismatch are fixable within the manuscript's scope. The theoretical contribution is modest and should be scoped carefully; the authors should not overclaim coverage of their function-approximation experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Strategy masking is a reasonable, clearly written extension of reward decomposition: train with decomposed Q-values, then adjust a weight vector at inference to suppress a behavior. The Coup experiments show the expected mechanical effect—negative weights on the Lie dimension drive lying actions toward zero—and the authors are honest that direct observability of the target behavior is a precondition. The convergence theorem is a standard contraction argument and is fine for the tabular case.\n\nWhat is genuinely new is the post-training inference-time mask sweep, which I don't think prior reward decomposition work does. The paper is also well organized and the limitations section is candid.\n\nThe soft spots are empirical. The central claim that win rate is 'minimally impacted' rests on Figure 4, which has no error bars, no seeds, and no comparison baseline. In a three-player game with baseline win around 33%, a drop of even a few points without variance estimates is uninterpretable. The robustness run with updated PFSP priorities has the same problem. The internal inconsistency the stress test flags is real: Table IV shows the top two collected reward dimensions for the Win-Lie agent are Win and Challenge, not Win and Lie, and for Win-Challenge it's Win and Lie, not Win and Challenge. That undercuts the 'controllable behavior shaping' sub-claim, though not the suppression direction, which is mechanistic.\n\nThe suppression direction itself is partly by construction—changing the scalarized Q-values changes action selection—so the independent content is the win-rate claim, and that is not yet demonstrated. I'd also like to see a baseline like retraining with a fixed negative weight, and some analysis of how mask weights interact with the league's opponent distribution.\n\nWho is this for? Researchers working on reward decomposition, interpretable RL, or inference-time guardrails in AI safety. It deserves a serious referee because the idea is novel and the paper is honestly written, but it needs major revision: statistical rigor, baselines, and ideally released code before the performance-preservation claim can be accepted.","headline":"A plausible inference-time guardrail idea in value-based RL, but the central win-rate preservation claim is not empirically supported as presented.","tokens_in":15025,"tokens_out":1639,"would_cite":false,"duration_ms":16979,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Strategy masking suppresses learned lying in RL agents after training, with win rate nearly unchanged.","keywords":["strategy masking","reward decomposition","guardrails","reinforcement learning","lying","social deception","DQN","value-based RL"],"falsifier":"Play the same post-training sweep in a variant of Coup where lying is frequently optimal, for example a rule change that makes bluffs much cheaper to attempt; if negative lie weights then cut win rate substantially, the claim that masking suppresses lying without compromising performance would fail. Alternatively, an environment where the lie reward is only partially observable, such as estimating hallucination in language-model outputs, would test whether the method still works when the reward channel must itself be inferred.","tokens_in":107,"feed_emoji":"🤥","tokens_out":4468,"duration_ms":129760,"temperature":0.7,"pith_summary":"The paper introduces strategy masking, a way to make a value-based reinforcement learning agent change its behavior after training by decomposing its reward into separate dimensions and tuning a mask vector that picks which dimensions the agent may use when choosing actions. The authors apply it to lying in the social-deception game Coup, where a lie is well-defined: completing an action or block without holding the corresponding card. They show that an agent trained only to win can be steered at inference time by giving the 'lie' reward dimension a negative weight, driving lying actions to near zero while leaving win rate roughly unchanged. The method matters because it offers a way to impose guardrails on already-trained agents without retraining or extra compute, provided the unwanted behavior can be observed and encoded as a reward channel. A convergence theorem is proved for masked Q-learning, showing the masked update converges to the optimal Q-values under standard stochastic-approximation conditions.","feed_headline":"Masking a reward dimension kills lying without costing wins","feed_subtitle":"Trained Coup agents' lies drop to near zero when the lie reward is negatively weighted; win rate holds.","key_machinery":"The strategy mask is a coefficient vector $\\vec{m}$ whose dot product with the decomposed state-action value vector $\\vec{Q}(s,a)$ selects and weights reward dimensions in the agent's action choice: $a^*(s) = \\arg\\max_a \\vec{Q}(s,a) \\cdot \\vec{m}$. During training, masked SARSA, masked expected SARSA, and masked Q-learning incorporate $\\vec{m}$ into the temporal-difference target, so the agent learns future values under the constraint that its own subsequent decisions will respect the mask. At inference the mask can be changed without retraining, which lets the user suppress or punish behaviors that were learned during training. The masked Q-learning update is shown to be a contraction mapping in the sup-norm, so the masked values converge with probability one to the optimal Q-values of the masked problem.","core_discovery":"The paper's central claim is that a value-based agent's learned incentives can be explicitly decomposed into reward dimensions, and that by masking or negatively weighting a dimension during inference the experimenter can suppress the associated behavior without materially harming performance on the primary task. Evidence comes from Coup: a win-only agent, when its lie-reward dimension is weighted negative after training, takes lying actions in nearly zero percent of its decisions while its win rate drops only slightly; positive weights, by contrast, push lying up to roughly 70 percent of actions and drag win rate down. The authors interpret this asymmetry as lying being advantageous in only a limited set of game states, so removing the incentive forces changes in few decisions, while inflating it pushes the agent into riskier lies. The result is presented as a general guardrail technique for any value-based RL setting with a decomposable reward.","pith_inferences":["In real applications like LLM hallucination, the limiting step will be constructing a reliable reward channel for the unwanted behavior; if that channel can be built, the same post-training mask sweep could serve as a cheap alignment intervention.","The asymmetry between positive and negative weights suggests a general recipe: suppressing a behavior only changes decisions where that behavior was marginally preferred, so it should generally be cheaper than forcing a behavior.","The technique could be extended to policy-gradient methods by using the mask to reweight a decomposed advantage or by reshaping the policy distribution, though the paper does not show how."],"forward_implications":["A trained agent's lying rate can be driven to near zero by a negative mask weight on the lie dimension, with win rate minimally affected.","Positive mask weights increase lying up to roughly 70 percent of actions and lower win rate, showing the mask can also amplify behaviors.","The same mask-sweep result holds when opponent-selection priorities are updated, though the overall win rate drops under harder opponents.","Masked Q-learning converges with probability one to the optimal Q-values under standard learning-rate conditions."],"supporting_citations":[{"why":"Supplies reward decomposition, the basis for the vector-valued Q-function that strategy masking operates on.","marker":"Juozapaitis et al. [2019]"},{"why":"Provides the DQN algorithm that the paper extends with reward decomposition and masking.","marker":"Mnih et al. [2013]"},{"why":"Provides DRQN, used to handle partial observability in the Coup environment.","marker":"Hausknecht and Stone [2015]"},{"why":"Supplies the league-play and prioritized fictitious self-play training structure used for multi-agent capability.","marker":"Vinyals et al. [2019]"},{"why":"Provides the convergence theorem that the masked Q-learning proof relies on.","marker":"Jaakkola et al. [1994]"}],"fun_headline_variants":["Strategy masking curbs AI lies, win rate stays high","Flip a reward weight to erase learned deception","Post-training guardrail: mask reward dimensions to stop cheating","Value-based RL: suppress lying by masking reward incentives","Reward masking lets you prune AI behaviors without performance loss"],"cache_read_input_tokens":17152,"weakest_assumption_plain":"The whole demonstration depends on the unwanted behavior—here, lying—being unambiguously detectable and scored as its own reward channel; in settings where the behavior cannot be observed and encoded that cleanly, strategy masking cannot be applied as stated.","fun_headline_variants_meta":{"raw":{"variants":["Strategy masking curbs AI lies, win rate stays high","Flip a reward weight to erase learned deception","Post-training guardrail: mask reward dimensions to stop cheating","Value-based RL: suppress lying by masking reward incentives","Reward masking lets you prune AI behaviors without performance loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2720,"prompt_tokens":847,"completion_tokens":1873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1796}},"tokens_in":463,"tokens_out":1873,"duration_ms":14556,"temperature":1.0,"reasoning_tokens":1796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:14:10.921365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Play the same post-training sweep in a variant of Coup where lying is frequently optimal, for example a rule change that makes bluffs much cheaper to attempt; if negative lie weights then cut win rate substantially, the claim that masking suppresses lying without compromising performance would fail. Alternatively, an environment where the lie reward is only partially observable, such as estimating hallucination in language-model outputs, would test whether the method still works when the reward channel must itself be inferred.","supporting_citations":[],"review_version":1}