{"id":"474bdf8e-5076-4452-a64c-70b3121d284a","arxiv_id":"2505.10670","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Steering LLaMA-3-8B's internal 'good faith/bad faith' feature shifts its defection probability in the iterated prisoner's dilemma by 28 percentage points.","lead":"A single-author preprint tests whether adding or removing specific feature directions inside an LLM's internal representations changes how an AI agent plays a repeated prisoner's dilemma game. Steering on a feature associated with 'good faith' moves the model's average defection probability between 47% and 75%, suggesting representation-level steering as a possible alignment tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 28pp shift is a uniform average of next-token probabilities over all 64 histories, not the expected defection rate in actual IPD play; steering changes both the policy and the history distribution, so the headline may not describe real game behavior.","rationale":"The reader's weakest assumption identifies essentially the same gap: the paper measures next-token probabilities on a fixed set of histories rather than sampling complete games, so multi-turn dynamics may decouple the reported numbers from actual agent behavior. My stress-test sharpens this into a precise technical issue: the reported average is a uniform average over 64 histories, not an expectation under the distribution induced by the steered policy. This is load-bearing because the central claim is phrased as a change in 'average defection probability' in an Iterated Prisoner's Dilemma, which should be an on-policy quantity. The proposed test—simulating full games and comparing empirical defection rates to the uniform-history averages—settles the concern directly. I do not see grounds to reject the paper: the controlled measurement is deterministic and reproduces a real effect on conditional probabilities, and the qualitative finding that some interpretable SAE features modulate cooperative behavior in this toy setting is plausible. However, the headline quantitative claim should be restated as a property of the 64-history test suite, or validated on-policy, before it is used to support generalizations to real-world agent systems. I also note two secondary issues that do not change the verdict but should be fixed: the sign of the 'good/bad faith' effect is reversed between Section 5 and Table 1, and the claim of prompt-independence is weakened by the method's deliberate filtering of features by their activation on prompt tokens.","tokens_in":20439,"tokens_out":7105,"duration_ms":70904,"concrete_test":"Reproduce the Section 5 measurement for LLaMA3-IT-8B feature 30695 at the reported steering strengths, then run full IPD games (e.g., 250 games, mirroring Section 4.1) under each steering direction, sampling fourth-round actions from the model's next-token distribution (or with temperature 0.1) against a fixed opponent policy. Compute the empirical average fourth-round defection rate and compare it with the reported 47% and 75%. Also reweight the 64 measured conditional probabilities by the empirical history frequencies observed in those simulations; if the resulting on-policy average differs from the uniform average by more than about 5 percentage points, the 28pp headline is an artifact of the uniform-history ensemble rather than a description of actual game-level defection behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 explicitly replaces full IPD simulation with an exhaustive sweep over the 64 possible three-round histories and reads off the last-token probability P('blue') as the fourth-round defection probability. Section 5 then reports the headline effect for the 'good/bad faith' feature (index 30695): average defection changes from 47% to 75% (or the reverse; the sign is inconsistent between the text and Table 1), a 28pp shift. The problem is that this average is (1/64) Σ_h P_θ_w(blue|h), a uniform average over a fixed set of histories. In a real IPD, the expected fourth-round defection probability is Σ_h P_θ_w(h) P_θ_w(blue|h), where P_θ_w(h) is the distribution over histories induced by the steered policy, the opponent's policy, and the sampling process. Steering with w>0 and w<0 changes not only the conditional action probabilities but also the agent's own earlier actions, and therefore the histories that actually arise. A history that is nearly impossible under the positively steered policy—say, one where the agent defected three times—counts equally in the reported average. The uniform-history average is a property of a set of 64 fixed prompts, not a property of the agent's strategy in the game. Consequently, the 28pp shift may substantially overstate or understate the change in observable defection behavior once multi-turn dynamics are included. The paper's own preliminary study (Section 4.1) uses full game simulations, but the main steering results do not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an inference-time steering method for LLM agents in an Iterated Prisoner's Dilemma (IPD) environment. The authors use sparse autoencoder (SAE) features on the residual streams of Gemma-2B, Gemma2-2B, and LLaMA3-IT-8B, and measure the effect of positive and negative steering on the next-token probability of the defection token ('blue') on the fourth round, averaged uniformly over all 64 possible three-round histories. They report that steering the LLaMA3-IT-8B 'good/bad faith' feature (index 30695) shifts this average from 47% to 75% (or the reverse, depending on the passage), that a 'sacrifice' feature in Gemma-2B shifts it by 47 percentage points, and that these effects are consistent with the semantic interpretations of the features. The paper also discusses monosemanticity and argues that such steering could generalize to real-world agent alignment.","tokens_in":20723,"tokens_out":6495,"duration_ms":58332,"significance":"If the quantitative claims were validated in full multi-turn play, this would be a valuable empirical demonstration that an interpretable feature-level intervention can shift an LLM agent's game behavior in a semantically coherent direction. The use of public SAEs, the comparison across three model families, and the promise of reproducible code and data are strengths. However, as reported, the headline effect is a uniform average of next-token probabilities over fixed prompts, not an expected defection rate under the steered policy in an actual game, so the central quantitative claim needs revalidation before the significance of the result can be assessed.","major_comments":[{"comment":"The headline 28 percentage point shift is an average of P('blue') over all 64 fixed histories, i.e., (1/64) Σ_h P_{θ,w}(blue|h), where h enumerates all possible three-round histories. In an actual IPD, the expected fourth-round defection probability is Σ_h P_{θ,w}(h) P_{θ,w}(blue|h), with P_{θ,w}(h) induced by the steered policy. Steering with positive and negative w changes the agent's own earlier actions and therefore the distribution over histories, so a uniform average over all 64 histories is not an estimate of the agent's defection rate in the game. Section 4.2 explicitly replaces full IPD simulation with this exhaustive sweep, and Section 5 then reports the result as 'average defection probability'. The authors should either re-run full game simulations under both steering directions (as in the §4.1 preliminary study) or clearly relabel the result as a fixed-prompt next-token metric and provide a reweighted estimate.","section":"§4.2, §5"},{"comment":"The paper equates the next-token probability P('blue') with the agent's defection probability, but the agent's actual choice is a generated response, often in a specified format such as JSON for the Mixtral prompt. Decoding with temperature, sampling, or multi-token outputs such as 'project blue' may decouple the top next-token probability from the final action. The manuscript provides no validation that P('blue') matches the probability of selecting 'project blue' in a complete response. At minimum, the authors should report agreement between the next-token probability and decoded/sampled actions, or explicitly restrict the claim to the next-token distribution.","section":"§4.2, §5"},{"comment":"There is an internal inconsistency in the reported direction of the effect for the 'good/bad faith' feature (index 30695). Section 5 states ⟨P('blue'|+good/bad faith)⟩ = 47% and ⟨P('blue'|−good/bad faith)⟩ = 75%, whereas Table 1 lists 0.75 and 0.47 for the same row. These two statements imply opposite steering directions. The abstract's claim that the 'good-faith negotiation feature lowers the average defection probability' depends on which convention is correct, and this inconsistency must be resolved before the central claim can be evaluated.","section":"§5 vs Table 1"},{"comment":"The magnitude of the reported effects is selected from a large scan and depends on the steering strength w, which is chosen empirically per model so that P('green')+P('blue')≈1. The paper scans thousands of features (2,339 for Gemma-2B), identifies features in the tails of the δ distribution, and then reports the largest effects without multiple-comparison correction or confidence intervals. Because w is a free parameter and the headline features are selected post hoc, the 28pp and 47pp numbers are not parameter-free estimates of intervention efficacy. The authors should report the distribution of δ with variance estimates, show the sensitivity of the headline features to w (Figure 7 does this for one feature), and state the total number of features scanned for each model.","section":"§4.2, §5"}],"minor_comments":[{"comment":"The sentence 'cannot be directly applied tor high-stakes tasks' contains a typo; 'tor' should be 'to'.","section":"§1"},{"comment":"The text states w ∈ (−10, 8) but does not define the scale or normalization of w; Figure 3 notes that w is 'scaled down by a factor of 10', which should be explained in the method.","section":"§4.2"},{"comment":"The text reports ⟨P('blue'|−blue)⟩ = 0.04% for the Gemma-2b 'blue' feature, while Table 1 gives 0.04; if this is a probability, 0.04 is 4%, not 0.04%, and the formatting should be made consistent.","section":"Appendix A.3"},{"comment":"Figure captions 8-11 refer to 'Gamma-2b' instead of 'Gemma-2b'; please correct the model name.","section":"Appendix A.4"},{"comment":"Section 5 says 'we also find the 'green' and 'blue' features' for LLaMA3, but Table 1 lists only 'Blue' for that model; the text and table should agree.","section":"§5, Table 1"},{"comment":"The phrase 'denouncement of violence' in Section 5 and 'denounce of violence' in Appendix A.3 are inconsistent; please standardize.","section":"§5, Appendix A.3"},{"comment":"The sign convention for δ is not stated explicitly for the headline features; for the 'sacrifice' feature, the reported 47 percentage point effect is an absolute change, but the sign of δ is left implicit. Please state δ for each headline feature.","section":"Equation (1)"},{"comment":"The Gemma prompt template shown ends at 'round: 2' and does not display the exact formatting for the third round or the current-round request; please confirm that the full 64-history prompt template is specified in the released code.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is presented as a game-behavior result but is measured as a fixed-prompt next-token average, and the sign inconsistency between Section 5 and Table 1 for the headline feature compounds this issue. I recommend major revision: the paper needs full-game validation or a substantial reframing of the central claim, plus a corrected sign convention and robustness analysis for the steering strength. I did not identify concerns about citation patterns or novelty disclosure; the study is a reasonable empirical contribution if the quantitative claims are brought in line with the actual measurement procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible proof-of-concept that SAE feature steering can nudge an LLM's cooperation in a toy prisoner's dilemma, but the headline 28-point shift is a uniformly averaged next-token probability over 64 fixed histories, not a measurement of what happens in actual play. The paper is worth engaging, but the central quantitative claim is softer than it looks.\n\nWhat's genuinely new: combining SAE feature steering with interactive game-theoretic LLM evaluation. I haven't seen that specific combination. The protocol for the 64-history sweep is clear and reproducible: they publish code and data, use public SAEs, and scan thousands of features. The 'green'/'blue' sanity checks work as expected, and the semantic tokens for features like 'sacrifice' and 'good/bad faith' line up with the direction of the steering effect. That is real evidence that the representation-level manipulation is doing something interpretable.\n\nSoft spots:\n\n– The stress-test note is on target. The 28pp is (1/64) Σ_h P_θ_w(blue|h), a uniform average over a fixed set of prompts. In an actual game, steering changes the agent's own earlier actions, hence the distribution over histories, so the expected defection under the steered policy is not that uniform average. The paper's own preliminary study used full rollouts; the steering experiments do not. The headline number may therefore not describe observable behavior in the game.\n\n– The feature is selected from the tails of a distribution over thousands of features, with no multiple-comparison correction, and the steering strength w is tuned per model. So the 28pp is a selected extreme, not a typical effect. Also, no error bars or confidence intervals anywhere in the steering results.\n\n– There's a sign inconsistency: Section 5 reports ⟨P(blue|+)⟩=47% and ⟨P(blue|−)⟩=75%, but Table 1 lists 0.75 for + and 0.47 for −. That has to be fixed.\n\n– The claim of being 'independent of the prompt' is overstated, since the feature filtering is based on activation on the prompt tokens, and w is tuned per model.\n\nWho this is for: researchers working on interpretability and alignment of LLM agents. The qualitative idea — that monosemantic features can modulate cooperative behavior in a controlled game setting — is credible and worth following up. The quantitative headline needs a lot more work before I'd trust it as a measure of real agent behavior.\n\nRecommendation: yes, send it to peer review, but as a conditional accept with major revision. The work is honest, reproducible, and the core observation is likely real. The author should rerun at least a subset with full game simulations, give confidence intervals, correct for multiple comparisons, and resolve the sign mismatch.","headline":"A credible proof-of-concept that SAE steering shifts LLM cooperation, but the headline 28pp figure is a selected uniform average over fixed histories, not a measurement of real game behavior.","tokens_in":21253,"tokens_out":3624,"would_cite":false,"duration_ms":31461,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Steering one sparse-autoencoder feature moves an LLM agent's defection probability from 47% to 75%.","keywords":["LLM agents","Iterated Prisoner's Dilemma","sparse autoencoders","feature steering","interpretability","AI alignment","defection probability","LLM safety"],"falsifier":"Run the same steering experiment with full autoregressive decoding and sample complete four-round games, then compare actual defection rates to $P(\\mathrm{blue})$; if the 47%-to-75% separation does not survive full-game sampling, or if under steering $P(\\mathrm{green})+P(\\mathrm{blue})$ drops far below 1, the next-token proxy is falsified.","tokens_in":20180,"feed_emoji":"🎲","tokens_out":13222,"duration_ms":114114,"temperature":0.7,"pith_summary":"Large language model agents behave unreliably in strategic settings: with an identical prompt and game history, a small non-zero temperature can make a Mixtral agent's defection probability jump from near 0.1 to near 0.9. The paper's proposed intervention is to modify the model's residual stream at inference time by adding a decoded sparse-autoencoder feature, a method it argues is independent of both the particular game and the prompt. In an Iterated Prisoner's Dilemma worded as an investment game, steering the 'sacrifice' feature of Gemma-2b moves the average fourth-round defection probability from 22% to 69%, and steering the monosemantic 'good faith/bad faith' feature of LLaMA3-IT-8B moves it by 28 percentage points, between 47% and 75%. The paper positions this as a step toward transparent, inference-time alignment of agent behavior, with game-theoretic evaluation serving as the testbed.","feed_headline":"One feature steers an LLM agent's defection rate by 28 points","feed_subtitle":"Steering one sparse-autoencoder feature moves the agent's chance of defecting in a prisoner's dilemma from 47% to 75%.","key_machinery":"Sparse autoencoder feature steering is the mechanism: at inference time, the model's residual-stream activation at a chosen layer is modified by adding the decoded feature vector, $x'_l = x_l + \\omega W_{\\mathrm{dec}}(f_{ID})$. The paper relies on pre-trained SAEs whose features are sparse and often monosemantic, and selects features using two criteria: the feature's activation-density histogram (bimodal for single-meaning features) and the preservation of $P(\\mathrm{green}) + P(\\mathrm{blue}) \\approx 1$ in the next-token distribution. The steering strength $\\omega$ is chosen empirically for each model. This machinery carries the argument because it connects a human-readable concept such as 'good faith' to a measurable change in strategic behavior without retraining or prompt redesign.","core_discovery":"On its own terms, the paper reports that a single sparse-autoencoder feature can act as a semantic steering dial for an LLM agent's strategy. In the LLaMA3-IT-8B model, the feature labeled 'good faith/bad faith' (index 30695) is monosemantic, and steering it in the residual stream changes the average fourth-round defection probability by 28 percentage points, from 47% to 75%, across all 64 possible three-round histories. The change is approximately monotonic in steering strength for each history. The same procedure applied to the 'sacrifice' feature of Gemma-2b moves the average defection probability from 22% to 69%, and an 'environment' feature appears to steer both the Gemma and LLaMA families. The paper concludes that game-theoretic evaluation, combined with representation-steering alignment, may generalize to real-world applications on end-user devices and embodied platforms.","pith_inferences":["Beyond the paper: the 28-percentage-point result is measured on next-token probability, not on full game trajectories; the natural next experiment is to sample complete games and see whether actual defection rates track $P(\\mathrm{blue})$ under the same steering strengths.","The near-monotonic good-faith axis suggests the model has an internal dimension that links negotiation 'faith' to cooperation; if that axis is general, the same feature should shift behavior in other trust- and negotiation-based games, which is testable.","Because polysemantic features like 'sacrifice' also steer strongly, the practical extension would be an automated pre-screen that ranks features by activation-density bimodality and by the stability of $P(\\mathrm{green})+P(\\mathrm{blue})$, replacing manual token inspection."],"forward_implications":["A monosemantic feature can serve as a continuous strategy dial: the 'good faith/bad faith' direction in LLaMA3-IT-8B moves average fourth-round defection between 47% and 75% with a near-monotonic response.","Because the intervention happens in the residual stream at inference time, it can be applied to open models with publicly available sparse autoencoders, avoiding fine-tuning cost and catastrophic forgetting.","Abstract features such as 'sacrifice' and 'environment' steer behavior even though they are not game tokens, and at least one feature ('environment') has effects across model families.","Feature effectiveness is selective: the monosemantic 'trust' feature leaves defection nearly unchanged (47% to 50%), so automatic feature selection matters for any deployment.","The hypothesized generalization is that the same steering procedure could align agents in real-world settings beyond the toy game, such as consumer devices and embodied platforms."],"supporting_citations":[{"why":"Supplies the sparse-autoencoder interpretability foundation: feature splitting and activation-density histograms that identify monosemantic features.","marker":"[11]"},{"why":"Establishes that sparse autoencoders trained on LLM activations find highly interpretable features, motivating their use for steering.","marker":"[18]"},{"why":"Provides the Iterated Prisoner's Dilemma baseline for LLM behavior and the logistic relationship between opponent and agent defection that the preliminary study reproduces.","marker":"[25]"},{"why":"Provides the residual-stream sparse autoencoder for the LLaMA-3 model, from which the 'good faith/bad faith' feature is drawn.","marker":"[35]"},{"why":"Software used to hook the sparse autoencoder to the residual stream and perform the steering at inference time.","marker":"[36]"},{"why":"Prior inference-time intervention method that modifies internal activations, setting the context for activation steering.","marker":"[39]"},{"why":"Provides the sparse autoencoders for the Gemma-family models used in the experiments.","marker":"[40]"},{"why":"Source of the prompt and problem setup adopted so results are comparable with earlier one-shot experiments.","marker":"[51]"},{"why":"Defines representation engineering as a general strategy of steering activations toward abstract concepts.","marker":"[73]"}],"fun_headline_variants":["Steering one feature cuts LLM agent defection by 28 points","Single sparse-autoencoder feature steers LLM's defection probability","Good-faith feature moves LLM defection from 47% to 75%","LLM agent strategy steered via interpretable feature in residual stream","Game-theoretic test shows one feature changes LLM defection by 28%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the next-token probability of the 'blue' token on a fixed fourth-round prompt equals the agent's real defection probability; if decoding and multi-turn dynamics break that link, the reported 28-point steering effect may not describe actual agent strategy.","fun_headline_variants_meta":{"raw":{"variants":["Steering one feature cuts LLM agent defection by 28 points","Single sparse-autoencoder feature steers LLM's defection probability","Good-faith feature moves LLM defection from 47% to 75%","LLM agent strategy steered via interpretable feature in residual stream","Game-theoretic test shows one feature changes LLM defection by 28%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1807,"prompt_tokens":887,"completion_tokens":920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":821}},"tokens_in":503,"tokens_out":920,"duration_ms":7325,"temperature":1.0,"reasoning_tokens":821,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:05:30.246651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same steering experiment with full autoregressive decoding and sample complete four-round games, then compare actual defection rates to $P(\\mathrm{blue})$; if the 47%-to-75% separation does not survive full-game sampling, or if under steering $P(\\mathrm{green})+P(\\mathrm{blue})$ drops far below 1, the next-token proxy is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sparse-autoencoder interpretability foundation: feature splitting and activation-density histograms that identify monosemantic features."},{"cited_title":"llama-3-8b-it-res (revision 53425c3), 2024","cited_arxiv_id":null,"evidence_quote":"Provides the residual-stream sparse autoencoder for the LLaMA-3 model, from which the 'good faith/bad faith' feature is drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Software used to hook the sparse autoencoder to the residual stream and perform the steering at inference time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the prompt and problem setup adopted so results are comparable with earlier one-shot experiments."}],"review_version":1}