{"id":"81460d3b-365f-4ef8-aafa-1dacdae05578","arxiv_id":"2505.03189","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Contrastive activation steering of LLMs is reliable only in-distribution and harms model perplexity, while larger models resist the damage better.","lead":"This paper tested contrastive activation engineering, a zero-cost way to steer language model behavior by editing internal states. It found that steering only works near the distribution it was built from, degrades model quality, and can be reversed by adversarial prompts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The out-of-distribution conclusion rests on an unvalidated LLM judge and a small synthetic dataset; the paper itself concedes this in Section 8, so the central claim that CAE fails OOD is not yet supported.","rationale":"The reader's weakest-assumption analysis identified exactly the same concern: the OOD evaluation depends on an unvalidated LLM judge over a small synthetic dataset, and the paper itself admits this assumption was not verified. I agree with that assessment and with the CONDITIONAL verdict. The central claim that CAE fails out of distribution would be the main contribution of the paper, but it is only as strong as the evaluation that produced Figures 8-11. Validating the judge is a concrete, feasible step that would either confirm the flat OOD curves or reveal an artifact. Other issues, such as the internal inconsistency in the claimed sample-size threshold (abstract says 80, Section 6 says 100, Section 5 mentions 89 and 55) and the lack of error bars, are secondary and do not by themselves undermine the main direction. The concern is not that the authors are wrong; it is that the key OOD claim is currently underdetermined by the evidence presented, and the authors' own limitation statement flags the missing verification.","tokens_in":12867,"tokens_out":1956,"duration_ms":20538,"concrete_test":"Validate the judge by having two independent human annotators score a stratified sample of roughly 100 steered outputs per model (spanning strengths -2, +2, +5 and both CAA and ActAdd) on the same behavior and coherency dimensions, then measure agreement between the LLM judge and human labels (e.g., Cohen's kappa or ICC). If agreement is below 0.6, re-plot Figures 8-11 using human scores; if the flat OOD curves persist, the claim is supported, and if they show a monotonic steering effect, the original conclusion was an evaluation artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline finding—that CAE is only reliably effective in-distribution—depends almost entirely on the Section 5 out-of-distribution evaluation. There, Llama 3 70B-Instruct is used to score both behavior and coherency of steered 8B and 70B outputs on 540 synthetic questions generated by DeepSeek-R1. Section 8 explicitly states: \"Our methodology relies on the assumption that LLMs can act in place of human evaluators to provide feedback signals (Bavaresco et al., 2024). We did not verify this assumption on the specific tasks we evaluated steered models on. Although the OOD evaluation dataset passes qualitative checks, it is not grounded in real user queries.\" If the judge is insensitive to the target behavior, or if its coherency score drops as soon as outputs become non-typical, then the flat combined-score curves in Figures 8–11 could be an artifact of judge bias or of the product-of-scores metric masking a behavior gain with a coherency penalty. The small per-condition sample size (30 prompts per behavior, 9 behaviors per split) and absence of error bars further weaken the inference. Because the central claim is precisely that steering makes no obvious difference OOD, the unvalidated judge is the most load-bearing assumption in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies contrastive activation addition, a form of contrastive activation engineering, applied to Llama 3 8B and 70B Instruct models across ten steering targets. It reports in-distribution layer and strength sweeps on Anthropic's model-written evaluations, a synthetic out-of-distribution evaluation with choice-qa and open-ended splits scored by Llama 3 70B-Instruct, MMLU degradation as a function of vector sample size, perplexity effects on question sets, and an adversarial prompt-optimization study. The abstract's five findings are that CAE is only reliably effective in-distribution, that sample-size gains diminish around 80 samples, that steering can be reversed by adversarial inputs, that steering harms perplexity, and that larger models resist degradation more.","tokens_in":13216,"tokens_out":7112,"duration_ms":65157,"significance":"If the findings were established, the paper would provide practically useful guardrails for CAE deployment and a lightweight, extensible out-of-distribution evaluation protocol. The strengths include a public OOD dataset, unusually explicit limitations, experiments at two model scales, and replication of previously reported optimal-layer phenomena. However, the central OOD claim is conditional on an unvalidated LLM judge, and several secondary findings are stated more strongly than the evidence supports. The current manuscript is not yet sufficient to carry the headline conclusions.","major_comments":[{"comment":"The central claim that CAE has no obvious out-of-distribution effect is currently supported only by an unvalidated LLM judge. Section 8 explicitly concedes that the assumption that LLMs can replace human evaluators 'was not verified' on the specific tasks evaluated. Because the combined score is the product of behavior and coherency scores, the flat curves in Figures 8–11 could arise from judge insensitivity to the target behavior or from a coherency penalty on unusual outputs, masking a genuine behavior signal. The evaluation uses 540 synthetic questions (30 per behavior per split) with no error bars or human validation, so the uncertainty around the flat curves is unknown. To support the headline conclusion, the authors should validate the judge against human annotations on a subset, report per-behavior and per-question-length breakdowns with uncertainty, or use a less subjective metric such as known-answer multiple-choice accuracy.","section":"Section 5; Figures 8–11; Section 8"},{"comment":"The sample-size threshold for diminishing returns is stated inconsistently: the abstract says 'around 80 samples', Section 4 says performance converges 'beyond 100 samples' and also notes that 20% of the MWE train split is 160 samples, and Figures 6–7 are described as converging 'beyond 100 samples'. Moreover, Section 5 states that steering vectors generated with 89 and 55 samples perform best for Llama 8B and 70B respectively, but Table 1 does not show a monotonic trend for Llama 8B (e.g., 21 samples gives 2.1, 34 gives 3.4, and 55 gives 3.2). The authors should reconcile these numbers and state the exact criterion used to define the convergence threshold and the 'best' sample size.","section":"Abstract; Section 4; Section 5; Table 1"},{"comment":"The abstract-level finding that steering vectors harm overall model perplexity is demonstrated for Llama 3 8B at steering strength +1 on MWE and DeepSeek-generated questions; no corresponding 70B perplexity sweep or strength sweep is shown in the main text. The illustrative French-feature example uses a different model (Gemma 2 2B) and a different feature, so it does not directly support the claim for the steering targets studied in Sections 4–5. If this is to stand as a general finding, the authors should provide perplexity results for both model sizes, across steering strengths, and for the same ten steering targets used elsewhere in the paper.","section":"Section 6; Figures 14–15"},{"comment":"The claim that steering vectors are 'susceptible' to adversarial inputs is supported only by four example strings with high cross-entropy; no success rate, number of optimization runs, or comparison to unsteered baselines is reported. The text does note that the EPO inputs have 'significantly high cross-entropy', which qualifies the finding, but the abstract's wording overstates the evidence. Please add quantitative success rates, the number of runs, and a baseline showing how often the same prompts flip the unsteered model, to substantiate the adversarial-susceptibility claim.","section":"Section 7; Table 2"},{"comment":"The in-distribution effectiveness metric is bounded by baseline behavior, as the authors acknowledge in Section 8 ('if the model is honest 95% of the time, it can be at most 5% more honest'). Because Figures 2–5 aggregate percentage changes across behaviors with very different baselines, the averaged curves may be dominated by ceiling or floor effects. The authors should report per-behavior baseline rates and either normalize the metric or demonstrate that aggregation is not driven by these bounding effects.","section":"Section 4; Section 8"}],"minor_comments":[{"comment":"The title promises 'Mechanisms', but the paper presents empirical patterns and does not test any mechanistic hypothesis; consider retitling to reflect this scope, for example 'Patterns and Limitations of Contrastive Activation Engineering'.","section":"Title"},{"comment":"The text states that the test split is 'in-distribution because we used the same dataset to generate the steering vectors'; it would be clearer to state explicitly that the vector-generation split and the evaluation split are disjoint subsets of MWE, which appears to be the intended meaning.","section":"Section 4"},{"comment":"The text refers to 'Table 1' for the adversarial examples, but the examples appear in Table 2; this cross-reference should be corrected.","section":"Section 7"},{"comment":"The sign convention in Table 1 is unclear: positive values for Llama 8B (e.g., +3.7) are described as 'degradation (%)', yet they suggest an improvement over baseline. Please define the direction of the change and explain why a positive value indicates less degradation.","section":"Table 1"},{"comment":"The captions mention 'column centering' versus 'without column centering', but the centering choice is not described in the text, and the color scales differ between the two figures; please explain the preprocessing and keep the scales consistent or justify the difference.","section":"Figures 14–15"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's OOD evaluation method and dataset could be a useful community resource, but the unvalidated judge is a serious gap that directly affects the headline claim. The explicit limitations section is a strength, but it also documents that the most load-bearing assumption was not checked. A major revision with human validation of the judge (or a softened central claim) seems appropriate. The title's 'Mechanisms' overpromises relative to the content, which may matter for fit with the journal's expectations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives us three things that are genuinely new: systematic sample-size convergence curves for steering vectors, a lightweight AI-feedback OOD evaluation dataset, and an EPO-based adversarial nullification demonstration. All three are useful. The sample-size result (diminishing returns around 80–100 examples) is the kind of concrete number practitioners want, and the EPO section shows that even in-distribution steering can be reversed, though the inputs are unnatural. Credit where due: the authors are transparent. Section 8 explicitly concedes that the LLM judge was not verified and that the OOD dataset is synthetic. They also cite Tan et al. and Turner et al. appropriately and do not oversell novelty.\n\nThe in-distribution layer and strength sweeps on Llama 3 8B/70B are competently done and align with prior work. The larger-model robustness finding is plausible and consistent with the graphs. The MMLU sample-size benchmark is a nice addition, though the sign convention in Table 1 is confusing—negative degradation for small samples looks like improvement.\n\nThe soft spots are real and mostly where the reader's report puts them. The central OOD conclusion—CAE is ineffective out of distribution—rests almost entirely on a 540-question synthetic set and an unvalidated judge. The paper admits this. If the judge is insensitive or biased, the flat curves in Figures 8–11 could be artifacts. The per-condition sample size is tiny (30 prompts per behavior) and there are no error bars, so we cannot tell how flat those curves really are. The sample-size threshold is also inconsistent: abstract says 80, figures show convergence near 100, text mentions 160, conclusion says \"a hundred.\" That needs a single consistent number.\n\nPerplexity harm is demonstrated for one steering strength (+1) and one model (8B) for the claims in Section 6, so it reads as a single data point rather than a sweep. The EPO adversarial results are explicitly exploratory and the authors note the inputs are unlikely to occur naturally.\n\nNone of this breaks the core qualitative direction. The paper is a deployment-oriented playbook, not a theoretical breakthrough, and it is honest about its gaps. Practitioners will get reasonable expectations from it. The missing code and reliance on a dataset link limit reproducibility, but the authors seem capable of fixing the major issues with additional analysis rather than new experiments.\n\nI would send this to peer review. The right outcome is major revision: validate the judge against human ratings or at least show calibration, add error bars or release full per-prompt scores, and clean up the sample-size messaging. The paper deserves that referee time.","headline":"A useful, honest empirical addendum to the CAE literature, with the headline OOD claim resting on an unvalidated judge.","tokens_in":13639,"tokens_out":2064,"would_cite":true,"duration_ms":20534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that contrastive activation engineering is only reliably effective in-distribution: steering vectors control behavior on familiar prompts, make little difference on new ones, and still degrade perplexity.","keywords":["contrastive activation engineering","activation steering","contrastive activation addition","out-of-distribution generalization","perplexity","adversarial prompts","LLM interpretability","AI safety"],"falsifier":"Give the paper's 540 synthetic questions to human raters who score the same steered outputs for behavior and coherence, then ask whether steering shifts human behavior scores at strengths +1 to +3. A clear human-measured steering effect where the judge model recorded none would falsify the paper's central out-of-distribution claim.","tokens_in":12649,"feed_emoji":"🎛️","tokens_out":7959,"duration_ms":66887,"temperature":0.7,"pith_summary":"This paper tries to establish a practical playbook for contrastive activation engineering (CAE), a zero-cost, inference-time family of methods that steer LLM behavior by adding a contrast vector to internal activations. Its central claim is that CAE is only reliably effective in-distribution: vectors computed from one dataset shift answers on held-out examples of that dataset, but show no obvious behavioral effect on a separate set of realistic user prompts. The paper also argues that steering-vector quality saturates around 80–100 contrastive samples, that steering generally worsens model perplexity, that larger models tolerate steering better, and that evolutionary prompt optimization can produce adversarial inputs that reverse the steered behavior. A sympathetic reader should care because these findings determine whether a lightweight, fine-tuning-free control method can actually be trusted outside carefully matched evaluation sets.","feed_headline":"Steering vectors only work in-distribution","feed_subtitle":"CAE nudges behavior on familiar prompts, then fades on new ones and quietly raises perplexity.","key_machinery":"The central object is the steering vector from contrastive activation addition: at a chosen layer $l$, the modified activation is $$A'_l(x) = A_l(x) + \\$\\alpha$ \\left( \\frac{1}{|D|} \\sum_{(x^+,x^-)\\in D} \\left[A_l(x^+)[-1] - A_l(x^-)[-1]\\right] \\right),$$ where $x^+$ and $x^-$ are desired and undesired prompts, $D$ is the contrastive dataset, and $\\alpha$ is the steering strength. The paper varies the layer, the strength, and the number of contrastive pairs used to estimate the vector, and measures behavior shifts in-distribution, on a synthetic out-of-distribution question set, and on perplexity benchmarks. The same vector object connects every result: in-distribution control, saturation near a hundred examples, out-of-distribution failure, adversarial reversal, and performance degradation.","core_discovery":"On the paper's own terms, the discovery is a boundary condition on contrastive activation addition: adding the mean difference between positive and negative residual activations at an early-mid layer (layer 15 for the 8B model, layer 29 for the 70B model) reliably increases answer-matching behavior only when the test prompts come from the same distribution as the contrastive dataset. On a new 540-question dataset spanning ten behavioral targets, the same vectors produce no right-skew in behavior or combined behavior-coherency scores across steering strengths, while single-pair vectors degrade even faster. Sample-size sweeps show the performance of the vector converges once roughly a hundred contrastive examples are used. Steering also lowers the likelihood of most held-out sequences: perplexity worsens on nearly all tested steering-target/question combinations, and the larger model shows smaller degradation. Finally, adversarial prompts found by evolutionary optimization—though with high cross-entropy and unlikely to occur naturally—can invert the behavior a steering vector was meant to enforce.","pith_inferences":["Beyond the paper: if the out-of-distribution null result holds, linear steering directions are context-relative, which would qualify the linear representation hypothesis and motivate computing vectors from multiple contexts rather than one contrast set.","Beyond the paper: because the judge-model assumption was not verified, the strongest disconfirmation path is a human-rated rerun of the 540-question set; a positive human result would overturn the no-out-of-distribution-effect reading while leaving the in-distribution and perplexity findings intact.","Beyond the paper: the perplexity cost suggests a measurable trade-off—behavior shift per unit perplexity increase—that could serve as a standard efficiency metric for comparing steering methods.","Beyond the paper: adversarial reversal implies CAE pairs naturally with input filtering; a detector that blocks high-cross-entropy adversarial prompts could neutralize the nullification attack without changing the steering mechanism."],"forward_implications":["Deploying CAE in practice requires collecting high-quality contrastive data from the actual deployment distribution; otherwise the expected behavioral benefit is near zero.","There is little reason to gather more than about one hundred contrastive pairs per behavior, because additional samples give diminishing returns.","Steering should be treated as a performance tax: it raises perplexity on nearly all tested steering-target and question combinations, even at small steering strengths.","Larger models are better subjects for steering because they show less steering-induced degradation than smaller models.","Adversarially crafted prompts can reverse the behavior a steering vector enforces, so CAE should not be relied on as a standalone safety control."],"supporting_citations":[{"why":"Introduces activation engineering and the injection mechanism on which all steering vectors here are built.","marker":"[Turner et al., 2024]"},{"why":"Defines contrastive activation addition, the exact steering-vector formula the paper sweeps.","marker":"[Panickssery et al., 2024]"},{"why":"Supplies the Model-Written Evaluations dataset used as the contrastive source and in-distribution test set.","marker":"[Perez et al., 2022]"},{"why":"Prior result that steering effectiveness is largely dataset- and target-dependent, framing the paper's distribution question.","marker":"[Tan et al., 2024]"},{"why":"Provides evolutionary prompt optimization, the method that finds the adversarial reversal inputs.","marker":"[Thompson et al., 2024]"},{"why":"Grounds the LLM-as-judge assumption the paper depends on but does not verify for its own tasks.","marker":"[Bavaresco et al., 2024]"},{"why":"Provides the Pile corpus used in the preliminary loss-perplexity analysis of steering effects.","marker":"[Gao et al., 2020]"},{"why":"States the linear representation hypothesis that motivates why adding a direction should steer behavior.","marker":"[Park et al., 2024]"}],"fun_headline_variants":["CAE steering works only in-distribution","Steering vectors fail out-of-distribution","Contrastive activation: OOD breakage, perplexity cost","Diminishing returns at 80 samples for CAE","Adversarial prompts can flip CAE steering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The out-of-distribution verdict rests on the unverified assumption that a larger judge model's behavior and coherence scores correctly measure steering on a synthetic question set; if the judge is insensitive or biased, the null OOD result could be an evaluation artifact rather than a property of steering.","fun_headline_variants_meta":{"raw":{"variants":["CAE steering works only in-distribution","Steering vectors fail out-of-distribution","Contrastive activation: OOD breakage, perplexity cost","Diminishing returns at 80 samples for CAE","Adversarial prompts can flip CAE steering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1611,"prompt_tokens":930,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":606}},"tokens_in":546,"tokens_out":681,"duration_ms":6651,"temperature":1.0,"reasoning_tokens":606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:56:53.906582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the paper's 540 synthetic questions to human raters who score the same steered outputs for behavior and coherence, then ask whether steering shifts human behavior scores at strengths +1 to +3. A clear human-measured steering effect where the judge model recorded none would falsify the paper's central out-of-distribution claim.","supporting_citations":[],"review_version":1}