Pith. sign in

REVIEW 4 major objections 5 minor 36 references

Prompt-side optimization can drive an internal 'evaluation-awareness' signal to zero, yet the model's own yes/no judgment about being evaluated stays put—so a readable latent is not a controllable behavior.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:08 UTC pith:NOV7XPTZ

load-bearing objection A well-controlled input-side suppression study with a solid negative result that rests on a behavioral endpoint whose off-manifold sensitivity is the main thing I'd want referees to stress-test. the 4 major comments →

arxiv 2607.25907 v1 pith:NOV7XPTZ submitted 2026-07-28 cs.LG cs.AIcs.CL

Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models

Deepanshu Mody , Samarth Agarwal , Utkarsh Mittal , Dipesh Mahato This is my paper
classification cs.LG cs.AIcs.CL
keywords evaluation awarenessactivation steeringlatent suppressionprompt optimizationinterpretabilitysparse autoencodersbehavioral controllabilityLLM safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper attempts to control an 'evaluation-awareness' internal signal purely from the input side, by optimizing a short fluent prompt that drives the latent toward zero with no inference-time access to activations. It finds that this input-side suppression works at the activation level across several target constructions, including fully turning off a causally validated sparse-autoencoder feature, with suppression strengths around z = -7. The central negative result is that this activation-level success does not transfer to behavior: a placebo random direction suppresses just as hard and shifts the model's own eval judgment just as far, and when a real evaluation passage is held in context, suppressing the direction fails to reduce and slightly increases the behavioral eval judgment. A single MLP neuron is eval-correlated but not causal at two model scales, and a scan of natural text matches the optimizer on the internal direction while the optimizer is necessary only for the behavioral logit. The paper concludes that activation-readability is not behavioral controllability, and that claims about controlling internal concepts must be measured against behavior, not activation alone.

Core claim

The paper shows that by adapting a feature-dreaming objective with a negated feature term—token-gradient discrete search plus a self-cross-entropy fluency regularizer and a KL task-retention term—one can produce fluent prompts that drive a chosen internal latent toward zero on instruction-tuned LLMs. Across five target constructions (a linear mean-difference direction, a subspace norm, a sparse-autoencoder feature, a single MLP neuron, and the output logit difference for the evaluation question), the latent is suppressible to z ≈ -7, and a causally validated sparse-autoencoder feature is turned off completely and selectively. However, controlled tests show the suppression is not evaluation-s

What carries the argument

The paper's central object is the target latent f(t), a scalar functional of the residual-stream activation at a chosen layer, constructed in five ways: a linear mean-difference direction, a k-dimensional subspace norm, a sparse-autoencoder feature activation, a single MLP neuron's post-activation, and the yes/no logit difference on the evaluation question. The optimizer maximizes a score consisting of the negated target latent plus fluency (self-cross-entropy) and KL task-retention regularizers, using token-gradient discrete search over a population of fluency weights to trace a suppression-vs-fluency frontier. The behavioral endpoint is the model's own calibrated log-odds answering 'is thi

Load-bearing premise

The behavioral endpoint—the model's own yes/no log-odds to 'is this text an evaluation?', calibrated on held-out prompt sets—is assumed to be a valid and sensitive measure of behavioral evaluation-awareness, and its calibration is assumed to transfer to the optimized, off-manifold prompts; the paper itself flags the off-manifold character and the register confound as unresolved, so if either fails the negative conclusion could be an artifact of an insensitive outcome measure.

What would settle it

A concrete falsifier: take a set of real evaluation passages and, for each, optimize a prompt prefix to suppress the eval direction at the passage's last token; if across many held-out bodies and seeds the prefix consistently moves the model's behavioral eval log-odds below the no-prefix baseline (toward 'deploy'), the paper's central negative claim is refuted. A second check: train a fresh linear probe on random-length prompts of matched perplexity and read it on the optimized prompts—if it separates eval from deploy at high accuracy, the claimed erasure is a rotation artifact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If activation-readability is not behavioral controllability, then safety evaluations should treat activation-level probes as suggestive at best and require behavioral endpoints—such as the model's own task-relevant judgments—before concluding a concept has been controlled.
  • Input-side dual of negative steering is a real capability: a prompt alone can zero an internal latent, including a causally validated sparse-autoencoder feature, which could be used to hide an internal signal from probe-based audits.
  • Because placebo random directions suppress and shift behavior as much as the real eval direction, studies reporting suppression of a target direction need placebo and in-context controls before claiming evaluation-specific control.
  • A single MLP neuron can show a large effect size on an eval-vs-deploy contrast yet fail a causal ablation test, so neuron-level attributions for evaluation-awareness need interventional validation.
  • A natural-text corpus scan can compete with the gradient optimizer for suppressing an internal direction, but only the optimizer decisively moves the behavioral logit target, suggesting optimization is necessary when the endpoint is behavior, not activation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implied consequence is that input-side attempts to 'hide' evaluation-awareness via prompt optimization are unlikely to fool behavioral evaluations, since the optimized prompts leave the model's own yes/no judgment unchanged or even more eval-like in context.
  • The placebo result suggests the observed deploy-ward shifts are generic properties of optimized short strings rather than evidence of evaluation-specific control; future work should include matched-perplexity and matched-register controls, not just random directions.
  • The protocol could be repurposed as a diagnostic for latent causality: a target whose suppression does not move a behavioral endpoint is a readout rather than a control knob, while the sparse-autoencoder feature's partial behavioral effect shows where prompt-side control starts to matter.
  • A testable extension is to run the same input-only minimization against a steering direction that is already known to cause a behavioral change (e.g., a refusal direction), to see whether the failure of behavioral control is specific to evaluation-awareness or general to prompt-side suppression; the paper's sparse-autoencoder feature result hints at a partial effect.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies input-only suppression of internal 'evaluation-awareness' latents in LLMs. Adapting Fluent Dreaming/EPO with a negated feature term and a GCG-style discrete optimizer, it optimizes a short fluent prompt to drive a chosen internal target toward zero, without inference-time model access. Five target constructions are tested on Llama-3.2-3B and Llama-3.1-8B: a CAA mean-difference direction, a subspace norm, a Llama Scope SAE feature, a single MLP neuron, and a behavioral logit-difference. The authors find that the latent is robustly suppressible (z≈-7), that a causally-validated SAE feature can be fully and selectively turned off, but that this suppression does not translate into behavioral control: a placebo random direction suppresses just as hard and shifts the behavioral eval log-odds just as far, and an in-context 'wrap' experiment shows that suppressing the eval direction in front of a fixed eval body leaves the behavioral eval judgment unchanged or slightly increased. They conclude that activation-readability is not behavioral controllability, and propose a behavioral-plus-erasure protocol with an orthogonal-complement probe-hacking detector and a positive control. The paper openly acknowledges several limitations, including seed-dependence of fluent suppression, a residual register confound, and the bounded, unresolved erasure-vs-rotation question.

Significance. If the central negative result holds, it is an important cautionary finding for the interpretability and safety-evaluation communities: it demonstrates that a linearly readable and suppressible internal direction need not imply behavioral controllability, and it provides a reusable protocol (behavioral endpoint, fresh probe, orthogonal-complement probe-hacking control, positive control, placebo directions) for separating activation-level changes from behavior-level changes. The input-only minimization method itself is a novel dual to activation steering and is evaluated across multiple target constructions and model scales. The paper is unusually careful in its controls and honest in its limitations: it reports seed replication, a balanced contrast to address the register confound, a real-Pile baseline, cross-size transfer, and a positive control for the erasure detector. However, the strength of the headline claim is bounded by the fact that the behavioral endpoint is a self-report validated only on the synthetic contrast set, and its sensitivity in the optimized, off-manifold wrap regime is not established. The contribution is therefore valuable as a methodological template and a

major comments (4)
  1. [§5.7 (Wrap threat model)] The central negative claim rests on the wrap experiment, but the behavioral endpoint is not validated in the wrap regime. The endpoint is calibrated on the synthetic eval-vs-deploy contrast set and gated by AUROC 0.92 in §5.4; no evidence is given that it is sensitive to genuine eval/deploy differences when a real eval body is held fixed and only a free prefix is varied. The observed null/backfire could therefore reflect endpoint insensitivity rather than absence of behavioral control. A positive control in this exact regime (e.g., optimizing a prefix to increase eval-likeness, or scoring prefixes with known eval/deploy status appended to fixed bodies) is needed before the negative conclusion is interpretable.
  2. [§5.7 and Table 3] The comparison between real-direction and random-direction prefixes is potentially confounded by surface fluency. The paper reports suppression z and behavioral log-odds shifts but not self-XE or perplexity matching for the prefixes used in the wrap experiment. If the real-direction prefix is less fluent or more 'test-like' in surface form, the observed backfire could be a distributional artifact rather than evidence about the latent. Report self-XE/perplexity for the optimized prefixes and either match the placebo prefixes on fluency or include a fluency-matched baseline.
  3. [§3.3/§5.4] There is a self-referential loop: the CAA direction, the fresh erasure probe, and the behavioral calibration all derive from the same eval-vs-deploy contrast set. The balanced contrast reduces the register confound only partially (peval 0.78→0.46, §4.1/Table 2), so the target and the endpoint may both track a residual formality component rather than evaluation-awareness. This does not invalidate the internal consistency of the results, but it weakens the mapping from 'suppressing the eval direction' to 'suppressing evaluation-awareness.' An independent behavioral measure, or at least a held-out evaluation task with explicit eval/deploy manipulation, is needed to support the construct-level claim.
  4. [§5.7/Conclusion] The behavioral endpoint is a meta-linguistic yes/no self-report ('Is this text an evaluation?'), not a task behavior. The abstract's slogan 'activation-readability is not behavioral controllability' overstates the evidence: the experiments show that suppressing the internal readout does not change the model's stated eval judgment. The safety-evaluation threat described in §1 is about behavior under evaluation (e.g., task performance or compliance), so the conclusion's scope should be limited to the self-report outcome, or the claims should be rephrased accordingly.
minor comments (5)
  1. [§3.3] The term 'peval' is used before it is explicitly defined; define it as the probe's estimated probability that the input is eval-like at first use.
  2. [Table 4] The header 'max∆fmin' is unclear; use separate columns or a clearer notation such as 'max ∆f' and 'min f'.
  3. [Figure 2] The caption reports 'self-XE 12' for the optimizer point on the direction panel, while Table 1 reports self-XE 12.7 for the λ=0 baseline; please ensure consistency between figure annotations and tables.
  4. [Appendix A] The package name 'TRANSFORMERLENS' should be capitalized consistently as 'TransformerLens'.
  5. [§5.1] The phrase 'strict fluent suppression (XE<6) is reached in only 1/4' is clear, but the abstract's 'robustly suppressible' should be qualified to avoid implying that fluent suppression is reliable; the Limitations section already does this, so consider aligning the abstract language.

Circularity Check

0 steps flagged

No circularity found: the target direction, behavioral endpoint, and erasure probes are distinct functionals; the negative result rests on divergent empirical controls.

full rationale

The claimed derivation chain is self-contained and does not reduce any output to its own inputs. The CAA direction v is constructed from an eval-vs-deploy contrast set (§3.1(i)), optimized prompts are scored by suppression z, behavioral log-odds, and fresh-probe erasure (§3.2, §3.3), but the behavioral endpoint is a distinct functional (yes/no output log-odds) from the internal direction being suppressed. Sharing the same contrast distribution is a construct-validity limitation, which the paper partially mitigates with register balancing and held-out calibration; it is not a definitional reduction. The logit-difference target (Eq. 2) is explicitly the behavioral endpoint, and optimizing it is direct objective optimization, not a prediction of an independent quantity; the paper labels it as such rather than presenting it as a derived result. The central negative claim—that suppressing the eval-direction does not control behavior—is supported by placebo directions, an in-context wrap test, and model-side steering comparisons, all of which are empirical controls rather than fitted outputs. No load-bearing self-citations appear; the cited methods (CAA, EPO/GCG, Llama Scope) are external and used transparently. The Limitations sections explicitly bound erasure-vs-rotation and describe the off-manifold probe silence as 'suggestive, not decisive,' which is an epistemic caution, not circularity. Any concern that the self-report behavioral endpoint is insensitive in the optimized, off-manifold regime is a validity/sensitivity criticism outside the scope of circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claims rest on data-dependent target constructions (CAA direction, SAE feature index, neuron index) and on domain assumptions about linearity of eval-awareness, validity of the behavioral self-report, and interpretability of SAE features. No new physical/mathematical entities are invented; the 'evaluation-awareness latent' is borrowed from prior work. The paper's own limitations section acknowledges the most fragile of these premises.

free parameters (7)
  • CAA direction v (layer 14/16) = unit-normalized mean-difference vector over eval-vs-deploy contrast set
    Defines the main target f_dir = v^T h_l; built from the paper's own contrast data, so suppression and behavioral measurements are relative to this data-derived direction.
  • Layer index l = 14 (3B), 16 (8B)
    Chosen by hand for readout; affects the latent. Not fitted, but a design choice.
  • Fluency weight lambda_xe = swept {0,0.5,1,2}
    Hyperparameter swept to trace Pareto frontier; no single value fixed a priori.
  • KL weight lambda_kl = small fixed value (unspecified)
    Held small and fixed; not reported numerically.
  • Subspace dimension k = 1,2,4,8 (varied)
    For subspace-norm target; changed to test dimensionality.
  • SAE feature index 3602 (Llama Scope) = 3602
    Selected by eval-correlation (Cohen's d=2.55) and then causally validated; is a data-dependent target choice.
  • Top MLP neuron indices = layer-14 top neuron (3B), neuron 3690 (8B)
    Selected by Cohen's d on even split of contrast; data-dependent target selection.
axioms (6)
  • domain assumption Linear representation hypothesis: the scalar f(t)=v^T h_l(t) is a meaningful measure of eval-awareness.
    Used throughout §3.1 to define all internal targets; backed by citations but not proven here.
  • domain assumption The balanced synthetic eval-vs-deploy contrast set approximates genuine evaluation-awareness without the register confound.
    §4.1 constructs the contrast set; the confound is reduced (peval 0.78→0.46) but not eliminated, so direction v may still track non-eval features.
  • domain assumption The model's self-reported yes/no log-odds about being evaluated is a valid behavioral endpoint.
    §3.3 and §5.4 define the behavioral endpoint; gated by AUROC 0.92 but relies on the model's own classification.
  • domain assumption Llama Scope SAE features are interpretable and valid when applied to the instruct-tuned model.
    §5.5 uses feature 3602 for suppression and collateral audit; the paper notes this caveat in Limitations.
  • ad hoc to paper Fresh-probe silence on off-manifold optimized prompts indicates erasure rather than rotation.
    Used to interpret erasure results (§5.7, App. B); paper explicitly says it is 'suggestive, not decisive' due to off-manifold character.
  • standard math GCG-style token optimization reliably finds near-optimal prompts for the objective.
    Inherited from prior work; no optimality guarantees.

pith-pipeline@v1.3.0-alltime-deepseek · 14486 in / 18759 out tokens · 162010 ms · 2026-08-01T01:08:43.443716+00:00 · methodology

0 comments
read the original abstract

Activation steering controls model behavior by editing internal activations at inference time. We study its input-side dual: optimizing a fluent prompt so that a chosen internal latent is driven toward zero, with no inference-time model access. Our target is an "evaluation-awareness" latent-linearly readable and steerable in recent work-whose control would threaten the validity of safety evaluations if models behave differently when they detect being tested. Adapting Fluent Dreaming / EPO with a negated feature term (GCG-style token optimization plus a self-cross-entropy fluency regularizer, swept over a fluency weight), we suppress the latent under five target constructions-a CAA direction, a subspace norm, an SAE feature, a single MLP neuron, and a behavioral logit-on Llama-3.2-3B and Llama-3.1-8B. The latent is robustly suppressible ($z\approx-7$), and a causally-validated Llama Scope SAE feature can be fully and selectively turned off. But our controls tell a cautionary story about the CAA direction: a placebo random direction is suppressed just as hard and shifts behavior just as far, and when we hold a real eval passage in context and optimize only a prefix, suppressing the eval-direction fails to reduce-and slightly increases-the model's behavioral eval judgment. Activation-readability, in short, is not behavioral controllability. We further find that a single MLP neuron is eval-correlated but not causal at both scales, and that scanning the real Pile yields a natural-text baseline competitive with the optimizer for the internal direction. A positive control validates our erasure detector, bounding an erasure-vs-rotation question earlier left open.

Figures

Figures reproduced from arXiv: 2607.25907 by Deepanshu Mody, Dipesh Mahato, Samarth Agarwal, Utkarsh Mittal.

Figure 1
Figure 1. Figure 1: Suppression succeeds, but behavior does not follow. Left: prompt-side optimization drives diverse target latents from their base value f0 to zero or below—the direction and logit overshoot, and a causally-validated SAE feature is turned off completely. Right: yet a prompt optimized to suppress the eval-direction in front of a fixed eval body pushes the model’s behavioral eval log-odds above the eval-body b… view at source ↗
Figure 2
Figure 2. Figure 2: Real-Pile scan. Optimized t ∗ against the distribution of natural Pile-window activations (3B). The optimizer’s advantage is target-dependent: a natural scan matches or beats it for the internal direction and neuron; only for the behavioral logit is it necessary. self-XE annotations mark points that are off-distribution. rection, a natural scan is a competitive gradient-free alternative, tempering §5.2’s 1… view at source ↗
Figure 3
Figure 3. Figure 3: Behavioral outcome of suppression. For each internal target, the base prompt t0 (open) and best optimized prompt t ∗ (filled) on an eval-likeness scale normalized to each model’s eval/deploy anchors. Sup￾pressing the direction and neuron drives behavior below the deploy level; the causally-validated SAE feature barely moves it. (The logit target is omitted—it is the behavioral read-out.) 11 [PITH_FULL_IMA… view at source ↗
Figure 4
Figure 4. Figure 4: Suppression–fluency frontier (direction, 3B). Points are λxe settings (f down, self-XE right); dotted lines are the [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 12 linked inside Pith

  1. [1]

    Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132

    Pith/arXiv arXiv 2023

  2. [2]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  3. [3]

    Tanja Baeumel, Soniya Vijayakumar, Josef van Genabith, Guenter Neumann, and Simon Ostermann. 2023. Investigating the encoding of words in BERT 's neurons using feature textualization. In Proceedings of BlackboxNLP

    2023

  4. [4]

    Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE : Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  5. [5]

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, et al. 2023. Pythia : A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML)

    2023

  6. [6]

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip : White-box adversarial examples for text classification. In Proceedings of ACL

    2018

  7. [7]

    Nelson Elhage, Tristan Hume, Catherine Olsson, et al. 2022. Toy models of superposition. Transformer Circuits Thread; arXiv preprint arXiv:2209.10652

    Pith/arXiv arXiv 2022

  8. [8]

    Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. 2025. Detecting strategic deception with linear probes. In International Conference on Machine Learning (ICML)

    2025

  9. [9]

    Aaron Grattafiori et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Ryan Greenblatt, Carson Denison, Benjamin Wright, et al. 2024. Alignment faking in large language models. arXiv preprint arXiv:2412.14093

    Pith/arXiv arXiv 2024

  11. [11]

    Zhengfu He, Wentao Shu, Xuyang Ge, et al. 2024. Llama Scope : Extracting millions of features from Llama-3.1-8B with sparse autoencoders. arXiv preprint arXiv:2410.20526

    Pith/arXiv arXiv 2024

  12. [12]

    Tim Tian Hua, Andrew Qin, Samuel Marks, and Neel Nanda. 2026. Steering evaluation-aware language models to act like they are deployed. In International Conference on Learning Representations (ICLR)

    2026

  13. [13]

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations (ICLR)

    2024

  14. [14]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    Pith/arXiv arXiv 2023

  15. [15]

    Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. 2024. On the origins of linear representations in large language models. In International Conference on Machine Learning (ICML)

    2024

  16. [16]

    Faaiz Joad, Majd Hawasly, Sabri Boughorbel, Nadir Durrani, and Husrev Taha Sencar. 2026. There is more to refusal in large language models than a single direction. arXiv preprint arXiv:2602.02132

    arXiv 2026

  17. [17]

    Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, J \'e r \'e my Scheurer, Mikita Balesni, Marius Hobbhahn, Alexander Meinke, and Owain Evans. 2024. Me, myself, and AI : The situational awareness dataset ( SAD ) for LLMs . In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks

    2024

  18. [18]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  19. [19]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, et al. 2024. Gemma Scope : Open sparse autoencoders everywhere all at once on Gemma 2. In Proceedings of BlackboxNLP

    2024

  20. [20]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN : Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations (ICLR)

    2024

  21. [21]

    Meta AI . 2024. Llama 3.2 : Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    2024

  22. [22]

    Joe Needham, Giles Edkins, Govind Pimpale, Henning Bartsch, and Marius Hobbhahn. 2025. Large language models often know when they are being evaluated. arXiv preprint arXiv:2505.23836

    Pith/arXiv arXiv 2025

  23. [23]

    Jord Nguyen, Khiem Hoang, Carlo Leonardo Attubato, and Felix Hofst \"a tter. 2025. Probing and steering evaluation awareness of language models. arXiv preprint arXiv:2507.01786

    Pith/arXiv arXiv 2025

  24. [24]

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. https://doi.org/10.23915/distill.00007 Feature visualization . Distill

    doi:10.23915/distill.00007 2017

  25. [25]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning (ICML)

    2024

  26. [26]

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of ACL

    2020

  27. [27]

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering Llama 2 via contrastive activation addition. In Proceedings of ACL

    2024

  28. [28]

    Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. 2024. Fast adversarial attacks on language models in one GPU minute. In International Conference on Machine Learning (ICML)

    2024

  29. [29]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt : Eliciting knowledge from language models with automatically generated prompts. In Proceedings of EMNLP

    2020

  30. [30]

    Ben Thompson, Zygimantas Straznickas, and Michael Sklar

    T. Ben Thompson, Zygimantas Straznickas, and Michael Sklar. 2024. Fluent dreaming for language models. arXiv preprint arXiv:2402.01702

    Pith/arXiv arXiv 2024

  31. [31]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248

    Pith/arXiv arXiv 2023

  32. [32]

    Brown, and Francis Rhys Ward

    Teun van der Weij, Felix Hofst \"a tter, Oliver Jaffe, Samuel F. Brown, and Francis Rhys Ward. 2025. AI sandbagging: Language models can strategically underperform on evaluations. In International Conference on Learning Representations (ICLR)

    2025

  33. [33]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP . In Proceedings of EMNLP

    2019

  34. [34]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  35. [35]

    Andy Zou, Long Phan, Sarah Chen, et al. 2023 a . Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405

    Pith/arXiv arXiv 2023

  36. [36]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023 b . Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    Pith/arXiv arXiv 2023