Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improved Representation Steering for Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reference-free preference steering beats LM training on Gemma models.

desk verdict RePS is a genuinely new steering objective with credible steering gains, but the suppression claims rest on a train/inference mismatch that needs fixing. read the letter →

arxiv 2505.20809 v1 pith:ZXET5ILL submitted 2025-05-27 cs.CL

Zhengxuan Wu , Qinan Yu , Aryaman Arora , Christopher D. Manning , Christopher Potts This is my paper
classification cs.CL
keywords representationsteeringpreferenceoptimizationvectorsconceptsuppressionjailbreakingrobustnesslow-rankadaptationreference-freeobjectivelinearhypothesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Representation steering — editing a model's internal activations at inference time to add or remove a behavior — has lagged behind prompting. This paper claims the shortfall is a training-objective problem, not a method problem: steering interventions trained with a language-modeling loss miss the preference signal that instruct-tuned models are built on. The authors propose RePS, a reference-free, bidirectional preference objective that rewards steered behavior under positive intervention and punishes it under negative intervention, and train three low-rank interventions (steering vectors, LoRA, LoReFT) with it. On the AxBench concept tasks across Gemma models from 2B to 27B, RePS consistently beats the language-modeling objective and the earlier BiPO baseline, substantially narrowing the gap with prompting. For suppressing a concept, RePS-trained rank-1 steering vectors match the LM objective on Gemma-2 and beat it on larger Gemma-3 models, and they resist prompt-based jailbreaks that defeat prompt defenses.

What carries the argument

The load-bearing object is the RePS loss, summed over two log-sigmoid terms: one for positive steering (increase the likelihood of the steered response under the learned intervention) and one for negative steering (increase the likelihood of the original response under a null projection that removes the steering direction). The null projection, $\Phi_{\mathrm{Null}}(h_l) = h_l - \frac{\mathrm{ReLU}(h_l \cdot w_1)}{\lVert w_1\rVert^2} w_1$, is the mechanism that turns a single learned direction into a suppression operator without retraining. A training-time factor-sampling trick, in which steering factors are drawn from a fixed set rather than fixed to one value, stabilizes optimization and is what makes preference training of interventions practical across model scales.

What would settle it

Run the same 500 AxBench concepts through RePS training while measuring how linearly separable the steered and unsteered activations are with a linear probe at the intervened layer; if a large fraction of concepts show low linear-probe separability yet still steer well, the linear-representation premise is not what carries the method. A direct alternative is to train RePS on a task set constructed to be nonlinear in representation space (e.g., disjunctive multi-concept targets) and compare rank-1 steering vectors with rank-4 LoRA interventions on steering and suppression scores.

Watch

Extended reading notes

Core claim

The central discovery is that a preference-optimization objective of a specific shape—asymmetric, reference-free, and bidirectional—makes rank-1 representation interventions genuinely competitive with prompting. Positive steering uses a SimPO-style length-normalized likelihood margin in which the steered response is up-weighted by a factor that grows when the unsteered model finds it improbable; negative steering is trained against a null intervention that projects out the learned steering direction. The same learned vector therefore transfers from eliciting a concept to suppressing it, and suppression survives both explicit instruction-following attacks and many-shot jailbreaking, where text-based defenses collapse. The result is evidence that steering performance is bounded less by the capacity of rank-1 interventions than by the training objective applied to them.

Load-bearing premise

Each steering concept can be represented by a single direction in the residual stream of the model, an assumption that the rank-1 steering vector and the null-projection suppression both rely on and that the paper does not separately validate across its 500 concepts.

Editorial extensions

If this is right

  • RePS-trained rank-1 steering vectors beat language-modeling-trained vectors on steering across all three intervention types and all four Gemma sizes tested, with the widest margins on the smaller Gemma-2 models.
  • For concept suppression, RePS-trained steering vectors outperform LM-objective-trained ones on Gemma-3-12B and 27B and match them on Gemma-2-2B and 9B, indicating the advantage grows with model scale.
  • RePS suppression is substantially more resistant than prompting to instruction-following and many-shot jailbreaking attacks; on larger models the prompt-based defenses degrade while the intervention-based defense stays roughly constant.
  • The gap between representation steering and prompting in the AxBench steering scores is narrowed but not closed, with RePS surpassing prompt scores on some Gemma-2 concept subsets and remaining below prompt scores on most.
  • Factor sampling during training lowers the variance of steering scores across hyperparameter runs, which the paper credits with making the preference objective practical to tune.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because RePS closes much of the gap with prompting using only a rank-1 vector, the paper's logic extends to a prediction: as base models get larger and better at following instructions, preference-trained interventions should continue to improve, possibly overtaking prompting on steering where prompt defenses leak system instructions.
  • The null-projection trick suggests a principled way to build 'erase' operators for any learned direction; one could extend it to higher-rank interventions such as LoRA by projecting onto the null spaces of multiple learned directions, which the paper did not test.
  • Using RePS as a refinement step on top of unsupervised directions (e.g., difference-in-means vectors or SAE features) might inherit both the interpretability of those features and the preference-aligned steering strength; this is a direct testable extension.
  • The resistance of intervention-based suppression to prompt attacks implies a defense-in-depth design: an invisible, auditable intervention could guard system prompts against user-induced leakage, since prompt-append defenses in the paper are shown to leak the system prompt under many-shot attacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RePS, a reference-free bidirectional preference-optimization objective for training low-rank representation-steering interventions (rank-1 steering vectors, LoRA, and LoReFT). RePS is derived from SimPO and BiPO and jointly trains positive steering (via the steering intervention) and negative/suppression behavior (via a null-projection intervention in Eq. 7). The authors evaluate RePS against the standard language-modeling objective and the BiPO baseline across Gemma-2/3 models from 2B to 27B on the AxBench concept-steering and suppression tasks, including prompt-based jailbreaking scenarios. They report that RePS outperforms existing steering methods trained with a language-modeling objective, narrows the gap with prompting, and is more robust than prompting for suppression under attack.

Significance. If the reported results hold, RePS would be a useful and practical contribution to representation steering: it is reference-free, parameter-efficient, and appears to improve suppression robustness for larger Gemma-3 models while maintaining interpretability. The paper provides a substantial evaluation across model scales and intervention types, and it ships a public code repository. However, the main comparative claims currently rest on single-seed numbers with no variance estimates, and the suppression evaluation does not match the intervention actually trained in the negative-steering objective; these issues are load-bearing for the paper's central claims and need to be addressed before the results can be regarded as established.

major comments (3)
  1. [§5.2, Table 1] The claim that "RePS consistently outperforms Lang." is contradicted by the reported LoReFT results on Gemma-3-12B/D100: Lang. scores 0.714 while RePS scores 0.651. The same table shows very small RePS advantages for SV (1.230 vs 1.219) and LoRA (0.950 vs 0.943) on this model. Given that the table reports no variance and the experiments use a single seed, I cannot tell whether these differences are meaningful, and the word "consistently" is not supported. Please add confidence intervals or replicate at least the critical comparisons over several seeds, and revise the overbroad claim.
  2. [§3.3 vs §5.3, Eqs. (6)--(7) and Eq. (9)] The suppression evaluation does not test the intervention that the negative-steering loss actually trains. Eq. (6) defines the negative loss using ΦNull(hl) = hl − ReLU(hl·w1)/‖w1‖² w1, which removes the projection of the activation onto w1. Section 5.3, however, states that suppression is applied by using negative coefficients α in ΦSteer(hl, α) = hl + α·w1 + b1, a translation along w1 with a bias term. These are different operations, and the paper never evaluates ΦNull nor demonstrates that the two interventions are behaviorally equivalent. Since the suppression results in Tables 2 and 3 and Figure 1 are used to conclude that RePS is a robust alternative to prompting, this train/inference mismatch is load-bearing. The authors should either evaluate suppression with the actual ΦNull intervention used in training, or provide empirical evidence (e.g., on a development set) that negative-α ΦSteer and ΦNull produce equivalent suppression behavior for the trained vectors.
  3. [§5.3, Tables 2--3, Figure 1] The suppression comparisons are reported without any error bars, confidence intervals, or multiple seeds. For example, the 12B and 27B differences between RePS and Lang. in Table 2 (1.035 vs 0.912 and 1.031 vs 0.940) are plausible but could easily be within run-to-run variability, especially given that Appendix D says all experiments use a single seed and that the model-generation and LM-judge pipeline is stochastic. The robustness claims in Section 5.4 would be stronger if the authors provided variance estimates or replication for at least the key attack conditions.
minor comments (5)
  1. [§5.3] The sentence "We how take the RePS-trained interventions..." contains a typo; it should likely read "We now take...".
  2. [§3.3, Eq. (7)] The phrase "from from hl" in the text defining ΦNull is a typo and should be corrected.
  3. [§1 and throughout] The benchmark name "AxBench" is typeset inconsistently (e.g., AXBENCH, A XBENCH, AxBench); please standardize.
  4. [Appendix D] The stability analysis in Figure 12 is helpful, but it reports only the maximum steering-score difference across three runs for two settings; reporting the full distribution or standard deviations of the overall score would give a better sense of evaluation noise.
  5. [§5.2] The table note "Φr=1 SV is rank-1 and has the fewest trainable parameters" is useful, but the main text could also state explicitly which layers were used for each model and whether the same supervised layer search was applied to all objectives with identical budgets.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction in the RePS derivation; suppression train/inference mismatch is a validity issue, not circularity.

full rationale

The RePS objective (Eqs. 5-8) is constructed from the external SimPO reward with reference-model log-probabilities (p(y|x), p(yc|x)) used only as fixed inputs for the beta weights; these are not fitted parameters, and the reported steering scores are measured on held-out instructions after selecting factors on a separate set, so no prediction reduces to the training fit by construction. Self-citations to AXBENCH (Wu et al., 2025) and ReFT (Wu et al., 2024) supply the benchmark, baseline numbers, and intervention parameterizations, but the central comparison is against published, independently reproducible baselines and the paper's own Lang. runs, so the self-citations are not load-bearing in a circular sense. The suppression experiments do exhibit a training/evaluation mismatch (negative steering is trained with PhiNull in Eq. 7, then evaluated with negative alpha on PhiSteer in Eq. 9), and the paper does not validate equivalence between these interventions; however, this is a correctness/validity limitation, not a circularity, because the suppression numbers are empirical measurements rather than consequences of the training loss by definition. The appendix's honest negative result (H.4: Lang.-trained vectors are better concept detectors) further supports that the analysis is not outcome-contrived. Overall, the derivation chain is self-contained with respect to its inputs, so no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the linear representation hypothesis and on the quality of synthetic preference pairs and LM judges. These are domain assumptions, not standard math. No new physical entities are introduced.

free parameters (2)
  • Steering factor ranges = e.g., {2,...,20} for Gemma-2; {20,...,200} for Gemma-3; {0.2,...,2.0} for LoRA/ReFT
    Chosen by hand to match layer norms; used at training and inference; the paper states these factors are fixed and affect performance (Appendix D).
  • Intervening layer = varies per model (e.g., 10,20 for 2B; 22 for 12B)
    Selected via grid search; layer selection drastically affects steerability (Figure 13).
assumptions (3)
  • domain assumption Linear representation hypothesis: concepts are encoded as directions in residual stream activations.
    Used to justify rank-1 SV and the null projection in Eq. (7) and Section 4.
  • domain assumption Preference pairs (steered vs. original) generated by gpt-4o-mini accurately represent the steering concept.
    Dataset construction in Section 5.1; if these are noisy, training signal is corrupted.
  • domain assumption LM judge scores (concept, instruction, fluency) are valid and unbiased.
    All evaluations rely on remote LM judges; no human validation reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improved Representation Steering for Language Models." pith.science (2026). https://pith.science/paper/ZXET5ILL

@misc{pith2026250520809,
  author       = {Pith},
  title        = {Pith review of: Improved Representation Steering for Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXET5ILL}},
  note         = {Machine review of arXiv:2505.20809}
}
read the original abstract

Steering methods for language models (LMs) seek to provide fine-grained and interpretable control over model generations by variously changing model inputs, weights, or representations to adjust behavior. Recent work has shown that adjusting weights or representations is often less effective than steering by prompting, for instance when wanting to introduce or suppress a particular concept. We demonstrate how to improve representation steering via our new Reference-free Preference Steering (RePS), a bidirectional preference-optimization objective that jointly does concept steering and suppression. We train three parameterizations of RePS and evaluate them on AxBench, a large-scale model steering benchmark. On Gemma models with sizes ranging from 2B to 27B, RePS outperforms all existing steering methods trained with a language modeling objective and substantially narrows the gap with prompting -- while promoting interpretability and minimizing parameter count. In suppression, RePS matches the language-modeling objective on Gemma-2 and outperforms it on the larger Gemma-3 variants while remaining resilient to prompt-based jailbreaking attacks that defeat prompting. Overall, our results suggest that RePS provides an interpretable and robust alternative to prompting for both steering and suppression.

Figures

Figures reproduced from arXiv: 2505.20809 by the authors.

Figure 1
Figure 1. Suppression scores for different defense methods under many-shot jailbreaking attacks with Gemma-3-12B LM. Our suppression score is defined as the harmonic mean of three individual scores measuring adherence to the system prompt (see appendix R), fluency, and instruction￾following. We compare our intervention-based defense, RePS-trained SV, with four prompt-based defenses, including variants of prepending or appendi… view at source ↗
Figure 2
Figure 2. Mean score breakdown for all methods on our unseen testing instruction set after selecting [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Distribution of optimal steering factors for each intervention-based methods (LoRA, ReFT [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Steering factor vs. scores for Gemma-2 models. 0 100 200 300 SV w/ Lang. SV w/ RePS Gemma-2b L10 0 100 200 300 Gemma-2b L20 0 100 200 300 Gemma-9b L20 50 40 30 20 10 0 0 100 200 300 50 40 30 20 10 0 Gemma-9b L31 Optimal steering factor Count [PITH_FULL_IMAGE:figures/f…
Figure 5
Figure 5. Figure 5: Distribution of optimal suppression factors for each intervention-based methods (LoRA, [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Suppression factor vs. scores for Gemma-2 models. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Mean score breakdown for all methods on our unseen testing instruction set after selecting [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Distribution of optimal steering factors for each intervention-based methods (LoRA, ReFT [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Steering factor vs. scores for Gemma-3 models. 0 25 50 SV w/ Lang. SV w/ RePS Gemma-12b L22 100 50 0 25 50 100 50 Gemma-27b L28 Optimal steering factor Count [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Suppression factor vs. scores for Gemma-3 models. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Suppression Mean score breakdown for all methods on our unseen testing instruction set [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Steering score distribution for three distinct runs with different random seeds under the [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Steering score vs. intervening layers of steering vectors on [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Generation lengths for different LMs from two [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Averaged layer-norm of two LMs from the Gemma-2 family. 0.04 0.08 6 12 w/ sampled factors 0.04 0.08 w/o sampled factors Learning rate Batch size Std Dev 0.1 0.2 0.3 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Variance of steering scores across hyperparameter-tuning runs for both with and without [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Distribution of cosine similarity scores between SV weights learned by RePS and language [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Logits lens rankings of output tokens with methods trained on [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Mean ROC curves over all concepts with steering vectors trained on [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Steering factor vs. scores for concepts with different genres with the training data from [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]
Figure 21
Figure 21. Figure 21: Steering factor vs. scores for concepts with different genres with new training data created [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: Rule-based suppression score break down on concept 1–10 [PITH_FULL_IMAGE:figures/full_fig_p036_22.png]
Figure 23
Figure 23. Figure 23: Rule-based suppression score break down on concept 10–20 [PITH_FULL_IMAGE:figures/full_fig_p037_23.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probabilistic Concept-Aware Steering for Trustworthy LLM Inference

    cs.AI 2026-05 reject novelty 4.0 of 10

    PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chos...

Reference graph

Works this paper leans on

58 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud

    Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Ja...

    work page 2024

  2. [2]

    A Practical Method for Generating String Counterfactuals

    Matan Avitan, Ryan Cotterell, Yoav Goldberg, and Shauli Ravfogel. What changed? C onverting representational interventions to natural language. In arXiv:2402.11355, 2024. URL https://arxiv.org/abs/2402.11355

    work page Pith review arXiv 2024

  3. [3]

    Comparing bad apples to good oranges: Aligning large language models via joint preference optimization

    Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. Comparing bad apples to good oranges: Aligning large language models via joint preference optimization. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. URL https://openreview.net/forum?id=AzMnkF0jRT

    work page 2024

  4. [4]

    LEACE : P erfect linear concept erasure in closed form

    Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE : P erfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2306.03819

    arXiv 2023

  5. [5]

    B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Association for Computational Linguistics (ACL), 2022. URL https://arxiv.org/abs/2106.10199

    arXiv 2022

  6. [6]

    Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization

    Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2406.00045

    arXiv 2024

  7. [7]

    Efficient prompting methods for large language models: A survey

    Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey. In Transactions on Machine Learning Research (TMLR), 2024. URL https://arxiv.org/abs/2404.01077

    arXiv 2024

  8. [8]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2310.08419

    arXiv 2023

Show all 58 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. In arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv

  2. [10]

    Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-ope...

    2023

  3. [11]

    R\' o bert Csord\' a s, Kazuki Irie, J\" u rgen Schmidhuber, Christopher Potts, and Christopher D. Manning. MoEUT : Mixture-of-experts Universal Transformers . In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Inf...

    2024

  4. [12]

    L earn-to- S hare: A hardware-friendly transfer learning framework exploiting computation and parameter sharing

    Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. L earn-to- S hare: A hardware-friendly transfer learning framework exploiting computation and parameter sharing. In International Conference on Machine Learning (ICML), 2021. URL http://proceedings.mlr.press...

    2021

  5. [13]

    Gemma: Open models based on G emini research and technology

    Gemma Team , Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on G emini research and technology. 2024. URL https://arxiv.org/abs/2403.08295

    2024 arXiv

  6. [14]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. In Transactions on Machine Learning Research (TMLR), 2024. URL https://arxiv.org/abs/2403.14608

    2024 arXiv

  7. [15]

    Understanding and minimising outlier features in Transformer training

    Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in Transformer training. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Pro...

    2024

  8. [16]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2110.04366

    2022 arXiv

  9. [17]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP . In International Conference on Machine Learning (ICML), 2019. URL https://arxiv.org/a...

    2019 arXiv

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2106.09685

    2022 arXiv

  11. [19]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSP y: Compiling declarative language model calls into self-impr...

    2024 arXiv

  12. [20]

    LoRA-Switch : Boosting the efficiency of dynamic llm adapters via system-algorithm co-design

    Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, and Yunxin Liu. LoRA-Switch : Boosting the efficiency of dynamic llm adapters via system-algorithm co-design. In arXiv:2405.17741, 2024. URL https://arxiv.org/abs/2405.17741

    2024 arXiv

  13. [21]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Empirical Methods in Natural Language Processing (EMNLP), 2021. URL https://arxiv.org/abs/2104.08691

    2021 arXiv

  14. [22]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems (NeurIPS), 2024 a . URL https://arxiv.org/abs/2306.03341

    2024 arXiv

  15. [23]

    Predicting vs

    Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling. In arXiv:2407.02446, 2024 b . URL https://arxiv.org/abs/2407.02446

    2024 arXiv

  16. [24]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021. URL https://arxiv.org/abs/2101.00190

    2021 arXiv

  17. [25]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval : An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  18. [26]

    In-context vectors: M aking in context learning more effective and controllable through latent space steering

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: M aking in context learning more effective and controllable through latent space steering. In International Conference on Machine Learning (ICML), 2024 a . URL https://arxiv.org/abs/2311.06668

    2024 arXiv

  19. [27]

    DoRA : Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA : Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024 b . URL https://arxiv.org/abs/2402.09353

    2024 arXiv

  20. [28]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. URL https://arxiv.org/abs/2310.06824

    2024 arXiv

  21. [29]

    SimPO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO : Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2405.14734

    2024 arXiv

  22. [30]

    Interpreting GPT : The logit lens

    Nostalgebraist . Interpreting GPT : The logit lens. In LessWrong blog post, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  23. [31]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https:/...

    2024 arXiv

  24. [32]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2311.03658

    2024 arXiv

  25. [33]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://a...

    2023 arXiv

  26. [34]

    Cotterell

    Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D. Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning (ICML), 2022. URL https://arxiv.org/abs/2201.12091

    2022 arXiv

  27. [35]

    Steering Llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering Llama 2 via contrastive activation addition. In Association for Computational Linguistics (ACL), 2024. URL https://arxiv.org/abs/2312.06681

    2024 arXiv

  28. [36]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arxiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  29. [37]

    Slora: Scalable serving of thousands of lora adapters

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. Slora: Scalable serving of thousands of lora adapters. In Proceedings of Machine Learning and Systems (MLSys), 2024

    2024

  30. [38]

    MiMiC : Minimally modified counterfactuals in the representation space

    Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. MiMiC : Minimally modified counterfactuals in the representation space. In arXiv:2402.09631, 2024. URL https://arxiv.org/abs/2402.09631

    2024 arXiv

  31. [39]

    Extracting latent steering vectors from pretrained language models

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of Association for Computational Linguistics (ACL), 2022. URL https://arxiv.org/abs/2205.05124

    2022 arXiv

  32. [40]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 technical report. 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  33. [41]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  34. [42]

    Activation addition: Steering language models without optimization

    Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. In arXiv:2308.10248, 2023 a . URL https://arxiv.org/abs/2308.10248

    2023 arXiv

  35. [43]

    Steering gemini using BiPO vectors

    Alex Turner, Mark Kurzeja, Dave Orr, and David Elson. Steering gemini using BiPO vectors. In The Pond, 2025. URL https://turntrout.com/gemini-steering

    2025

  36. [44]

    Understanding and controlling a maze-solving policy network

    Alexander Matt Turner, Peli Grietzer, Ulisse Mini, Monte M, and David Udell. Understanding and controlling a maze-solving policy network. In Alignment Forum, 2023 b . URL https://shorturl.at/XGtmh

    2023

  37. [45]

    DyLoRA : Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA : Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In European Chapter of the Association for Computational Linguistics (EACL), 2023. URL https://arxiv.o...

    2023 arXiv

  38. [46]

    Extending activation steering to broad skills and multiple behaviours

    Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. In arXiv:2403.05767, 2024. URL https://arxiv.org/abs/2403.05767

    2024 arXiv

  39. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL http://papers.nips.cc/paper/7181-attention-is-all-yo...

    2017

  40. [48]

    repeng, 2024

    Theia Vogel. repeng, 2024. URL https://github.com/vgel/repeng/

    2024

  41. [49]

    AdaMix : Mixture-of-adaptations for parameter-efficient model tuning

    Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix : Mixture-of-adaptations for parameter-efficient model tuning. In Empirical Methods in Natural Language Processing (EMNLP), 2022. URL http://arxiv.org/abs...

    2022 arXiv

  42. [50]

    Orthogonal negation in vector spaces for modelling word-meanings and document retrieval

    Dominic Widdows. Orthogonal negation in vector spaces for modelling word-meanings and document retrieval. In Association for Computational Linguistics (ACL), 2003. URL https://aclanthology.org/P03-1018/

    2003

  43. [51]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. ReFT : Representation finetuning for language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2404.03592

    2024 arXiv

  44. [52]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. AxBench : Steering LLMs ? Even simple baselines outperform sparse autoencoders. In International Conference on Machine Learning (ICML), 2025. URL h...

    2025 arXiv

  45. [53]

    Composing parameter-efficient modules with arithmetic operation

    Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operation. In Advances in Neural Information Processing Systems (NeurIPS), 2024 a . URL https://arxiv.org/abs/2306.14870

    2024 arXiv

  46. [54]

    AdaLoRA : Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA : Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), 2023. URL https://...

    2023 arXiv

  47. [55]

    AutoLoRA : Automatically tuning matrix ranks in low-rank adaptation based on meta learning

    Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. AutoLoRA : Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In North American Chapter of the Association for Computational Linguistics (NAACL), 2024 b . URL https://arxiv.org/abs/...

    2024 arXiv

  48. [56]

    Lora land: 310 fine-tuned llms that rival gpt-4, a technical report

    Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. In arXiv:2405.00732, 2024. URL https://arxiv.org/abs/2405.00732

    2024 arXiv

  49. [57]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. In arxiv:2311.07911, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  50. [58]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.