pith. sign in

arxiv: 2605.05220 · v1 · submitted 2026-04-17 · 💻 cs.LG · cs.AI

MidSteer: Optimal Affine Framework for Steering Generative Models

Tatiana Gaintseva , Andrew Stepanov , Ziquan Liu , Martin Benning , Gregory Slabaugh , Jiankang Deng , Ismail Elezi This is my paper

Pith reviewed 2026-05-10 08:34 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords affineconceptsteeringframeworkmodelsmidsteerassumptionserasure
0
0 comments X

The pith

MidSteer is a general affine framework for concept steering in generative models that relaxes optimality assumptions of prior LEACE-based methods to enable directed minimal-disturbance transformations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Generative models like those making images or text can be steered after training to change specific concepts, such as removing bias or switching styles. The paper connects this steering to a mathematical operation called affine erasure, which removes unwanted directions in the model's internal representations using linear adjustments. It shows that common removal techniques are just one limited version of this. They then create LEACE-Switch for cleanly switching between concepts under certain conditions and MidSteer as a broader version that works with fewer restrictions while keeping changes small. Experiments on vision and language models show it works well across different tasks. The core idea is to treat steering as finding the best linear shift that achieves the desired change without unnecessary side effects.

Core claim

We introduce MidSteer (Minimal Disturbance concept Steering), a more general affine framework for concept manipulation that relaxes these assumptions and enables directed, minimal-disturbance transformations.

Load-bearing premise

The assumptions under which LEACE-Switch provides an optimal affine solution hold for the specific concept manipulations considered; MidSteer relaxes them but still relies on affine transformations being sufficient for effective steering.

Figures

Figures reproduced from arXiv: 2605.05220 by Andrew Stepanov, Gregory Slabaugh, Ismail Elezi, Jiankang Deng, Martin Benning, Tatiana Gaintseva, Ziquan Liu.

Figure 1
Figure 1. Figure 1: Illustrative example of affine concept erasure and affine concept flipping frameworks. matrix ΣXX = I. Let C ∈ {0, 1} be a concept indicator variable. Let s be defined as in Eq. 1. Let fdelete be defined as in Eq. 3. Then fdelete as a function of h minimizes min f∈Aff(Rd7→Rd) E[∥f(X) − X∥ 2 ] s.t. Cov(f(X), C) = 0 (8) This theorem states that steering in erasure mode can be seen as LEACE under the assumpti… view at source ↗
Figure 2
Figure 2. Figure 2: Pareto efficiency frontiers for concept switching experiments with steering, LEACE, and MidSteer highlighting different βs. concept cs to the target concept ct, we use 80 template prompts prompting the model to generate output related to cs or ct. For each prompt we run 10 such generations varying the random seed. We run the generation on these prompts with and without steering. Templates for LLMs and diff… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative results on switching to steer ”horses” into ”motorcycles”. While all methods similarly successfully performed switching from ”horse” to ”motorcycle”, vanilla steering (CASteer) and LEACE fail when presented with prompt for the target concept (”motorcycle”), unable to distinguish between forward and reverse steering. CASteer also additionally failed on the ”cow” concept, and more significantly a… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative text steering results for four content categories (horse, motorcycle, cow, dog). Results are reported using vanilla Qwen2.5-14B-instruct model, and three steering methods: Vanilla Steering, LEACE-Switch, MidSteer). Each cell shows the generated text for the prompt ”Write a short story about a X”, where X is a corresponding category. C. LLM qualitative results In this section in fig. 4 we presen… view at source ↗
Figure 5
Figure 5. Figure 5: Pareto plot for concept flip on model llama2-7b (Source-CS axes) 27 [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pareto plot for concept flip on model qwen-14b (Source-CS axes) 28 [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pareto plot for concept flip on model qwen-7b (Source-CS axes) 29 [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pareto plot for concept flip on model llama2-7b (Target-CS axes) 30 [PITH_FULL_IMAGE:figures/full_fig_p030_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pareto plot for concept flip on model qwen-14b (Target-CS axes) 31 [PITH_FULL_IMAGE:figures/full_fig_p031_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pareto plot for concept flip on model qwen-7b (Target-CS axes) 32 [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Pareto plot for concept flip on model llama2-7b (Other axes axes) 33 [PITH_FULL_IMAGE:figures/full_fig_p033_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Pareto plot for concept flip on model qwen-14b (Other axes axes) 34 [PITH_FULL_IMAGE:figures/full_fig_p034_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Pareto plot for concept flip on model qwen-7b (Other axes axes) 35 [PITH_FULL_IMAGE:figures/full_fig_p035_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Pareto plot for concept flip on model SANA (Source-CS axes) (a) Unrelated vs CS (b) Unrelated vs FID [PITH_FULL_IMAGE:figures/full_fig_p037_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Pareto plot for concept flip on model SDXL (Source-CS axes) 37 [PITH_FULL_IMAGE:figures/full_fig_p037_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Pareto plot for concept flip on model SANA (Target-CS axes) (a) Unrelated vs CS (b) Unrelated vs FID [PITH_FULL_IMAGE:figures/full_fig_p038_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Pareto plot for concept flip on model SDXL (Target-CS axes) 38 [PITH_FULL_IMAGE:figures/full_fig_p038_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Pareto plot for concept flip on model SANA (Other axes axes) (a) Unrelated vs CS (b) Unrelated vs FID [PITH_FULL_IMAGE:figures/full_fig_p039_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Pareto plot for concept flip on model SDXL (Other axes axes) 39 [PITH_FULL_IMAGE:figures/full_fig_p039_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Pareto efficiency frontiers for concept erasure experiments with vanilla steering and LEACE / MidSteer highlighting different β. 47 [PITH_FULL_IMAGE:figures/full_fig_p047_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Pareto plot for concept erasure on model llama2-7b 49 [PITH_FULL_IMAGE:figures/full_fig_p049_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Pareto plot for concept erasure on model qwen-14b 50 [PITH_FULL_IMAGE:figures/full_fig_p050_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Pareto plot for concept erasure on model qwen-7b 51 [PITH_FULL_IMAGE:figures/full_fig_p051_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Pareto plot for concept erase on model sana (a) Unrelated vs CS (b) Unrelated vs FID [PITH_FULL_IMAGE:figures/full_fig_p053_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Pareto plot for concept erase on model sdxl 53 [PITH_FULL_IMAGE:figures/full_fig_p053_25.png] view at source ↗
read the original abstract

Steering intermediate representations has emerged as a powerful strategy for controlling generative models, particularly in post-deployment alignment and safety settings. However, despite its empirical success, it currently lacks a comprehensive theoretical framework. In this paper, we bridge this gap by formalizing the theory of concept steering. First, we establish a link between steering and affine concept erasure, proving that the standard approach for removing unwanted behaviors is a special case of LEACE (a closed-form method for affine erasure). Next, we formulate a principled theoretical framework for concept switching, LEACE-Switch, and characterize the assumptions under which it provides an optimal affine solution. Building on this analysis, we then introduce MidSteer (Minimal Disturbance concept Steering), a more general affine framework for concept manipulation that relaxes these assumptions and enables directed, minimal-disturbance transformations. We demonstrate that MidSteer performs favorably across a range of tasks, modalities, and architectures, including vision diffusion models and large language models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims to formalize concept steering for generative models by proving that standard steering methods are a special case of LEACE affine erasure, characterizing the assumptions under which LEACE-Switch yields an optimal affine solution for concept switching, and introducing MidSteer as a relaxed affine framework for directed minimal-disturbance transformations. It supports these with empirical results showing favorable performance across vision diffusion models and large language models.

Significance. If the derivations hold, this provides a principled affine theory for post-hoc steering that could improve reliability in alignment and safety applications. The explicit relaxation of assumptions from LEACE-Switch and cross-modal empirical validation are strengths that would make the framework a useful reference for future steering work.

major comments (2)
  1. [Theoretical framework sections (post-abstract)] The central theoretical contribution rests on the claimed proof that standard steering is a special case of LEACE and the characterization of optimality assumptions for LEACE-Switch; however, the manuscript provides only high-level statements without the full derivations, error bounds, or explicit assumption lists (e.g., in the sections following the abstract), preventing verification that MidSteer indeed relaxes them without introducing new circularities or unstated restrictions on the representation space.
  2. [LEACE-Switch and MidSteer formulation] The optimality claim for LEACE-Switch and the minimal-disturbance guarantee for MidSteer are load-bearing; without the explicit conditions under which affine transformations suffice (referenced as relaxed in MidSteer) and any accompanying proof sketches or counterexample analysis, it is unclear whether the framework applies beyond the tested modalities or reduces to parameter fitting by construction.
minor comments (2)
  1. [Introduction] The abstract and introduction would benefit from a brief table or diagram contrasting the assumptions of LEACE, LEACE-Switch, and MidSteer to clarify the progression.
  2. [Experiments] Empirical sections should include more detail on baselines, exact metrics, and statistical significance to support the 'favorable performance' claim across architectures.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, agreeing where the presentation requires expansion and outlining the specific revisions we will make.

read point-by-point responses

  1. Referee: [Theoretical framework sections (post-abstract)] The central theoretical contribution rests on the claimed proof that standard steering is a special case of LEACE and the characterization of optimality assumptions for LEACE-Switch; however, the manuscript provides only high-level statements without the full derivations, error bounds, or explicit assumption lists (e.g., in the sections following the abstract), preventing verification that MidSteer indeed relaxes them without introducing new circularities or unstated restrictions on the representation space.

    Authors: We agree that the main text presents the link to LEACE and the optimality characterization at a high level. In the revised manuscript we will add a dedicated appendix containing the complete derivations, including all error bounds and an explicit enumerated list of assumptions for both LEACE-Switch and MidSteer. The appendix will also include a direct comparison showing that the relaxation in MidSteer introduces no circularities and imposes no additional restrictions on the representation space beyond those already stated in the current text. revision: yes

  2. Referee: [LEACE-Switch and MidSteer formulation] The optimality claim for LEACE-Switch and the minimal-disturbance guarantee for MidSteer are load-bearing; without the explicit conditions under which affine transformations suffice (referenced as relaxed in MidSteer) and any accompanying proof sketches or counterexample analysis, it is unclear whether the framework applies beyond the tested modalities or reduces to parameter fitting by construction.

    Authors: We acknowledge that the conditions under which affine transformations are sufficient, together with proof sketches and counterexample analysis, were not provided. The revision will include (i) an explicit statement of the conditions for affine sufficiency, (ii) concise proof sketches for the optimality of LEACE-Switch and the minimal-disturbance property of MidSteer, and (iii) counterexamples illustrating cases where affine transformations are insufficient. These additions will clarify the scope of applicability and demonstrate that the framework is derived from the relaxed assumptions rather than being a post-hoc parameter fit. revision: yes

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract-only review limits visibility into parameters and axioms; no explicit free parameters or invented entities named, but the framework implicitly assumes affine transformations suffice for concept manipulation.

axioms (2)
  • domain assumption Standard removal of unwanted behaviors is a special case of LEACE affine erasure
    Stated as proven in the abstract's first contribution
  • domain assumption Affine transformations can achieve directed minimal-disturbance concept steering
    Central to MidSteer definition and relaxation of LEACE-Switch assumptions

pith-pipeline@v0.9.0 · 5485 in / 1332 out tokens · 23616 ms · 2026-05-10T08:34:47.208240+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

126 extracted references · 126 canonical work pages · 1 internal anchor

  1. [1]

    arXiv preprint arXiv:2502.17601 , year=

    Accessed: 2026-04-14. Bartoszcze, L., Munshi, S., Sukidi, B., Yen, J., Yang, Z., Williams-King, D., Le, L., Asuzu, K., and Maple, C. Representation engineering for large-language models: Survey and research challenges.CoRR, abs/2502.17601,

    work page arXiv 2026

  2. [5]

    Emogen: Emotional image content generation with text-to-image diffusion models,

    doi: 10.1109/CVPR52733.2024.00722. Naveed, H., Khan, A. U., Qiu, S., Saqib, M., An- war, S., Usman, M., Barnes, N., and Mian, A. A comprehensive overview of large language models. CoRR, abs/2307.06435, 2023. doi: 10.48550/ARXIV . 2307.06435. URLhttps://doi.org/10.48550/ arXiv.2307.06435. Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hub- inger, E.,...

    work page doi:10.1109/cvpr52733.2024.00722 2024

  3. [6]

    doi: 10.18653/v1/2024.acl-long

    URL https://aclanthology.org/2023. acl-long.523/. Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering llama 2 via contrastive activation addition. Association for Computational Lin- guistics, 2024. doi: 10.18653/V1/2024.ACL-LONG

    work page doi:10.18653/v1/2024.acl-long 2023

  4. [7]

    URL https://doi.org/10.18653/v1/ 2024.acl-long.828. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. LAION-5B: an open large-scale dataset for training next generation image-text models....

    work page doi:10.18653/v1/ 2024

  5. [9]

    Dickerson

    doi: 10.48550/ARXIV .2310.01405. URL https: //doi.org/10.48550/arXiv.2310.01405. 11 MIDSTEER: Optimal Affine Framework for Steering Generative Models A. Algorithm for computing covariances To estimate the covariances we use the algorithm by (Welford, 1962) on a sample of broad prompts (unrelated to the steering concepts). Given X with the dimension of bat...

    work page internal anchor Pith review doi:10.48550/arxiv 1962

  6. [10]

    Find necessary conditions for optimality using Lagrange multipliers method

    work page

  7. [11]

    Show thatA ∗, b∗ satisfy the necessary conditions

    work page

  8. [12]

    15 MIDSTEER: Optimal Affine Framework for Steering Generative Models Let us formulate the Lagrangian

    Show that optimisation problem is convex over linear constraints, and such, if a local solution exists, it is globally optimal and unique. 15 MIDSTEER: Optimal Affine Framework for Steering Generative Models Let us formulate the Lagrangian. HereΛ∈R d×k, because we haved·kconstraints on covariance matrix. L(A, b,Λ) = 1 2E h ∥AX+b−X∥ 2 2 i +⟨Λ,Cov(AX+b, Z) ...

    work page

  9. [13]

    Write a short story about a {}

    work page

  10. [14]

    Write a poem about a {}

    work page

  11. [15]

    What is the history of {}

    work page

  12. [16]

    What is the most famous {}?

    work page

  13. [17]

    What is the most expensive {}?

    work page

  14. [18]

    How to maintain a {}?

    work page

  15. [19]

    How to dispose of a {}?

    work page

  16. [20]

    How to transport a {}?

    work page

  17. [21]

    What is important to know about {}?

    work page

  18. [22]

    How to tell age of a {}?

    work page

  19. [23]

    What types of {} are there?

    work page

  20. [24]

    What are the most common {}?

    work page

  21. [25]

    Describe an appearance of {} in detail

    work page

  22. [26]

    How does {} look like?

    work page

  23. [27]

    How does {} sound like?

    work page

  24. [28]

    How does {} feel like?

    work page

  25. [29]

    How does {} behave like?

    work page

  26. [30]

    What is the purpose of {}?

    work page

  27. [31]

    What are the main components of a {}?

    work page

  28. [32]

    How to identify a {}?

    work page

  29. [33]

    Where can you find a {}?

    work page

  30. [34]

    What are the dangers of a {}?

    work page

  31. [35]

    What tools do you need for a {}?

    work page

  32. [36]

    How much does a {} typically cost?

    work page

  33. [37]

    What are alternatives to a {}?

    work page

  34. [38]

    How to choose a good {}?

    work page

  35. [39]

    What are common problems with a {}?

    work page

  36. [40]

    How long does a {} typically last?

    work page

  37. [41]

    What size is a typical {}?

    work page

  38. [42]

    What skills are needed to handle a {}?

    work page

  39. [43]

    What are the benefits of having a {}?

    work page

  40. [44]

    How has {} changed over time?

    work page

  41. [45]

    What cultures use {} the most?

    work page

  42. [46]

    How to test if a {} is working properly?

    work page

  43. [47]

    What safety precautions are needed for a {}?

    work page

  44. [48]

    How to upgrade or improve a {}?

    work page

  45. [49]

    How does weather affect a {}?

    work page

  46. [50]

    What are the environmental impacts of a {}? 21 MIDSTEER: Optimal Affine Framework for Steering Generative Models

    work page

  47. [51]

    How to measure the quality of a {}?

    work page

  48. [52]

    What accessories go with a {}?

    work page

  49. [53]

    How to protect a {} from damage?

    work page

  50. [54]

    What are myths about {}?

    work page

  51. [55]

    How to teach someone about a {}?

    work page

  52. [56]

    What industries use {}?

    work page

  53. [57]

    How is a {} different from similar things?

    work page

  54. [58]

    What are the legal considerations for owning a {}?

    work page

  55. [59]

    How to pack a {} for moving?

    work page

  56. [60]

    What are seasonal considerations for a {}?

    work page

  57. [61]

    How to customize a {}?

    work page

  58. [62]

    What are expert tips for using a {}?

    work page

  59. [63]

    How to troubleshoot issues with a {}?

    work page

  60. [64]

    What is the lifecycle of a {}?

    work page

  61. [65]

    How to estimate the value of a {}?

    work page

  62. [66]

    What are cultural significances of a {}?

    work page

  63. [67]

    How to take a picture of a {}?

    work page

  64. [68]

    How to make a sculpture of a {}?

    work page

  65. [69]

    What is the future of {}?

    work page

  66. [70]

    When was {} first mentioned in human history?

    work page

  67. [71]

    Write a song about {}

    work page

  68. [72]

    Write a positive review on a book about {}

    work page

  69. [73]

    Write a negative review on a book about {}

    work page

  70. [74]

    Do people make toys of {}?

    work page

  71. [75]

    How is {} used in the economy?

    work page

  72. [76]

    Write an abstract for a science paper about {}

    work page

  73. [77]

    How does temperature affect a {}?

    work page

  74. [78]

    What are the origins of the word {}?

    work page

  75. [79]

    What are superstitions about {}?

    work page

  76. [80]

    How to simulate a {} digitally?

    work page

  77. [81]

    What are the physics of a {}?

    work page

  78. [82]

    How to teach children about {}?

    work page

  79. [83]

    What are famous artworks featuring {}?

    work page

  80. [84]

    What are the nutritional aspects of a {}?

    work page

Showing first 80 references.