Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Line of Sight: On Linear Representations in VLLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ImageNet classes are linearly decodable from a vision-language model's residual stream, and steering those one-dimensional directions causally changes generated captions.

desk verdict Solid VLLM interpretability paper with a strong linear-probing/steering core and a weak manual-rating basis for the 'increasingly shared across modalities' headline. read the letter →

arxiv 2506.04706 v1 pith:2LXGIWHQ submitted 2025-06-05 cs.CV cs.AI

Achyuta Rajaram , Sarah Schwettmann , Jacob Andreas , Arthur Conmy This is my paper · ORCID
classification cs.CVcs.AI
keywords linearrepresentationsvision-languagemodelssparseautoencoderssteeringvectorsinterpretabilityImageNetclassificationLLaVA-Nextresidualstream
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a vision-language model encodes image concepts as simple linear directions inside its transformer, and whether those directions actually drive generation. Working with LLaVA-Next, the authors train linear probes on the mean residual-stream activation of image tokens and find that ImageNet class is readable with accuracy competitive with dedicated image encoders. They then compute steering vectors between classes and show that adding such a vector to image tokens predictably shifts the model's captions toward the target class, while ablating image tokens after layer 12 barely affects outputs. They also train multimodal sparse autoencoders on image and text tokens and find interpretable, monosemantic features, with the share of features shared across modalities increasing in deeper layers. If correct, visual semantics in a VLM are largely one-dimensional, causally effective directions in the early decoder.

What carries the argument

The central object is the contrastive activation-addition (CAA) steering vector: a single residual-stream direction computed by subtracting the mean image-token activation of one ImageNet class from another. It acts both as a probe of linearly readable information and as an intervention added back into image tokens to test causality. The supporting instruments are linear probes on mean image-token activations, zero-ablation of image tokens to localize information transfer, and sparse autoencoders—two-layer MLPs with a 32k-unit dictionary trained under an L1 sparsity penalty on a mix of image and text tokens from ShareGPT4V. The SAE decoder matrix is then used as a sparse dictionary, with a HardConcrete binary-masking optimization, to approximate steering vectors with few active features. This dictionary is what generalizes the linear-representation claim from a few hand-picked classes to a broad geometric statement about visual concepts.

What would settle it

Steer an image from class A by the sum of the steering vectors from A to B and from B to C at the same layer and strength, and compare the resulting caption with the caption produced by the direct A-to-C steering vector; if the combined direction does not produce output close to the direct caption, class directions do not compose linearly, undercutting the linear-representation claim.

Watch

Extended reading notes

Core claim

ImageNet class is represented by one-dimensional vectors in the residual stream of LLaVA-Next: linear probes trained on mean image-token activations recover the class at accuracy on par with self-supervised and supervised image encoders, and contrastive steering vectors applied to image tokens causally redirect captions toward the target class. Image information is transferred to text tokens in the early-to-middle layers of the decoder, since zero-ablating image tokens stops affecting output coherence after layer 12. Sparse autoencoders trained jointly on image and text tokens produce a 32k-feature dictionary of interpretable, monosemantic features in which ImageNet-class directions are sparse (about 20 features suffice), and the proportion of features shared across modalities increases with depth. Text-derived steering vectors, built from logit-lens or text embeddings, do not steer image representations effectively, indicating that image and text spaces are not trivially aligned in the early residual stream.

Load-bearing premise

The claim that features become increasingly shared across modalities rests on a small human rating study—25 SAE features and 50 neurons per layer from only five layers labeled by raters whose judgments are assumed unbiased; if later-layer features are simply harder to describe and raters default to 'multimodal,' the rising trend could be an artifact.

Editorial extensions

If this is right

  • If ImageNet class is linearly decodable from mean image-token activations, a linear probe can recover coarse image semantics from any layer of the residual stream without fine-tuning.
  • If steering vectors are causally effective in early-to-middle layers, test-time control of VLLM captions—forcing, removing, or blending visual classes—requires adding a single vector to image tokens.
  • If image information has been transferred to text tokens by layer 12, interpretability and editing studies of this architecture should concentrate on early-to-middle decoder layers.
  • If SAE dictionaries approximate steering vectors in roughly 20 features, sparse autoencoder latents can serve as a causal interface for editing visual concepts in VLLMs.
  • If text-derived steering vectors fail while image-derived ones succeed, methods that assume a shared linear basis between image and text representations in the early decoder will underperform.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if linear visual directions generalize across VLLM architectures, concept-level intervention (amplifying or deleting a visual class during generation) could become a practical tool for content control and safety in multimodal models.
  • Beyond the paper, the observation that raw neurons are more interpretable than SAE features for image information suggests the advantage of dictionary learning over neurons may be modality- and architecture-dependent, not universal.
  • Beyond the paper, the depth-increasing multimodality trend could be tested automatically by clustering text and image activation patterns of the same SAE feature in an embedding space; if they cluster together, the stages-of-inference reading gains support independent of human ratings.
  • Beyond the paper, the failure of text-derived steering hints that modality alignment may occur in the projection MLP or via cross-attention rather than in the residual stream; probing the projection output directly would settle where the shared space lives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies how visual concepts are represented in the residual stream of LLaVA-Next, a vision-language model. The authors train linear probes on mean image-token activations and report ImageNet-1K classification accuracy competitive with self-supervised encoders. They use contrastive activation addition (CAA) to build steering vectors for 50 random ImageNet class pairs, evaluate the resulting outputs with a gpt4o-mini judge (validated against human raters), and find that steering is most effective at early-middle layers; zero-ablation experiments show that image-token information is transferred to text tokens by around layer 12. They also train multimodal sparse autoencoders (SAEs) on residual-stream activations and evaluate them via loss-recovered and VQA benchmarks, and show that ImageNet-class steering vectors can be approximated with 20–100 SAE features. Finally, a manual annotation study (25 SAE features and 50 neurons per layer from layers 8–24) is used to argue that SAE features are slightly more interpretable than neurons and that the fraction of multimodal features grows with depth. The paper concludes that visual concepts are encoded as linear, causally effective directions and that representations become increasingly shared across modalities in deeper layers.

Significance. If the central claims hold, the paper would extend the linear representation hypothesis to vision-language models, showing that coarse visual semantics are linearly decodable and causally editable via single residual-stream directions, and that multimodal sharing increases with layer depth. The paper's strengths are its multi-method design (probing, causal steering, SAE reconstruction, sparse approximation), its use of human validation for the LLM judge, and its transparent reporting of negative results (text-derived steering and adversarial attacks). The main weakness is that the cross-modal sharing trend, which is a headline claim, rests on a small, single-annotator subjective rating study without inter-rater reliability or error bars. If the rating-bias concern is resolved, the paper would be a solid contribution, but as presented the abstract-level 'increasingly shared in deeper layers' claim is not yet convincingly supported.

major comments (4)
  1. [§5.1, Appendix C] The claim that representations become increasingly shared across modalities with depth is supported only by a small manual annotation study (25 SAE features and 50 neurons per layer from layers 8, 12, 16, 20, 24; the appendix's stated total of 250 is actually 375 items). No inter-rater reliability is reported, and the single-annotator protocol of writing text and image descriptions and then judging whether they match is vulnerable to a default-to-'multimodal' bias when later-layer features are harder to describe and both descriptions become vague. Since this trend is a headline claim in the abstract and Section 6, the authors should either collect a larger multi-rater dataset with agreement metrics or supply an independent quantitative measure of cross-modal sharing (e.g., activation-overlap statistics) before this claim is accepted.
  2. [Figures 5, 6, 8, 9] Quantitative comparisons lack error bars and confidence intervals. For example, 'competitive with modern SSL techniques' (Figure 5), 'slightly more interpretable' (Figure 9a), and 'most of the effect of the dense steering vector with only 20 SAE features' (Figure 8b) cannot be assessed for statistical reliability. The authors should report variance across the 50 random class pairs (or other random seeds) and, where relevant, perform significance tests.
  3. [Section 4] The section opens with 'We find that one-dimensional vectors in the residual stream represent the ImageNet class,' but the probing evidence is a multiclass linear regression (one weight vector per class, jointly trained) and the steering evidence is a difference of two class-mean directions. These support 'linearly decodable' or 'linear directions for class contrasts,' not necessarily a single one-dimensional representation per class. Please clarify the precise geometric claim and consider testing it directly (e.g., comparing the probe weight vectors to the top PCA directions or random directions).
  4. [Appendix A] The human validation of the gpt4o-mini judge is described only qualitatively ('aligns well'); no correlation coefficients, sample sizes, or agreement metrics are reported for either the coherence or the steering-score validation. Because the main steering results are filtered through this judge, these numbers are essential for assessing the reliability of the steering evaluations.
minor comments (6)
  1. [Appendix C] The appendix says the sample comprises '25 SAE features and 50 Neurons from layers 8,12,16,20,24, for a total of 250 features,' but the actual total is 375; correct the arithmetic.
  2. [Throughout] There are naming inconsistencies: 'LlaV A-Next' vs. 'LLaVA-Next' and 'ShareGPT-V' vs. 'ShareGPT4V' appear in different places; please standardize.
  3. [Appendix C] The text alternates between 'a group of expert human raters' and 'the user' (singular); specify exactly how many raters labeled each feature and whether descriptions were elicited independently.
  4. [Section 3.4] The statement that splicing in the SAE at all token positions drops accuracy to near chance 'likely due to a distributional shift' is an untested explanation; label it as a hypothesis or test it (e.g., by training the SAE on instruction-following data as well).
  5. [Algorithm 1] The algorithm leaves the number of iterations T and the sparsity hyperparameter λ used in the experiments unspecified; add the actual values.
  6. [Section 6] The Discussion's statement that 'neurons form a highly interpretable basis over image information, comparing favorably to SAE features' is stronger than what Figure 9a shows (a small difference); either temper the claim or report the underlying percentages and a statistical comparison.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity found: probing, steering, and SAE claims are externally grounded; self-citations are methodological and not load-bearing.

full rationale

The derivation chain is self-contained on its central claims. Section 4 measures linear probes against external ImageNet-1K labels (Fig. 5), and steering vectors are evaluated by intervening on model outputs and scoring completions with an LLM judge validated against human raters (Section 4.2, Appendix A), so the causal claim is not determined by how the steering vector was fit. Section 5.2 uses Algorithm 1 only to test whether a known steering direction is sparsely representable in the SAE dictionary, and the behavioral transfer of the sparse approximation is again tested by external steering evaluations rather than by the reconstruction loss itself. The SAE is trained generically to reconstruct residual-stream activations, not to reproduce the probe or steering results, and the SAE's general fidelity is checked on held-out loss and VQA benchmarks (Section 3.4). Self-citations (e.g., Chalnev et al. [9], Kissane et al. [25], Rajamanoharan et al. [45], which include author A. Conmy) appear only as methodological precedents for LLM judging, sparse approximation, and manual interpretability protocol; no uniqueness theorem or central ansatz is imported from prior author work. The one legitimate caveat is the Section 5.1 / Appendix C multimodality trend, which rests on a small single-rater study (25 SAE features and 50 neurons per layer at layers 8, 12, 16, 20, 24) and could be affected by rater bias in later layers; that is a robustness and external-validity concern, not a circular reduction, because 'multimodal' is operationally defined by rater description-matching and the layer trend is not used to define the probes, steering vectors, or SAE objective. Score 2 reflects the presence of minor methodological self-citations only, not substantive circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper relies on standard interpretability assumptions: residual stream directions correspond to concepts, CAA steering isolates a concept, SAE reconstruction faithfulness reflects informativeness, and an LLM judge approximates human preferences. It introduces no new entities or fitted constants beyond standard SAE hyperparameters.

free parameters (4)
  • SAE L1 density coefficient lambda = 5
    Chosen from Templeton et al. and ramp-up schedule; affects sparsity of features (L0 ~5) and therefore the interpretability results.
  • SAE expansion factor = 8 (32k features)
    Selected to match standard SAE widths; determines dictionary size and feature granularity.
  • SAE learning rate = 5e-5
    Training hyperparameter; not fitted to downstream interpretability.
  • Number of SAE training tokens = 1.5 billion
    Chosen training budget; affects feature quality and the distribution shift observed.
assumptions (4)
  • domain assumption The residual stream is a linear representation space where concepts are directions
    Assumed from prior work (Park et al. 2024); the paper probes only ImageNet class, but interventions support causality.
  • domain assumption Contrastive activation addition isolates the targeted concept direction
    Steered outputs could mix other factors (e.g., style); human-validated judge mitigates but does not eliminate.
  • domain assumption SAE reconstruction quality transfers to feature interpretability
    They use loss recovered and VQA benchmarks as evidence that SAE features capture model behavior, but interpretability is rated by humans on a small sample.
  • domain assumption gpt4o-mini judge scores correlate with human judgments for all steering rollouts
    Validated on a subset, but assumed to generalize across 200 class pairs and 200-token completions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Line of Sight: On Linear Representations in VLLMs." pith.science (2026). https://pith.science/paper/2LXGIWHQ

@misc{pith2026250604706,
  author       = {Pith},
  title        = {Pith review of: Line of Sight: On Linear Representations in VLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2LXGIWHQ}},
  note         = {Machine review of arXiv:2506.04706}
}
read the original abstract

Language models can be equipped with multimodal capabilities by fine-tuning on embeddings of visual inputs. But how do such multimodal models represent images in their hidden activations? We explore representations of image concepts within LlaVA-Next, a popular open-source VLLM. We find a diverse set of ImageNet classes represented via linearly decodable features in the residual stream. We show that the features are causal by performing targeted edits on the model output. In order to increase the diversity of the studied linear features, we train multimodal Sparse Autoencoders (SAEs), creating a highly interpretable dictionary of text and image features. We find that although model representations across modalities are quite disjoint, they become increasingly shared in deeper layers.

Figures

Figures reproduced from arXiv: 2506.04706 by the authors.

Figure 1
Figure 1. Linear Representations of Image Information A schematic diagram of linear probes, steering vectors, and Sparse Autoencoders as methods for analysing linear features derived from image tokens. Image Input Text Prompt Patchify Tokenizer Wemb Describe This Image. Transformer Decoder Block Transformer Decoder Block Transformer Decoder Block Transformer Decoder Block The image shows a historical site with a preserved sec… view at source ↗
Figure 2
Figure 2. Schematic of LlaVA-Next SAE Interventions: text and image embeddings are processed separately, with the results projected into a shared latent space. If a water-related SAE feature is activated on all image tokens, the generated description of a cannon on dry land contains a hallucinated moat filled with water. to perform intervention-level analysis on generative image models [53]. More recently, work has been done … view at source ↗
Figure 3
Figure 3. Sparse Autoencoder Training: From an image-text dataset D. we sample activations from random image and text tokens. We train a Sparse Autoencoder (SAE) to reconstruct the residual stream activations, using a sparse latent dictionary (L1 penalized). from Templeton et al. [50], with no major changes. We use a ReLU activation function, and train to minimize reconstruc￾tion error (as measured by mean-squared error), as … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: SAE Evaluations. We measure a). loss recovered and b). benchmark scores for the model with the SAE spliced in at various positions. We find that the SAEs reconstruct the information present in the image tokens well. 4 b.). However, upon splicing in the SAE at all token…
Figure 5
Figure 5. Figure 5: ImageNet Accuracy. We benchmark linear probes trained on mean embeddings across image tokens on ImageNet Classification. Performance is competitive with modern SSL tech￾niques indicating linear representations of the class variable. 4.2. Can the residual stream be used…
Figure 7
Figure 7. Figure 7: Text-Based Steering Vectors We measure coherence vs. steering score for steering vectors across several steering strengths. We find that both a). using the unembedding matrix (Logit Lens, abbreviated LL). and b). using the hidden layer activations as Text Embeddings (T…
Figure 6
Figure 6. Figure 6: Steering Vector Interventions a). Schematic Diagram of Steering Vectors. We compute them using activations from images of two classes, and add the resulting vectors in at image tokens. b). We compute coherence and steering scores as measures of steering vector efficacy…
Figure 8
Figure 8. Figure 8: SAE Steering Vector Approximation a). A schematic diagram of the approximation process. We optimize SAE hidden latents, using the decoder matrix as a dictionary, for sparsity (Lsp) and reconstruction accuracy (Lcos). b).To measure approximation quality, we compute cohe…
Figure 9
Figure 9. Figure 9: Manual Interpretability Experiments a). We find that SAE features are slightly more interpretable than neurons, but both are highly interpretable throughout the model. b). We find that our human raters are more likely to label both neurons and SAE features from later l…
Figure 10
Figure 10. Figure 10: Example of Coherence Evaluation A random sample from our dataset. Clearly, Answer 1 is more coherent. Then, we compute the probability that the turker will prefer one answer to another, based on their respective grader scores. Steering Score: To evaluate this, we firs…
Figure 11
Figure 11. Figure 11: Example of Coherence Evaluation A random sample from our dataset. Clearly, the completion is most aligned with the class “shield” We then compute the probability that the user selects the target and original class, given a steering score for the prompt. Using this, we…
Figure 12
Figure 12. Figure 12: Steering Examples Several selected examples of image-token steering. Three image descriptions are taken, with the simple prompt: “What is this image?”, at three steering levels. The first sample is steered towards “pizza”, the second is steered towards “backpack”, the…
Figure 13
Figure 13. Figure 13: Example of a browser: This SAE feature appears to activate on images of crust and baked goods. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 48 canonical work pages

  1. [1]

    Understanding interme- diate layers using linear classifier probes, 2018

    Guillaume Alain and Yoshua Bengio. Understanding interme- diate layers using linear classifier probes, 2018

    work page 2018

  2. [2]

    Flamingo: a visual language model for few-shot learning, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Shar- ifzadeh, Mikolaj Binkow...

    work page 2022

  3. [3]

    Refusal in lan- guage models is mediated by a single direction, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in lan- guage models is mediated by a single direction, 2024

    work page 2024

  4. [4]

    Understanding the role of individual units in a deep neural network

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020

    work page 2020

  5. [5]

    Probing classifiers: Promises, shortcom- ings, and advances, 2021

    Yonatan Belinkov. Probing classifiers: Promises, shortcom- ings, and advances, 2021

    work page 2021

  6. [6]

    Towards monosemanticity: Decomposing language mod- els with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yi- fan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Ka- rina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, an...

    work page

  7. [7]

    Steven Cao, Victor Sanh, and Alexander M. Rush. Low- complexity probing via finding subnetworks, 2021

    work page 2021

  8. [8]

    Emerg- ing properties in self-supervised vision transformers, 2021

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers, 2021

    work page 2021

Show all 58 references
  1. [9]

    Improv- ing steering vectors by targeting sparse autoencoder features, 2024

    Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. Improv- ing steering vectors by targeting sparse autoencoder features, 2024

    2024

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023

    2023

  3. [11]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023

    2023

  4. [12]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2401.01345, 2024

    2024 arXiv

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  6. [14]

    Softmax linear units

    Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfi...

    2022

  7. [15]

    Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark

    Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024

    2024

  8. [16]

    Concept sliders: Lora adaptors for precise control in diffusion models, 2023

    Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Anto- nio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models, 2023

    2023

  9. [17]

    The pile: An 800gb dataset of diverse text for language mod- eling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language mod- eling, 2020

    2020

  10. [18]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupr´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024

    2024

  11. [19]

    Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information, 2021

    Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information, 2021

    2021

  12. [20]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015

    2015

  13. [21]

    Natural language descriptions of deep visual features

    Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. InInternational Conference on Learning Representations, 2022

    2022

  14. [22]

    Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rockt¨aschel, and David Scott Krueger

    Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rockt¨aschel, and David Scott Krueger. Mechanistically ana- lyzing the effects of fine-tuning on procedurally defined tasks, 2024

    2024

  15. [23]

    Interpreting and editing vision-language representa- tions to mitigate hallucinations, 2024

    Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gan- delsman. Interpreting and editing vision-language representa- tions to mitigate hallucinations, 2024

    2024

  16. [24]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  17. [25]

    Saes are highly dataset dependent: a case study on the refusal direction

    Connor Kissane, robertzk, Neel Nanda, and Arthur Conmy. Saes are highly dataset dependent: a case study on the refusal direction. 2024

    2024

  18. [26]

    The remarkable robustness of llms: Stages of inference?, 2024

    Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of inference?, 2024

    2024

  19. [27]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    2024

  20. [28]

    Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l0 regularization, 2018. 9

    2018

  21. [29]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables, 2017

    2017

  22. [30]

    Michaud, Yonatan Be- linkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Be- linkov, David Bau, and Aaron Mueller. Sparse feature cir- cuits: Discovering and editing interpretable causal graphs in language models, 2024

    2024

  23. [31]

    Mass-editing memory in a trans- former, 2023

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a trans- former, 2023

    2023

  24. [32]

    Linearly mapping from image to text space, 2023

    Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space, 2023

    2023

  25. [33]

    Emer- gent linear representations in world models of self-supervised sequence models, 2023

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emer- gent linear representations in world models of self-supervised sequence models, 2023

    2023

  26. [34]

    Towards interpreting visual in- formation processing in vision-language models, 2024

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual in- formation processing in vision-language models, 2024

    2024

  27. [35]

    Sparse autoencoder

    Andrew Ng. Sparse autoencoder. CS294A Lecture Notes,

  28. [36]

    interpreting GPT: the logit lens — LessWrong — lesswrong.com

    nostalgebraist. interpreting GPT: the logit lens — LessWrong — lesswrong.com. https : / / www . lesswrong . com / posts / AcKRB8wDpdaN6v6ru / interpreting-gpt-the-logit-lens . [Accessed 26-01-2025]

    2025

  29. [37]

    Kai Nylund, Suchin Gururangan, and Noah A. Smith. Time is encoded in the weights of finetuned language models, 2023

    2023

  30. [38]

    Olshausen and David J

    Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1?Vision Research, 37(23):3311–3325, 1997

    1997

  31. [39]

    Dinov2: Learning robust visual features without supervision, 2024

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Ra...

    2024

  32. [40]

    Steering llama 2 via contrastive activation addition, 2024

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024

    2024

  33. [41]

    The linear representation hypothesis and the geometry of large language models, 2024

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models, 2024

    2024

  34. [42]

    Instruction tuning with gpt-4, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023

    2023

  35. [43]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024

    2024

  36. [44]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  37. [45]

    Improving dictionary learning with gated sparse autoencoders, 2024

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J´anos Kram´ar, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders, 2024

    2024

  38. [46]

    Automatic discovery of visual circuits, 2024

    Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, and Sarah Schwettmann. Automatic discovery of visual circuits, 2024

    2024

  39. [47]

    A-okvqa: A bench- mark for visual question answering using world knowledge, 2022

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A bench- mark for visual question answering using world knowledge, 2022

    2022

  40. [48]

    Multimodal neurons in pretrained text-only transformers, 2023

    Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers, 2023

    2023

  41. [49]

    Effi- cient sparse coding algorithms, pages 801–808

    Bernhard Sch ¨olkopf, John Platt, and Thomas Hofmann. Effi- cient sparse coding algorithms, pages 801–808. 2007

    2007

  42. [50]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lind- sey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte Mac- Diarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, J...

    2024

  43. [51]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024

    2024

  44. [52]

    Manning, and Christo- pher Potts

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christo- pher Potts. Reft: Representation finetuning for language models, 2024

    2024

  45. [53]

    Linear semantics in generative adversarial networks, 2021

    Jianjin Xu and Changxi Zheng. Linear semantics in generative adversarial networks, 2021

    2021

  46. [54]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  47. [55]

    Mmmu- pro: A more robust multi-discipline multimodal understand- ing benchmark, 2024

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu- pro: A more robust multi-discipline multimodal understand- ing benchmark, 2024

    2024

  48. [56]

    feature browser

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. 10 A. Steering V ector Evaluation Details We evaluate steering vectors on steering model answers to general VQA questions. To do this, we select 200 random p...

    2023

  49. [2011]

    Unpublished lecture notes

  50. [2023]

    pub / 2023/monosemantic-features/index.html

    https : / / transformer - circuits . pub / 2023/monosemantic-features/index.html

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.