REVIEW 13 cited by
Analyzing Transformers in Embedding Space
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks. In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different models that share a vocabulary, and (b) constructing a classifier without training by ``translating'' the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.
Forward citations
Cited by 13 Pith papers
-
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
Layer-normalized averaging of all decoder-only LLM hidden states, rather than last-layer embeddings, improves text-to-image compositional alignment and beats T5 on GenAI-Bench.
-
Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
Attribution-based Parameter Decomposition splits a network's parameters into faithful, minimal, and simple components and recovers ground-truth mechanisms in toy models of superposition and compressed computation.
-
Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders
A lightweight Q-Former proxy trained on VLM hidden states reveals that localization signals peak in input-dependent intermediate layers, not the final layers used by standard editing pipelines.
-
Unraveling Syntax: Language Modeling and the Substructure of Grammars
Language-modeling loss decomposes linearly over the sub-grammars of a probabilistic context-free grammar, and models learn these sub-grammars in parallel rather than in stages.
-
Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones
Language models fail at balanced parentheses because unreliable internal components that promote wrong tokens can outvote reliable ones, and amplifying reliable components fixes the errors.
-
Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers
Neurons that respond abnormally to adversarial inputs are concentrated in early ViT layers, and suppressing them with a fixed mask improves robustness across attacks without retraining.
-
Task Vectors in In-Context Learning: Emergence, Formation, and Benefit
Small transformers naturally encode task information in specific layers under limited conditions; a new auxiliary loss places a strong task vector at a chosen layer and improves out-of-distribution robustness.
-
Rethinking Associative Memory Mechanism in Induction Head
A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.
-
Cross-modal Information Flow in Multimodal Large Language Models
In LLaVA multimodal models, visual information flows into question token representations in two stages, global then object-specific, before propagating to the final answer position.
-
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
The paper shows LLaVA's visual QA mechanism parallels textual QA: visual embeddings encode animal and color features, attention heads extract and match them, and visual instruction tuning refines existing Vicuna heads.
-
LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
A modular LoRA-transfer pipeline reuses old LoRA weights across LLM upgrades, claiming gains over small-scale retraining and sometimes over full retraining.
-
Joint Knowledge Editing for Information Enrichment and Probability Promotion
Jointly editing low-layer information enrichment and high-layer probability promotion stages yields the top overall knowledge-editing scores on zsRE and Multi-COUNTERFACT.
-
NEAT: Concept driven Neuron Attribution in LLMs
NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.
Discussion (0). Continue with ORCID to comment.