REVIEW 26 cited by
Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation "queries" the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.
Forward citations
Cited by 26 Pith papers
-
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?
A shortcut-filtered benchmark shows LLMs genuinely compose facts internally for country-bridge queries (over 80% for the best models) but almost never for year-bridge queries (about 5-6%).
-
Verbalizable Representations Form a Global Workspace in Language Models
Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.
-
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.
-
Sparse Autoencoders Do Not Find Canonical Units of Analysis
Across different dictionary sizes, SAE latents are neither complete nor atomic, so SAEs do not learn a canonical set of features.
-
What's in the Image? A Deep-Dive into the Vision of Vision Language Models
Vision-language models store a global image summary in the query text tokens, rely on the middle transformer layers for vision-to-text transfer, and fetch fine details from image tokens in a spatially localized way.
-
Prototype Transformer: Towards Language Model Architectures Interpretable by Design
ProtoT is an autoregressive language model whose attention is replaced by learned prototype channels that are claimed to capture nameable concepts and allow targeted edits, at linear sequence cost but with slightly lo...
-
Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
Spectral features of attention are claimed to classify proof validity with near-perfect effect sizes, but the main evaluation relabels proofs using the classifier's own outputs.
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
Potemkin Understanding in Large Language Models
LLMs frequently pass definition questions yet fail to use the same concepts in classification, generation, and editing tasks, a gap the authors call potemkin understanding.
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
Mamba Knockout for Unraveling Factual Information Flow
Fine-grained token-to-token knockout reveals that Mamba models, like Transformers, rely on subject-token information flow in late-intermediate layers, with architecture-specific variations in relation-token and first-...
-
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.
-
Emergent Response Planning in LLMs
Hidden representations of LLM prompts encode global attributes of the upcoming response, and simple probes can predict length, content choices, and answer confidence before generation begins.
-
Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability
A position paper unifying feature, data, and component attribution under three shared techniques, perturbation, gradient, and linear approximation, and proposing cross-attribution research directions.
-
Rethinking Associative Memory Mechanism in Induction Head
A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.
-
Cross-modal Information Flow in Multimodal Large Language Models
In LLaVA multimodal models, visual information flows into question token representations in two stages, global then object-specific, before propagating to the final answer position.
-
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
The paper shows LLaVA's visual QA mechanism parallels textual QA: visual embeddings encode animal and color features, attention heads extract and match them, and visual instruction tuning refines existing Vicuna heads.
-
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
A variational Bayesian inverse RL objective, AVA, models token-level rewards and outperforms reward-difference baselines in LLM alignment on standard benchmarks.
-
Linear Relational Decoding of Morphology in Language Models
A single matrix multiplication on middle-layer word states reproduces morphological transformations in GPT-J and Llama-7b with roughly 90% faithfulness.
-
On the Fundamental Impossibility of Hallucination Control in Large Language Models
The paper claims a mathematical impossibility: every capable LLM must violate at least one of four idealized response properties, so hallucination is structurally inevitable.
-
Understanding the learned look-ahead behavior of chess neural networks
The Leela Chess Zero policy network encodes information about destination squares of moves up to seven plies ahead, with attention heads that copy future-square information backward in time in a pattern-dependent way.
-
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
MEMAT combines MEMIT weight edits with optimized attention-head corrections, improving cross-lingual success and magnitude metrics over MEMIT in English and Catalan.
-
Joint Knowledge Editing for Information Enrichment and Probability Promotion
Jointly editing low-layer information enrichment and high-layer probability promotion stages yields the top overall knowledge-editing scores on zsRE and Multi-COUNTERFACT.
-
NEAT: Concept driven Neuron Attribution in LLMs
NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
-
Towards a Theory of AI Personhood
The paper outlines agency, theory of mind, and self-awareness as necessary conditions for AI personhood, reviews inconclusive evidence, and argues that AI personhood would make control-focused alignment ethically problematic.
Discussion (0). Continue with ORCID to comment.