The paper defines STE and SPID, two information-theoretic measures of semantic flow and decomposition in language exchanges, and applies them to four dialogue datasets.
hub
How to use and interpret activation patching
37 Pith papers cite this work, alongside 5 external citations. Polarity classification is still indexing.
37
Pith papers citing it
5
external citations · Pith
abstract
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.
hub tools
citation-role summary
method 2
background 1
dataset 1
citation-polarity summary
representative citing papers
Re-derivation of activation patching NIE reveals it captures interaction effects in addition to direct causal effects, demonstrated via GPT-2 IOI circuit where INT explains component ranking issues and faithfulness instability.
SWITCH uses explicit <swi> and </swi> boundary tokens to make latent chain-of-thought compatible with on-policy RL (GRPO) and open to causal mechanistic probing, outperforming prior hidden-state recurrence methods.
Structurally distinct circuits for literal sequence copying across token frequency bands implement the same computation, shown by broad transfer of band-specific edges, a shared core recovering 99% performance, and interchangeable representations via causal interventions.
Single-bucket probes in Mamba-2 recover only the small BOS-specialist execution layer of the state sink while missing the larger dual-head detection layer with the same representational signature.
Text tokens in FLUX.2 absorb reference image properties like color and style to influence outputs while pixel-exact details bypass them, localized to padding tokens via causal interventions.
Persuasion in LLMs works by redirecting a small set of attention heads to copy the target option token instead of reasoning over evidence, via a rank-one routing feature that can be directly edited or removed.
Instruction-tuned language models stabilize their next-token predictions later in the forward pass than pretrained models, with late MLP layers providing the strongest tested control point under matched histories.
Transformer circuits show free evolution during SFT, rendering static mechanistic localization inadequate for future parameter updates due to inherent temporal latency.
Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.
Llama-3.1-8B computes sums for cyclic concepts using base-10 addition via task-agnostic Fourier features with periods 2, 5, and 10 rather than modular arithmetic in the concept period.
Zero-ablation overstates register content dependence in DINO ViTs because mean, noise, and cross-image shuffle replacements preserve performance while zeroing does not.
Latent reasoning faithfulness is a property of training stage and answer format, not of architecture or the final checkpoint alone.
Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.
Graph-PRefLexOR fine-tunes graph-native models with GRPO to organize reasoning into phases, yielding 40-65% gains in traceable hypothesis generation and 2-3x semantic diversity on 100 materials science questions.
Perturbation experiments across six LLMs show activation robustness follows L^p norm with p>2 for feature directions (contrastive, MELBO, SAE) but p≈2 for random/PCA controls, indicating feature-specific error correction.
IGSD uses signed differences of Sobol indices from matched replacement and ablation to identify task-specific content channels that standard importance scores miss in GPT-2 and Qwen2.5.
Dominant error in attribution patching arises from downstream non-linearities; a single HVP correction removes the leading error term and matches Integrated Gradients accuracy at lower cost across 124M-9B models.
RMCT matches the rate of target behaviors like bias-following across input perturbations to reduce sycophancy in LLMs while preserving verbalization of bias cues.
Sparse Python code transformer develops dedicated circuits for 106 concepts organized by computational atomicity and token ambiguity, with AST nodes showing up to 62.5% concept-only neurons distinct from token activation.
Temporal knowledge drift is encoded as a geometrically orthogonal direction in LLM residual streams, independent of correctness and uncertainty.
A classification-trained ViT encodes patch boundaries at layers 5-6 and depth at layer 8, with causal interventions showing the depth signal is actively re-derived rather than passively carried.
LLMs implement a second-order confidence architecture where the PANL activation encodes both error likelihood and the ability to correct it, beyond verbal confidence or log-probabilities.
LLM 2-bit quantization fails via either cumulative signal degradation or early computation collapse in key components.
citing papers explorer
-
Information Dynamics of Language Communication
The paper defines STE and SPID, two information-theoretic measures of semantic flow and decomposition in language exchanges, and applies them to four dialogue datasets.
-
The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching
Re-derivation of activation patching NIE reveals it captures interaction effects in addition to direct causal effects, demonstrated via GPT-2 IOI circuit where INT explains component ranking issues and faithfulness instability.
-
Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning
SWITCH uses explicit <swi> and </swi> boundary tokens to make latent chain-of-thought compatible with on-policy RL (GRPO) and open to causal mechanistic probing, outperforming prior hidden-state recurrence methods.
-
Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery
Structurally distinct circuits for literal sequence copying across token frequency bands implement the same computation, shown by broad transfer of band-specific edges, a shared core recovering 99% performance, and interchangeable representations via causal interventions.
-
Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
Single-bucket probes in Mamba-2 recover only the small BOS-specialist execution layer of the state sink while missing the larger dual-head detection layer with the same representational signature.
-
Vision-Language Binding in In-Context Image Generation
Text tokens in FLUX.2 absorb reference image properties like color and style to influence outputs while pixel-exact details bypass them, localized to padding tokens via causal interventions.
-
How LLMs Are Persuaded: A Few Attention Heads, Rerouted
Persuasion in LLMs works by redirecting a small set of attention heads to copy the target option token instead of reasoning over evidence, via a rank-one routing feature that can be directly edited or removed.
-
The Convergence Gap: Instruction-Tuned Language Models Stabilize Later in the Forward Pass
Instruction-tuned language models stabilize their next-token predictions later in the forward pass than pretrained models, with late MLP layers providing the strongest tested control point under matched histories.
-
Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
Transformer circuits show free evolution during SFT, rendering static mechanistic localization inadequate for future parameter updates due to inherent temporal latency.
-
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.
-
Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Llama-3.1-8B computes sums for cyclic concepts using base-10 addition via task-agnostic Fourier features with periods 2, 5, and 10 rather than modular arithmetic in the concept period.
-
Zero-Ablation Overstates Register Content Dependence in DINO Vision Transformers
Zero-ablation overstates register content dependence in DINO ViTs because mean, noise, and cross-image shuffle replacements preserve performance while zeroing does not.
-
Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories
Latent reasoning faithfulness is a property of training stage and answer format, not of architecture or the final checkpoint alone.
-
Temporal Preference Concepts and their Functions in a Large Language Model
Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.
-
Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination
Graph-PRefLexOR fine-tunes graph-native models with GRPO to organize reasoning into phases, yielding 40-65% gains in traceable hypothesis generation and 2-3x semantic diversity on 100 materials science questions.
-
Evidence for feature-specific error correction in LLMs
Perturbation experiments across six LLMs show activation robustness follows L^p norm with p>2 for feature directions (contrastive, MELBO, SAE) but p≈2 for random/PCA controls, indicating feature-specific error correction.
-
Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components
IGSD uses signed differences of Sobol indices from matched replacement and ablation to identify task-specific content channels that standard importance scores miss in GPT-2 and Qwen2.5.
-
When Attribution Patching Lies: Diagnosis and a Second-Order Correction
Dominant error in attribution patching arises from downstream non-linearities; a single HVP correction removes the leading error term and matches Integrated Gradients accuracy at lower cost across 124M-9B models.
-
Consistency Training while Mitigating Obfuscation via Rate Matching
RMCT matches the rate of target behaviors like bias-following across input perturbations to reduce sycophancy in LLMs while preserving verbalization of bias cues.
-
CSP-Atlas: Concept-Specific Neural Circuits in a Sparse Python Transformer
Sparse Python code transformer develops dedicated circuits for 106 concepts organized by computational atomicity and token ambiguity, with AST nodes showing up to 62.5% concept-only neurons distinct from token activation.
-
The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
Temporal knowledge drift is encoded as a geometrically orthogonal direction in LLM residual streams, independent of correctness and uncertainty.
-
From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers
A classification-trained ViT encodes patch boundaries at layers 5-6 and depth at layer 8, with causal interventions showing the depth signal is actively re-derived rather than passively carried.
-
How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
LLMs implement a second-order confidence architecture where the PANL activation encodes both error likelihood and the ability to correct it, beyond verbal confidence or log-probabilities.
-
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
LLM 2-bit quantization fails via either cumulative signal degradation or early computation collapse in key components.
-
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Token-level contrastive attribution yields informative signals for some LLM benchmark failures but is not universally applicable across datasets and models.
-
Hallucination as Trajectory Commitment: Causal Evidence for Asymmetric Attractor Dynamics in Transformer Generation
Hallucination is an early trajectory commitment in transformers governed by asymmetric attractor dynamics, with prompt encoding selecting the basin and correction needing multi-step intervention.
-
Weight Patching: Toward Source-Level Mechanistic Localization in LLMs
Weight Patching localizes capabilities to specific parameter modules in LLMs by replacing weights from a behavior-specialized model into a base model and validating recovery via a vector-anchor interface, revealing a hierarchy of source, routing, and execution components.
-
How do LLMs Compute Verbal Confidence
Mechanistic experiments on Gemma 3 27B, Qwen 2.5 7B and Magistral Small 24B show verbal confidence is cached at post-answer positions from answer tokens and captures richer answer-quality information beyond token log-probabilities.
-
On Emergent Social World Models -- Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models
Suggestive evidence indicates language models develop interconnected social world models by functionally integrating theory of mind and pragmatic reasoning.
-
Understanding Counting Mechanisms in Large Language and Vision-Language Models
LLMs and LVLMs encode latent positional count information in individual tokens or visual features, with an internal counter mechanism that updates per item and emerges progressively across layers, relying on structural cues like separators.
-
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Emergent capabilities arise stochastically from abrupt learning of sparse attention patterns on synthetic linear map and cellular automata tasks, with larger models learning them earlier on average.
-
Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
Reasoning in large output spaces proceeds via shortlisting then fine-grained reasoning; this characterization enables a mechanistic distillation strategy that outperforms standard distillation.
-
Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?
Causal mediation analysis on SpiritLM reveals discrepancies in factual recall between text-to-text and speech-to-text paths, indicating only partial carry-over of mechanisms from text to speech modality.
-
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
Across 15 LLMs, first-answer accuracy on arithmetic procedural execution falls from 63% at 5 steps to 20% at 95 steps, with under-execution increasing.
-
Curvature-Guided Module Localization for Low-Rank Detoxification of Backdoored Large Language Models
Curvature-guided localization and low-rank repair detoxifies backdoored LLMs by suppressing trigger responses while preserving normal behavior.
- Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
- Same Targets, Different Computation: How Post-Training Divides Work Across Model Layers