Pith. sign in

REVIEW 11 cited by

An Interpretability Illusion for BERT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.07143 v1 pith:G2VPENNR submitted 2021-04-14 cs.CL cs.LG

Tolga Bolukbasi , Adam Pearce , Ann Yuan , Andy Coenen , Emily Reif , Fernanda Viégas , Martin Wattenberg This is my paper · ORCID
classification cs.CLcs.LG
keywords bertillusioninterpretabilityactivationsfactwhenanalyzingappear
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We describe an "interpretability illusion" that arises when analyzing the BERT model. Activations of individual neurons in the network may spuriously appear to encode a single, simple concept, when in fact they are encoding something far more complex. The same effect holds for linear combinations of activations. We trace the source of this illusion to geometric properties of BERT's embedding space as well as the fact that common text corpora represent only narrow slices of possible English sentences. We provide a taxonomy of model-learned concepts and discuss methodological implications for interpretability research, especially the importance of testing hypotheses on multiple data sets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation

    cs.LG 2026-08 accept novelty 7.0 of 10

    In ablation-based SAE evaluation, the measurement token is selected by the dictionary under test, and holding that token fixed collapses most of the variance that is usually attributed to differences between dictionaries.

  2. Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

    cs.LG 2026-05 conditional novelty 7.0 of 10

    SA-GSAE with Bi-Jump-ReLU enables one latent to encode both polarities of anticorrelated features, Pareto-dominating or matching full-width gated SAEs while reducing dead latents by up to 500x on some LLM hookpoints.

  3. Recovering Event Probabilities from Large Language Model Embeddings via Axiomatic Constraints

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A sign-flip constraint on one latent dimension of a VAE trained on LLM embeddings yields complementary event probabilities that sum to near one and track true probabilities on held-out dice events.

  4. The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Fixed activation probes keep near-ceiling accuracy on harmful-vs-benign corpus contrasts but fall to AUROC 0.59-0.69 on topic- and surface-matched harmful/benign pairs, so they behave as broad-risk detectors, not cont...

  5. Evaluating Neuron Explanations: A Unified Framework with Sanity Checks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Most commonly used neuron explanation evaluation metrics fail two new sanity checks, and only Correlation, Cosine, AUPRC, F1-score, and IoU pass.

  6. The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    The Heap is a new multilingual code dataset of 32.7M copyleft-licensed files with exact and near-duplicate flags relative to The Stack, Red Pajama, GitHub Code, and CodeParrot.

  7. Inferring Functionality of Attention Heads from their Parameters

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Attention-head functions can be inferred from parameters alone via vocabulary-space top-k scoring, and these inferences match head outputs and survive causal ablation across six LLMs.

  8. Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Autoregressive language models represent multiple competing readings of garden path sentences via interpretable syntactic features, and do not reuse those features to answer follow-up questions.

  9. Towards Utilising a Range of Neural Activations for Comprehending Representational Associations

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Mid-level (near-zero) logit activations contain information about spurious correlations and mislabels that maximal activations hide, enabling a no-group-label retraining method, MID, that improves worst-group accuracy...

  10. Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The paper introduces an Explanatory Virtues Framework and argues, via a qualitative rubric, that Compact Proofs are the most promising method for mechanistic interpretability.

  11. Perspectives for Direct Interpretability in Multi-Agent Deep Reinforcement Learning

    cs.AI 2025-02 unverdicted novelty 4.0 of 10

    A perspective paper that advocates direct, post hoc interpretability for multi-agent deep reinforcement learning and offers a taxonomy of where those methods might apply.

Pith tools