Disentangling Polysemantic Channels in Convolutional Neural Networks

Jonas Fischer; Robin Hesse; Simone Schaub-Meyer; Stefan Roth

REVIEW 4 major objections 6 minor 24 references

Disentangling Polysemantic Channels in Convolutional Neural Networks

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pretrained CNN's polysemantic channels can be explicitly split into single-concept channels by rewiring the weights through a new intermediate layer, without changing the network's predictions.

desk verdict A neat workshop-scale paper: explicit residual-corrected splitting of polysemantic channels that preserves outputs and makes MACO visualizations interpretable; the flagship example convinces, the quantitative evaluation does not. read the letter →

arxiv 2504.12939 v1 pith:LHG4XRQJ submitted 2025-04-17 cs.CV cs.LG

Robin Hesse , Jonas Fischer , Simone Schaub-Meyer , Stefan Roth This is my paper · ORCID

classification cs.CVcs.LG

keywords mechanisticinterpretabilitypolysemanticchannelschanneldisentanglementfeaturevisualizationinput-gradientattributiongamma-polysemanticityResNet-50ImageNet

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that polysemantic channels in convolutional neural networks—channels that fire for several unrelated concepts, such as 'digital clock' and 'cauliflower'—are not an unavoidable property of the trained model. It proposes a weight-rewiring procedure that splits such a channel into two channels, each responding to one concept, plus a residual channel that exactly restores the original activation. Because the disentanglement is explicit, standard tools such as feature visualization can be applied to each concept separately instead of producing a blended image. If the claim holds, pretrained CNNs can be made more interpretable without retraining and without changing their predictive behavior.

What carries the argument

The machinery is the average relevance vector (ARV): for a channel $c$ in layer $l$ and a class $t$, the vector of Input×Gradient attributions from every channel in layer $l-1$ to $c$, averaged over class-$t$ images. Cosine similarity between the two class ARVs defines $\gamma$-polysemanticity (below threshold $\gamma$), and the scaled comparison $\mathrm{ARV}(l,c,t_2)_i\cdot\rho > \mathrm{ARV}(l,c,t_1)_i$ assigns each previous-layer channel $i$ to concept 2, allowing the incoming edges of $c$ to be partitioned into concept-1, concept-2, and shared parts. The residual channel, wired to $c$ with weight $-1$, removes the double encoding of shared features and recovers the original activation exactly. This three-way weight partition is what makes the disentanglement explicit rather than virtual.

What would settle it

Take a channel flagged as $\gamma$-polysemantic, apply the split with the paper's automatic $\rho$ selection, and feed held-out class images through the edited network. If the residual channel's mean absolute activation is comparable to the original channel's activation rather than near zero, or if the concept-matched disentangled channel responds weakly on its own class or strongly on the other, the 'only activates for one concept' claim is falsified. Comparing the original and edited logits also settles the recovery guarantee: any material change contradicts the claim that the original activation is exactly recoverable.

Watch

Extended reading notes

Core claim

The central claim is that a channel $c$ in layer $l$ that is $\gamma$-polysemantic for two classes $t_1$ and $t_2$ can be replaced by three channels in a new layer between $l-1$ and $l$: one channel that keeps only the incoming edges mainly responsible for $t_1$, one that keeps only those mainly responsible for $t_2$, and a residual channel that keeps the shared edges and connects to the original channel $c$ with weight $-1$. Summing the three recovers $c$'s original activation, so the edit is prediction-preserving. On a ResNet-50 trained on ImageNet, the paper reports that for penultimate-layer channels with $\gamma=0.5$, the concept-matched disentangled channels retain roughly 114% and 125% of the original activation on their own class while staying near zero on the other, and the residual channel stays near zero. This is presented as evidence that each disentangled channel 'only activates for one of the respective class concepts' and that the residual does not need to be explained.

Load-bearing premise

The approach assumes that the two concepts are cleanly separable in the immediately preceding layer: each previous-layer channel must be attributable mostly to one concept, so cutting edges by the scaled ARV ($\mathrm{ARV}(l,c,t_2)_i\cdot\rho > \mathrm{ARV}(l,c,t_1)_i$) comparison isolates the concepts rather than removing shared or distributed features that the residual channel cannot compensate for.

Editorial extensions

If this is right

Feature visualizations such as MACO applied to the disentangled channels should show one concept each rather than a blend; the paper demonstrates this for the 'digital clock' and 'cauliflower' channel #1660.
The edited network's outputs are unchanged by construction, so the procedure can be applied to deployed pretrained models without retraining or fine-tuning.
gamma-polysemanticity gives a quantitative criterion for when a channel is worth splitting, and the relative-activation table gives a quantitative check of whether the split succeeded.
The construction is not tied to ResNet-50: any CNN layer where Input×Gradient attributions can be computed is eligible, so the method can be transferred to earlier layers or other architectures.
Multi-concept channels beyond two concepts can be handled by recursive or repeated application, since the paper decomposes multi-concept polysemanticity into pairwise cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's own evidence, the residual channel's activation level could be used as a general diagnostic: if the residual stays small across many splits, the network's late-layer representations are closer to a privileged basis than superposition theorizing suggests; if it grows, the concepts are distributed and edge-masking alone cannot isolate them.
The same 'two concept channels plus a negative residual' wiring is a generic identity for decomposing any mixed signal into components plus a correction term, so the construction may transfer to transformers or other architectures where a separation criterion analogous to the ARV inequality can be defined.
Because polysemanticity is defined through class labels, concepts that appear in many classes or none, such as textures or shapes, may evade detection; an unsupervised variant based on clustering activation patterns, rather than classes, would test whether the class-based definition is essential or merely convenient.
The near-zero residual in the paper's experiments suggests a testable claim: for penultimate-layer channels at gamma=0.5, pairwise concept separation is almost complete, which could be checked by applying the procedure to all flagged channels and measuring the distribution of residual activations.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about arXiv:2504.12939. The core construction is a genuinely neat trick: insert a new layer before the last convolution, split the polysemantic channel into two concept channels plus a residual channel connected back with weight −1, and the identity c1 + c2 − residual recovers the original activation exactly. Because the split is done in weight space, you can run MACO or any feature visualization on the disentangled channels directly, which is something PURE's virtual disentanglement cannot do. The digital clock/cauliflower example is convincing, and the activation density plots in Fig. 3 show real separation between the two channels.

The genuinely new bit is the explicit, output-preserving rewrite. Sparse autoencoders approximate the network in a larger space; this edits the network so its outputs are unchanged by construction. The γ-polysemanticity criterion (low cosine similarity between average relevance vectors in the previous layer, validated against WordNet) is a reasonable, cheap heuristic. Related work is handled fairly — PURE, sparse autoencoders, CAVs, and concept whitening are all cited and clearly differentiated. Code is released, which for this kind of work matters.

The soft spots are in the evaluation, not the idea. Table 1 aggregates across channels with no error bars, no per-channel spread, no random-split baseline, and no comparison to PURE. The ρ threshold is fitted on training images by maximizing essentially the same activation-ratio objective the table reports on validation. That is not leakage — the split is honest, train versus validation — but it is a fitted hyperparameter, and without error bars the headline numbers overstate the generality. The claim that each disentangled channel 'only activates for one concept' is also stronger than what is shown; the paper itself concedes light cauliflower leakage in Fig. 1, which is exactly what you expect when the two concepts share or distribute over the same previous-layer channels. That separability condition is the real scope limit. The residual channel absorbs shared edges, but if the shared mass is large the concept channels still co-activate. I read the stress-test note as claiming this is unvalidated; I would soften that — it is under-validated, and mostly acknowledged in the conclusion. Two minor gaps: normalization layers are never discussed, and output preservation is argued by construction but never verified numerically on task accuracy, a cheap check that would settle it.

This is a workshop-scale proof of concept for CNN mechanistic interpretability. The construction is correct, the idea is new, and with error bars, a random-split baseline, and a PURE comparison it becomes a solid technique paper. I would send it to peer review and accept after revision.

Referee Report

4 major / 6 minor

Summary. The paper proposes a post-hoc method to identify and explicitly disentangle polysemantic channels in a pretrained CNN. A channel is flagged as γ-polysemantic when the average Input×Gradient relevance vectors (ARVs) from the previous layer differ between two classes (cosine similarity below γ, Definition 3.1). The disentanglement inserts an auxiliary layer l′ in which channel c is replaced by two concept channels plus a residual channel; the residual is connected to c with weight −1 so that c1 + c2 − residual linearly reconstructs the original channel's pre-activation. Experiments on the penultimate layer of an ImageNet-trained ResNet-50 show, for a case study (channel #1660, digital clock/cauliflower) and aggregated over polysemantic channels, that the two new channels activate almost exclusively for one class each while the residual stays near zero.

Significance. The contribution is genuinely useful if the claims hold. The explicit weight-space rewiring is a clean alternative to PURE's virtual disentanglement, and the c1 + c2 − residual construction is an elegant linear device that provably recovers the original pre-activation; the authors ship code and define γ-polysemanticity crisply. The case study is convincing: the activation densities in Fig. 3 and the MACO visualizations in Fig. 1 show clear separation, and the WordNet-similarity heuristic for choosing γ (Fig. 4) is a reasonable empirical calibration. Three caveats temper the result: the monosemanticity claim is only partial because shared edges remain in both channels (as the authors concede in the Fig. 1 discussion); Table 1 evaluates essentially the same objective used to select ρ; and the asserted output preservation is never measured empirically. None of these flaws is fatal, but the claims currently outrun the evidence.

major comments (4)

[Sec. 3.2, Ineq. (5), Table 1] The central claim that each disentangled channel 'only activates for one of the respective class concepts' (Sec. 3.2) is not guaranteed by Ineq. (5): edges for which neither ARV(l,c,t2)_i·ρ > ARV(l,c,t1)_i nor the symmetric condition holds are copied into both c1 and c2, so class-t2 signal continues to reach c1 through those shared edges. The manuscript itself concedes residual cross-concept signal in the Fig. 1 discussion ('some cauliflower patterns are still lightly visible'). Table 1 quantifies mean cross-activations of −9%/−21% but reports no error bars, no number of aggregated channels or class pairs, and no statistic on the mass of shared edges. I ask for per-channel distributions and confidence intervals, the number of channels and class pairs aggregated, a shared-edge mass or activation-purity metric, and a wording change from 'only activating for one concept' to a qualified claim such as 'mainly activating for one concept'.
[Appendix A, Eq. (6); Table 1] ρ is selected by maximizing, on training images, the ratio of |c′1| to |c′2| activations for the two classes (Eq. 6); Table 1 then reports essentially the same relative-activation quantities on validation images. The main quantitative evaluation therefore re-measures the selection objective, which limits its confirmatory value for the claim of effective disentanglement. I suggest adding at least one outcome that is not the selection objective, such as classification accuracy before and after rewiring, logit reconstruction error, or a MACO-based concept purity measured on channels held out from ρ selection, and a sensitivity analysis over the ρ grid.
[Sec. 3.2, output-preservation claim] The claim that rewiring 'does not affect the output or predictive performance' rests on pre-activation(c) = c1 + c2 − residual. This identity holds only if the inserted layer l′ is purely linear (no BatchNorm or ReLU) and if the two masked edge sets are disjoint, which is true for positive ARVs and ρ ≤ 1 but can fail with negative ARVs. The paper does not state whether l′ contains normalization or an activation, nor how the insertion interacts with the residual skip connections of ResNet-50, and no empirical output check is reported. Please specify the implementation of l′, report the reconstruction error of c1 + c2 − residual versus the original channel on the validation set (maximum and quantiles), and report top-1 accuracy or logit outputs before and after rewiring.
[Sec. 4] The experiments contain no baseline comparison. Since PURE [5] is the closest prior work and is discussed at length in Sec. 2, a direct comparison (e.g., per-concept activation purity) would substantiate the positioning of the explicit approach. In addition, a random edge-split baseline for Ineq. (5) would show whether the ARV-based masking, rather than the mere act of splitting the channel into two, is responsible for the observed separation. Without such comparisons, the claim that the proposed algorithm 'is effective' (Sec. 4) is supported only by the method's own numbers.

minor comments (6)

[Sec. 4; Appendix A] The text refers twice to 'Theorem 3.1' (in the qualitative analysis of Sec. 4 and in the quantitative analysis of Appendix A); Definition 3.1 is a definition, not a theorem.
[Table 1] Please report the number of polysemantic channels and class pairs aggregated in Table 1, and clarify how channels relevant to three or more classes are reduced to a two-concept disentanglement; the paper only states that this 'can be broken down' into two-concept cases without specifying the pairing rule.
[Appendix A, Eq. (6); Ineq. (5)] Please specify the discretization and range of the ρ grid used to solve Eq. (6), and report the selected ρ values; additionally, state whether ARVs are nonnegative in practice, since for negative attribution values the scaled comparison in Ineq. (5) changes its meaning and can even make the two masking sets overlap, breaking the exact reconstruction.
[Appendix A, quantitative analysis] The quantitative evaluation is restricted to validation images whose relative attribution exceeds τ, but this filter is described only in Appendix A; please state it in the main text, since Table 1 otherwise appears to average over all images of the two classes.
[Appendix A, hyperparameters] The paper states that τ, γ, and p are handpicked; given that the aggregate analysis depends on γ through Definition 3.1, a small sensitivity analysis over γ (and ideally τ and p) would help establish that the reported results are not brittle to these choices.
[Sec. 2; Ineqs. (2), (5)] Both channel identification (Ineq. 2) and edge assignment (Ineq. 5) rely on Input×Gradient attributions, and the paper states only that this choice 'works sufficiently well'; a small faithfulness sanity check (e.g., comparing ARVs from Input×Gradient with LRP or Integrated Gradients on the case-study channel) would increase confidence in the whole pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: residual recovery is an exact identity, rho is tuned on training and evaluated on validation, and MACO visualizations are independent.

full rationale

The paper's derivation chain is self-contained. The residual-channel construction is an exact algebraic identity: since the first two new channels keep the original l-1 to l weights except for edges assigned to the other concept, and the third keeps only shared edges, c1 + c2 - c3 equals the original channel activation by construction; this is a design guarantee, not a fitted prediction. Polysemanticity is introduced as an explicit definition (Definition 3.1, gamma-polysemanticity via ARV cosine similarity), and the disentanglement algorithm is a constructive procedure built from the same ARVs, so the method is internally consistent rather than circular. The only fitted hyperparameter, rho, is selected on the training split via Eq. (6) and evaluated on the validation split in Table 1; although the evaluation metric resembles the fitted separation objective, the held-out evaluation and the independent MACO feature visualizations (Fig. 1) provide non-tautological evidence. The paper itself acknowledges partial leakage ('some cauliflower patterns are still lightly visible' in Fig. 1) and notes that the class-label formulation 'may work better in later layers,' which are honest limitations, not circular steps. There are no load-bearing self-citations or imported uniqueness claims; references to prior work are external (e.g., superposition in [6]). Thus no step reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four free hyperparameters and four domain assumptions about attribution faithfulness, separability of concepts in layer l-1, and class labels as concept proxies. No new physical or ontological entities are introduced; the disentangled and residual channels are engineered rewirings of existing weights.

free parameters (4)

tau (relevance threshold) = 0.03
Handpicked in Appendix A; controls whether a channel is relevant for a class in Ineq. (2), thus determining the candidate channel set.
p (fraction threshold) = 0.75
Handpicked; a channel must be relevant for at least 75% of a class's images to count as relevant for that class.
gamma (polysemanticity threshold) = 0.5
Handpicked using the WordNet similarity plot in Fig. 4; defines which channels qualify as gamma-polysemantic under Definition 3.1.
rho (edge-masking ratio) = automatic per channel via Eq. (6)
Selected by maximizing class separation of disentangled activations on training images; the same style of separation is reported in Table 1, so part of the quantitative success is tuned rather than predicted.

assumptions (4)

domain assumption Input x Gradient is a faithful channel-level relevance attribution.
Eq. (1) defines A via Input x Gradient; if this attribution mis-identifies relevant earlier channels, the ARVs and the resulting masks are unreliable.
domain assumption Concepts within a polysemantic channel induce distinct activation patterns in the previous layer.
Sec. 3.1 assumes similar concepts give similar ARVs and that polysemantic concepts give dissimilar ARVs; this is the core detection premise.
domain assumption A previous-layer channel can be assigned to one of two concepts by an ARV ratio threshold.
Ineq. (5) classifies a channel i as belonging to concept t2 when ARV_t2 * rho > ARV_t1; if concepts are distributed or overlapping, this linear separability fails.
domain assumption Class labels are a sufficient proxy for discrete concepts.
Polysemanticity is defined relative to class pairs, and the authors note in Limitations that this may work better in later, more semantically meaningful layers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Disentangling Polysemantic Channels in Convolutional Neural Networks." pith.science (2026). https://pith.science/paper/LHG4XRQJ

@misc{pith2026250412939,
  author       = {Pith},
  title        = {Pith review of: Disentangling Polysemantic Channels in Convolutional Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHG4XRQJ}},
  note         = {Machine review of arXiv:2504.12939}
}

read the original abstract

Mechanistic interpretability is concerned with analyzing individual components in a (convolutional) neural network (CNN) and how they form larger circuits representing decision mechanisms. These investigations are challenging since CNNs frequently learn polysemantic channels that encode distinct concepts, making them hard to interpret. To address this, we propose an algorithm to disentangle a specific kind of polysemantic channel into multiple channels, each responding to a single concept. Our approach restructures weights in a CNN, utilizing that different concepts within the same channel exhibit distinct activation patterns in the previous layer. By disentangling these polysemantic features, we enhance the interpretability of CNNs, ultimately improving explanatory techniques such as feature visualizations.

Figures

Figures reproduced from arXiv: 2504.12939 by the authors.

**Figure 1.** Strongly activating image patches and MACO [8] visualizations for channel #1660 of an ImageNet-trained ResNet-50 and the two corresponding disentangled channels (see Sec. 3.2). The original channel #1660 is polysemantic as it strongly activates for unrelated concepts occurring in “digital clock” and “cauliflower” images. After disentangling, we obtain channels that are significantly stronger activating either to con… view at source ↗

**Figure 2.** Illustration of our proposed disentanglement approach. We show two input samples from two different classes (left), the original neural network (middle), and our disentangled neural network (right). The bar plots in each channel indicate how active that channel is for each of the two input images. The color of the edge indicates if that edge is mainly propagating information from the first image, from the second ima… view at source ↗

**Figure 3.** Density plot of the activations for the original channel #1660 and the corresponding disentangled channels for images from the classes “digital clock” and “cauliflower”. In both plots, one disentangled channel mimics the activation of the original channel #1660 while the other disentangled channel is mostly inactive, indicating that the disentanglement was successful. #1660 to obtain two channels that detect the cor… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Intuitively, similar classes that share concepts, such as different dog species, have a high WordNet similarity while semantically different classes have a low similarity. Thus, we can use the WordNet similarity as a proxy for semantic similarity, which we use as a pro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

[5]

PURE: Turning polysemantic neurons into pure features by identifying rele- vant circuits

Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wo- jciech Samek, and Sebastian Lapuschkin. PURE: Turning polysemantic neurons into pure features by identifying rele- vant circuits. In CVPR XAI4CV Workshop, pages 8212–8217,

work page
[1]

On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

Sebastian Bach, Alexander Binder, Gr´egoire Montavon, Fred- erick Klauschen, Klaus-Robert M¨uller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46,

work page
[2]

Concept whitening for interpretable image recognition

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nat. Mach. Intell., 2(12): 772–782, 2020. 2

work page 2020
[3]

ImageNet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 3, 6

work page 2009
[4]

Visual and semantic similarity in ImageNet

Thomas Deselaers and Vittorio Ferrari. Visual and semantic similarity in ImageNet. In CVPR, pages 1777–1784, 2011. 6

work page 2011
[6]

Toy models of superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wat- tenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. 1, 2

work page 2022
[7]

Bengio, Aaron Courville, and Pascal Vin- cent

Dumitru Erhan, Y . Bengio, Aaron Courville, and Pascal Vin- cent. Visualizing higher-layer features of a deep network. Technical Report, Universit´e de Montr´eal, 2009. 1, 4

work page 2009
[8]

Unlocking feature visualization for deep net- work with magnitude constrained optimization

Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, R´emi Cad `ene, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deep net- work with magnitude constrained optimization. In NeurIPS, pages 37813–37826, 2023. 1, 4

work page 2023

Show all 24 references

[9]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778, 2016. 3, 6

2016
[10]

Sparse autoencoders can interpret randomly ini- tialized transformers

Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. Sparse autoencoders can interpret randomly ini- tialized transformers. arXiv:2501.17727 [cs.LG], 2025. 2

2025
[11]

Sparse autoencoders find highly interpretable features in language models

Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In ICLR, 2024. 2

2024
[12]

Cai, James Wexler, Fernanda B

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Vi´egas, and Rory Sayres. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (TCA V). In ICML, pages 2673–2682, 2018. 1, 2

2018
[13]

Donald E. Knuth. Two notes on notation. The American Mathematical Monthly, 99(5):403–422, 1992. 2

1992
[14]

Compositional explanations of neurons

Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In NeurIPS, pages 17153–17163, 2020. 1, 2

2020
[15]

Multi- faceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Multi- faceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv:1602.03616 [cs.NE], 2016. 2

2016 arXiv
[16]

Oikarinen and Tsui-Wei Weng

Tuomas P. Oikarinen and Tsui-Wei Weng. Linear explanations for individual neurons. arXiv:2405.06855 [cs.LG], 2024. 1, 2

2024 arXiv
[17]

Feature visualization

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. 4

2017
[18]

Disentangling neuron representations with concept vectors

Laura O’Mahony, Vincent Andrearczyk, Henning M¨uller, and Mara Graziani. Disentangling neuron representations with concept vectors. In CVPRW, pages 3770–3775, 2023. 1, 2

2023
[19]

Automatic differentiation in PyTorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017. 6

2017
[20]

Sparse autoencoders trained on the same data learn different features

Gonc ¸alo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. arXiv:2501.16615 [cs.LG], 2025. 2

2025 arXiv
[21]

Towards a fuller understanding of neurons with clustered compositional explanations

Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco. Towards a fuller understanding of neurons with clustered compositional explanations. In NeurIPS, pages 70333–70354,
[22]

Learning important features through propagating activation differences

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, pages 3145–3153, 2017. 2

2017
[23]

Axiomatic attribution for deep networks

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pages 3319–3328,
[24]

Man- ning, and Christopher Potts

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Man- ning, and Christopher Potts. AxBench: Steering LLMs? Even simple baselines outperform sparse autoencoders. arXiv:2501.17148 [cs.CL], 2025. 2 A. Experimental Details For our e...

2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [5]

PURE: Turning polysemantic neurons into pure features by identifying rele- vant circuits

Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wo- jciech Samek, and Sebastian Lapuschkin. PURE: Turning polysemantic neurons into pure features by identifying rele- vant circuits. In CVPR XAI4CV Workshop, pages 8212–8217,

work page

[2] [1]

On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

Sebastian Bach, Alexander Binder, Gr´egoire Montavon, Fred- erick Klauschen, Klaus-Robert M¨uller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46,

work page

[3] [2]

Concept whitening for interpretable image recognition

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nat. Mach. Intell., 2(12): 772–782, 2020. 2

work page 2020

[4] [3]

ImageNet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 3, 6

work page 2009

[5] [4]

Visual and semantic similarity in ImageNet

Thomas Deselaers and Vittorio Ferrari. Visual and semantic similarity in ImageNet. In CVPR, pages 1777–1784, 2011. 6

work page 2011

[6] [6]

Toy models of superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wat- tenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. 1, 2

work page 2022

[7] [7]

Bengio, Aaron Courville, and Pascal Vin- cent

Dumitru Erhan, Y . Bengio, Aaron Courville, and Pascal Vin- cent. Visualizing higher-layer features of a deep network. Technical Report, Universit´e de Montr´eal, 2009. 1, 4

work page 2009

[8] [8]

Unlocking feature visualization for deep net- work with magnitude constrained optimization

Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, R´emi Cad `ene, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deep net- work with magnitude constrained optimization. In NeurIPS, pages 37813–37826, 2023. 1, 4

work page 2023