Pith. sign in

REVIEW 2 major objections 5 minor 24 references

Image classifiers routinely fire when a concept is missing, and standard explanations miss that signal until you look for it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 23:59 UTC pith:SE3JCAMA

load-bearing objection Clean causal framing of encoded absences plus two simple complementary XAI tools, backed by progressive experiments that actually change how you would run attributions and debiasing. the 2 major comments →

arxiv 2603.09787 v2 pith:SE3JCAMA submitted 2026-03-10 cs.CV cs.LG

What is Missing? Explaining Neurons Activated by Absent Concepts

Robin Hesse , Simone Schaub-Meyer , Janina Hesse , Bernt Schiele , Stefan Roth This is my paper
classification cs.CV cs.LG
keywords encoded absenceXAIattributionfeature visualizationdebiasingImageNetinhibitory signalsconcept absence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most explanation tools for vision models assume neurons light up because something is present in the image. This paper argues that many neurons also fire harder when a concept is absent, and that this “encoded absence” is a real causal relationship the model uses, especially when telling similar classes apart. Standard attribution maps and max-activation visualizations only show what is in the input or what pushes a neuron up, so they systematically hide the inhibitory side of the story. Two simple flips—attributing the target class on images that do not belong to it, and minimizing rather than maximizing a neuron—surface those missing concepts. With those tools the authors show that ImageNet networks encode absences of related species, and that debiasing a model for a spurious colorful patch works better when you suppress both the patch’s presence and its absence.

Core claim

Encoded absences—concepts whose presence lowers a neuron’s activation—are common in modern image classifiers; mainstream attribution and feature visualization miss them in their usual form, while non-target attribution and minimization-based visualization recover them, ImageNet models exploit them for fine-grained decisions, and debiasing improves when both presence and absence are constrained.

What carries the argument

Encoded absence (Definition 2.1): a concept whose presence decreases a neuron’s activation under an idealized intervention. Operationally recovered by non-target attribution (attributing class t on non-t images) and by feature visualization through minimization.

Load-bearing premise

Negative attributions on other-class images and least-activating patches are taken as reliable stand-ins for the ideal causal test that inserting a concept truly suppresses the neuron.

What would settle it

In a controlled setup where a hypothesized missing concept can be cleanly added or removed without out-of-distribution artifacts, show that the neuron’s activation does not drop when the concept is inserted, or that non-target negative attributions and least-activating patches fail to identify any such concept.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper formalizes encoded absences: a neuron encodes the absence of a concept if the concept’s presence decreases that neuron’s activation (Definition 2.1). It gives a mechanistic construction (negative weights plus positive potential; Proposition 2.2), argues that standard target attribution and maximization-based feature visualization miss this relationship, and proposes two complementary extensions—non-target attribution and feature visualization by minimization. Controlled experiments (hand-crafted Hassenstein–Reichardt detector; trained toy CNN), ImageNet-scale patch-insertion and channel analyses (VGG19, ResNet-50, ViT-B/16), and an ISIC-style debiasing study with presence vs. presence+absence attribution priors support the claims that absences are common, explainable by the proposed tools, used especially for fine-grained discrimination, and useful for debiasing.

Significance. If the results hold, the paper fills a genuine gap in mainstream vision XAI: explanations that only associate high activation with concept presence are incomplete, and absences can be both a discriminative signal and a source of bias. Strengths include a clean causal definition independent of the proposed methods, an explicit mechanistic existence argument, simple drop-in extensions rather than new architectures, multi-scale evidence from known-weight models through ImageNet backbones, controlled interventions in Appendix B.5 that align practical approximations with data-generation interventions, prediction-level flip rates under a suppression-only constraint, and released code. The debiasing result is practically relevant: presence-only attribution priors can leave absence-based shortcuts intact. The contribution is conceptual and methodological rather than a large new benchmark, but it is well scoped and falsifiable.

major comments (2)
  1. [§5.3, Figure 6] §5.3 and Figure 6: the claim that ImageNet channels encode absences of “closely related classes” rests on manual selection of three monosemantic patches from eight least-activating candidates, plus a post-hoc check that at least one patch is classified as a related class. Appendix Figure 11 shows the full sets are more polysemantic. A quantitative relatedness measure (e.g., class-label co-occurrence, embedding distance, or automatic concept labels over all least-activating patches, not a curated subset) is needed so the fine-grained-discrimination narrative is not driven by selection.
  2. [Definition 2.1, §4, Appendix B.5] Definition 2.1 vs. §4 methods: the central causal claim is an idealized do-intervention inequality, while non-target attributions and least-activating patches are practical approximations. Appendix B.5 and Table 7 provide the right controls and should be treated as load-bearing evidence in the main narrative (briefly in §4/§5.3), with main-text wording distinguishing “consistent with encoded absence under these interventions” from a fully identified do-effect on natural ImageNet images. Without that, residual OOD/gradient-noise concerns remain more prominent than the appendix warrants.
minor comments (5)
  1. [Table 1, §5.3] Table 1 ViT row: the absolute drop for least-activating patches is small (−0.14 → −0.24). Briefly justify the min-over-tokens aggregation and whether CLS-token or other pooling changes the conclusion.
  2. [§5.4, Table 2] §5.4 / Table 2: report how λ was chosen more explicitly in the main text (grid search on unbiased validation is in the appendix) and note that the experiment is binary with a synthetic patch bias, so multi-class natural-bias transfer remains open.
  3. [§5.1–5.2] Figure 3–4: activation values on patches are helpful; adding a short note that pre-activation (pre-ReLU) is used for minimization would avoid confusion with post-ReLU collapse discussed in related work.
  4. [§3] Related work: the distinction from Mu & Andreas (2020) logical NOTs is important and well made in §5.3; a one-sentence pointer in the main related-work section that IoU-based NOT is not causal inhibition would help readers who stop early.
  5. [Figures 3–4, §2.1] Typos/clarity: “resp.” abbreviations and occasional long sentences in §2.1; “QUBA Score” in Figures 3–4 is unexplained in the caption.

Circularity Check

0 steps flagged

No significant circularity: definition, constructive existence, method extensions, and external empirical checks are independent of one another.

full rationale

Definition 2.1 is a pure causal inequality on do-interventions and does not presuppose the proposed XAI modifications. Proposition 2.2 is an existence construction (negative weights plus positive potential) that is independent of any data fit. Non-target attribution and minimization-based feature visualization are algorithmic duals of standard methods, not redefinitions of the target phenomenon. All quantitative claims (Table 1, layer-wise ablations, prediction flips under suppression-only constraints, ISIC debiasing accuracies) are evaluated on external models (ImageNet VGG/ResNet/ViT, ISIC) or hand-crafted/toy networks whose weights are known a priori; controlled interventions in Appendix B.5 further align the practical approximations with the data-generating process. Self-citations (Hesse et al. 2021/2023/2025) supply only tooling or prior experimental setups and are not load-bearing for the existence or utility of encoded absences. No fitted parameter is renamed a prediction, no uniqueness theorem is imported from the authors, and no known empirical pattern is merely re-labeled. The derivation chain is therefore self-contained.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claim rests on a causal definition of encoded absence (do-intervention inequality), the modeling assumption that concepts can be approximated by patches or optimized patterns, and standard DNN/XAI machinery. Free parameters are experimental choices (patch size, λ, channel-importance threshold). No new physical entities are postulated; “encoded absence” is a named causal relationship, not an invented particle or force.

free parameters (4)
  • patch size for insertion experiments = 48×48 (default)
    48×48 (and ablations 32/64) chosen for quantitative Table 1; affects measured suppression magnitude.
  • attribution-prior strength λ = chosen per run for best unbiased performance
    Selected by grid search over {1,10,100,1000,10000} for best unbiased accuracy in debiasing.
  • channel importance threshold = 0.05
    Relative attribution ≥0.05 used to select channels important for a class in qualitative analysis.
  • number of most/least activating images/patches = 100 / 8 / 3
    100 images, 8 candidate patches, manual selection of 3 for figures; affects qualitative claims.
axioms (4)
  • ad hoc to paper A neuron encodes the absence of concept x̂ if inserting x̂ decreases its activation (Definition 2.1 do-intervention inequality).
    Core definition introduced in §2.1; operationalized via approximate interventions.
  • domain assumption Concepts are families of input patterns that share a common effect on a neuron; semantic labels are post-hoc.
    Stated in Appendix A.2; standard in concept-based XAI.
  • ad hoc to paper Negative non-target attributions and least-activating patterns approximate the idealized absence intervention sufficiently for the claims.
    Acknowledged as approximation in §4, §6, A.2; supported by B.5 controlled checks.
  • domain assumption Standard feed-forward DNN as structural causal model with deterministic structural assignments and zero noise.
    §2.1; common XAI causal framing.
invented entities (1)
  • encoded absence independent evidence
    purpose: Name and formalize the causal relationship where concept presence suppresses neuron activation, enabling systematic study and method design.
    New named construct; independent evidence is the empirical suppression effects and debiasing gains, not an external physical prediction.

pith-pipeline@v1.1.0-grok45 · 30457 in / 2915 out tokens · 26583 ms · 2026-07-14T23:59:54.776519+00:00 · methodology

0 comments
read the original abstract

Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 3 linked inside Pith

  1. [1]

    Egelhaaf, M., Borst, A., and Reichardt, W

    Medium (accessed: January 2026). Egelhaaf, M., Borst, A., and Reichardt, W. Computational structure of a biological motion-detection system as re- vealed by local detector analysis in the fly’s nervous sys- tem. Journal of the Optical Society of America , 6(7): 1070–1087,

    2026

  2. [2]

    Toy models of superpo- sition

    Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superpo- sition. https://transformer-circuits.pub/ 2022/toy_model/index.html,

    2022

  3. [3]

    Erhan, D., Bengio, Y ., Courville, A., and Vincent, P

    Trans- former Circuits Thread (accessed: January 2026). Erhan, D., Bengio, Y ., Courville, A., and Vincent, P. Visual- izing higher-layer features of a deep network. Technical Report, Universit´e de Montr´eal,

    2026

  4. [4]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv:1606.08415 [cs.LG],

    Pith/arXiv arXiv

  5. [5]

    Captum: A unified and generic model interpretability library for PyTorch

    Kokhlikyan, N., Miglani, V ., Martin, M., Wang, E., Al- sallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., and Reblitz-Richardson, O. Captum: A unified and generic model interpretability library for PyTorch. arXiv:2009.07896 [cs.LG],

    Pith/arXiv arXiv 2009

  6. [6]

    Fea- ture Visualization

    Olah, C., Mordvintsev, A., and Schubert, L. Fea- ture Visualization. https://distill.pub/2017/ feature-visualization/,

    2017

  7. [7]

    Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S

    Distill (ac- cessed: January 2026). Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduc- tion to circuits. https://distill.pub/2020/ circuits/zoom-in/,

    2026

  8. [8]

    O’Mahony, L., Andrearczyk, V ., M¨uller, H., and Graziani, M

    Distill (accessed: Jan- uary 2026). O’Mahony, L., Andrearczyk, V ., M¨uller, H., and Graziani, M. Disentangling neuron representations with concept vectors. In CVPR Workshop on Explainable AI for Com- puter Vision, pp. 3770–3775,

    2026

  9. [9]

    and AlRegib, G

    Prabhushankar, M. and AlRegib, G. Contrastive reasoning in neural networks. arXiv:2103.12329 [cs.CL],

    Pith/arXiv arXiv

  10. [10]

    Counterfactual-based saliency map: Towards visual contrastive explanations for neural networks

    Wang, X., Wang, Z., Weng, H., Guo, H., Zhang, Z., Jin, L., Wei, T., and Ren, K. Counterfactual-based saliency map: Towards visual contrastive explanations for neural networks. In ICCV, pp. 2042–2051,

    2042

  11. [11]

    Theoretical Elaborations The main text focuses on presenting our core theoretical insights

    12 Explaining Neurons Activated by Absent Concepts A. Theoretical Elaborations The main text focuses on presenting our core theoretical insights. Here, we provide additional elaborations to complement the main paper. A.1. Hassenstein-Reichardt detector In the visual system of the fruit fly Drosophila melanogaster (Borst & Groschner, 2023), lobula plate ta...

    2023

  12. [12]

    dog snouts

    (cf. Figure 8). Consequently, the output neuron of the Hassenstein–Reichardt detector encodes the presence and absence of two concepts alike (rightward, resp. leftward motion). A.2. Implicit assumptions and approximations For clarity and completeness, we make explicit several assumptions and approximations underlying our formulation and methods. These ass...

    2023

  13. [13]

    The activation of two subunits – encoding right-to-left (R) and left-to-right (L) movements – is subtracted

    Simplified illustration of the Hassenstein-Reichardt detector in Drosophila. The activation of two subunits – encoding right-to-left (R) and left-to-right (L) movements – is subtracted. The output neuron encodes the presence of left-to-right movements while encoding the absence of right-to-left movements (L ∧ ¬R). A.3. Feature visualization and counterfac...

    2023

  14. [14]

    The above mechanistic processes work for unnormalized and ReLU (Fukushima,

    The positive potential can not only be supplied via (a) using the activation of another concept ˜x in l − 1, but also via (b) a learned averaging over the previous layer (Hesse et al., 2021), or via (c) the bias term. The above mechanistic processes work for unnormalized and ReLU (Fukushima,

    2021

  15. [15]

    and GeLU (Hendrycks & Gimpel, 2016), can be seen as smooth variants of ReLU, and thus, are capable of implementing the same mechanisms for encoded absences as outlined for ReLU activation functions. Since self-attention layers in Transformer architectures can be viewed as a generalization of linear layers, Vision Transformers can encode absences through s...

    2016

  16. [16]

    neurons” with “feature space directions

    Three mechanistic processes to encode the absence of a feature. A neuron encoding the absence of concept ˆx (i.e., ¬ˆx) can be implemented by having a negative connection to a neuron encoding ˆx and a positive potential through (a) another activating concept ˜x, (b) some form of averaging, or (c) the bias. So far, for simplicity, we have assumed that conc...

    2022

  17. [17]

    For each channel/neuron in the last 16 Explaining Neurons Activated by Absent Concepts Table

    torchvision models (VGG19 (Simonyan & Zisserman, 2015), ResNet-50 (He et al., 2016), ViT-B/16 (Dosovitskiy et al., 2021)). For each channel/neuron in the last 16 Explaining Neurons Activated by Absent Concepts Table

    2015

  18. [18]

    As we found the CLS token of the ViT to be largely insensitive to local patch insertions, we instead use a localized measure based on token-level activations

    32 100 0.18 ± 0.06 0.17 ± 0.06 0.16 ± 0.06 → 0.16± 0.06 0.21 ± 0.06 0.12 ± 0.07 ResNet-50 48 100 0.18 ± 0.06 0.16 ± 0.06 0.15 ± 0.06 → 0.15± 0.06 0.25 ± 0.07 0.03 ± 0.11 ResNet-50 64 100 0.18 ± 0.06 0.14 ± 0.07 0.13 ± 0.07 → 0.14± 0.07 0.29 ± 0.08 -0.09 ± 0.11 ResNet-50 48 50 0.21 ± 0.05 0.19 ± 0.06 0.15 ± 0.06 → 0.15± 0.06 0.29 ± 0.07 0.06 ± 0.11 ResNet-...

    2020

  19. [19]

    We compute the average channel/neuron activation (after GAP/min) across all modified images and all channels/neurons

    For identifying logical NOTs from (Mu & Andreas, 2020), we use the default hyperparameters with the only exception of reducing the beam search limit to 50, which was recommended by the authors for getting good explanations in a reasonable time. We compute the average channel/neuron activation (after GAP/min) across all modified images and all channels/neu...

    2020

  20. [20]

    + Least act

    Across all layers, we observe evidence of suppressive signals/encoded absences, as indicated by lower scores for “+ Least act.” compared to the random baseline. However, this effect varies across layers: in the first and last blocks, the difference is pronounced (approximately factors of 3 and 5, respectively), while in intermediate blocks it is notably s...

    2036

  21. [21]

    eft” (amphibian), for channel 1470 as “German shepherd,

    as was discussed as a limitation in the main paper. To further validate that these minimally activating patches carry meaningful semantics from the model’s perspective, and are not merely an artifact, we classify each patch using the same ResNet-50 under inspection. In all three groups, at least one minimally activating patch is assigned to a semantically...

    2026

  22. [22]

    The eight least activating patches for channels 2026 and

    2026

  23. [23]

    For the most and least activating patches in Section 5.3, we obtain eight candidate patches and manually select a monosemantic subset of three patches for more interpretable visualizations. Inspecting all eight patches for channels 2026 and 1470 in ResNet-50 reveals that these channels encode the absence of multiple concepts, consistent with prior work on...

    2026

  24. [24]

    We finetune each model with a binary cross-entropy loss, using an Adam optimizer (Kingma & Ba,

    is pre-trained on ImageNet-1k (Russakovsky et al., 2015), with weights obtained from (Hesse et al., 2021, Apache-2.0 license). We finetune each model with a binary cross-entropy loss, using an Adam optimizer (Kingma & Ba,

    2015