Pith. sign in

REVIEW 4 major objections 6 minor 96 references

Towards Utilising a Range of Neural Activations for Comprehending Representational Associations

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read This paper claims that the samples a classifier is least confident about—those with near-zero logits—concentrate the examples that break spurious correlations, and that retraining the classifier's last layer on a clustered subset of those…

desk verdict A promising no-group-label method for finding spurious correlations via mid-level logits, with strong WGA gains but an unmeasured core mechanism and a human-in-the-loop cluster selection that deserves scrutiny. read the letter →

arxiv 2411.10019 v1 pith:WUXFKXDT submitted 2024-11-15 cs.CV cs.LG

classification cs.CVcs.LG MSC 68T0768T45
keywords mid-levelactivationsspuriouscorrelationsworst-groupaccuracyinterpretabilitylast-layerretrainingmid-rangelogitsconceptentanglementlogitinterceptdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Interpretability research has mostly asked which inputs maximally activate a neuron or concept direction, implicitly treating activation as binary. This paper argues that the unexamined middle of the activation range carries the signal that matters: samples with near-zero logits for a class are disproportionately either in-class examples that lack the spurious attribute the model relies on, or out-of-class examples that contain it. The authors turn this observation into a practical method, MID (Modifying with logit Intercept Data): select mid-range logit samples, filter mislabels, cluster their frozen penultimate embeddings, identify the cluster whose poor classification is inappropriate (a spurious pattern), and retrain only the classification layer on that cluster. On the Waterbirds and CelebA benchmarks this lifts worst-group accuracy from 68.4 and 47.8 under standard training to 87.8 and 85.5, matching or beating methods that require group labels. The paper's broader claim is that non-extremal activations are a general resource for locating entangled and confounding concepts in representations.

What carries the argument

The load-bearing object is the 'intercept data': the set of training samples whose logit for a chosen class lies near zero, where the model's prediction confidence is lowest. The paper argues that this band is enriched for two kinds of examples that break the spurious trend—in-class samples lacking the spurious attribute and out-of-class samples carrying it—and that k-means clustering of their penultimate-layer embeddings separates a coherent group whose poor accuracy is caused by the spurious association rather than by benign ambiguity. Carrying the argument further, the last layer of the ERM model is replaced and retrained by $\ell^1$-regularised logistic regression on this selected cluster, exploiting the finding that ERM still learns core features and only the final mapping needs correction.

What would settle it

Measure, on CelebA or Waterbirds, the density of minority-group samples (blonde males, or water birds on land backgrounds) inside the near-zero-logit band selected by MID; if it is no higher than the base rate in the training set, the mid-range selection does not target spurious correlations and the method's mechanism is wrong. A second check: retrain the last layer on a randomly chosen cluster instead of the cluster judged to encode the spurious pattern—the paper's own ablation shows random-cluster retraining does not reliably improve worst-group accuracy, so a reader could verify this holds across seeds and datasets.

Watch

Extended reading notes

Core claim

The paper's central claim is that intermediate representations encode information at every level of stimulation, not just at the extremes, and that the mid-range of output-neuron activations is where a model's learned associations between class labels and confounding attributes become visible. Concretely, when images are ranked by their logit for a class, the maximally activating examples are typical of the majority spurious trend, while examples near the zero-logit intercept include shapes in a spurious location for the wrong class, low-spuriosity instances of the class, and mislabels—associations that maximum-activation analysis hides. MID operationalises this: for each class it keeps the near-zero logit samples, removes disputed labels using a vision-language model, clusters the encoder embeddings with k-means, and selects the cluster whose poor performance cannot be excused by image quality or labelling ambiguity (for CelebA, blonde faces with masculine features). Retraining the final layer on that cluster with $\ell^1$-regularised logistic regression, on top of the frozen feature extractor, improves worst-group accuracy to 87.8 on Waterbirds and 85.5 on CelebA without access to spurious-attribute labels for training or validation. The same logic, the paper argues, extends beyond output neurons to any concept direction in latent space found by methods such as TCAV or SVD-based decomposition.

Load-bearing premise

The method assumes that near-zero-logit samples are enriched for examples that break the spurious trend rather than dominated by mislabels, label noise, or low-quality images, and that this enrichment, demonstrated on synthetic DSpritesUnfair data, transfers to CelebA and Waterbirds.

Editorial extensions

If this is right

  • Maximal-activation interpretability is incomplete: analyses that only inspect the highest-activating examples will systematically miss the associations and confounds a model encodes at mid-range stimulation.
  • Last-layer retraining on automatically selected mid-logit clusters can match or approach the worst-group accuracy of methods that require group labels, while needing none for training or validation.
  • The pipeline is cheap: it reuses the frozen penultimate features of a standard ERM model and only re-fits the final layer with logistic regression.
  • The same mid-range inspection applies to any concept direction in latent space, not just output neurons, so confounding concepts found by concept-vector methods could be audited and mitigated the same way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test: iterate MID—retrain, re-select the mid-range band, and re-cluster—to see whether repeated rounds keep removing spurious reliance or converge to a stable selection.
  • The mid-logit band is a free, unsupervised proxy for the samples a model has memorised against its own trend; the same filter could be reused to flag out-of-distribution inputs or label noise in other benchmarks, since the paper shows it captures 55.1% of the ERM model's errors on CelebA.
  • Because the cluster-interpretation step is the only human-in-the-loop part, automating it with a vision-language model (which the paper mentions but does not evaluate) is a direct testable extension.
  • The near-zero-logit heuristic suggests a broader principle: for any neuron or direction, the decision boundary in activation space may be where entangled concepts live, connecting this work to geometric accounts of polysemanticity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that interpretability analyses based on maximally activating examples miss information encoded at non-maximal activation levels, and proposes a practical method, MID (Modifying with logit Intercept Data). MID first trains a standard ERM classifier, then selects samples whose class logits are near zero (the logit 'intercept'), filters suspected mislabels with BLIP, clusters the penultimate-layer embeddings of the remaining samples with k-means, manually selects a cluster that exhibits a harmful spurious pattern, and retrains the classification layer using DFR-style logistic regression. On Waterbirds and CelebA, MID reports worst-group accuracies of 87.8 and 85.5, respectively, compared with ERM baselines of 68.4 and 47.8, and is competitive with methods that use group labels while requiring none.

Significance. If the mechanism claimed by the paper holds, this is a valuable bridge between interpretability and robustness: it would show that non-maximal activations can expose confounds without group labels, and the proposed last-layer retraining is simple and cheap. The paper is honest about the human-in-the-loop nature of the method and provides ablations suggesting that both the mid-logit filter and the choice of cluster matter. The reported WGA gains are large, and the use of DFR-style retraining is well motivated by prior work. However, the central enrichment assumption -- that near-zero logit samples are enriched for low-spuriosity and counter-spurious examples -- is only qualitatively demonstrated on a synthetic dataset and never directly measured on the real benchmarks, so the significance of the empirical results is conditional on that assumption being validated.

major comments (4)
  1. [Sec. 4.2, Fig. 3 and Sec. 5.1] The core claim that near-zero class logits are enriched for low-spuriosity and counter-spurious examples is supported only by a qualitative inspection of six images on DSpritesUnfair (bias 0.4). No quantitative enrichment statistic is reported for Waterbirds or CelebA, and Fig. 10 shows the group composition only of the final cluster, which is already the product of human selection of k and of the 'spurious' cluster. Because the Table 1 gains could in principle arise from DFR-style retraining on any hard or balanced subset rather than from a property specific to mid-level activations, the paper should measure the group composition of the mid-logit selection before clustering (e.g., the fraction of minority-group examples among selected samples versus random samples matched by logit rank) and report this for both benchmarks.
  2. [Sec. 5.1 and Appendix A.6] The cluster selection step is not prespecified: k is increased 'until a pattern emerges', and the cluster used for retraining is chosen by manual inspection of cluster contents. For CelebA this selects the cluster corresponding to the known spurious attribute (blonde with masculine-associated features). This introduces human selection based on prior knowledge of each benchmark's known bias and undercuts the claim of operating without group information. Please specify a preselectable or automated rule for choosing k and the retraining cluster, and report sensitivity to both choices (e.g., WGA for every cluster and every k tried).
  3. [Appendix A.7] The ablation of the mid-logit filter is qualitative only. The paper states that clustering the full CelebA embeddings did not reveal a spurious pattern and that retraining on one ambiguous cluster does not help WGA, but no quantitative worst-group accuracies are given for these ablations. A quantitative comparison of MID against (a) retraining on a random balanced subset, (b) retraining on BLIP-filtered mid-logit data without clustering, and (c) retraining on each individual k-means cluster would isolate the contribution of the mid-logit selection and the cluster choice. This is needed to support the claim that the mid-level activation lens, rather than the last-layer retraining mechanism, is responsible for the improvements.
  4. [Sec. 5.1 and Appendix A.5] The mid-range selection rule is underspecified. For CelebA the paper reports 2,000 logit-intercept points per class, but for Waterbirds no analogous count or threshold is given, and the exact definition of 'near-zero logit' is not operationalized (e.g., a window around zero, a fixed number per class, or a sign-change region). Without a precise selection rule the method cannot be reproduced or compared across datasets; please specify the selection criterion and report the number of selected points for both benchmarks.
minor comments (6)
  1. [Table 1] The entry 'DFRTrTr' is not defined in the table or caption; the text in Sec. 6.3 mentions 'DFR T r T r' but it should be spelled out as a specific DFR variant.
  2. [Sec. 6.3 and Table 1] No variance or number of seeds is reported for the Waterbirds and CelebA ERM and MID runs, despite the DSprites pilot reporting means over five seeds. Since Table 1 is the main quantitative claim, please include standard deviations or error bars over multiple initializations.
  3. [Appendix A.6] The sentence 'we train on the available validation data with additional randomly selected training set data' is ambiguous and appears to contradict Sec. 6.4's statement that group information is used only for evaluation. Please clarify whether unlabeled validation data are used during retraining and how this affects the comparison with baselines that use group-balanced validation sets.
  4. [Sec. 7] The claim that points 'right above' the zero logit are more likely to be low-spuriosity memorized points while points 'right below' are more likely to be counter-spurious examples is presented as a finding but is not measured anywhere in the paper; it should be framed as a hypothesis or supported with data.
  5. [Table 2 and References] There is a LaTeX artifact in Table 2 ('L n(L)'), and several reference formatting issues such as 'Hermann et Lampinen' and 'TCA V' with inconsistent spacing; these should be cleaned up.
  6. [Figs. 3 and 6] The example images in Figs. 3 and 6 are very small, making the qualitative claims about position and shape difficult to verify; larger crops or additional annotations would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MID's WGA improvements are measured on held-out benchmark test sets, and the paper's assumptions, while empirically under-supported, are not definitionally tied to its outputs.

full rationale

The paper's central claim is empirical: mid-range logit samples, after k-means clustering and human selection of a 'spurious' cluster, can be used to retrain the classification layer and improve worst-group accuracy on Waterbirds and CelebA. These improvements (Table 1) are evaluated on external held-out test sets with group labels used only for evaluation, so they do not reduce by construction to the selection procedure. The key assumption that near-zero logits are enriched for low-spuriosity and counter-spurious samples is supported only qualitatively on DSpritesUnfair and is not directly measured on the real benchmarks; this is a validity and evidential weakness, not a circularity. The human-in-the-loop choice of k and cluster, and the tuning of the regularization strength on a split of the selected retraining data, are explicitly disclosed (Sec. 5.1, Sec. 6.3, Appendix A.6) and do not constitute a fitted parameter being renamed as a prediction. The last-layer retraining step is borrowed from DFR [39] with citation, and the paper does not claim that step as a novel derivation. Self-citations in the related work are not load-bearing for the main result. No equation or definition in the paper makes the reported WGA equal to an input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The main load-bearing elements are the free parameters of the data-selection and clustering pipeline (number of mid-logit samples, number of clusters, manual cluster choice, regularization strength) and the domain assumptions about last-layer retraining and the informativeness of mid-level logits. These assumptions are empirically motivated but not proven.

free parameters (4)
  • Mid-range logit count per class = 2000 for CelebA; unspecified for Waterbirds
    The number of samples taken near the zero-logit intercept per class is set by the user; the paper uses 2000 for CelebA and does not report the value for Waterbirds, so the threshold is a free parameter.
  • Number of clusters k = 3 for CelebA (k=2..4 explored)
    The paper starts at k=2 and increases until a pattern emerges; k=3 is selected for CelebA. This hand-chosen parameter affects which grouping is later used for retraining.
  • Spurious cluster selection = Cluster judged to reflect an unacceptable spurious pattern (e.g., masculine-featured blondes for CelebA)
    After k-means, a human inspects cluster contents and decides which cluster to use for retraining. This choice is not determined by a fixed rule and depends on the dataset and human judgment.
  • Regularization strength of logistic regression = 1.0 from {1.0, 0.7, 0.3, 0.1, 0.07, 0.03, 0.01} for both datasets
    Tuned on one half of the selected retraining data, so it is fit to data rather than set a priori.
assumptions (4)
  • domain assumption Last-layer retraining is sufficient to reduce spurious feature reliance
    The method assumes that the frozen penultimate-layer embeddings retain core features, so retraining only the classification layer can drop spurious features. This is borrowed from Kirichenko et al. (DFR) and Izmailov et al. and is invoked in Sec 5.1 and Step 4.
  • domain assumption BLIP VQA can reliably identify mislabeled or ambiguous hair-color images
    The filter step uses BLIP in a VQA setting to remove disputed labels; the paper assumes the model's answers are accurate enough for this curation, which is not independently verified.
  • ad hoc to paper Near-zero logit intercept sets are enriched for low-spuriosity and counter-spurious examples
    The central hypothesis of the paper. It is demonstrated empirically on one synthetic setting (DSpritesUnfair, bias 0.4) and assumed to transfer to CelebA and Waterbirds without a formal justification.
  • domain assumption Frozen penultimate-layer embeddings preserve core features necessary for last-layer retraining
    The paper relies on the prior finding that ERM models learn core features despite relying on spurious shortcuts, so that the last layer alone can be retrained. This is cited from Kirichenko et al. and Izmailov et al.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Utilising a Range of Neural Activations for Comprehending Representational Associations." pith.science (2026). https://pith.science/paper/WUXFKXDT

@misc{pith2026241110019,
  author       = {Pith},
  title        = {Pith review of: Towards Utilising a Range of Neural Activations for Comprehending Representational Associations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUXFKXDT}},
  note         = {Machine review of arXiv:2411.10019}
}
read the original abstract

Recent efforts to understand intermediate representations in deep neural networks have commonly attempted to label individual neurons and combinations of neurons that make up linear directions in the latent space by examining extremal neuron activations and the highest direction projections. In this paper, we show that this approach, although yielding a good approximation for many purposes, fails to capture valuable information about the behaviour of a representation. Neural network activations are generally dense, and so a more complex, but realistic scenario is that linear directions encode information at various levels of stimulation. We hypothesise that non-extremal level activations contain complex information worth investigating, such as statistical associations, and thus may be used to locate confounding human interpretable concepts. We explore the value of studying a range of neuron activations by taking the case of mid-level output neuron activations and demonstrate on a synthetic dataset how they can inform us about aspects of representations in the penultimate layer not evident through analysing maximal activations alone. We use our findings to develop a method to curate data from mid-range logit samples for retraining to mitigate spurious correlations, or confounding concepts in the penultimate layer, on real benchmark datasets. The success of our method exemplifies the utility of inspecting non-maximal activations to extract complex relationships learned by models.

Figures

Figures reproduced from arXiv: 2411.10019 by the authors.

Figure 1
Figure 1. An illustration of the spurious correlation data analysis and automatic selection for retraining to mitigate the found spurious bias [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. (a) Maximally activating examples for the neuron corre [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. UMAP projections for encoder embeddings of (a) all [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Representational similarity matrices plotted for encoders trained with varying levels of bias (below each image). The order of [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: (a) Maximally activating examples for the neuron corre [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Logits in descending order for the class non-blonde (in [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Middle logit images with y labels on top disputed by BLIP labels shown below each image. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Group composition of k-means clusters for [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Group composition of k-means clustering on the fil [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Group composition of k-means clustering on the entire [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 65 canonical work pages

  1. [1]

    Understanding inter- mediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding inter- mediate layers using linear classifier probes. InInternational conference on learning representations, 2017. 3

  2. [2]

    Network dissection: Quantifying inter- pretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 6541–6549, 2017. 1, 3

  3. [3]

    Gan dissection: Visualizing and understanding gener- ative adversarial networks

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Tor- ralba. Gan dissection: Visualizing and understanding gener- ative adversarial networks. In International Conference on Learning Representations, 2019. 1

  4. [4]

    Rep- resentation learning: A review and new perspectives

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Rep- resentation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798–1828, 2013. 1, 3

  5. [5]

    An interpretability illusion for bert

    Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Co- enen, Emily Reif, Fernanda Vi ´egas, and Martin Watten- berg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. 3

  6. [6]

    Towards monose- manticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. 3

  7. [7]

    Gender shades: Inter- sectional accuracy disparities in commercial gender classifi- cation

    Joy Buolamwini and Timnit Gebru. Gender shades: Inter- sectional accuracy disparities in commercial gender classifi- cation. In Conference on fairness, accountability and trans- parency, pages 77–91. PMLR, 2018. 3

  8. [8]

    Labeling neural representations with in- verse recognition

    Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina H¨ohne. Labeling neural representations with in- verse recognition. Advances in Neural Information Process- ing Systems, 36, 2023. 1

Show all 96 references
  1. [9]

    Isolating sources of disentanglement in varia- tional autoencoders

    Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in varia- tional autoencoders. Advances in neural information pro- cessing systems, 31, 2018. 3

  2. [10]

    Fair prediction with disparate im- pact: A study of bias in recidivism prediction instruments

    Alexandra Chouldechova. Fair prediction with disparate im- pact: A study of bias in recidivism prediction instruments. Big data, 5(2):153–163, 2017. 3

  3. [11]

    Independent component analysis, a new con- cept? Signal processing, 36(3):287–314, 1994

    Pierre Comon. Independent component analysis, a new con- cept? Signal processing, 36(3):287–314, 1994. 1, 3

  4. [12]

    Flexibly fair representation learning by disentan- glement

    Elliot Creager, David Madras, J ¨orn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentan- glement. In International conference on machine learning , pages 1436–1445. PMLR, 2019. 4

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6

  6. [14]

    Toy- modelsof superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy- modelsof superposition. Transformer Circuits Thread, 2022. 3

  7. [15]

    Visualizing higher-layer features of a deep network

    Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009. 1

  8. [16]

    A holistic approach to unifying automatic concept extraction and concept importance estimation

    Thomas Fel, Victor Boutin, Louis B ´ethune, R ´emi Cad `ene, Mazda Moayeri, L ´eo And ´eol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Ad- vances in Neural Information Processing Systems,...

  9. [17]

    Craft: Concept recursive activation factoriza- tion for explainability

    Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, R ´emi Cad`ene, and Thomas Serre. Craft: Concept recursive activation factoriza- tion for explainability. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

  10. [18]

    Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neu- ral networks

    Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neu- ral networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730–8738,

  11. [19]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr ´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. 3

  12. [20]

    Shortcut learning in deep neural networks

    Robert Geirhos, J ¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Fe- lix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. 2, 3

  13. [21]

    Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In Interna- tional Conference on Learning Representations, 2019. 3, 13

  14. [22]

    Don’t trust your eyes: on the (un) reliability of feature visualizations

    Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don’t trust your eyes: on the (un) reliability of feature visualizations. In Forty-first In- ternational Conference on Machine Learning, 2024. 3

  15. [23]

    Concept discovery and dataset exploration with singular value decomposition

    Mara Graziani, An-phi Nguyen, Laura O’Mahony, Henning M¨uller, and Vincent Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and compu- tation for Trustworthy ML, 2023. 3

  16. [24]

    Uncovering unique con- cept vectors through latent space decomposition

    Mara Graziani, Laura O’Mahony, An-phi Nguyen, Henning M¨uller, and Vincent Andrearczyk. Uncovering unique con- cept vectors through latent space decomposition. Transac- tions on Machine Learning Research, 2023. 1, 3, 8

  17. [25]

    Fairness without demo- graphics in repeated loss minimization

    Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning , pages 1929–1938. PMLR, 2018. 3

  18. [26]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6 9

  19. [27]

    A baseline for detect- ing misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Repre- sentations, 2017. 3, 13

  20. [28]

    The origins and prevalence of texture bias in convolutional neu- ral networks

    Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neu- ral networks. Advances in Neural Information Processing Systems, 33:19000–19015, 2020. 3

  21. [29]

    What shapes feature representations? exploring datasets, architectures, and training

    Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995–10006, 2020. 2, 3, 13

  22. [30]

    beta-vae: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017. 4

  23. [31]

    Disentanglement via latent quantiza- tion

    Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantiza- tion. Advances in Neural Information Processing Systems , 36, 2023. 3

  24. [32]

    Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037

    Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037. PMLR, 2018. 3

  25. [33]

    Independent component analysis: algorithms and applications

    Aapo Hyv ¨arinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411–430, 2000. 3

  26. [34]

    Simple data balancing achieves com- petitive worst-group-accuracy

    Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves com- petitive worst-group-accuracy. In Conference on Causal Learning and Reasoning, pages 336–351. PMLR, 2022. 3, 6

  27. [35]

    Shape or texture: Understanding discriminative features in cnns

    Md Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Bj¨orn Ommer, Konstantinos G Derpanis, and Neil Bruce. Shape or texture: Understanding discriminative features in cnns. In International Conference on Learning Representa- tions, 2021. 3

  28. [36]

    On feature learning in the presence of spuri- ous correlations

    Pavel Izmailov, Polina Kirichenko, Nate Gruver, and An- drew G Wilson. On feature learning in the presence of spuri- ous correlations. Advances in Neural Information Processing Systems, 35:38516–38532, 2022. 3, 6, 7, 13

  29. [37]

    Removing spurious fea- tures can hurt accuracy and affect groups disproportionately

    Fereshte Khani and Percy Liang. Removing spurious fea- tures can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM conference on fairness, ac- countability, and transparency, pages 196–205, 2021. 3

  30. [38]

    Interpretability be- yond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability be- yond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on ma- chine learning, pages 2668–2677. PMLR, 2...

  31. [39]

    Last layer re-training is sufficient for robustness to spu- rious correlations

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wil- son. Last layer re-training is sufficient for robustness to spu- rious correlations. In The Eleventh International Conference on Learning Representations, 2023. 3, 6, 7, 13

  32. [40]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. 8

  33. [41]

    Towards a fuller understanding of neurons with clustered compositional explanations

    Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco. Towards a fuller understanding of neurons with clustered compositional explanations. Advances in Neural Informa- tion Processing Systems, 36:70333–70354, 2023. 1, 3

  34. [42]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 6

  35. [43]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghu- nathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Confer- ence on Machine Learning, pages 6781–6792. PMLR,...

  36. [44]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. 2, 6

  37. [45]

    Challenging common assumptions in the unsuper- vised learning of disentangled representations

    Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Sch ¨olkopf, and Olivier Bachem. Challenging common assumptions in the unsuper- vised learning of disentangled representations. In Interna- tional conference on machine learning , pages 411...

  38. [46]

    Weakly-supervised disentanglement without compromises

    Francesco Locatello, Ben Poole, Gunnar R ¨atsch, Bern- hard Sch ¨olkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International conference on machine learning , pages 6348–6359. PMLR, 2020. 4

  39. [47]

    Understanding and mitigating human-labelling er- rors in supervised contrastive learning

    Zijun Long, Lipeng Zhuang, George Killick, Richard Mc- Creadie, Gerardo Aragon Camarasa, and Paul Hender- son. Understanding and mitigating human-labelling er- rors in supervised contrastive learning. arXiv preprint arXiv:2403.06289, 2024. 6

  40. [48]

    On interpretability of deep learning based skin lesion classifiers using concept activation vectors

    Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. In 2020 international joint conference on neural networ...

  41. [49]

    Understanding deep image representations by inverting them

    Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196, 2015. 1

  42. [50]

    Promises and pitfalls of black-box concept learning models

    Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314,

  43. [51]

    Is this the subspace you are looking for? an inter- pretability illusion for subspace activation patching

    Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? an inter- pretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representa- tions, 2023. 3

  44. [52]

    Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021

    Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021. 8 10

  45. [53]

    Catastrophic inter- ference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic inter- ference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , vol- ume 24, pages 109–165. Elsevier, 1989. 13

  46. [54]

    Acquisition of chess knowledge in alphazero

    Thomas McGrath, Andrei Kapishnikov, Nenad Toma ˇsev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022. 3

  47. [55]

    Umap: Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018. 7

  48. [56]

    Linguis- tic regularities in continuous space word representations

    Tom ´aˇs Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguis- tic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Hu- man language technologies, pages 746...

  49. [57]

    A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes

    Mazda Moayeri, Phillip Pope, Yogesh Balaji, and So- heil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1908...

  50. [58]

    Spuriosity rankings: sorting data to measure and miti- gate biases

    Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi. Spuriosity rankings: sorting data to measure and miti- gate biases. Advances in Neural Information Processing Sys- tems, 36:41572–41600, 2023. 3

  51. [59]

    Compositional explanations of neurons

    Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Sys- tems, 33:17153–17163, 2020. 1

  52. [60]

    Beyond distribution shift: Spurious features through the lens of training dynamics

    Nihal Murali, Aahlad Manas Puli, Ke Yu, Rajesh Ranganath, et al. Beyond distribution shift: Spurious features through the lens of training dynamics. Transactions on Machine Learning Research, 2023. 2, 3, 13, 14

  53. [61]

    Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation

    Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Confer- ence on Learning Representations, 2022. 3, 6

  54. [62]

    Explaining explainability: Understanding concept activation vectors

    Angus Nicolson, Lisa Schut, J Alison Noble, and Yarin Gal. Explaining explainability: Understanding concept activation vectors. arXiv preprint arXiv:2404.03713, 2024. 3, 8

  55. [63]

    Zoom in: An in- troduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An in- troduction to circuits. Distill, 5(3):e00024–001, 2020. 1, 3

  56. [64]

    Feature visualization

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017. 1, 3

  57. [65]

    Disentangling neuron representations with concept vectors

    Laura O’Mahony, Vincent Andrearczyk, Henning M ¨uller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3769–3774, 2023. 1

  58. [66]

    The linear rep- resentation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear rep- resentation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024. 3

  59. [67]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

  60. [68]

    Scikit-learn: Machine learning in python

    Fabian Pedregosa, Ga ¨el Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011. 15

  61. [69]

    Identifying mislabeled data using the area under the margin ranking

    Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Pro- cessing Systems, 33:17044–17056, 2020. 6

  62. [70]

    Why Should I Trust You?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY , USA,

  63. [71]

    Parallel distributed processing, volume 1: Ex- plorations in the microstructure of cognition: Foundations

    David E Rumelhart, James L McClelland, PDP Research Group, et al. Parallel distributed processing, volume 1: Ex- plorations in the microstructure of cognition: Foundations . The MIT press, 1986. 3

  64. [72]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 6

  65. [73]

    Distributionally robust neural networks

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In In- ternational Conference on Learning Representations , 2020. 3

  66. [74]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 ,

  67. [75]

    A mathematical theory of semantic development in deep neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):11537–11546, 2019. 13

  68. [76]

    The pitfalls of simplicity bias in neural networks

    Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Pro- cessing Systems, 33:9573–9585, 2020. 3

  69. [77]

    Weakly supervised disentanglement with guarantees

    Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Rep- resentations, 2020. 4

  70. [78]

    Opening the black box of deep neural networks via information

    Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017. 13

  71. [79]

    On the proper treatment of connectionism

    Paul Smolensky. On the proper treatment of connectionism. Behavioral and brain sciences, 11(1):1–23, 1988. 3

  72. [80]

    No subclass left behind: Fine- grained robustness in coarse-grained classification prob- lems

    Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher R ´e. No subclass left behind: Fine- grained robustness in coarse-grained classification prob- lems. Advances in Neural Information Processing Systems , 33:19339–19352, 2020. 3, 6, 17 11

  73. [81]

    In- triguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. In- triguing properties of neural networks. In International con- ference on learning representations, 2014. 3

  74. [82]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lind- sey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte Mac- Diarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, J...

  75. [83]

    Lin- ear spaces of meanings: compositional structures in vision- language models

    Matthew Trager, Pramuditha Perera, Luca Zancato, Alessan- dro Achille, Parminder Bhatia, and Stefano Soatto. Lin- ear spaces of meanings: compositional structures in vision- language models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15...

  76. [84]

    Thiagarajan

    Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan. A closer look at model adaptation using feature distortion and simplicity bias. In The Eleventh International Conference on Learning Representations, 2023. 2

  77. [85]

    Neuro- match academy: Teaching computational neuroscience with global accessibility

    Tara van Viegen, Athena Akrami, Kathryn Bonnen, Eric De- Witt, Alexandre Hyafil, Helena Ledmyr, Grace W Lindsay, Patrick Mineault, John D Murray, Xaq Pitkow, et al. Neuro- match academy: Teaching computational neuroscience with global accessibility. Trends in cognitive science...

  78. [86]

    Inves- tigating gender bias in language models using causal medi- ation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Inves- tigating gender bias in language models using causal medi- ation analysis. Advances in neural information processing systems, 33:12388–12401, 2020. 3

  79. [87]

    Concept algebra for score-based conditional model

    Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for score-based conditional model. InICML 2023 Workshop on Structured Probabilistic Inference {\&} Generative Modeling, 2023. 3

  80. [88]

    Caltech-ucsd birds 200

    Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. 2, 6

  81. [89]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 2, 6

  82. [90]

    Noise or signal: The role of image back- grounds in object recognition

    Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image back- grounds in object recognition. In International Conference on Learning Representations, 2021. 3

  83. [91]

    Spurious correlations in machine learning: A survey

    Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, Xia Hu, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.12715, 2024. 3

  84. [92]

    Contin- ual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR,

  85. [93]

    Coping with label shift via distributionally robust optimisation

    Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Sri- nadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. InIn- ternational Conference on Learning Representations , 2021. 3

  86. [94]

    Correct-n-contrast: a con- trastive approach for improving robustness to spurious corre- lations

    Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-n-contrast: a con- trastive approach for improving robustness to spurious corre- lations. In International Conference on Machine Learning , pages 26484–26516. PMLR, 2022. 3, 6

  87. [95]

    trifea- ture

    Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 119–134, 2018. 3 12 A. Appendix A.1. Further Discussion on Related Work Machine learni...

  88. [2016]

    Association for Computing Machinery. 2

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.