REVIEW 4 major objections 6 minor 96 references
Towards Utilising a Range of Neural Activations for Comprehending Representational Associations
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that the samples a classifier is least confident about—those with near-zero logits—concentrate the examples that break spurious correlations, and that retraining the classifier's last layer on a clustered subset of those…
desk verdict A promising no-group-label method for finding spurious correlations via mid-level logits, with strong WGA gains but an unmeasured core mechanism and a human-in-the-loop cluster selection that deserves scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 'intercept data': the set of training samples whose logit for a chosen class lies near zero, where the model's prediction confidence is lowest. The paper argues that this band is enriched for two kinds of examples that break the spurious trend—in-class samples lacking the spurious attribute and out-of-class samples carrying it—and that k-means clustering of their penultimate-layer embeddings separates a coherent group whose poor accuracy is caused by the spurious association rather than by benign ambiguity. Carrying the argument further, the last layer of the ERM model is replaced and retrained by $\ell^1$-regularised logistic regression on this selected cluster, exploiting the finding that ERM still learns core features and only the final mapping needs correction.
What would settle it
Measure, on CelebA or Waterbirds, the density of minority-group samples (blonde males, or water birds on land backgrounds) inside the near-zero-logit band selected by MID; if it is no higher than the base rate in the training set, the mid-range selection does not target spurious correlations and the method's mechanism is wrong. A second check: retrain the last layer on a randomly chosen cluster instead of the cluster judged to encode the spurious pattern—the paper's own ablation shows random-cluster retraining does not reliably improve worst-group accuracy, so a reader could verify this holds across seeds and datasets.
Extended reading notes
Core claim
The paper's central claim is that intermediate representations encode information at every level of stimulation, not just at the extremes, and that the mid-range of output-neuron activations is where a model's learned associations between class labels and confounding attributes become visible. Concretely, when images are ranked by their logit for a class, the maximally activating examples are typical of the majority spurious trend, while examples near the zero-logit intercept include shapes in a spurious location for the wrong class, low-spuriosity instances of the class, and mislabels—associations that maximum-activation analysis hides. MID operationalises this: for each class it keeps the near-zero logit samples, removes disputed labels using a vision-language model, clusters the encoder embeddings with k-means, and selects the cluster whose poor performance cannot be excused by image quality or labelling ambiguity (for CelebA, blonde faces with masculine features). Retraining the final layer on that cluster with $\ell^1$-regularised logistic regression, on top of the frozen feature extractor, improves worst-group accuracy to 87.8 on Waterbirds and 85.5 on CelebA without access to spurious-attribute labels for training or validation. The same logic, the paper argues, extends beyond output neurons to any concept direction in latent space found by methods such as TCAV or SVD-based decomposition.
Load-bearing premise
The method assumes that near-zero-logit samples are enriched for examples that break the spurious trend rather than dominated by mislabels, label noise, or low-quality images, and that this enrichment, demonstrated on synthetic DSpritesUnfair data, transfers to CelebA and Waterbirds.
Editorial extensions
If this is right
- Maximal-activation interpretability is incomplete: analyses that only inspect the highest-activating examples will systematically miss the associations and confounds a model encodes at mid-range stimulation.
- Last-layer retraining on automatically selected mid-logit clusters can match or approach the worst-group accuracy of methods that require group labels, while needing none for training or validation.
- The pipeline is cheap: it reuses the frozen penultimate features of a standard ERM model and only re-fits the final layer with logistic regression.
- The same mid-range inspection applies to any concept direction in latent space, not just output neurons, so confounding concepts found by concept-vector methods could be audited and mitigated the same way.
Reading between the lines
- A natural extension the paper does not test: iterate MID—retrain, re-select the mid-range band, and re-cluster—to see whether repeated rounds keep removing spurious reliance or converge to a stable selection.
- The mid-logit band is a free, unsupervised proxy for the samples a model has memorised against its own trend; the same filter could be reused to flag out-of-distribution inputs or label noise in other benchmarks, since the paper shows it captures 55.1% of the ERM model's errors on CelebA.
- Because the cluster-interpretation step is the only human-in-the-loop part, automating it with a vision-language model (which the paper mentions but does not evaluate) is a direct testable extension.
- The near-zero-logit heuristic suggests a broader principle: for any neuron or direction, the decision boundary in activation space may be where entangled concepts live, connecting this work to geometric accounts of polysemanticity.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that interpretability analyses based on maximally activating examples miss information encoded at non-maximal activation levels, and proposes a practical method, MID (Modifying with logit Intercept Data). MID first trains a standard ERM classifier, then selects samples whose class logits are near zero (the logit 'intercept'), filters suspected mislabels with BLIP, clusters the penultimate-layer embeddings of the remaining samples with k-means, manually selects a cluster that exhibits a harmful spurious pattern, and retrains the classification layer using DFR-style logistic regression. On Waterbirds and CelebA, MID reports worst-group accuracies of 87.8 and 85.5, respectively, compared with ERM baselines of 68.4 and 47.8, and is competitive with methods that use group labels while requiring none.
Significance. If the mechanism claimed by the paper holds, this is a valuable bridge between interpretability and robustness: it would show that non-maximal activations can expose confounds without group labels, and the proposed last-layer retraining is simple and cheap. The paper is honest about the human-in-the-loop nature of the method and provides ablations suggesting that both the mid-logit filter and the choice of cluster matter. The reported WGA gains are large, and the use of DFR-style retraining is well motivated by prior work. However, the central enrichment assumption -- that near-zero logit samples are enriched for low-spuriosity and counter-spurious examples -- is only qualitatively demonstrated on a synthetic dataset and never directly measured on the real benchmarks, so the significance of the empirical results is conditional on that assumption being validated.
major comments (4)
- [Sec. 4.2, Fig. 3 and Sec. 5.1] The core claim that near-zero class logits are enriched for low-spuriosity and counter-spurious examples is supported only by a qualitative inspection of six images on DSpritesUnfair (bias 0.4). No quantitative enrichment statistic is reported for Waterbirds or CelebA, and Fig. 10 shows the group composition only of the final cluster, which is already the product of human selection of k and of the 'spurious' cluster. Because the Table 1 gains could in principle arise from DFR-style retraining on any hard or balanced subset rather than from a property specific to mid-level activations, the paper should measure the group composition of the mid-logit selection before clustering (e.g., the fraction of minority-group examples among selected samples versus random samples matched by logit rank) and report this for both benchmarks.
- [Sec. 5.1 and Appendix A.6] The cluster selection step is not prespecified: k is increased 'until a pattern emerges', and the cluster used for retraining is chosen by manual inspection of cluster contents. For CelebA this selects the cluster corresponding to the known spurious attribute (blonde with masculine-associated features). This introduces human selection based on prior knowledge of each benchmark's known bias and undercuts the claim of operating without group information. Please specify a preselectable or automated rule for choosing k and the retraining cluster, and report sensitivity to both choices (e.g., WGA for every cluster and every k tried).
- [Appendix A.7] The ablation of the mid-logit filter is qualitative only. The paper states that clustering the full CelebA embeddings did not reveal a spurious pattern and that retraining on one ambiguous cluster does not help WGA, but no quantitative worst-group accuracies are given for these ablations. A quantitative comparison of MID against (a) retraining on a random balanced subset, (b) retraining on BLIP-filtered mid-logit data without clustering, and (c) retraining on each individual k-means cluster would isolate the contribution of the mid-logit selection and the cluster choice. This is needed to support the claim that the mid-level activation lens, rather than the last-layer retraining mechanism, is responsible for the improvements.
- [Sec. 5.1 and Appendix A.5] The mid-range selection rule is underspecified. For CelebA the paper reports 2,000 logit-intercept points per class, but for Waterbirds no analogous count or threshold is given, and the exact definition of 'near-zero logit' is not operationalized (e.g., a window around zero, a fixed number per class, or a sign-change region). Without a precise selection rule the method cannot be reproduced or compared across datasets; please specify the selection criterion and report the number of selected points for both benchmarks.
minor comments (6)
- [Table 1] The entry 'DFRTrTr' is not defined in the table or caption; the text in Sec. 6.3 mentions 'DFR T r T r' but it should be spelled out as a specific DFR variant.
- [Sec. 6.3 and Table 1] No variance or number of seeds is reported for the Waterbirds and CelebA ERM and MID runs, despite the DSprites pilot reporting means over five seeds. Since Table 1 is the main quantitative claim, please include standard deviations or error bars over multiple initializations.
- [Appendix A.6] The sentence 'we train on the available validation data with additional randomly selected training set data' is ambiguous and appears to contradict Sec. 6.4's statement that group information is used only for evaluation. Please clarify whether unlabeled validation data are used during retraining and how this affects the comparison with baselines that use group-balanced validation sets.
- [Sec. 7] The claim that points 'right above' the zero logit are more likely to be low-spuriosity memorized points while points 'right below' are more likely to be counter-spurious examples is presented as a finding but is not measured anywhere in the paper; it should be framed as a hypothesis or supported with data.
- [Table 2 and References] There is a LaTeX artifact in Table 2 ('L n(L)'), and several reference formatting issues such as 'Hermann et Lampinen' and 'TCA V' with inconsistent spacing; these should be cleaned up.
- [Figs. 3 and 6] The example images in Figs. 3 and 6 are very small, making the qualitative claims about position and shape difficult to verify; larger crops or additional annotations would help.
Circularity Check
No significant circularity: MID's WGA improvements are measured on held-out benchmark test sets, and the paper's assumptions, while empirically under-supported, are not definitionally tied to its outputs.
full rationale
The paper's central claim is empirical: mid-range logit samples, after k-means clustering and human selection of a 'spurious' cluster, can be used to retrain the classification layer and improve worst-group accuracy on Waterbirds and CelebA. These improvements (Table 1) are evaluated on external held-out test sets with group labels used only for evaluation, so they do not reduce by construction to the selection procedure. The key assumption that near-zero logits are enriched for low-spuriosity and counter-spurious samples is supported only qualitatively on DSpritesUnfair and is not directly measured on the real benchmarks; this is a validity and evidential weakness, not a circularity. The human-in-the-loop choice of k and cluster, and the tuning of the regularization strength on a split of the selected retraining data, are explicitly disclosed (Sec. 5.1, Sec. 6.3, Appendix A.6) and do not constitute a fitted parameter being renamed as a prediction. The last-layer retraining step is borrowed from DFR [39] with citation, and the paper does not claim that step as a novel derivation. Self-citations in the related work are not load-bearing for the main result. No equation or definition in the paper makes the reported WGA equal to an input by construction.
Assumptions & free parameters
free parameters (4)
- Mid-range logit count per class =
2000 for CelebA; unspecified for Waterbirds
- Number of clusters k =
3 for CelebA (k=2..4 explored)
- Spurious cluster selection =
Cluster judged to reflect an unacceptable spurious pattern (e.g., masculine-featured blondes for CelebA)
- Regularization strength of logistic regression =
1.0 from {1.0, 0.7, 0.3, 0.1, 0.07, 0.03, 0.01} for both datasets
assumptions (4)
- domain assumption Last-layer retraining is sufficient to reduce spurious feature reliance
- domain assumption BLIP VQA can reliably identify mislabeled or ambiguous hair-color images
- ad hoc to paper Near-zero logit intercept sets are enriched for low-spuriosity and counter-spurious examples
- domain assumption Frozen penultimate-layer embeddings preserve core features necessary for last-layer retraining
Cite this review
Pith. "Pith review of Towards Utilising a Range of Neural Activations for Comprehending Representational Associations." pith.science (2026). https://pith.science/paper/WUXFKXDT
@misc{pith2026241110019,
author = {Pith},
title = {Pith review of: Towards Utilising a Range of Neural Activations for Comprehending Representational Associations},
year = {2026},
howpublished = {\url{https://pith.science/paper/WUXFKXDT}},
note = {Machine review of arXiv:2411.10019}
}
read the original abstract
Recent efforts to understand intermediate representations in deep neural networks have commonly attempted to label individual neurons and combinations of neurons that make up linear directions in the latent space by examining extremal neuron activations and the highest direction projections. In this paper, we show that this approach, although yielding a good approximation for many purposes, fails to capture valuable information about the behaviour of a representation. Neural network activations are generally dense, and so a more complex, but realistic scenario is that linear directions encode information at various levels of stimulation. We hypothesise that non-extremal level activations contain complex information worth investigating, such as statistical associations, and thus may be used to locate confounding human interpretable concepts. We explore the value of studying a range of neuron activations by taking the case of mid-level output neuron activations and demonstrate on a synthetic dataset how they can inform us about aspects of representations in the penultimate layer not evident through analysing maximal activations alone. We use our findings to develop a method to curate data from mid-range logit samples for retraining to mitigate spurious correlations, or confounding concepts in the penultimate layer, on real benchmark datasets. The success of our method exemplifies the utility of inspecting non-maximal activations to extract complex relationships learned by models.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Understanding inter- mediate layers using linear classifier probes
Guillaume Alain and Yoshua Bengio. Understanding inter- mediate layers using linear classifier probes. InInternational conference on learning representations, 2017. 3
2017
-
[2]
Network dissection: Quantifying inter- pretability of deep visual representations
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 6541–6549, 2017. 1, 3
2017
-
[3]
Gan dissection: Visualizing and understanding gener- ative adversarial networks
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Tor- ralba. Gan dissection: Visualizing and understanding gener- ative adversarial networks. In International Conference on Learning Representations, 2019. 1
2019
-
[4]
Rep- resentation learning: A review and new perspectives
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Rep- resentation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798–1828, 2013. 1, 3
2013
-
[5]
An interpretability illusion for bert
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Co- enen, Emily Reif, Fernanda Vi ´egas, and Martin Watten- berg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. 3
arXiv 2021
-
[6]
Towards monose- manticity: Decomposing language models with dictionary learning
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. 3
2023
-
[7]
Gender shades: Inter- sectional accuracy disparities in commercial gender classifi- cation
Joy Buolamwini and Timnit Gebru. Gender shades: Inter- sectional accuracy disparities in commercial gender classifi- cation. In Conference on fairness, accountability and trans- parency, pages 77–91. PMLR, 2018. 3
2018
-
[8]
Labeling neural representations with in- verse recognition
Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina H¨ohne. Labeling neural representations with in- verse recognition. Advances in Neural Information Process- ing Systems, 36, 2023. 1
2023
Show all 96 references
-
[9]
Isolating sources of disentanglement in varia- tional autoencoders
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in varia- tional autoencoders. Advances in neural information pro- cessing systems, 31, 2018. 3
2018
-
[10]
Fair prediction with disparate im- pact: A study of bias in recidivism prediction instruments
Alexandra Chouldechova. Fair prediction with disparate im- pact: A study of bias in recidivism prediction instruments. Big data, 5(2):153–163, 2017. 3
2017
-
[11]
Independent component analysis, a new con- cept? Signal processing, 36(3):287–314, 1994
Pierre Comon. Independent component analysis, a new con- cept? Signal processing, 36(3):287–314, 1994. 1, 3
1994
-
[12]
Flexibly fair representation learning by disentan- glement
Elliot Creager, David Madras, J ¨orn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentan- glement. In International conference on machine learning , pages 1436–1445. PMLR, 2019. 4
2019
-
[13]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6
2009
-
[14]
Toy- modelsof superposition
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy- modelsof superposition. Transformer Circuits Thread, 2022. 3
2022
-
[15]
Visualizing higher-layer features of a deep network
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009. 1
2009
-
[16]
A holistic approach to unifying automatic concept extraction and concept importance estimation
Thomas Fel, Victor Boutin, Louis B ´ethune, R ´emi Cad `ene, Mazda Moayeri, L ´eo And ´eol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Ad- vances in Neural Information Processing Systems,...
2023
-
[17]
Craft: Concept recursive activation factoriza- tion for explainability
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, R ´emi Cad`ene, and Thomas Serre. Craft: Concept recursive activation factoriza- tion for explainability. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...
2023
-
[18]
Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neu- ral networks
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neu- ral networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730–8738,
-
[19]
Scaling and evaluating sparse autoencoders
Leo Gao, Tom Dupr ´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. 3
2024 arXiv
-
[20]
Shortcut learning in deep neural networks
Robert Geirhos, J ¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Fe- lix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. 2, 3
2020
-
[21]
Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In Interna- tional Conference on Learning Representations, 2019. 3, 13
2019
-
[22]
Don’t trust your eyes: on the (un) reliability of feature visualizations
Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don’t trust your eyes: on the (un) reliability of feature visualizations. In Forty-first In- ternational Conference on Machine Learning, 2024. 3
2024
-
[23]
Concept discovery and dataset exploration with singular value decomposition
Mara Graziani, An-phi Nguyen, Laura O’Mahony, Henning M¨uller, and Vincent Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and compu- tation for Trustworthy ML, 2023. 3
2023
-
[24]
Uncovering unique con- cept vectors through latent space decomposition
Mara Graziani, Laura O’Mahony, An-phi Nguyen, Henning M¨uller, and Vincent Andrearczyk. Uncovering unique con- cept vectors through latent space decomposition. Transac- tions on Machine Learning Research, 2023. 1, 3, 8
2023
-
[25]
Fairness without demo- graphics in repeated loss minimization
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning , pages 1929–1938. PMLR, 2018. 3
1929
-
[26]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6 9
2016
-
[27]
A baseline for detect- ing misclassified and out-of-distribution examples in neural networks
Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Repre- sentations, 2017. 3, 13
2017
-
[28]
The origins and prevalence of texture bias in convolutional neu- ral networks
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neu- ral networks. Advances in Neural Information Processing Systems, 33:19000–19015, 2020. 3
2020
-
[29]
What shapes feature representations? exploring datasets, architectures, and training
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995–10006, 2020. 2, 3, 13
2020
-
[30]
beta-vae: Learning basic visual concepts with a constrained variational framework
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017. 4
2017
-
[31]
Disentanglement via latent quantiza- tion
Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantiza- tion. Advances in Neural Information Processing Systems , 36, 2023. 3
2023
-
[32]
Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037. PMLR, 2018. 3
2018
-
[33]
Independent component analysis: algorithms and applications
Aapo Hyv ¨arinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411–430, 2000. 3
2000
-
[34]
Simple data balancing achieves com- petitive worst-group-accuracy
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves com- petitive worst-group-accuracy. In Conference on Causal Learning and Reasoning, pages 336–351. PMLR, 2022. 3, 6
2022
-
[35]
Shape or texture: Understanding discriminative features in cnns
Md Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Bj¨orn Ommer, Konstantinos G Derpanis, and Neil Bruce. Shape or texture: Understanding discriminative features in cnns. In International Conference on Learning Representa- tions, 2021. 3
2021
-
[36]
On feature learning in the presence of spuri- ous correlations
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and An- drew G Wilson. On feature learning in the presence of spuri- ous correlations. Advances in Neural Information Processing Systems, 35:38516–38532, 2022. 3, 6, 7, 13
2022
-
[37]
Removing spurious fea- tures can hurt accuracy and affect groups disproportionately
Fereshte Khani and Percy Liang. Removing spurious fea- tures can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM conference on fairness, ac- countability, and transparency, pages 196–205, 2021. 3
2021
-
[38]
Interpretability be- yond feature attribution: Quantitative testing with concept activation vectors (tcav)
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability be- yond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on ma- chine learning, pages 2668–2677. PMLR, 2...
2018
-
[39]
Last layer re-training is sufficient for robustness to spu- rious correlations
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wil- son. Last layer re-training is sufficient for robustness to spu- rious correlations. In The Eleventh International Conference on Learning Representations, 2023. 3, 6, 7, 13
2023
-
[40]
Concept bottleneck models
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. 8
2020
-
[41]
Towards a fuller understanding of neurons with clustered compositional explanations
Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco. Towards a fuller understanding of neurons with clustered compositional explanations. Advances in Neural Informa- tion Processing Systems, 36:70333–70354, 2023. 1, 3
2023
-
[42]
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 6
2022
-
[43]
Just train twice: Improving group robustness without training group information
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghu- nathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Confer- ence on Machine Learning, pages 6781–6792. PMLR,...
2021
-
[44]
Deep learning face attributes in the wild
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. 2, 6
2015
-
[45]
Challenging common assumptions in the unsuper- vised learning of disentangled representations
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Sch ¨olkopf, and Olivier Bachem. Challenging common assumptions in the unsuper- vised learning of disentangled representations. In Interna- tional conference on machine learning , pages 411...
2019
-
[46]
Weakly-supervised disentanglement without compromises
Francesco Locatello, Ben Poole, Gunnar R ¨atsch, Bern- hard Sch ¨olkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International conference on machine learning , pages 6348–6359. PMLR, 2020. 4
2020
-
[47]
Understanding and mitigating human-labelling er- rors in supervised contrastive learning
Zijun Long, Lipeng Zhuang, George Killick, Richard Mc- Creadie, Gerardo Aragon Camarasa, and Paul Hender- son. Understanding and mitigating human-labelling er- rors in supervised contrastive learning. arXiv preprint arXiv:2403.06289, 2024. 6
2024 arXiv
-
[48]
On interpretability of deep learning based skin lesion classifiers using concept activation vectors
Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. In 2020 international joint conference on neural networ...
2020
-
[49]
Understanding deep image representations by inverting them
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196, 2015. 1
2015
-
[50]
Promises and pitfalls of black-box concept learning models
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314,
-
[51]
Is this the subspace you are looking for? an inter- pretability illusion for subspace activation patching
Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? an inter- pretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representa- tions, 2023. 3
2023
-
[52]
Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021. 8 10
2021 arXiv
-
[53]
Catastrophic inter- ference in connectionist networks: The sequential learning problem
Michael McCloskey and Neal J Cohen. Catastrophic inter- ference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , vol- ume 24, pages 109–165. Elsevier, 1989. 13
1989
-
[54]
Acquisition of chess knowledge in alphazero
Thomas McGrath, Andrei Kapishnikov, Nenad Toma ˇsev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022. 3
2022
-
[55]
Umap: Uniform manifold approximation and projection
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018. 7
2018
-
[56]
Linguis- tic regularities in continuous space word representations
Tom ´aˇs Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguis- tic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Hu- man language technologies, pages 746...
2013
-
[57]
A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes
Mazda Moayeri, Phillip Pope, Yogesh Balaji, and So- heil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1908...
2022
-
[58]
Spuriosity rankings: sorting data to measure and miti- gate biases
Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi. Spuriosity rankings: sorting data to measure and miti- gate biases. Advances in Neural Information Processing Sys- tems, 36:41572–41600, 2023. 3
2023
-
[59]
Compositional explanations of neurons
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Sys- tems, 33:17153–17163, 2020. 1
2020
-
[60]
Beyond distribution shift: Spurious features through the lens of training dynamics
Nihal Murali, Aahlad Manas Puli, Ke Yu, Rajesh Ranganath, et al. Beyond distribution shift: Spurious features through the lens of training dynamics. Transactions on Machine Learning Research, 2023. 2, 3, 13, 14
2023
-
[61]
Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation
Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Confer- ence on Learning Representations, 2022. 3, 6
2022
-
[62]
Explaining explainability: Understanding concept activation vectors
Angus Nicolson, Lisa Schut, J Alison Noble, and Yarin Gal. Explaining explainability: Understanding concept activation vectors. arXiv preprint arXiv:2404.03713, 2024. 3, 8
2024 arXiv
-
[63]
Zoom in: An in- troduction to circuits
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An in- troduction to circuits. Distill, 5(3):e00024–001, 2020. 1, 3
2020
-
[64]
Feature visualization
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017. 1, 3
2017
-
[65]
Disentangling neuron representations with concept vectors
Laura O’Mahony, Vincent Andrearczyk, Henning M ¨uller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3769–3774, 2023. 1
2023
-
[66]
The linear rep- resentation hypothesis and the geometry of large language models
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear rep- resentation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024. 3
2024
-
[67]
Pytorch: An im- perative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...
2019
-
[68]
Scikit-learn: Machine learning in python
Fabian Pedregosa, Ga ¨el Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011. 15
2011
-
[69]
Identifying mislabeled data using the area under the margin ranking
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Pro- cessing Systems, 33:17044–17056, 2020. 6
2020
-
[70]
Why Should I Trust You?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY , USA,
-
[71]
Parallel distributed processing, volume 1: Ex- plorations in the microstructure of cognition: Foundations
David E Rumelhart, James L McClelland, PDP Research Group, et al. Parallel distributed processing, volume 1: Ex- plorations in the microstructure of cognition: Foundations . The MIT press, 1986. 3
1986
-
[72]
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 6
2015
-
[73]
Distributionally robust neural networks
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In In- ternational Conference on Learning Representations , 2020. 3
2020
-
[74]
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 ,
-
[75]
A mathematical theory of semantic development in deep neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):11537–11546, 2019. 13
2019
-
[76]
The pitfalls of simplicity bias in neural networks
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Pro- cessing Systems, 33:9573–9585, 2020. 3
2020
-
[77]
Weakly supervised disentanglement with guarantees
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Rep- resentations, 2020. 4
2020
-
[78]
Opening the black box of deep neural networks via information
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017. 13
2017 arXiv
-
[79]
On the proper treatment of connectionism
Paul Smolensky. On the proper treatment of connectionism. Behavioral and brain sciences, 11(1):1–23, 1988. 3
1988
-
[80]
No subclass left behind: Fine- grained robustness in coarse-grained classification prob- lems
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher R ´e. No subclass left behind: Fine- grained robustness in coarse-grained classification prob- lems. Advances in Neural Information Processing Systems , 33:19339–19352, 2020. 3, 6, 17 11
2020
-
[81]
In- triguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. In- triguing properties of neural networks. In International con- ference on learning representations, 2014. 3
2014
-
[82]
Daniel Freeman, Theodore R
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lind- sey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte Mac- Diarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, J...
2024
-
[83]
Lin- ear spaces of meanings: compositional structures in vision- language models
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessan- dro Achille, Parminder Bhatia, and Stefano Soatto. Lin- ear spaces of meanings: compositional structures in vision- language models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15...
-
[84]
Thiagarajan
Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan. A closer look at model adaptation using feature distortion and simplicity bias. In The Eleventh International Conference on Learning Representations, 2023. 2
2023
-
[85]
Neuro- match academy: Teaching computational neuroscience with global accessibility
Tara van Viegen, Athena Akrami, Kathryn Bonnen, Eric De- Witt, Alexandre Hyafil, Helena Ledmyr, Grace W Lindsay, Patrick Mineault, John D Murray, Xaq Pitkow, et al. Neuro- match academy: Teaching computational neuroscience with global accessibility. Trends in cognitive science...
2021
-
[86]
Inves- tigating gender bias in language models using causal medi- ation analysis
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Inves- tigating gender bias in language models using causal medi- ation analysis. Advances in neural information processing systems, 33:12388–12401, 2020. 3
2020
-
[87]
Concept algebra for score-based conditional model
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for score-based conditional model. InICML 2023 Workshop on Structured Probabilistic Inference {\&} Generative Modeling, 2023. 3
2023
-
[88]
Caltech-ucsd birds 200
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. 2, 6
2010
-
[89]
Sun database: Large-scale scene recognition from abbey to zoo
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 2, 6
2010
-
[90]
Noise or signal: The role of image back- grounds in object recognition
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image back- grounds in object recognition. In International Conference on Learning Representations, 2021. 3
2021
-
[91]
Spurious correlations in machine learning: A survey
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, Xia Hu, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.12715, 2024. 3
2024
-
[92]
Contin- ual learning through synaptic intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR,
-
[93]
Coping with label shift via distributionally robust optimisation
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Sri- nadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. InIn- ternational Conference on Learning Representations , 2021. 3
2021
-
[94]
Correct-n-contrast: a con- trastive approach for improving robustness to spurious corre- lations
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-n-contrast: a con- trastive approach for improving robustness to spurious corre- lations. In International Conference on Machine Learning , pages 26484–26516. PMLR, 2022. 3, 6
2022
-
[95]
trifea- ture
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 119–134, 2018. 3 12 A. Appendix A.1. Further Discussion on Related Work Machine learni...
2018
-
[2016]
Association for Computing Machinery. 2
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.