Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Adversarial Attacks Leverage Interference Between Features in Superposition

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adversarial vulnerability can arise simply from how networks compress many features into few dimensions: superposed features interfere, and adversarial perturbations exploit that interference.

desk verdict A clean toy-model demonstration that superposition produces adversarial vulnerability and explains transferability; the ViT extension overreaches and the headline validation is partly circular. read the letter →

arxiv 2510.11709 v2 pith:47CLSN7M submitted 2025-10-13 cs.LG cs.AIcs.CV

Edward Stevinson , Lucas Prieto , Melih Barsbey , Tolga Birdal This is my paper
classification cs.LGcs.AIcs.CV
keywords adversarialexamplessuperpositionlinearrepresentationhypothesisattacktransferabilityfeatureinterferencemechanisticinterpretabilityrobustnessprojectedgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that adversarial vulnerability can be a byproduct of how neural networks pack information efficiently: when a network represents more features than it has dimensions (superposition), the features interfere, and adversarial perturbations exploit exactly that interference. In a controlled linear setting, the authors prove that the optimal attack from class j to class k is δ ∝ W_e^T(v_k − v_j), meaning each input coordinate is perturbed in proportion to how much its feature interferes with the two class representations. They show that PGD-discovered attacks align almost perfectly with this theoretically optimal direction, that input correlations constrain the learned feature geometry, and that models trained on similarly correlated data share enough geometry to make attacks transfer (from roughly 18% transfer on uncorrelated data to 94% on globally correlated data). They reproduce the main trends in a vision transformer on CIFAR-10 with an engineered bottleneck that forces superposition, and they show superposition is sufficient but not necessary: a modular-addition network with orthogonal features is still vulnerable, via a frequency-based "algorithmic brittleness." If correct, the paper supplies a representation-level mechanism that predicts which perturbations succeed, explains transferability, and reframes part of adversarial robustness as a cost of representational compression.

What carries the argument

The central object is the encoder's overcomplete dictionary of feature directions: the columns v_i of W_e, with the decoder approximately equal to W_e^T, so each class logit is a dot product between a feature direction and the latent activation. The load-bearing identity is the decision-boundary normal n = v_k − v_j, whose pullback through the encoder gives the optimal perturbation δ ∝ W_e^T(v_k − v_j). The associated corollary — that feature i is perturbed with magnitude proportional to v_i^T(v_k − v_j) — is what turns interference into vulnerability. Proposition 2 shows this attack direction is invariant under orthogonal transformations Q of the feature directions, which is the mechanism t

What would settle it

Find a pair of models with low geometric similarity between their learned feature cosine matrices but high attack transferability; the paper's mechanism predicts transfer tracks geometry, so such a pair would refute it. Conversely, in an unmodified image classifier, measure whether PGD perturbations align with W_e^T(v_k − v_j) computed from class-representation directions at the final layer; near-zero alignment would falsify the claim that attacks exploit interference.

Watch

Extended reading notes

Core claim

The paper claims that adversarial vulnerability can be created by superposition alone, without invoking non-robust input features or decision-boundary bugs. In a two-layer linear model where the encoder's columns v_i are superposed feature directions and the decoder is its transpose, the logit for class j is z_j = v_j^T h, so moving a sample from class j to class k requires maximizing the margin z_k − z_j. The optimal perturbation is δ ∝ W_e^T(v_k − v_j), and the per-coordinate magnitude is |δ_i| ∝ |v_i^T(v_k − v_j)| — the differential interference between feature i and the class representations. The paper verifies that PGD attacks align with this direction, that feature geometry is constrai

Load-bearing premise

The load-bearing premise is that superposition induced by an engineered low-rank bottleneck on frozen image features behaves like naturally emerging superposition in unmodified networks; the paper's own limitation states its insights derive from simplified settings with class features in engineered superposition, so if real superposition differs qualitatively, the predicted attack patterns and transferability results may not hold.

Editorial extensions

If this is right

  • Given the geometry of learned features, PGD attacks become predictable: the perturbation profile can be read off from v_i^T(v_k − v_j) without running an attack.
  • Attack transferability between models is governed by shared interference patterns, so models trained on similarly correlated data will share vulnerabilities; decorrelating or making features more orthogonal should reduce transfer.
  • Superposition pressure (more features than dimensions) monotonically degrades robust accuracy, making representational compression a controllable driver of robustness and a candidate explanation for accuracy–robustness trade-offs.
  • Adversarial vulnerability does not require non-robust input features or decision-boundary bugs: compression alone suffices, and removing superposition can eliminate class-flipping attacks (zero successful attacks when m = k).
  • Robustness certification that only enforces norm bounds raises the required perturbation budget but does not remove the underlying algorithmic vulnerability in the orthogonal-feature modular-addition case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the formula δ ∝ W_e^T(v_k − v_j) suggests a layer-wise diagnostic for real classifiers — compute the unit-gradient direction at every layer and compare it with actual PGD updates; layers with high alignment are where the superposition mechanism is active, a localization the paper does not run on unmodified networks.
  • Editorial inference: if interference is the causal driver, then defenses that encourage feature orthogonality or sparser, more separable representations should reduce both attack success and transferability; this is a concrete experiment implied but not performed and would separate correlation from causation.
  • Editorial inference: the modular-addition result points to a taxonomy of two vulnerability mechanisms — geometric interference and algorithmic brittleness — which may combine in deep networks when upstream superposition feeds noisy features into downstream orthogonal circuits; the paper conjectures this reconciliation in its appendix, and it is a natural next test.
  • Editorial inference: in large models whose features are extracted by sparse autoencoders, features with high mutual cosine similarity should be the ones that adversarial perturbations most consistently activate; measuring that would connect the toy mechanism to large-scale behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes that adversarial vulnerability can arise from interference between features represented in superposition. In a controlled linear toy model, it shows that input correlations shape the geometry of learned feature directions; that PGD-discovered perturbations align with the closed-form optimum δ ∝ W_e^T(v_k − v_j) derived from that geometry; that geometric similarity across random seeds tracks attack transferability; and that removing or reducing superposition suppresses successful attacks. The paper then extends the setup to a ViT on CIFAR-10 by inserting a trained linear bottleneck on frozen features, and to a modular-addition MLP to argue that superposition is sufficient but not necessary for vulnerability, attributing the latter to 'algorithmic brittleness.' The central thesis is that adversarial examples are predictable from representation geometry and can be a byproduct of efficient representational compression.

Significance. If the central claims hold, the paper offers a mechanistic, representation-level account of adversarial examples and transferability that complements data-centric or boundary-geometry explanations. The main strengths are the carefully controlled toy model, the exact linear analysis (Propositions 1 and 2 with proofs in Appendix B), the explicit code release, and the demonstration that input-correlation manipulations yield monotone changes in geometric similarity and attack transferability. The informed, non-gradient attack in §5 is a useful falsifiable consequence. However, two load-bearing issues weaken the paper as written: the Tab. 1 alignment between PGD and the 'theoretical optimum' is close to built-in for linear models, and the ViT experiments only show that an imposed bottleneck inherits the toy-model behavior, not that an unmodified ViT exhibits the proposed mechanism. These issues do not invalidate the core toy-model derivation, but they materially affect the scope of the claims made in the abstract and introduction.

major comments (3)
  1. [§3.1, Tab. 1 and Prop. 1] The claimed validation that 'PGD attacks align with theoretically optimal perturbations' is close to a tautology in the linear setting. Proposition 1 derives δ ∝ W_e^T(v_k − v_j), which is exactly the gradient of the logit margin z_k − z_j with respect to the input. PGD is a gradient-ascent method on the same margin/loss geometry, so a near-perfect cosine similarity between PGD directions and this 'optimum' is expected by construction rather than an independent confirmation that the mechanism is superposition. To make the empirical claim load-bearing, the paper should either (a) use a non-gradient attack (e.g., random search or the analytic informed attack from §5) and show it succeeds at or near the PGD budget, or (b) explicitly reframe Tab. 1 as a consistency check, not as evidence that PGD 'exploits' superposition.
  2. [§4 and Abstract] The ViT experiments impose superposition by replacing the classification head with a linear bottleneck on frozen features. With this design, the observed decreases in robust accuracy and increases in transferability as m shrinks are predictable from Proposition 1 applied to the engineered head; they do not demonstrate that an unmodified ViT represents class features in superposition or that PGD on such a network exploits naturally occurring interference. The abstract's statement that 'these findings persist in a vision transformer (ViT) trained on CIFAR-10' overstates the evidence. This is acknowledged in the Sec. 7 limitation ('Our insights derive from simplified settings with class features in engineered superposition'), but the caveat is absent from the abstract and contribution list. The authors should either soften the ViT claim or add experiments on the unmodified backbone, e.g., u
  3. [§3.2 / Appendix B, Prop. 1] The exact form of Proposition 1 relies on the assumption W_d = W_e^T, justified only as 'empirically observed' in the toy model. This is an ad hoc structural assumption, and the main text does not state it prominently. The appendix notes that an arbitrary decoder changes the result to δ ∝ W_e^T(w_k − w_j), so the main-text formula is not the general linear optimum. Since the paper's central prediction depends on this formula, the assumption should be stated in the main text and discussed: does it hold only for the specific CE, no-ReLU, no-bias setup, or is there a reason to expect encoder-transpose decoders beyond these toy models?
minor comments (6)
  1. [Abstract] Typo: 'superpositionsuffices' should read 'superposition suffices'.
  2. [Table 1] The row format, e.g. '6 20.97±0.02', conflates the k and m columns; use explicit columns so the reader can see the (k,m) pairs being compared.
  3. [§3.1] The sentence 'We find zero successful adversarial examples across 1000 attempts at all ϵ values tested' should report the actual ϵ values and the m=k configuration; otherwise the claim is hard to verify.
  4. [§5.1] Typo in Appendix E.3: 'We would contain a row corresponding...' should begin 'W_e would contain a row...'.
  5. [Tables 5 vs. §3.1] The correlation conditions are called 'Global' in the main text and 'Fully Correlated' in Appendix Table 5; unify terminology.
  6. [§4, Fig. 4] The right panel's legend appears to mix units: attack transferability is in percent and robust accuracy is normalized; please label the axes/curves explicitly so the monotonic relationship is unambiguous.

Circularity Check

1 steps flagged · score 6.0 of 10

Main 'prediction' that PGD matches theory is definitional: Prop. 1's optimum is the same logit-margin gradient that PGD follows; other findings are independent.

  1. self definitional [Sec. 3.1, Table 1; Sec. 3.2, Prop. 1; App. A, Eq. (1)]
    "To evaluate whether PGD attacks specifically exploit superposition geometry, we compare PGD-generated attacks against theoretically optimal perturbations that we show leverage superposition (derived in Sec. 3.2). ... Tab. 1 shows near-perfect alignment between PGD and optimal attacks across various dimensionalities. ... Proposition 1. The optimal input perturbations δ that maximise movement from class j to class k under constraint ∥δ∥2 = ϵ satisfy δ ∝ W⊤e n, where n = (v_k − v_j) is the normal to the decision boundary between classes."

    In this linear model z_k − z_j = (v_k − v_j)^T W_e x, so ∇_x(z_k − z_j) = W_e^T(v_k − v_j). Proposition 1's 'theoretically optimal perturbation' is exactly this gradient. PGD (App. A, Eq. 1) is defined as iteratively following the normalized gradient of the loss, and for a successful j→k attack the relevant near-boundary gradient is the same normal. Hence the near-perfect cosine similarity in Table 1 is guaranteed by construction: it re-verifies that PGD follows gradients. It does not independently confirm that attacks specifically exploit superposition, since the identity holds for any linear model, superposed or orthogonal. The random baseline only shows the direction is nonzero; it does not provide an alternative mechanism.

full rationale

The only load-bearing reduction I can exhibit is the Table 1 alignment test. In the paper's linear model, the logit margin z_k − z_j has gradient W_e^T(v_k − v_j), exactly the quantity Proposition 1 derives as the 'theoretically optimal perturbation.' PGD (App. A, Eq. 1) is defined as iterated normalized gradient ascent on the loss, so a successful PGD attack ending at the j/k boundary is expected from the paper's own equations to align with that direction. That part of the paper is self-definitional: the 'prediction' reduces to the identity between the optimum and the gradient that PGD follows. Other elements are not circular: the m=k no-superposition control, the transferability-vs-geometry correlation, the modular addition informed attacks, and the SAE appendix are controlled comparisons or externally checkable phenomena. The ViT bottleneck experiment is an imposed construction whose natural-superposition bridge is admittedly unverified in Sec. 7; that is a scope/external-validity limitation, not circularity. Self-citations are background and not load-bearing. Overall, partial circularity: the headline 'attacks are predictable' claim is partly built into the definitions, but the paper still contains independent empirical content.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on the linear representation hypothesis, the empirically observed encoder-transpose decoder, and input sparsity. The paper uses no global fitted constants; the model weights are learned objects under study. No new entities (particles/forces/dimensions) are postulated.

assumptions (4)
  • domain assumption Linear Representation Hypothesis (LRH): features are represented as linear directions in activation space (Definition 1).
    Invoked throughout; logits are written as z_j = v_j^T h in Sec. 3.2. If class-relevant features are not linear directions, the interference mechanism and the derived attack formula do not apply.
  • ad hoc to paper Decoder equals encoder transpose (W_d = W_e^T) in trained toy models.
    Stated as 'as empirically observed' in Sec. 3.2 and used to derive δ ∝ W_e^T(v_k−v_j). The paper notes the framework extends to arbitrary decoders but does not test that extension empirically.
  • domain assumption Input features are sparse (Superposition Hypothesis, Definition 2).
    The toy data uses Bernoulli sparsity S; the paper argues sparsity makes superposition possible. Real image inputs are dense, so the mechanism's applicability to raw pixels is questionable.
  • standard math The logit margin z_k − z_j is the correct objective for targeted attacks.
    Used in Proposition 1 to derive the optimal perturbation; standard in the adversarial-attack literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Attacks Leverage Interference Between Features in Superposition." pith.science (2026). https://pith.science/paper/47CLSN7M

@misc{pith2026251011709,
  author       = {Pith},
  title        = {Pith review of: Adversarial Attacks Leverage Interference Between Features in Superposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/47CLSN7M}},
  note         = {Machine review of arXiv:2510.11709}
}
read the original abstract

Why do adversarial examples exist, and why do they transfer between models? Existing explanations appeal to high-dimensional geometry, non-robust patterns in the input, and decision boundary structure, but none provides a representation-level mechanism that explains why specific perturbations succeed and why attacks transfer between models. In this paper, we show that adversarial vulnerability can stem from efficient information encoding in neural networks. Specifically, vulnerability can arise from superposition - the phenomenon where networks represent more concepts than they have dimensions, forcing non-orthogonal representation and thus interference. This interference causes perturbations targeting one representation to affect others, creating vulnerabilities determined by interference patterns. In synthetic settings with precisely controlled superposition, we establish that superposition suffices to create adversarial vulnerability. The resulting attacks are predictable: PGD-discovered perturbations align with theoretically optimal perturbations derived from the interference geometry. Models trained on similar data develop similar interference patterns, explaining attack transferability. We then show that successful attacks on image classifiers exhibit the structure predicted by our proposed mechanism. These findings reveal that adversarial vulnerability can be a byproduct of networks' representational compression, complementing existing explanations based on data properties or architectural factors.

Figures

Figures reproduced from arXiv: 2510.11709 by the authors.

Figure 1
Figure 1. An adversarial attack exploiting superposition geometry (k = 7, m = 2). (a) The original sample. (b) The adversarially perturbed sample, whose ground truth remains the same but is misclassified. The sign and magnitude of an input perturbation is determined by the configuration of latent representations. (c) The original and adversarial sample in activation space. The arrows are the column vectors of We, the latent r… view at source ↗
Figure 2
Figure 2. Greater input correlations create more con [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An adversarial attack (from class 5 to class 3) does not perturb the input features for a class represented orthogonally (class 1). Does reducing superposition suppress these attacks? If superposition creates vulnerability through interference, then removing superposi￾tion should limit adversarial vulnerability. We test this via three experiments that limit interfer￾ence. First we set m = k, and the network learns t… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Left: CIFAR-10 class representation structure remains similar between models across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualisations of AExs in the toy model, supplementing [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Left: Each column is the mean attack δ added to a to successfully attack a + b mod p. Right: The norms of the Fourier components in the δ and WMLPout [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Histogram of activation counts for CLS token SAE features (threshold 0.1). [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Feature overlap between original vs. attacked and original vs. noisy images for CLS token [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Jaccard similarity for CLS token SAE features (threshold 0.1). [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Histogram of mean thresholds for CLS token SAE features (threshold 0.1). [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Histogram of activation counts for CLS token SAE features (threshold 0.001), with noise. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Feature overlap between original vs. attacked and original vs. noisy images for CLS token [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Comparison of Jaccard similarity for CLS token SAE features (threshold 0.001). [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Histogram of mean activations for CLS token SAE features (threshold 0.001), with noise. [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Histogram of activation counts for patch token SAE features (threshold 0.1), with noise. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Histogram of distinct patch features for patch token SAE features (threshold 0.1), with [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Feature overlap between original vs. attacked and original vs. noisy images for patch [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Comparison of Jaccard similarity for patch token SAE features (threshold 0.1). [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Histogram of mean thresholds for patch token SAE features (threshold 0.1), with noise. [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Histogram of activation counts for patch token SAE features (threshold 0.001), with noise. [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Histogram of distinct patch features for patch token SAE features (threshold 0.001), with [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: Feature overlap between original vs. attacked and original vs. noisy images for patch [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Comparison of Jaccard similarity for patch token SAE features (threshold 0.001). [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]
Figure 24
Figure 24. Figure 24: Histogram of mean activations for patch token SAE features (threshold 0.001), with noise. [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 7 linked inside Pith

  1. [1]

    On the interaction of com- pressibility and adversarial robustness.arXiv preprint arXiv:2507.17725,

    Melih Barsbey, Antônio H Ribeiro, Umut ¸ Sim¸ sekli, and Tolga Birdal. On the interaction of com- pressibility and adversarial robustness.arXiv preprint arXiv:2507.17725,

    arXiv

  2. [3]

    •H 1: Adversarial perturbations systematically exploit geometric relationships between superposed representations

    Research Q1: Do adversarial perturbations exploit superposition geometry? •H 0: Adversarial perturbations are random with respect to feature geometry. •H 1: Adversarial perturbations systematically exploit geometric relationships between superposed representations. Research Q2: Do data correlations determine superposition geometry? •H 0: Input correlation...

    2025

  3. [6]

    Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572,

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572,

    arXiv

  4. [10]

    Seeds 20/30/40’)

    to three different target models, each trained with one of the seeds listed in the sub-header (e.g., ’vs. Seeds 20/30/40’). The three slash-separated values in each cell correspond to the transferability to these three target seeds, respectively. All models within a row share the same bottleneck dimension, m. The ‘Mean ± Std’ column averages transferabili...

    2023

  5. [11]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  6. [12]

    Pawan Kumar, and Robert Stanforth

    Alessandro De Palma, Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar, and Robert Stanforth. IBP regularization for verified adversarial robustness via branch-and-bound.CoRR, abs/2206.14772,

    arXiv

  7. [13]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

    2024

  8. [14]

    Correlations in the data lead to semantically rich feature geometry under superposition.Mechanistic Interpretability Workshop at NeurIPS 2025, 2025a

    L Prieto, E Stevinson, M Barsbey, T Birdal, and P Mediano. Correlations in the data lead to semantically rich feature geometry under superposition.Mechanistic Interpretability Workshop at NeurIPS 2025, 2025a. Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability. InThe Thirteenth International Conf...

    2025

Show all 24 references
  1. [18]

    Szegedy, W

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In2nd International Conference on Learning Representations, ICLR 2014,

    2014

  2. [19]

    13 APPENDIX A GENERATION OFADVERSARIALEXAMPLES We use PGD to generate AExs, which is an iterative method to generate attacks (Madry et al., 2018). For untargeted attacks, it maximises the classifier’s loss function L(f(x ′), ytrue); for targeted, it minimisesL(f(x ′), ytarget)...

    2018

  3. [21]

    The leftmost plot now shows the cosine similarity matrix between pairs of latent representations for each of the classes. Figure 5: Visualisations of AExs in the toy model, supplementing Figure 1 from the main paper by illustrating attack mechanisms in activation space and inp...

    2023

  4. [24]

    The table compares a standard (pre-robustification) model with models certifiably trained using RSIP-IBP with different training perturbation budgets ( ϵtrain)

    Table 11: Verified robust accuracy (%) for modular addition models underℓ∞-norm perturbations. The table compares a standard (pre-robustification) model with models certifiably trained using RSIP-IBP with different training perturbation budgets ( ϵtrain). Accuracies are evalua...

    2023

  5. [113]

    Inputs are integer pairs (a, b), a, b∈ {0, . . . ,112}. Each integer is mapped to a 100-dimensional vector via a shared embedding WE ∈R 113×100. Embeddings Wa E and Wb E are concatenated into a 200-dim. feature vector. This vector feeds a 3-hidden-layer MLP (200 ReLU neurons/l...

    2018

  6. [2009]

    Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, and Sanmi Koyejo

    CIFAR-10 dataset. Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, and Sanmi Koyejo. What causes polysemanticity? an alternative origin story of mixed selectivity from incidental causes.arXiv preprint arXiv:2312.03096,

    arXiv

  7. [2014]

    A geometric notion of causal probing.arXiv preprint arXiv:2307.15054,

    Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing.arXiv preprint arXiv:2307.15054,

    arXiv

  8. [2017]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,

    arXiv

  9. [2018]

    Hamprecht, Yoshua Bengio, and Aaron C

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. InProceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Be...

    2019

  10. [2019]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025,

    2025

  11. [2020]

    Formal verification of piece-wise linear feed-forward neural networks

    Rüdiger Ehlers. Formal verification of piece-wise linear feed-forward neural networks. In Deepak D’Souza and K. Narayan Kumar (eds.),Automated Technology for Verification and Analysis - 15th International Symposium, ATVA 2017, Pune, India, October 3-6, 2017, Proceedings, volum...

    2017

  12. [2021]

    Mark Niklas Müller, Franziska Eckert, Marc Fischer, and Martin T. Vechev. Certified training: Small boxes are all you need. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  13. [2022]

    Adver- sarially robust generalization requires more data

    Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. InAdvances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2...

    2018

  14. [2023]

    Sparse autoencoders reveal selective remapping of visual concepts during adaptation.arXiv preprint arXiv:2412.05276,

    Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation.arXiv preprint arXiv:2412.05276,

    arXiv

  15. [2024]

    From flat to hierarchical: Extracting sparse representations with matching pursuit.arXiv preprint arXiv:2506.03093,

    Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit.arXiv preprint arXiv:2506.03093,

  16. [2025]

    Polysemanticity and capacity in neural networks.arXiv preprint arXiv:2210.01892,

    Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks.arXiv preprint arXiv:2210.01892,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.