Pith. sign in

REVIEW 3 major objections 4 minor 48 references

Can Interpretation Predict Behavior on Unseen Data?

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Simple attention-pattern readings on in-distribution data can predict which generalization rule a Transformer will follow out-of-distribution, even when the pattern is not the mechanism.

desk verdict Solid proof-of-concept for predicting OOD behavior from ID internals; the decoupling claim outruns the evidence and needs targeted ablation support. read the letter →

arxiv 2507.06445 v3 pith:BGC2WEQE submitted 2025-07-08 cs.LG cs.AIcs.CL

Victoria R. Li , Jenny Kaufmann , Tian Qin , Martin Wattenberg , David Alvarez-Melis , Naomi Saphra This is my paper
classification cs.LGcs.AIcs.CL
keywords interpretabilityout-of-distributiongeneralizationattentionpatternshierarchicalTransformersmechanisticcausalvscorrelationalanalysissyntheticDycktask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reading a model's internals can predict how it will behave on inputs outside its training distribution, rather than only how it responds to interventions. The authors train a population of small Transformers on an ambiguous parenthesis-classification task in which perfect in-distribution accuracy is compatible with two different rules: counting open and close parentheses, or checking proper nesting. They find that attention heads observed only on in-distribution data, specifically heads that track tree depth, predict which rule the model applies out-of-distribution. This predictive value survives even when ablations show those same heads do not cause the rule, and in one subtype they suppress it. The proof of concept suggests interpretability could become a reliability test for models under distribution shift, independent of causal mechanistic accounts.

What carries the argument

The central machinery is the hierarchical head, defined by depth-tracking attention at the end-of-sequence token. Depth at position $j$ is $d(j)=o(j)-c(j)$, where $o(j)$ and $c(j)$ are the cumulative counts of open and close parentheses up to that position. A head is depth-tracking on an input when it favors either negative-depth or non-negative-depth tokens, and a hierarchical head on a dataset when it does so on at least 80% of mixed-depth sequences (sequences containing both negative and non-negative depth tokens). Hierarchical heads split into negative-depth detectors, which always favor negative-depth tokens, and sign-matching heads, which favor the sign of the final token's depth. These heads, read from in-distribution activations, are the predictor; uniform attention ablation, which replaces all attention with equal weights, is the intervention that tests causal support.

What would settle it

Train the same model population while varying the 80% threshold (for example to 60% or 95%) and check whether the ID-head/OOD-NESTED correlation survives; alternatively, find a 2- or 3-layer model with a clear ID sign-matching head whose OOD labels follow EQUAL-COUNT, which would break the claimed predictive link.

Watch

Extended reading notes

Core claim

The paper's central claim is that observational interpretation of internal attention can forecast a model's out-of-distribution (OOD) generalization rule, and that this predictive power does not require the observed pattern to be causally load-bearing. In a population of small Transformers trained on data compatible with both the EQUAL-COUNT rule (equal numbers of open and close parentheses) and the NESTED rule (proper nesting), the authors define a hierarchical head as one whose end-of-sequence attention consistently favors either negative-depth or non-negative-depth tokens on at least 80% of mixed-depth sequences. Models with at least one in-distribution hierarchical head are more likely to apply NESTED on OOD inputs, and the correlation holds separately for 2- and 3-layer models. Uniform attention ablation shows the two subtypes behave differently: negative-depth detector heads support NESTED under ablation, while sign-matching heads, equally correlated with NESTED, suppress it; yet the same ablation barely changes in-distribution accuracy and its ID and OOD effects are only weakly correlated ($\rho = 0.24$). The paper therefore argues that correlational, instrumentalist interpretability can be valuable even when causal analysis would mislead.

Load-bearing premise

The load-bearing premise is the operational definition of a hierarchical head—depth as cumulative open minus close count and an 80% threshold on mixed-depth sequences—chosen without a sensitivity analysis; if that definition is arbitrary or unstable, the correlation between ID hierarchical heads and OOD NESTED behavior could be an artifact of the classifier rather than a property of model internals.

Editorial extensions

If this is right

  • A model with at least one in-distribution hierarchical head is more likely to generalize with NESTED on OOD data, and this holds separately for 2- and 3-layer models, adding predictive power beyond hyperparameters such as depth and weight decay.
  • Attention patterns can be correlated with a rule yet causally opposed to it: ablating sign-matching heads improves OOD NESTED accuracy, while ablating negative-depth detectors reduces it.
  • Causal conclusions drawn from in-distribution ablations need not transfer under distribution shift: uniform attention ablation barely changes ID validation accuracy but substantially changes OOD accuracy, and the two effects are only weakly correlated ($\rho = 0.24$).
  • OOD generalization rules are categorical, not continuous: models cluster into EQUAL-COUNT, NESTED, and FIRST-SYMBOL groups, and a vestigial FIRST-SYMBOL circuit can govern OOD judgments while having no detectable ID effect.
  • Correlation across a model population can validate an interpretation even when causal faithfulness fails, so mechanistic intervention is not the only standard for interpretability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same protocol could be applied to language models by replacing parenthesis depth with a syntactic depth proxy, then checking whether ID depth-tracking attention predicts OOD composition failures.
  • Editorial extension: a negative ID ablation result should be read as weak evidence against an interpretation, because the paper shows models can be nearly ID-immune to an ablation while remaining OOD-sensitive to it.
  • Editorial extension: if the 80% threshold is stable across settings, scanning attention heads on a small ID probe set could flag models likely to misgeneralize before any OOD labels are collected.
  • Editorial extension: the reusable experimental template is the population-correlation design itself, comparing many identically trained models to separate predictive signal from individual training luck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper asks whether observational interpretability tools can predict model behavior on out-of-distribution (OOD) inputs, rather than only predicting responses to mechanistic interventions. The authors train 270 small Transformers on a synthetic Dyck-style classification task whose in-distribution (ID) data is compatible with two rules, EQUAL-COUNT and NESTED, and measure which rule each model follows on an OOD test set. They report that models with at least one ID 'hierarchical' attention head--defined as a head whose EOS attention tracks token depth on at least 80% of mixed-depth sequences--tend to generalize hierarchically on OOD data, with the relationship holding across hyperparameter splits. They further report ablation results suggesting that some hierarchical heads correlate with NESTED behavior while actually suppressing it, leading them to argue for an 'instrumentalist' evaluation of interpretability in which predictive value need not align with causal support. The paper releases model weights, data, and code.

Significance. If the central correlational claim holds, this is a useful proof-of-concept for a new interpretability objective: predicting behavior on unseen inputs from passive inspection of internals. The study's strengths include a clearly described synthetic setup with a large, released population of trained models, explicit statistical tests (Mann-Whitney U) that are repeated within hyperparameter strata, and appendices that disclose many of the limitations of the ablation analysis. The finding that ID hierarchical attention heads predict OOD NESTED behavior is plausible and supported by the presented distributions. However, the paper's stronger decoupling claim--that prediction works 'even when the rule's implementation does not rely on these hierarchical patterns'--is not currently supported by the ablation evidence, because the ablation classification uses OOD head labels while the predictive analysis uses ID head labels, and the two classifications differ materially. This is fixable within the paper's scope, but it is load-bearing for the abstract's strongest claim.

major comments (3)
  1. [Section 4.3.2 and Appendix H] The causal 'suppression' result is not established for the heads used in the predictive analysis. Section 4.2 and Figures 4b/15/16 classify heads by their ID behavior, but Figure 5 and the surrounding text in Section 4.3.2 classify heads by their OOD behavior (Appendix E). The paper itself notes that 23% of ID hierarchical heads are not OOD hierarchical and that 25% of ID sign-matching heads become OOD negative-depth detectors, so the two classifications are far from interchangeable. Appendix H (Figure 18) directly shows that when heads are classified by ID behavior, uniform ablation tends to decrease OOD accuracy for sign-matching heads as well as for negative-depth heads. Consequently, the abstract's claim that ID hierarchical patterns predict NESTED 'even when the rule's implementation does not rely on these hierarchical patterns' overstates what the ablation experiments show; the ablation evidence applies to OOD-classified heads, not to the ID heads that did the predicting.
  2. [Section 4.3.1] The uniform all-head attention ablation confounds the interpretation of 'suppression.' Replacing every attention activation with uniform attention removes all attention patterns, not just the depth-tracking pattern, and the paper itself cites Wen et al. [41] to note that uniform attention is sufficient to implement NESTED. Thus an improvement in OOD accuracy after ablation could mean that the uniform replacement is a better NESTED implementation than the model's actual attention, rather than that the sign-matching head actively suppresses NESTED. Appendix G's one-head-at-a-time ablation partially addresses the co-occurrence concern, but it still classifies heads by OOD behavior. A targeted ablation that removes only the depth-tracking signal, or a re-analysis that classifies heads strictly from ID validation data, would be needed to support the causal decoupling claim.
  3. [Section 4.1.2] The definition of a 'hierarchical head' depends on the 80% threshold on mixed-depth sequences and on the specific choice of depth d(j) = o(j) - c(j) as the relevant latent structure, but no sensitivity analysis is provided for either choice. Since the central predictive result is defined entirely through this threshold, the paper should show that the ID-head/OOD-NESTED correlation is stable over a reasonable range of thresholds (e.g., 60% to 95%) and, ideally, that a comparable depth-based definition is not a knife-edge choice. Without such an analysis, it is difficult to rule out that the reported predictive power is an artifact of the particular operationalization.
minor comments (4)
  1. [Limitations] The Limitations section contains a typo: 'it it would take further effort' should read 'it would take further effort.'
  2. [Appendix B.1] The text contains the typo 'In-distibution'; this should be 'In-distribution.'
  3. [Figure 1] The schematic text includes an apparent rendering artifact, 'T est effect', which should be cleaned up.
  4. [Appendix G] The first sentence of Appendix G reads 'We found effects that were generally generally very similar'; the duplicated word should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central prediction is a held-out correlational claim, with no parameter fitted to the OOD target labels.

full rationale

The derivation is not circular. The central predictive claim is correlational: attention heads are classified as hierarchical using only ID validation data (Section 4.1.2), and the target rule is then read off from held-out OOD outputs. No parameter is fit to the OOD labels; the 80% mixed-depth threshold is a fixed definitional choice rather than a fitted parameter, and its arbitrariness is a robustness concern rather than circularity. The depth feature d(j)=o(j)-c(j) is admittedly motivated by the NESTED rule, but the observed correlation between depth-tracking attention and OOD behavior is an empirical result that could have failed, so the prediction is not equivalent to its input by construction. The decoupling claim in the abstract is supported by uniform attention ablation and by head types classified according to OOD behavior, while Appendix H shows ID sign-matching heads behave differently under ablation; this mismatch is an evidential gap in the causal-interpretation argument, not a circular reduction of the predictive claim. Self-citations ([28], [43], [15], [29]) are used only for background observations about rule clustering, random-seed effects, and recurrent inductive bias; they are not load-bearing, they do not supply a uniqueness theorem, and they do not smuggle in the central ansatz. No equation or fitted parameter is renamed as a prediction, so no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the operational definition of hierarchical heads (with two hand-picked thresholds), the assumption that EOS attention patterns are meaningful traces, the choice of depth as the relevant feature, and the validity of uniform attention ablation for causal conclusions. The vestigial circuit is a speculative explanatory entity. No parameters are fitted to OOD labels, so circularity burden is low.

free parameters (2)
  • Hierarchical head threshold = 80% (0.8) of mixed-depth sequences
    Hand-chosen threshold in Section 4.1.2 to classify a head as 'hierarchical'. No sensitivity analysis is provided, and the central correlation depends on this binary classification.
  • Depth-favoring threshold t = Unspecified positive threshold
    In Equations (5)-(6), a head 'favors' negative or non-negative depth tokens if attention is separable by some t > 0. The existence of any such t is used to define depth-tracking; the precise value is not reported and may vary by input.
assumptions (5)
  • domain assumption Attention activations at the EOS token carry meaningful information about the model's implemented algorithm.
    The entire approach inspects aEOS(i) (Equation 3) and assumes these are interpretable traces; Section 2.2.
  • domain assumption The synthetic Dyck-1 style task is a useful model for studying OOD generalization and interpretability.
    The paper builds on Dyck-1 [33, 12, 24, 41] and uses it to claim a proof of concept; Section 2.1.
  • ad hoc to paper Uniform attention ablation removes only the influence of attention patterns and leaves other computation intact enough to infer causal roles.
    Section 4.3.1 states this intervention 'strips any influence of depth-tracking attention or other attention activation patterns', but also acknowledges co-occurrence issues.
  • ad hoc to paper Token depth d(j)=o(j)-c(j) is the relevant latent structure for hierarchical processing.
    The depth definition (Equation 4) and the focus on negative versus non-negative depth tokens is motivated by the NESTED rule, but no evidence is given that this is the unique or correct structural description.
  • standard math Statistical tests (Mann-Whitney U, Spearman correlation) are appropriately applied to the model population.
    Used in Sections 3.2, 4.3.3, and appendices; assumes independence across models, which is plausible since models are independently trained.
invented entities (1)
  • Vestigial circuit
    purpose: To explain why some models apply the FIRST-SYMBOL heuristic on OOD data but not ID data.
    The paper hypothesizes that an early-training circuit persists and affects OOD judgments (Section 3.1.1). It is inferred from checkpoint behaviors, not directly observed, and is not a falsifiable prediction outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Interpretation Predict Behavior on Unseen Data?." pith.science (2026). https://pith.science/paper/BGC2WEQE

@misc{pith2026250706445,
  author       = {Pith},
  title        = {Pith review of: Can Interpretation Predict Behavior on Unseen Data?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGC2WEQE}},
  note         = {Machine review of arXiv:2507.06445}
}
read the original abstract

Interpretability research often predicts model responses to targeted mechanistic interventions. But can we predict responses to unseen input data? We propose and demonstrate this alternate objective by using model internals to predict their out-of-distribution (OOD) behavior. We train hundreds of Transformers on simple synthetic tasks, where perfect in-distribution accuracy is compatible with multiple OOD generalization rules. We successfully use attention patterns -- observed only on in-distribution data -- to predict which rule each model follows on OOD data. Our experiments decouple the mechanistic faithfulness of our interpretation from its predictive value; ablations reveal such internal patterns can suppress rather than support the rule they predict, showing observational analysis can forecast behavior even when causal analysis fails to support a simple cause-effect link. Our findings are a proof-of-concept for a new interpretability objective: understanding model internals to predict behavior and assess reliability under distribution shift.

Figures

Figures reproduced from arXiv: 2507.06445 by the authors.

Figure 1
Figure 1. Our approach. In a population of independently trained classifiers, we correlate internal structures with OOD behaviors. When can we claim to understand a system? One standard, that of the classic scientific method [14], re￾quires testable predictions of behavior under unseen conditions. Accordingly, interpretability research of￾ten assesses proposed mechanisms by predicting the effect of a test-time mechanistic int… view at source ↗
Figure 2
Figure 2. Similarly trained models vary in systematic generalization rules. 3.1 Models cluster around systematic rules We visualize all models according to their OOD output probabilities in Figure 2a. One cluster has near-0% OOD accuracy and another has high OOD accuracy, supporting existing claims [28, 43] that rule-following is a categorical phenomenon, not a continuum. Note, however, a third cluster in which models exhibit… view at source ↗
Figure 3
Figure 3. Unregularized models can enter tran￾sient heuristic stages during training. Interme￾diate checkpoints of all one-layer models without weight decay, sorted by final OOD accuracy. The gradient represents an intermediate checkpoint’s OOD accuracy. Each cell is left white if ID accu￾racy falls below 0.99, i.e., if the checkpoint can￾not model its training distribution. The color bar is marked in magenta for sections cor… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Hierarchical attention patterns correlate with hierarchical generalization rules. • An attention head favors negative depth tokens on input s if there exists threshold t > 0 such that: ∀j ∈ {1, . . . , n} : aEOS(j) ≥ t iff d(j) < 0 (5) • An attention head favors non-ne…
Figure 5
Figure 5. Figure 5: Some hierarchical attention patterns damage the implementation of hierarchical rules. OOD accuracy before and after applying uniform attention ablation to all attention heads. Each point represents a single model, colored by presence of an OOD sign-matching and/or nega…
Figure 6
Figure 6. Figure 6: Last OOD test accuracy for LSTM and Transformer models that achieve 99%+ ID accuracy. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Unlike weight decay and depth (Figure 2b), width is not a substantial factor in final OOD rule selection in this setting. The Mann-Whitney U test finds no statistically significant differences in the distribution of last OOD accuracy over width (all p > 0.05). C.3 Dept…
Figure 8
Figure 8. Figure 8: Final accuracy on OOD-test for Transformers of varying depths and weight decays. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: T-SNE of models’ final OOD classifications colored by model depth. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Data order (left) and weight initialization (right) are equally influential random factors in [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Illustration of generalization rules across training. ID convergence occurs when models [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Heatmaps showing model training dynamics broken down by depth, where purple and [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Percentage of 2- and 3-layer models containing each head type, by weight decay. Head [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: OOD accuracy of 2- and 3-layer models with and without OOD hierarchical heads, [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Last OOD test accuracy of 2 and 3-layer models with and without ID hierarchical heads. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: OOD accuracy of 2 and 3-layer models with and without ID hierarchical heads, by number [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: OOD accuracy before and after applying uniform attention ablation to one head in each [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: For heads classified by type based on their ID behavior, accuracy ID and OOD after [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: For heads classified by type based on their OOD behavior, accuracy ID and OOD after [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [41]

    K. Wen, Y . Li, B. Liu, and A. Risteski. Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023. URL https://par.nsf.gov/biblio/10489627

    work page arXiv 2023

  2. [1]

    Abnar, M

    S. Abnar, M. Dehghani, and W. Zuidema. Transferring inductive biases through knowledge distillation, 2020. URL https://arxiv.org/abs/2006.00555

    arXiv 2020

  3. [2]

    D. B. Arnold and M. R. Sleep. Uniform random generation of balanced parenthesis strings. ACM Trans. Program. Lang. Syst., 2(1):122–128, 1980. ISSN 0164-0925. doi: 10.1145/357084. 357091. URL https://doi.org/10.1145/357084.357091

    arXiv 1980

  4. [3]

    Bibal, R

    A. Bibal, R. Cardon, D. Alfter, R. Wilkens, X. Wang, T. François, and P. Watrin. Is attention explanation? an introduction to the debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3889–3900, 2022

    work page 2022

  5. [4]

    S. C. Y . Chan, A. Santoro, A. K. Lampinen, J. X. Wang, A. Singh, P. H. Richemond, J. Mc- Clelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers, 2022. URL https://arxiv.org/abs/2205.05055

    arXiv 2022

  6. [5]

    X. Chen, R. Pan, X. Wang, F. Tian, and C.-Y . Tsui. Late breaking results: Weight decay is all you need for neural network sparsification. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–2, 2023. doi: 10.1109/DAC56929.2023.10247950

    arXiv 2023

  7. [6]

    P. M. Churchland. Eliminative materialism and the propositional attitudes. the Journal of Philosophy, 78(2):67–90, 1981

    work page 1981

  8. [7]

    D. C. Dennett. The intentional stance. MIT press, 1989

    work page 1989

Show all 48 references
  1. [8]

    D. C. Dennett. Real patterns. The journal of Philosophy, 88(1):27–51, 1991

    1991

  2. [9]

    Dodge, G

    J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping, 2020. URL https://arxiv.org/abs/2002.06305

    2020 arXiv

  3. [10]

    Doshi, A

    D. Doshi, A. Das, T. He, and A. Gromov. To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets, 2024. URL https://arxiv.org/abs/ 2310.13061. 10

    2024 arXiv

  4. [11]

    P. Duhem. The aim and structure of physical theory. na, 1954

    1954

  5. [12]

    Ebrahimi, D

    J. Ebrahimi, D. Gelda, and W. Zhang. How can self-attention networks recognize Dyck-n languages? In T. Cohn, Y . He, and Y . Liu, editors,Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4301–4306, Online, Nov. 2020. Association for Computational Li...

    2020 doi

  6. [13]

    Geiger, D

    A. Geiger, D. Ibeling, A. Zur, M. Chaudhary, S. Chauhan, J. Huang, A. Arora, Z. Wu, N. Good- man, C. Potts, and T. Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability, 2024. URL https://arxiv.org/abs/2301.04709

    2024 arXiv

  7. [14]

    Hepburn and H

    B. Hepburn and H. Andersen. Scientific Method. In E. N. Zalta, editor, The Stanford Encyclo- pedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021

    2021

  8. [15]

    Juneja, R

    J. Juneja, R. Bansal, K. Cho, J. Sedoc, and N. Saphra. Linear connectivity reveals generalization strategies. In International Conference on Learning Representations, 2023

    2023

  9. [16]

    Karpathy

    A. Karpathy. MinGPT transformer model, 2020. URL https://github.com/karpathy/ minGPT

    2020

  10. [17]

    Kissane, robertzk, N

    C. Kissane, robertzk, N. Nanda, and A. Conmy. SAEs are highly dataset dependent: a case study on the refusal direction. Alignment Forum ,

  11. [18]

    Kramár, T

    J. Kramár, T. Lieberum, R. Shah, and N. Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URL https://arxiv.org/abs/2403.00745

    2024 arXiv

  12. [19]

    S. Liu, H. Ye, L. Xing, and J. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering, 2024. URL https://arxiv.org/abs/2311. 06668

    2024

  13. [20]

    Makelov, G

    A. Makelov, G. Lange, and N. Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching, 2023. URL https://arxiv.org/abs/2311.17030

    2023 arXiv

  14. [21]

    D. Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010

    2010

  15. [22]

    R. T. McCoy, R. Frank, and T. Linzen. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140, 2020. doi: 10.1162/tacl_a_00304. URL https: //aclantholog...

    2020 doi

  16. [23]

    R. T. McCoy, J. Min, and T. Linzen. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance, 2020. URL https://arxiv.org/abs/1911.02969

    2020 arXiv

  17. [24]

    Murty, P

    S. Murty, P. Sharma, J. Andreas, and C. Manning. Grokking of hierarchical structure in vanilla transformers. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),...

    2023 doi

  18. [25]

    S. Okasha. Philosophy of science: very short introduction. Oxford University Press, 2016

    2016

  19. [26]

    Petty, S

    J. Petty, S. van Steenkiste, I. Dasgupta, F. Sha, D. Garrette, and T. Linzen. The impact of depth on compositional generalization in transformer language models, 2024. URL https: //arxiv.org/abs/2310.19956

    2024 arXiv

  20. [27]

    H. Putnam. Mathematics, Matter and Method: Volume 1, Philosophical Papers, volume 1. cup Archive, 1975

    1975

  21. [28]

    T. Qin, N. Saphra, and D. Alvarez-Melis. Sometimes I am a tree: Data drives unstable hierarchical generalization, 2024. URL https://arxiv.org/abs/2412.04619

    2024

  22. [29]

    Saphra and A

    N. Saphra and A. Lopez. LSTMs compose—and Learn—Bottom-up. In T. Cohn, Y . He, and Y . Liu, editors,Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2797–2809, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.find...

    2020 doi

  23. [30]

    Smith, S

    L. Smith, S. Rajamanoharan, A. Conmy, CallumMcDougall, T. Lieberum, J. Kramár, R. Shah, and N. Nanda. Negative results for SAEs on downstream tasks and deprioritising SAE research (GDM mech interp team progress update #2), 2025. URL https://www.alignmentforum. org/posts/4uXCAJ...

    2025

  24. [31]

    Stander, Q

    D. Stander, Q. Yu, H. Fan, and S. Biderman. Grokking group multiplication with cosets, 2024. URL http://arxiv.org/abs/2312.06581

    2024 arXiv

  25. [32]

    Subramani, N

    N. Subramani, N. Suresh, and M. E. Peters. Extracting latent steering vectors from pretrained language models, 2022. URL https://arxiv.org/abs/2205.05124

    2022 arXiv

  26. [33]

    Suzgun, S

    M. Suzgun, S. Gehrmann, Y . Belinkov, and S. M. Shieber. Memory-augmented recurrent neural networks can learn generalized Dyck languages, 2019. URL https://arxiv.org/abs/1911. 03329

    2019

  27. [34]

    D. Tan, D. Chanin, A. Lynch, D. Kanoulas, B. Paige, A. Garriga-Alonso, and R. Kirk. Analyzing the generalization and reliability of steering vectors, 2025. URL https://arxiv.org/abs/ 2407.12404

    2025 arXiv

  28. [35]

    Y . Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. URL https://arxiv.org/abs/2109.10686

    2022 arXiv

  29. [36]

    Tessier, V

    H. Tessier, V . Gripon, M. Léonardon, M. Arzel, T. Hannagan, and D. Bertrand. Rethinking weight decay for efficient neural network pruning. Journal of Imaging, 8(3):64, Mar. 2022. ISSN 2313-433X. doi: 10.3390/jimaging8030064. URL http://dx.doi.org/10.3390/ jimaging8030064

    2022 doi

  30. [37]

    E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models, 2024. URL https://arxiv.org/abs/2310.15213

    2024 arXiv

  31. [38]

    K. Tran, A. Bisazza, and C. Monz. The importance of being recurrent for modeling hierarchical structure, 2018. URL https://arxiv.org/abs/1803.03585

    2018 arXiv

  32. [39]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  33. [40]

    J. Vig, S. Gehrmann, Y . Belinkov, S. Qian, D. Nevo, Y . Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing ...

    2020

  34. [42]

    Zhang and N

    F. Zhang and N. Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024. URL https://arxiv.org/abs/2309.16042

    2024 arXiv

  35. [43]

    R. Zhao, N. Saphra, and S. M. Kakade. Distributional scaling laws for emergent capabilites,

  36. [45]

    12 A Glossary Term Definition and Usage Depth Parentheses sequence token chracteristic

    URL https://openreview.net/pdf?id=e8eo9iEFaO. 12 A Glossary Term Definition and Usage Depth Parentheses sequence token chracteristic. At index j, the token depth is o(j) − c(j) where o(j) and c(j) are the cumulative counts of ( and ) up to j (Equation 4). Hierarchical head Hea...

  37. [46]

    These properties ensure that our samples are concentrated around a reasonable center, reducing extreme sequence lengths that could occur with other distributions like the Uniform

    Sample a sequence length n from a Binomial (40, 0.5) distribution, with mean 20 and variance 10. These properties ensure that our samples are concentrated around a reasonable center, reducing extreme sequence lengths that could occur with other distributions like the Uniform. ...

  38. [47]

    • To generate a uniformly random sequence that is neither EQUAL -COUNT nor NESTED , we choose each character independently from the set { (, ) }

    Generate a uniformly random parentheses sequence of length n with the desired attributes. • To generate a uniformly random sequence that is neither EQUAL -COUNT nor NESTED , we choose each character independently from the set { (, ) }. If the resulting sequence satisfies EQUAL...

  39. [48]

    shuffle seed,

    If the sequence generated does not already appear in the dataset, add it to the dataset. Thus, each length-n sequence s with the desired attributes is equally likely to be chosen, and it is chosen at most once. Since we discard repeats, the empirical distribution of sequence l...

  40. [2024]

    URL https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/ saes-are-highly-dataset-dependent-a-case-study-on-the

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.