Pith. sign in

REVIEW 3 major objections 7 minor 43 references

How Do Transformers Learn Variable Binding in Symbolic Programs?

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transformer trained from scratch on symbolic programs learns to bind variables to values by using its residual stream as an addressable memory, with a small set of attention heads routing the queried value along the assignment chain to…

desk verdict A solid behavioral developmental study with a plausible but under-validated mechanistic account; the unexplained 4-hop patching gap is the main soft spot. read the letter →

arxiv 2505.20896 v2 pith:CFNYOKQM submitted 2025-05-27 cs.LG cs.AIcs.CL

Yiwei Wu , Atticus Geiger , Raphaël Millière This is my paper
classification cs.LGcs.AIcs.CL
keywords variablebindingmechanisticinterpretabilitycausaltracinginterchangeinterventionsresidualstreamattentionheadssyntheticprogramslearningdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Variable binding—associating a name with a value—is the operation that lets symbolic programs work, and it is unclear whether a neural network without explicit memory can learn it. The paper trains a 37.8M-parameter transformer from scratch on 17-line programs where variables are assigned numbers or other variables and the model must output the value of a queried variable after following up to four hops, with distractor chains included. The model reaches over 99.9% accuracy and, along the way, passes through three distinct phases: random number prediction, a line-position heuristic, and a systematic dereferencing mechanism. Using causal interventions (replacing activations with counterfactual values), the authors show that the final mechanism exploits the residual stream as an addressable memory, with specific attention heads moving the root value along the query's chain to the output token. The result matters because it demonstrates that a symbol-manipulation capability can emerge from continuous vector computation without built-in binding operations, and that the systematic solution accumulates on top of the earlier heuristics rather than replacing them.

What carries the argument

The load-bearing tools are the interchange intervention and the residual stream itself, viewed as a 512-dimensional addressable memory. The interchange intervention replaces a component's activation—a residual-stream vector or an attention head output—with its value under a counterfactual input whose root number has been changed, and measures how often the model's top prediction flips to the new number. PCA followed by L1-regularized linear classifiers selects 10 components for numerical constants and 26 for variable names, and swapping only those subspaces between original and counterfactual inputs flips the model's answer 92.17% of the time for numbers and 87.08% for variable names. The circuit that carries the argument routes the counterfactual root value from the RHS token of the root assignment through the referential chain's RHS tokens in layers 6–9 to the query variable and colon tokens, with a rough layer-per-hop correspondence.

What would settle it

Train the same architecture and distribution but with the correct answer never on line 1; if the model still reaches above 99% accuracy without passing through a 56%-accuracy phase dominated by a line-1 heuristic, the claim that the systematic mechanism is necessarily built on that specific heuristic is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that a transformer can implement variable binding as a dynamic information-routing process rather than by storing a complete program state. Causal tracing with counterfactual root-value replacement shows that, in the trained model, the numerical value at the end of the query chain is propagated along the right-hand-side tokens of the assignment chain through layers 6–9, matched at the query variable token near layer 10, and moved to the final colon token. Individual attention heads mediate specific hops: heads 6.5 and 7.7 handle the first hop, heads 7.2, 8.3, and 9.4 handle the second and third hops, and heads 11.2, 11.3, and 11.7 transfer the result to the output position. Linear probes find no complete program state in any single vector (best layer, 6, reaches 30.87% variable-tracking accuracy), while swapping PCA-selected subspaces flips the model's answer 92.17% of the time for numerical constants and 87.08% for variable names, evidence that numbers and variables live in distinct, causally active subspaces. Interventions on checkpoints across training show the systematic circuit emerging after step ~34000 while the line-1 heuristic remains active for line-1 answers throughout, which the authors interpret as the systematic mechanism being layered on top of the earlier heuristics rather than replacing them.

Load-bearing premise

The conclusions assume that patching a single component's activation with a counterfactual value cleanly isolates that component's causal role; if the model's behavior relies on distributed, nonlinear, or compensating activity across many units, the identified heads and subspaces could be artifacts of the patching procedure.

Editorial extensions

If this is right

  • A transformer can acquire a symbol-manipulation operation—dereferencing chains of variable assignments—purely from next-token prediction, without architectural support for binding.
  • The learned circuit generalizes beyond its training envelope: accuracy stays high on unseen variable/value combinations and on chains up to 13 hops, although the line-1 heuristic itself fails on programs shorter or longer than the trained 16-line format.
  • The trajectory contradicts a strict 'grokking' narrative: the systematic solution does not replace the earlier line-position heuristic but is overlaid on it, so phase transitions in such models can be cumulative rather than competitive.
  • The model tracks only the bindings needed for the query rather than maintaining a complete program state, which is a memory-efficient strategy for symbolic tasks.
  • The layer-per-hop correspondence suggests that a transformer's depth may determine the length of dependency chains it can learn to trace without additional machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the residual-stream-as-addressable-memory account is right, then changing the residual stream dimension or head count should change how many hops a fixed-depth model can track; that prediction could be tested by measuring patching success on chains longer than four hops under modified widths.
  • The build-on-heuristics finding suggests a training-curriculum experiment: if the early-line answer is made unreliable from the start, the systematic circuit might emerge earlier or on a different trajectory, which the paper does not test.
  • The paper's setting uses single digits and single-letter variables; a natural extension is to check whether the same routing pattern survives multi-token values and longer variable names, where per-token routing would need to generalize.
  • Because head 8.3 participates in both the second and third hops, the mechanism may implement a reusable loop rather than a fresh circuit per hop; perturbing that head should then affect both hops jointly, consistent with the paper's patching data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper trains a 37.8M-parameter GPT-2-style Transformer from scratch on a synthetic variable-dereferencing task. Each program is a sequence of 16 assignment statements (var = const or var = var) plus a final query #var:; the model must output the numerical value bound to the queried variable, following chains of up to four assignments while ignoring distractor chains. The authors report a three-phase developmental trajectory: random number prediction, early-line heuristics (lines 1 and 2), and a systematic dereferencing mechanism. Using interchange interventions (activation patching) on the residual stream and on individual attention heads, they argue that the model stores and routes root values through the residual stream, with heads 6.5, 7.7, 8.3, 9.4, and 11.x mediating different hops. They also identify 10 PCA components encoding numerical constants and 26 encoding variable names, and report high swap-intervention success rates (92.17% and 87.08%). The paper includes held-out generalization experiments (longer programs, up to 13 hops, compositional generalization), a negative linear-probing result, and an interactive web platform.

Significance. If the mechanistic account holds, the paper makes a useful contribution to mechanistic interpretability and the connectionist-symbolic debate: it shows a Transformer can learn variable binding without explicit architectural support, that this learning is compositional and generalizes beyond training depth, and that early heuristics persist alongside the systematic circuit. Strengths include multi-seed training runs (Appendix F), held-out generalization to unseen combinations (Appendix G), longer programs (Appendix H), and 13-hop chains (Appendix I), as well as the negative linear-probing result (Appendix J), which meaningfully rules out a complete program-state representation. The interactive platform is a commendable reproducibility effort. However, as detailed below, the central mechanistic claim is not yet fully supported because the causal-intervention evidence has a critical gap for 4-hop programs and lacks standard faithfulness checks.

major comments (3)
  1. [Section 3.1, Fig. 3(b)] For 4-hop programs, the patching signal is reported as lost entirely after the Ref. Depth 2 RHS token. Since 4-hop programs are part of the training distribution and the paper's central claim is a complete mechanistic account of dereferencing assignment chains, this leaves the mechanism for the hardest in-distribution cases unexplained. The phrase 'increased difficulty' in Section 3.2 is not a mechanistic explanation. Please provide a quantitative account: for instance, report patching success at depths 3 and 4 separately for layers 8 and 9, or test whether a simultaneous multi-component patch of the hypothesized circuit recovers the signal. The fact that the model generalizes to 13 hops (Appendix I) makes the absence of a depth-3/4 signal especially puzzling and needs an explicit resolution.
  2. [Section 3, 'At the core of our approach is the interchange intervention method'] All mechanistic conclusions rest on single-component activation patching, but the manuscript does not validate the faithfulness of this intervention. Standard stress tests are missing: no simultaneous patching of the full hypothesized circuit, no ablation of the identified heads (zero- or mean-ablation), and no corruption/denoising baseline to show that the patching effect is specific to the hypothesized information rather than a generic sensitivity to any activation change. Without such checks, the identified heads (6.5, 7.7, 8.3, 9.4, 11.2/11.3/11.7) could be correlates rather than causes. Please add at least one validation experiment—for example, patch all heads in the proposed circuit simultaneously and show the output reliably flips, or ablate the heads and show target-task accuracy drops—or explicitly temper the causal claims to 'causal-tracing correlates.'
  3. [Section 3.3, subspace selection and intervention] The procedure appears to select the 10 numerical and 26 variable PCA components using L1-regularized linear classifiers trained on test-set activations, and then to intervene on those same components, apparently on the same or overlapping data. This creates a selection-circularity risk: components chosen to be predictive on a given set will tend to show high swap-intervention success on that set even if they are not the components the model actually uses. Please clarify whether the component selection was performed on a held-out subset disjoint from the intervention set, and report intervention success rates on programs not used for selection. If no such split was used, provide a cross-validated selection and re-run the intervention on held-out programs.
minor comments (7)
  1. [Section 2.1] The description of the four-token line structure applies to assignment statements but not to the final query line (#var:); the grammar in Appendix B clarifies this, but the main-text sentence should be qualified.
  2. [Figure 2 caption] The caption reads 'with rapid improvements at and steps'; the numeric training-step values appear to be missing due to a rendering issue. Please fix.
  3. [Figure 3(a) caption] The caption contains garbled text (e.g., '/uni...' encodings) and does not render the actual heatmap labels; please regenerate the figure so that the layer and token labels are readable.
  4. [Section 3.3] The method for selecting the 10 and 26 components is underspecified: please state the L1 regularization strength, whether the components are selected per class or globally, and how the number of components was chosen.
  5. [Appendix D] The claim 'Early experiments with simpler distributions showed that models could solve the task without developing genuine variable binding mechanisms' is not accompanied by any details or results; either provide a brief description or remove the unsupported claim.
  6. [Appendix A] The benchmark name 'hashhop' should be capitalized consistently, and the sentence beginning 'while these works focus on length generalization...' has a grammatical issue that should be corrected.
  7. [Appendix I] In the text, 'fig. 9' should be 'Fig. 9' for consistency with the other figure references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's causal intervention outcomes are independent behavioral tests rather than restatements of fitted inputs.

full rationale

The paper's derivation chain is self-contained, and its central mechanistic claims do not reduce to their own inputs by construction. The causal-tracing analysis (Section 3.1) uses interchange interventions: activations at specific (layer, token) positions are replaced with counterfactual values, and the success metric is whether the model's top-1 prediction flips to the counterfactual root value. This is a behavioral outcome independent of any fitted quantity, so identifying heads 6.5, 7.7, 8.3, and the residual-stream routing pattern is an empirical discovery, not a restatement of an input. The subspace analysis (Section 3.3) has a mild selection flavor, since PCA components are selected by L1-regularized classifiers that predict numerical constants and variable names, and the same test programs are used for selection and intervention; however, the causal claim is tested by a distinct operation that swaps only the spanned subspace and measures output flips (92.17% and 87.08%), which is not a re-derivation of classifier accuracy and could in principle have failed. The developmental claim that the final solution builds on early line-specific heuristics is supported by persistently high patching success at the layer-2 colon token in Fig. 4(e), again a behavioral measure rather than a fitted parameter. The paper's self-citations (Geiger et al. 2020, 2021, 2024) are to the interchange-intervention and causal-abstraction methodology, which is also externally established (Vig et al. 2020; Meng et al. 2022) and is not fitted to this paper's results, so the self-citations are not load-bearing. The acknowledged limitation that the patching signal is lost entirely for 4-hop programs after the Ref. Depth 2 RHS token, and the attenuated activation patterns at depths 3-4 (Fig. 3(b), Section 3.2), is a validity or completeness concern about whether the deepest chains are explained, not a circularity, since the reported measurements are not defined in terms of the conclusions they support. External behavioral benchmarks, including 10% held-out variable/number combinations (Appendix G), program-length generalization (Appendix H), and generalization to 13 hops (Appendix I), provide independent checks that the model has acquired a systematic mechanism, further indicating the findings are not tautological.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central empirical phenomenon depends on hand-chosen sampling parameters (documented in Appendix D) and on the faithfulness of activation-patching assumptions. The invented representational subspaces are empirically identified and causally validated within the paper, so they carry independent evidence rather than being unsupported postulates.

free parameters (3)
  • RHS constant-assignment probability = 0.30
    Each assignment line has a 30% chance of a numerical constant on the right-hand side. Chosen by hand; affects chain depth distribution and how often early lines contain root values, shaping Phase 2 heuristics.
  • Chain-extension cube weighting exponent = 3
    Chain extension is sampled with probability proportional to chain length cubed. Chosen to create long distractor chains and prevent length-based shortcuts, thereby forcing genuine binding tracking.
  • Referential depth rejection balancing = balanced 1-4 hops
    Rejection sampling equalizes the frequency of referential depths. Chosen to ensure all depths are tested and to make early-line root values more common, which the paper identifies as a driver of Phase 2 heuristics.
assumptions (3)
  • domain assumption Interchange interventions faithfully identify causal information flow in the trained model.
    Section 3 relies on the assumption that replacing activations with counterfactual values isolates the causal role of components; nonlinear interactions or redundant representations could confound localization.
  • domain assumption PCA plus L1-regularized linear probes recover the subspaces that actually encode numerical constants and variable names.
    Section 3.3 assumes the selected 10 and 26 principal components span the relevant encodings. If the encoding is nonlinear or more distributed, the subspace interventions may be misleading.
  • domain assumption The synthetic program distribution captures the essential features of variable binding relevant to the claimed conclusions.
    The generality of the 'Transformers can implement variable binding' claim depends on the task being representative; the paper's own sampling choices were tuned to prevent shortcuts, so the resulting behavior may be specific to this distribution.
invented entities (2)
  • Numerical constant subspace (10 PCA components) independent evidence
    purpose: Hypothesized subspace of the residual stream that encodes numerical constants at RHS token positions, enabling routing of root values.
    Identified via L1-linear probes on principal components, then validated by subspace swap interventions with 92.17% success (Section 3.3).
  • Variable name subspace (26 PCA components) independent evidence
    purpose: Hypothesized subspace encoding variable identities for binding and dereferencing.
    Identified via L1-linear probes and validated by subspace swap interventions with 87.08% success (Section 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Transformers Learn Variable Binding in Symbolic Programs?." pith.science (2026). https://pith.science/paper/CFNYOKQM

@misc{pith2026250520896,
  author       = {Pith},
  title        = {Pith review of: How Do Transformers Learn Variable Binding in Symbolic Programs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CFNYOKQM}},
  note         = {Machine review of arXiv:2505.20896}
}
read the original abstract

Variable binding -- the ability to associate variables with values -- is fundamental to symbolic computation and cognition. Although classical architectures typically implement variable binding via addressable memory, it is not well understood how modern neural networks lacking built-in binding operations may acquire this capacity. We investigate this by training a Transformer to dereference queried variables in symbolic programs where variables are assigned either numerical constants or other variables. Each program requires following chains of variable assignments up to four steps deep to find the queried value, and also contains irrelevant chains of assignments acting as distractors. Our analysis reveals a developmental trajectory with three distinct phases during training: (1) random prediction of numerical constants, (2) a shallow heuristic prioritizing early variable assignments, and (3) the emergence of a systematic mechanism for dereferencing assignment chains. Using causal interventions, we find that the model learns to exploit the residual stream as an addressable memory space, with specialized attention heads routing information across token positions. This mechanism allows the model to dynamically track variable bindings across layers, resulting in accurate dereferencing. Our results show how Transformer models can learn to implement systematic variable binding without explicit architectural support, bridging connectionist and symbolic approaches. To facilitate reproducible research, we developed Variable Scope, an interactive web platform for exploring our findings at https://variablescope.org

Figures

Figures reproduced from arXiv: 2505.20896 by the authors.

Figure 1
Figure 1. Example 3-Hop Program. While our programs have 17 lines, this example only has 7 lines for illustration. The variable chain of the query variable (w) includes 3 variable assignments or “hops.” This program also includes 3 irrelevant variable assign￾ments that act as distractors. An interactive version of this plot for any program can be viewed on variablescope.org. tation and cognition. It enables systems to represe… view at source ↗
Figure 2
Figure 2. Behavioral results showing the learning dynamics of our model across training steps. (a) Overall test set accuracy demonstrates three distinct learning phases, with rapid improvements at and steps. (b) Accuracy breakdown by correct answer line position (1–16) (c) Average accuracy of model checkpoints depending on the number of “hops” of the query variable chain in test set programs. (d) Distribution of model predict… view at source ↗
Figure 3
Figure 3. Patching analysis results on the final model checkpoint. In all experiments, logits are computed by passing the patched hidden states through the unembedding matrix to measure how interventions affect token predictions. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Evolution of patching success rates across training steps, revealing the development of line-specific heuristics and a general variable binding mechanism. Vertical dashed lines at steps 800 and 14,000 mark the transitions between major phases identified in our behavior…
Figure 5
Figure 5. Figure 5: Evolution of 2D UMAP for numerical-constant and variable-name residual stream subspaces (input to layer 6) across training. provides concrete evidence for how symbolic computation can emerge from continuous vector operations. Importantly, our probing experiments sugges…
Figure 7
Figure 7. Figure 7: Training accuracy comparison for evaluating composi￾tional generalization. The model trained with 10% of variable/num￾ber combinations held out achieves comparable test accuracy throughout training to the model trained on the full dataset. H. Generalization to Longer P…
Figure 6
Figure 6. Figure 6: Comparison of test set accuracy curves across multiple training runs initiated with different random seeds. All runs exhibit similar three-phase learning dynamics and transition points with slightly different convergence speeds. G. Generalization to Unseen Combinations…
Figure 8
Figure 8. Figure 8: Generalization performance across varying program lengths (2–25 lines). Performance is broken down by the line number containing the correct answer. While the model (trained on length 16) generalizes well when the answer is on line 2 or later, accuracy drops significan…
Figure 9
Figure 9. Figure 9: Generalization performance across varying program hop counts. The model maintains high accuracy on programs up to 13 hops (far exceeding the maximum of 4 seen during training) when the correct answer is not on lines 1 or 2, indicating a systematic solution. In contrast…
Figure 10
Figure 10. Figure 10: Patching results targeting individual attention heads across all 12 layers of the model. For each (head, token) position, we replace only that head’s contribution to the residual stream with its counterfactual value and compute logits. 16 [PITH_FULL_IMAGE:figures/ful…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 26 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Alhama, R. G. and Zuidema, W. A review of computational models of basic rule learning: The neural-symbolic debate and beyond. Psychonomic bulletin & review, 26 0 (4): 0 1174--1194, 2019

    work page 2019

  3. [3]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  4. [4]

    Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: Circuits. Distill, 2020. doi:10.23915/distill.00024. URL https://distill.pub/2020/circuits/

    doi:10.23915/distill.00024 2020

  5. [5]

    D., Schlichtkrull, M

    Cao, N. D., Schlichtkrull, M. S., Aziz, W., and Titov, I. How do decisions emerge across layers in neural models? I nterpretation with differentiable masking. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 3243--3255, Online, November 2020. Associati...

    doi:10.18653/v1/2020.emnlp-main.262 2020

  6. [6]

    D., Schmid, L., Hupkes, D., and Titov, I

    Cao, N. D., Schmid, L., Hupkes, D., and Titov, I. Sparse interventions in language models with differentiable masking. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2022. URL https://doi.org/10.18653/v1/2022.blackboxnlp-1.2

    doi:10.18653/v1/2022.blackboxnlp-1.2 2022

  7. [7]

    Causal scrubbing, a method for rigorously testing interpretability hypotheses

    Chan, L., Garriga-Alonso, A., Goldwosky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 2022. https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing

    work page 2022

  8. [8]

    Are neural nets modular? inspecting functional modularity through differentiable weight masks

    Csord \'a s, R., van Steenkiste, S., and Schmidhuber, J. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD

    work page 2021

Show all 43 references
  1. [9]

    Representational Analysis of Binding in Language Models

    Dai, Q., Heinzerling, B., and Inui, K. Representational Analysis of Binding in Language Models . In Al-Onaizan , Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pp.\ 17468--17493, Miami, Florida, ...

    2024

  2. [10]

    R., and Bau, D

    Davies, X., Nadeau, M., Prakash, N., Shaham, T. R., and Bau, D. Discovering variable binding circuitry with desiderata, 2023. URL https://arxiv.org/abs/2307.03637

    2023 arXiv

  3. [11]

    Elman, J. L. Generalization, rules, and neural networks: A simulation of M arcus et. al. HTML document, 1999

    1999

  4. [12]

    and Steinhardt, J

    Feng, J. and Steinhardt, J. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 a . URL https://openreview.net/forum?id=zb3b6oKO77

    2024

  5. [13]

    and Steinhardt, J

    Feng, J. and Steinhardt, J. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 b . URL https://openreview.net/forum?id=zb3b6oKO77

    2024

  6. [14]

    Monitoring Latent World States in Language Models with Propositional Probes , June 2024

    Feng, J., Russell, S., and Steinhardt, J. Monitoring Latent World States in Language Models with Propositional Probes , June 2024

    2024

  7. [15]

    and King, A

    Gallistel, C. and King, A. Memory and the Computational Brain: Why Cognitive Science will Transform Neuroscience. Blackwell/Maryland Lectures in Language and Cognition. Wiley, 2011. ISBN 9781444359763. URL https://books.google.com/books?id=o0jpHcgwkEoC

    2011

  8. [16]

    Neural natural language inference models partially embed theories of lexical entailment and negation

    Geiger, A., Richardson, K., and Potts, C. Neural natural language inference models partially embed theories of lexical entailment and negation. In Alishahi, A., Belinkov, Y., Chrupa a, G., Hupkes, D., Pinter, Y., and Sajjad, H. (eds.), Proceedings of the Third BlackboxNLP Work...

    2020 doi

  9. [17]

    Causal abstractions of neural networks

    Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing ...

    2021

  10. [18]

    C., and Potts, C

    Geiger, A., Carstensen, A., Frank, M. C., and Potts, C. Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review, 130 0 (2): 0 308--333, 2023. ISSN 1939-1471. doi:10.1037/rev0000371

    2023 doi

  11. [19]

    Causal abstraction: A theoretical foundation for mechanistic interpretability, 2024

    Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., and Icard, T. Causal abstraction: A theoretical foundation for mechanistic interpretability, 2024. URL https://arxiv.org/abs/2301.04709

    2024 arXiv

  12. [20]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  13. [21]

    S., Michaud, E., Tegmark, M., and Williams, M

    Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35: 0 34651--34663, 2022

    2022

  14. [22]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101

    2017 arXiv

  15. [23]

    Predicting inductive biases of pre-trained models

    Lovering, C., Jha, R., Linzen, T., and Pavlick, E. Predicting inductive biases of pre-trained models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=mNtmhaDkAr

    2021

  16. [24]

    Hashhop: Long context evaluation

    Magic. Hashhop: Long context evaluation. https://github.com/magicproduct/hash-hop, 2024

    2024

  17. [25]

    Marcus, G. F. Rethinking eliminative connectionism. Cognitive psychology, 37 0 (3): 0 243--282, 1998

    1998

  18. [26]

    Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science. The MIT Press, 04 2001. ISBN 9780262279086. doi:10.7551/mitpress/1187.001.0001. URL https://doi.org/10.7551/mitpress/1187.001.0001

    2001 doi

  19. [27]

    F., Vijayan, S., Rao, S

    Marcus, G. F., Vijayan, S., Rao, S. B., and Vishton, P. M. Rule learning by seven-month-old infants. Science, 283 0 (5398): 0 77--80, 1999

    1999

  20. [28]

    Umap: Uniform manifold approximation and projection for dimension reduction

    McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  21. [29]

    J., and Belinkov, Y

    Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y. Locating and editing factual associations in GPT . In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=-h6WAS6eE4

    2022

  22. [30]

    A tale of two circuits: Grokking as competition of sparse and dense subnetworks

    Merrill, W., Tsilivis, N., and Shukla, A. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023

    2023

  23. [31]

    S., Sun, J., Todd, E., Bau, D., and Belinkov, Y

    Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sharma, A. S., Sun, J., Todd, E., Bau, D., and Belinkov, Y. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024. UR...

    2024

  24. [32]

    Progress measures for grokking via mechanistic interpretability

    Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://...

    2023

  25. [33]

    R., Haklay, T., Belinkov, Y., and Bau, D

    Prakash, N., Shaham, T. R., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. ...

    2024

  26. [34]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  27. [35]

    and Wiegreffe, S

    Saphra, N. and Wiegreffe, S. Mechanistic? In The 7th BlackboxNLP Workshop, 2024. URL https://openreview.net/forum?id=schAf4BPtD

    2024

  28. [36]

    Seidenberg, M. S. and Elman, J. L. Networks are not `hidden rules'. Trends in Cognitive Sciences, 3 0 (8): 0 288--289, 1999 a

    1999

  29. [37]

    Seidenberg, M. S. and Elman, J. L. Do infants learn grammar with algebra or statistics? Science, 284 0 (5413): 0 433--433, 1999 b

    1999

  30. [38]

    J., Casper, S., Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland, J., Murfet, D., and McGrath, T

    Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garriga-Alonso, A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg, M., Schoots, N., Miller, J., Michaud, E. J., Casper, S., Tegma...

    2025 arXiv

  31. [39]

    Tensor product variable binding and the representation of symbolic structures in connectionist systems

    Smolensky, P. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46 0 (1): 0 159--216, 1990. ISSN 0004-3702. doi:https://doi.org/10.1016/0004-3702(90)90007-M. URL https://www.sciencedirect.com/scienc...

    1990

  32. [40]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  33. [41]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, pp.\ 5998--6008, 2017

    2017

  34. [42]

    Investigating gender bias in language models using causal mediation analysis

    Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Proces...

    2020

  35. [43]

    Unveiling transformers with LEGO : A synthetic reasoning task, 2023

    Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gunasekar, S., and Wagner, T. Unveiling transformers with LEGO : A synthetic reasoning task, 2023. URL https://openreview.net/forum?id=1jDN-RfQfrb

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.