Pith. sign in

REVIEW 3 major objections 5 minor 67 references

Can Input Attributions Explain Inductive Reasoning in In-Context Learning?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Conventional input attributions can locate the single disambiguating 'aha' example in in-context learning, but the simplest gradient-based method—gradient norm—does this best, and larger models are generally harder to interpret with…

desk verdict A useful and honest benchmark for input attribution in ICL, but the headline GN-over-IG ranking is confounded by items where the aha example is not causally needed. read the letter →

arxiv 2412.15628 v5 pith:N754FZFN submitted 2024-12-20 cs.CL

Mengyu Ye , Tatsuki Kuribayashi , Goro Kobayashi , Jun Suzuki This is my paper · ORCID
classification cs.CL
keywords inputattributionin-contextlearninginterpretabilitylargelanguagemodelsgradientnormintegratedgradientsinductivereasoningahaexample
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether standard input-attribution (IA) methods—tools that score how much each input token contributed to a prediction—can explain how a language model performs inductive reasoning from a few in-context examples. To make the question answerable, the authors build synthetic tasks in which most demonstrations are ambiguous between two rules and exactly one 'aha' demonstration resolves the ambiguity. They find that a simple gradient-norm score is the most reliable gradient-based method for pointing to that deciding example, while the more elaborate integrated gradients method consistently does worst among the gradient-based methods. They also find that larger models are generally harder to interpret with gradient-based attributions, although asking the model to name the helpful example works better at larger scale. If this is right, the practical choice of interpretability tool for language models depends on model size and context length, not just on methodological sophistication.

What carries the argument

The load-bearing object is the 'aha example' benchmark: a set of synthetic inductive-reasoning tasks in which every demonstration except one is consistent with two competing rules, and the single remaining demonstration disambiguates which rule is correct. Attribution methods are scored by whether that example receives the highest (top-1) or one of the two highest (top-2) attribution scores in the prompt, with the top-2 metric reflecting that the model must combine the aha example with some other example to infer the rule. The attribution scores themselves are computed at token level and summed within each demonstration; for gradient norm, input×gradient, and input erasure the paper uses a contrastive formulation that targets the correct answer against a foil answer corresponding to the losing rule. This machinery converts the vague question 'which example mattered?' into a measurable localization task, and it is what makes the ranking of methods comparable across tasks and models.

What would settle it

Run the same six tasks in a modified version where the disambiguating information is split across two complementary examples so that no single example is necessary; if gradient-norm attribution still assigns one of them a top score as often as it does in the original benchmark, then the method is keying on position or surface similarity rather than on the disambiguating information itself.

Watch

Extended reading notes

Core claim

The paper's central discovery, on its own terms, is that conventional input attribution methods can track at least part of the inductive reasoning process in in-context learning, but the ordering of methods defies the usual assumption that more sophisticated gradient-based methods are better. Across six synthetic tasks, the simplest gradient-based score—gradient norm computed on the contrastive objective of raising the correct answer's probability while lowering a foil's—outperforms input×gradient and integrated gradients, and input erasure outperforms all of them because the tasks are deliberately unsolvable without the disambiguating example. The paper further establishes that interpretability does not scale uniformly with model size: gradient-based attribution accuracy generally falls for larger models, while a self-answer baseline that asks the model to name the helpful example improves with size but degrades sharply when the number of demonstrations grows. Some tasks resembling linguistic generalization (verb-object, tense-article, part-of-speech/title) remain hard for every tested approach, which the authors attribute to the models' own inductive biases rather than to a single flaw in any one method.

Load-bearing premise

The benchmark assumes that the single 'aha' example is the unique ground-truth locus of the model's inductive reasoning, so top-1 and top-2 attribution accuracy against that example measures whether an explanation captures the reasoning; if the model actually leans on distributed demonstrations, surface cues, or its own inductive biases, the accuracy figures no longer measure explanatory faithfulness.

Editorial extensions

If this is right

  • Gradient norm is a cheap, implementation-light diagnostic for finding which demonstration in a few-shot prompt is steering the model's answer, and it remains usable when the prompt is lengthened to 50 or 100 examples.
  • The added computation of integrated gradients does not pay for itself in this setting; researchers wanting to localize influential examples should try the simplest gradient-based score first.
  • Interpreting larger models with gradient-based attributions becomes harder, so interpretability results obtained on small models should not be assumed to transfer to larger ones.
  • Asking the model itself to say which example it used behaves oppositely: it improves with model size but collapses with longer contexts, so the two families of explanation cover different regimes.
  • Current attribution methods are not yet general-purpose in-context-learning explainers, since several linguistically flavored tasks resisted all tested approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gradient norm's edge over input×gradient really comes from ignoring embedding norms, then tasks with high-variance token frequencies should widen that gap; that is a testable prediction the paper does not run.
  • The single-aha-example design may understate attribution quality in realistic few-shot prompts where the decisive information is spread across several demonstrations; a distributed-cue variant of the benchmark would separate 'identifying a critical example' from 'explaining the whole inference.'
  • Because the paper's own control shows models adopt default rules when the aha example is absent, attribution accuracy may partly measure how well the model's preferred rule aligns with the intended rule; conditioning the benchmark on cases where the model has no default preference would sharpen the comparison.
  • A hybrid explanation pipeline—using gradient norm to prune demonstrations and then asking the model to verbalize its reasoning on the shortlist—could combine the scale trend of self-answer with the long-context robustness of gradient methods; the paper's results suggest such a combination is worth testing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether conventional input attribution (IA) methods can identify the single disambiguating 'aha' example in in-context learning (ICL). To make this question tractable, the authors design synthetic tasks (LINEAR-OR-DISTINCT, ADD-OR-MULTIPLY, VERB-OBJECT, TENSE-ARTICLE, POS-TITLE, and ASSOCIATIVE-RECALL) in which most demonstrations are ambiguous between two rules and one example resolves the ambiguity. They evaluate four IA methods (input erasure, gradient norm, input×gradient, integrated gradients) plus baselines (edit distance, attention, self-answer, chance) on six open-weight LLMs under 10-, 50-, and 100-shot settings, measuring top-1 and top-2 accuracy for locating the aha example. The main reported findings are that input erasure performs best, that simple gradient norm (GN) is the best gradient-based method, that integrated gradients is consistently worst among gradient methods, and that gradient-based interpretability tends to worsen with model scale while remaining relatively robust to the number of demonstrations.

Significance. If the evaluation is valid, the paper provides practically useful guidance for ICL interpretability: the simplest gradient method can outperform more sophisticated ones, and scaling behavior differs across explanation families. The study is broad in scope, covering six models, six tasks, three context lengths, multiple token-aggregation schemes, base-model comparisons, and a chain-of-thought extension; the appendices report assets and compute in detail, and the code is linked. The central caveat is that the benchmark's validity depends on the aha example being the causal locus of the model's decision. The paper's own control results (Table 2) show that for many items the model answers correctly without the aha example, so the reported attribution accuracies are computed on a mixture of aha-dependent and aha-independent items. This makes the headline quantitative comparisons hard to interpret without a re-analysis on the causal subset, although the authors qualitatively acknowledge the issue in Section 6 and the Limitations section.

major comments (3)
  1. [§4.2 and Table 2] The top-1/top-2 evaluation against the aha example assumes that the model's correct prediction is causally dependent on that example. Table 2 shows this fails for a large fraction of Gemma-2-2B items: task accuracy without the aha example is 98.0% for LD (Rule A), 100.0% for VO (Rule A), and 98.0% for TA (Rule A). For those items the correct answer is determined by the model's inductive bias or by other demonstrations, so an attribution score for the aha example has no well-defined ground truth. Appendix E excludes only incorrect-answer instances, not aha-independent correct instances. The main results in Section 5 and the scaling claims in Section 5.2 therefore compare methods on a mixture of items where the target variable is causally relevant and items where it is not. The discussion in Section 6 and the Limitations acknowledges this qualitatively, but the paper does not re-analyze the data on the subset where removing the aha example changes the answer. I request this conditioning as a required revision, because the GN-versus-IG comparison and the model-size conclusions could differ substantially on the causal subset.
  2. [§5 and Figures 2–3] All reported attribution accuracies are point estimates with no confidence intervals or significance tests. For example, in Figure 2, differences between GN and I×G top-2 accuracy are often in the single digits over 360 items per task, and the claim that 'IG consistently yielded the lowest attribution accuracy' (Section 5.1) is based on rankings that may be within sampling noise. The paper would be much stronger if each bar carried a bootstrap confidence interval or if pairwise differences were tested (e.g., McNemar's test for paired items). Without such statistics, it is difficult to judge whether the headline finding that GN outperforms I×G and IG is robust or an artifact of a particular seed or item set.
  3. [§5.2 and Figure 3] The scaling claim that gradient-based methods 'maintain accuracy or rather improve against the longer context' is made on a single model (Gemma-2-2B) without error bars. The 10-shot chance rate is 20% for top-2, but the 100-shot chance rate is 2%, so raw accuracy comparisons across context lengths conflate method quality with the decreasing chance baseline. A normalized metric (e.g., accuracy minus chance, or a lift measure) or at least a statistical test against chance would make the robustness claim more convincing. This is load-bearing for the paper's second practical finding.
minor comments (5)
  1. [Table 1] The ADD-OR-MULTIPLY row contains a typo: 'numder' should be 'number'.
  2. [Figure 6 caption] The caption says 'can be obsered'; this should be 'observed'.
  3. [Appendix E] The phrase 'attribution accurices' should be 'attribution accuracies.'
  4. [§2.1, Eq. (5)] The notation x_i is used both for the token index and as a vector in R^d; the distinction between the token at position i and its embedding should be made explicit in the text.
  5. [§4.2] The justification for top-2 accuracy states that the model 'should at least consider the e* plus any other example,' but this presumes the model performs pairwise comparison. The authors later acknowledge in Section 6 that models may instead rely on inductive biases, so the top-2 rationale should be softened or explicitly conditioned on the model actually using the disambiguating example.

Circularity Check

1 steps flagged · score 2.0 of 10

One disclosed by-construction result (IE) and an acknowledged validity confound, but the central GN-versus-IG and scaling findings are empirical and independent; overall mild circularity.

  1. self definitional [Section 5.1 (Main results); see also Section 2.2 (Aha example), Section 4.2 (Metrics), Eq. (1)]
    "This is somewhat obvious because our task is designed to be unsolvable by removing the aha example and thus rather serves as a quick check for our experimental design."

    The evaluation target e* is defined in Section 2.2 as the example that disambiguates the rule: 'the correct rule f can be identified by comparing the aha example with one of the other examples in the prompt.' IE's score is exactly the drop in target-token probability when an example is erased (Eq. 1). Because the tasks are designed to be unsolvable without e*, for any item in which the model actually relies on e*, IE must assign e* the largest or near-largest score by construction: erasing e* removes answerability while erasing other examples does not. Thus the reported result that 'IE works the best' is an arithmetic consequence of aligning IE's definition with the benchmark's definition of e*, not an independent empirical discovery about IE's merits.

full rationale

This paper is an empirical evaluation rather than a derivation, and no predicted quantity is fitted from the data used to test it. The central claim that simple gradient norm (GN) is the best gradient-based input attribution method, and the scaling findings, are measured against a fixed benchmark and do not reduce to the benchmark construction: the ranking of GN over IG and IxG is not forced by how e* is defined. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The one genuine by-construction element is the IE 'works best' result, which the paper itself flags as obvious because the tasks are unsolvable without the aha example; this is a disclosed sanity check rather than a central predictive claim. The Table 2 confound (models often answer correctly without e* due to inductive bias) is a real threat to the construct validity of top-1/top-2 accuracy on those items, but it is not a circular reduction: the attribution scores are still computed from the model, not derived from the label. The paper acknowledges this limitation in Section 6 and in the Limitations section. Overall, the central empirical findings have independent content, so the circularity burden is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted in the central evaluation; the LoRA hyperparameters are training details and do not enter the attribution calculations. The 'aha example' is a benchmark construct, not a new theoretical entity. The axioms are benchmark-design assumptions that the evaluation relies on, and the paper partially addresses their validity in appendices.

assumptions (3)
  • domain assumption The single 'aha' example is the unique ground-truth disambiguating example; all other examples are ambiguous with respect to the rule.
    This defines the benchmark's ground truth and the evaluation metric. If models use a different set of examples, the top-1/top-2 accuracy measure is not a valid interpretability score.
  • domain assumption Summing token-level attribution scores within an example yields a valid example-level importance score.
    Used in Section 2.2 to convert token attributions to example attributions. The paper tests max and coalition aggregation in appendices, but the central results use sum aggregation.
  • domain assumption Fine-tuned models that solve the synthetic tasks are appropriate proxies for studying in-context learning interpretability.
    Models are fine-tuned to ensure they can solve the tasks; the paper reports similar trends for base models in Appendix D, but the main analysis relies on fine-tuned models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Input Attributions Explain Inductive Reasoning in In-Context Learning?." pith.science (2026). https://pith.science/paper/N754FZFN

@misc{pith2026241215628,
  author       = {Pith},
  title        = {Pith review of: Can Input Attributions Explain Inductive Reasoning in In-Context Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N754FZFN}},
  note         = {Machine review of arXiv:2412.15628}
}
read the original abstract

Interpreting the internal process of neural models has long been a challenge. This challenge remains relevant in the era of large language models (LLMs) and in-context learning (ICL); for example, ICL poses a new issue of interpreting which example in the few-shot examples contributed to identifying/solving the task. To this end, in this paper, we design synthetic diagnostic tasks of inductive reasoning, inspired by the generalization tests typically adopted in psycholinguistics. Here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates it. The question is whether conventional input attribution (IA) methods can track such a reasoning process, i.e., identify the influential example, in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

Figures

Figures reproduced from arXiv: 2412.15628 by the authors.

Figure 1
Figure 1. Overview of our experimental setup. The majority of in-context examples (gray) are ambiguous, supporting either Rule A of adding two tokens or Rule B of doubling tokens. A single disambiguating example (blue) reveals that Rule A is correct. We investigate whether input attribution (IA) methods can track such an inductive reasoning process. do conventional IA methods still empirically work in the modern NLP setting, … view at source ↗
Figure 2
Figure 2. IA scores (attribution accuracies) for each task/model in the 10-shot setting (thus, the chance rate is 20% [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. IA score (attribution accuracy) for interpreting Gemma-2-2B models across all six tasks. Gradient-based [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: IA score (attribution accuracy) obtained with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: IA score (attribution accuracy) for different disambiguating, aha example positions. Gemma-2-2B model [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Attribution accuracies for each task for base models. Similar patterns to those observed in the fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Attribution accuracies for each task use max aggregation. The overall trend for all IA methods is consistent [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Coalition attribution accuracies for each task by treating each example as one coalition. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Distribution of the position of the example with the highest attribution scores across IA methods (Gemma [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Distribution of the position of the example with the highest attribution scores across IA methods [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 28 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    David Alvarez-Melis and Tommi Jaakkola. 2017. https://doi.org/10.18653/v1/D17-1042 A causal framework for explaining the predictions of black-box sequence-to-sequence models . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412--421, Copenhagen, Denmark. Association for Computational Linguistics

    work page doi:10.18653/v1/d17-1042 2017

  4. [4]

    Leila Arras, Franziska Horn, Gr \'e goire Montavon, Klaus-Robert M \"u ller, and Wojciech Samek. 2016. https://doi.org/10.18653/v1/W16-1601 Explaining predictions of non-linear classifiers in NLP . In Proceedings of the 1st Workshop on Representation Learning for NLP , pages 1--7, Berlin, Germany. Association for Computational Linguistics

    work page doi:10.18653/v1/w16-1601 2016

  5. [5]

    Leila Arras, Ahmed Osman, Klaus-Robert M \"u ller, and Wojciech Samek. 2019. https://doi.org/10.18653/v1/W19-4813 Evaluating recurrent neural network explanations . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 113--126, Florence, Italy. Association for Computational Linguistics

    work page doi:10.18653/v1/w19-4813 2019

  6. [6]

    Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.64 `` will you find these shortcuts? '' a protocol for evaluating the faithfulness of input salience methods for text classification . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language ...

    doi:10.18653/v1/2022.emnlp-main.64 2022

  7. [7]

    Leonard Bereska and Stratis Gavves. 2024. https://openreview.net/forum?id=ePUVetPKu6 Mechanistic Interpretability for AI Safety - A Review . Transactions on Machine Learning Research (TMLR)

    work page 2024

  8. [8]

    Dan Braun, Jordan Taylor, Nicholas Goldowsky - Dill, and Lee Sharkey. 2024. https://doi.org/10.48550/ARXIV.2405.12241 Identifying functionally important features with end-to-end sparse dictionary learning . CoRR, abs/2405.12241

    arXiv doi:10.48550/arxiv.2405.12241 2024

Show all 67 references
  1. [9]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen,...

    2023

  2. [10]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jef...

    2020

  3. [11]

    Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. 2019. https://proceedings.neurips.cc/paper/2019/hash/c61f571dbd2fb949d3fe5ae1608dd48b-Abstract.html Input similarity from the neural network perspective . In Advances in Neural Information Processing Sys...

    2019

  4. [12]

    Noam Chomsky. 1980. Rules and Representations. Columbia University Press, New York

    1980

  5. [13]

    Misha Denil, Alban Demiraj, and Nando de Freitas. 2014. https://arxiv.org/abs/1412.6815 Extraction of salient sentences from labelled documents . CoRR, abs/1412.6815

    2014 arXiv

  6. [14]

    Shuoyang Ding and Philipp Koehn. 2021. https://doi.org/10.18653/v1/2021.naacl-main.399 Evaluating saliency methods for neural language models . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    2021 doi

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  8. [16]

    G \'a llego, Ioannis Tsiamas, and Marta R

    Javier Ferrando, Gerard I. G \'a llego, Ioannis Tsiamas, and Marta R. Costa-juss \`a . 2023. https://doi.org/10.18653/v1/2023.acl-long.301 Explaining how transformers use context to build predictions . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  9. [17]

    Leo Gao, Tom Dupr \' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://doi.org/10.48550/ARXIV.2406.04093 Scaling and evaluating sparse autoencoders . CoRR, abs/2406.04093

    2024 doi arXiv

  10. [18]

    Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. 2021. https://openreview.net/forum?id=9uvhpyQwzM\_ Evaluation of similarity-based explanations . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    2021

  11. [19]

    Andrej Karpathy, Justin Johnson, and Li Fei - Fei. 2015. https://arxiv.org/abs/1506.02078 Visualizing and understanding recurrent networks . CoRR, abs/1506.02078

    2015 arXiv

  12. [20]

    Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. 2024. https://openreview.net/forum?id=qzsDKwGJyB Measuring progress in dictionary learning for language model interpretability with...

    2024

  13. [21]

    Eugene Kharitonov and Rahma Chaabouni. 2021. https://openreview.net/forum?id=YmA86Zo-P\_t What they do when in doubt: a study of inductive biases in seq2seq learners . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021...

    2021

  14. [22]

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. 2024. https://openreview.net/forum?id=fewUBDwjji Interpreting attention layer outputs with sparse autoencoders . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  15. [23]

    Pang Wei Koh and Percy Liang. 2017. http://proceedings.mlr.press/v70/koh17a.html Understanding black-box predictions via influence functions . In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume ...

    2017

  16. [24]

    Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. https://arxiv.org/abs/2009.07896 Captum: A unified and generic model interpreta...

    2020 arXiv

  17. [25]

    Barbara Landau, Linda B Smith, and Susan S Jones. 1988. The importance of shape in early lexical learning. Cognitive development, 3(3):299--321

    1988

  18. [26]

    Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. https://doi.org/10.18653/v1/D16-1011 Rationalizing neural predictions . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107--117, Austin, Texas. Association for Computational L...

    2016 doi

  19. [27]

    Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016 a . https://doi.org/10.18653/v1/N16-1082 Visualizing and understanding neural models in NLP . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...

    2016 doi

  20. [28]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016 b . https://doi.org/10.18653/v1/N16-1014 A diversity-promoting objective function for neural conversation models . In Proceedings of the 2016 Conference of the North A merican Chapter of the Associatio...

    2016 doi

  21. [29]

    Jiwei Li, Will Monroe, and Dan Jurafsky. 2016 c . https://arxiv.org/abs/1612.08220 Understanding neural networks through representation erasure . CoRR, abs/1612.08220

    2016 arXiv

  22. [30]

    Dragan, Rohin Shah, and Neel Nanda

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \' a nos Kram \' a r, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. https://doi.org/10.48550/ARXIV.2408.05147 Gemma scope: Open sparse autoencoders everywhere all at once ...

    2024 doi arXiv

  23. [31]

    Hui Liu, Qingyu Yin, and William Yang Wang. 2019. https://doi.org/10.18653/v1/P19-1560 Towards explainable NLP : A generative explanation framework for text classification . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5570-...

    2019 doi

  24. [32]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  25. [33]

    Lundberg and Su - In Lee

    Scott M. Lundberg and Su - In Lee. 2017. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html A unified approach to interpreting model predictions . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural In...

    2017

  26. [34]

    Aleksandar Makelov. 2024. https://openreview.net/forum?id=JdrVuEQih5 Sparse autoencoders match supervised features for model steering on the IOI task . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  27. [35]

    Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew E. Peters. 2022. https://doi.org/10.18653/V1/2022.FINDINGS-NAACL.31 Few-shot self-rationalization with natural language prompts . In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United S...

    2022 doi

  28. [36]

    R Thomas McCoy, Robert Frank, and Tal Linzen. 2018. https://escholarship.org/uc/item/12w6n3xn Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks . In 40th Annual Meeting of the Cognitive Science Society:...

    2018

  29. [37]

    Thomas McCoy, Robert Frank, and Tal Linzen

    R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. https://doi.org/10.1162/tacl_a_00304 Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks . Transactions of the Association for Computational Linguistics, 8:125--140

    2020 doi

  30. [38]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  31. [39]

    Tom M Mitchell. 1980. https://www.cs.cmu.edu/ tom/pubs/NeedForBias_1980.pdf The need for biases in learning generalizations . Citeseer

    1980

  32. [40]

    Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024. In-context learning generalizes, but not always robustly: The case of syntax. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2024

  33. [41]

    James Murdoch, Peter J

    W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. https://openreview.net/forum?id=rkRwGg-0Z Beyond word importance: Contextual decomposition to extract interactions from lstms . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, Apri...

    2018

  34. [42]

    Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. https://arxiv.org/abs/2004.14546 Wt5?! training text-to-text models to explain their predictions . CoRR, abs/2004.14546

    2020 arXiv

  35. [43]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. https://doi.org/10.23915/distill.00024.001 Zoom in: An introduction to circuits . Distill. Https://distill.pub/2020/circuits/zoom-in

    2020 doi

  36. [44]

    Amy Perfors, Joshua B Tenenbaum, and Terry Regier. 2011. https://www.sciencedirect.com/science/article/abs/pii/S0010027710002593 The learnability of abstract syntactic principles . Cognition, 118(3):306--338

    2011

  37. [45]

    Nina Poerner, Hinrich Sch \"u tze, and Benjamin Roth. 2018. https://doi.org/10.18653/v1/P18-1032 Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement . In Proceedings of the 56th Annual Meeting of the Association for Computational ...

    2018 doi

  38. [46]

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. 2024 a . https://openreview.net/forum?id=Ppj5KvzU8Q Improving sparse decomposition of language model activations with gated sparse autoencoders . In ICML...

    2024

  39. [47]

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J \' a nos Kram \' a r, and Neel Nanda. 2024 b . https://doi.org/10.48550/ARXIV.2407.14435 Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders . CoRR, abs/2407.14435

    2024 doi arXiv

  40. [48]

    Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. https://doi.org/10.18653/v1/P19-1487 Explain yourself! leveraging language models for commonsense reasoning . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistic...

    2019 doi

  41. [49]

    Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. https://doi.org/10.18653/v1/N16-3020 `` why should I trust you? '' : Explaining the predictions of any classifier . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational...

    2016 doi

  42. [50]

    Morgane Rivi \` e re, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \' e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \' e , Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le ...

    2024 doi arXiv

  43. [51]

    Soumya Sanyal and Xiang Ren. 2021. Discretized integrated gradients for explaining language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10285--10299, Stroudsburg, PA, USA. Association for Computational Linguistics

    2021

  44. [52]

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. http://proceedings.mlr.press/v70/shrikumar17a.html Learning Important Features Through Propagating Activation Differences . In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 o...

    2017

  45. [53]

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. http://arxiv.org/abs/1312.6034 Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps . In 2nd International Conference on Learning Representations (ICLR) Workshop Track Proceedings

    2014 arXiv

  46. [54]

    Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, and Yanjun Qi. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.33 Perturbing inputs for fragile interpretations in deep natural language processing . In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing a...

    2021 doi

  47. [55]

    Riedmiller

    Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2015. http://arxiv.org/abs/1412.6806 Striving for simplicity: The all convolutional net . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2...

    2015 arXiv

  48. [56]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. http://proceedings.mlr.press/v70/sundararajan17a.html Axiomatic attribution for deep networks . In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , v...

    2017

  49. [57]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  50. [58]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  51. [59]

    Johannes Treutlein, Dami Choi, Jan Betley, Samuel Marks, Cem Anil, Roger Baker Grosse, and Owain Evans. 2024. Connecting the dots: LLMs can infer and verbalize latent structure from disparate training data. In The Thirty-eighth Annual Conference on Neural Information Processin...

    2024

  52. [60]

    Oriol Vinyals and Quoc V. Le. 2015. https://arxiv.org/abs/1506.05869 A neural conversational model . CoRR, abs/1506.05869

    2015 arXiv

  53. [61]

    Mittelstadt, and Chris Russell

    Sandra Wachter, Brent D. Mittelstadt, and Chris Russell. 2017. https://arxiv.org/abs/1711.00399 Counterfactual explanations without opening the black box: Automated decisions and the GDPR . CoRR, abs/1711.00399

    2017 arXiv

  54. [62]

    Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.16 Learning which features matter: R o BERT a acquires a preference for linguistic generalizations (eventually) . In Proceedings of the 2020 Conference ...

    2020 doi

  55. [63]

    Colin Wilson. 2006. https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog0000_89 Learning phonology with substantive bias: an experimental and computational study of velar palatalization . Cogn. Sci., 30(5):945--982

    2006 doi

  56. [64]

    Jialin Wu and Raymond Mooney. 2019. https://doi.org/10.18653/v1/W19-4812 Faithful multimodal explanation for visual question answering . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 103--112, Florence, Italy. As...

    2019 doi

  57. [65]

    Aditya Yedetore, Tal Linzen, Robert Frank, and R Thomas McCoy. 2023. https://aclanthology.org/2023.acl-long.521/ How poor is the stimulus? evaluating hierarchical generalization in neural networks trained on child-directed speech . In Proceedings of ACL 2023, pages 9370--9393

    2023

  58. [66]

    Kayo Yin and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.14 Interpreting language models with contrastive explanations . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 184--198, Abu Dhabi, United Arab Emira...

    2022 doi

  59. [67]

    Zintgraf, Taco S

    Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. https://openreview.net/forum?id=BJ5UeU9xx Visualizing deep neural network decisions: Prediction difference analysis . In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, A...

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.