Pith. sign in

REVIEW 2 major objections 5 minor 36 references

Fine-tuned LLMs memorize new facts yet fail to use them because the knowledge is stored in the wrong layers for multi-step reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 08:09 UTC pith:F5RU5PBO

load-bearing objection Solid empirical map of a real memorization–use gap plus a usable patching diagnostic; the misalignment story is the natural reading, not a uniquely locked mechanism. the 2 major comments →

arxiv 2607.08393 v1 pith:F5RU5PBO submitted 2026-07-09 cs.AI cs.CL

Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Lu Dai , Ziyang Rao , Yili Wang , Hanqing Wang , Hao Liu , Hui Xiong This is my paper
classification cs.AI cs.CL
keywords Knowing-Using Gapself-patchingknowledge injectionLLM fine-tuningmechanistic interpretabilityknowledge-circuit misalignmentmulti-hop reasoningactivation patching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When large language models are fine-tuned on genuinely new facts, they quickly learn to recall those facts in isolation but often cannot apply them to multi-hop reasoning questions. The paper names this the Knowing–Using Gap: both an accuracy gap and a time lag between memorization and usable generalization. With self-patching—an intervention that copies an entity’s hidden representation from one layer into another—the authors show that the needed information is frequently already inside the model after memorization saturates, yet is not routed into the mid-layer positions that reasoning uses. Manually relocating those representations recovers large fractions of failed cases across model families and domains, and a simple fixed early-to-mid and late-to-mid heuristic recovers most of that oracle headroom. The result reframes fine-tuning’s generalization failure as a routing and alignment problem rather than a pure capacity or learning problem.

Core claim

The Knowing–Using Gap is driven by knowledge–circuit misalignment: fine-tuning first encodes new facts in easy-to-fit storage states, often early or very late layers that support direct recall, but does not reliably route them into the mid-layer computation positions multi-step reasoning needs. Self-patching shows that after memorization saturates, answer-relevant representations can already exist off the natural path; relocating them into effective mid layers immediately restores substantial generalization accuracy, even when ordinary fine-tuning has already stalled.

What carries the argument

Self-patching: an activation intervention that copies the residual-stream representation of an anchor entity from a source layer into a target layer of the same forward pass, then measures whether the correct multi-hop answer becomes recoverable. It yields layer-pair permeation maps that locate where memorized knowledge is causally usable and how that usability expands—or stalls—during training.

Load-bearing premise

The paper treats a sudden accuracy gain from copying one layer’s entity representation into another as proof that the fact was already present and merely mis-routed, rather than that the intervention itself builds or enriches a usable signal.

What would settle it

After memorization saturates but multi-hop use still fails, exhaustive entity-position layer-to-layer self-patching either never raises generalization accuracy, or patching representations of unrelated facts matches the gains from patching the correct fact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A failed multi-hop answer after fine-tuning need not mean the fact is missing; it can mean the fact is stranded in storage-oriented layers.
  • A fixed early→mid and late→mid patch rule can recover 58–75% of oracle recovery without per-instance search.
  • Larger models and more injected facts do not by themselves close the temporal lag or the final accuracy gap.
  • Alignment-aware training that drives knowledge into mid-layer circuits is a concrete path to better knowledge injection.
  • The same misalignment pattern appears across biomedical and academic knowledge domains and across Qwen and LLaMA families.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Objectives that keep gradients alive after memorization saturates—or that explicitly reward mid-layer routing—could shrink the gap without post-hoc patching.
  • The same storage-versus-computation split may help explain why single-fact knowledge edits often break multi-hop consistency.
  • If early-layer storage is routinely usable when forced into mid layers, pretraining may already contain latent multi-hop circuits that fine-tuning fails to engage.
  • Layer-pair patchability could serve as an online readiness signal for when injected knowledge is safe to rely on in reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper formalizes a Knowing–Using Gap in LLM knowledge injection: after fine-tuning on novel facts, memorization accuracy saturates early while multi-hop use (chaining and intersection) is delayed and often remains much lower. Using novel STaRK-derived biomedical and academic QA pairs with leakage filtering, the authors introduce self-patching—copying residual-stream entity representations across layers/contexts—and map when and where injected knowledge becomes causally usable. They propose a knowledge–circuit misalignment hypothesis: facts are stored in early/late layers that support recall but are not reliably routed into mid-layer reasoning computation. Oracle self-patching recovers large generalization headroom across Qwen/LLaMA scales and two domains; a fixed early→mid / late→mid heuristic recovers 58–75% of that headroom. Controls include token-position ablations, CoT, irrelevant-fact patching, and cross-context transfer.

Significance. If the gap and the misalignment reading hold, the paper reframes a widely observed fine-tuning failure as a routing/alignment problem rather than pure capacity or missing reasoning circuits, with direct implications for knowledge editing, continual learning, and alignment-aware fine-tuning. Strengths include a carefully novelty-checked multi-hop dataset, cross-domain and cross-architecture replication, Wilson CIs and McNemar tests, and a practical non-oracle heuristic that partially closes the gap. The work is diagnostic rather than a full training fix, but the combination of dynamics heatmaps, causal interventions, and a usable heuristic is a concrete contribution to mechanistic understanding of knowledge injection.

major comments (2)
  1. [§5.1–5.3, Alg. 1, Table 4, Fig. 5] §5.1–5.3, Algorithm 1, Table 4, Figure 5: The central causal claim treats positive ΔI from residual copy as evidence that knowledge already exists off-path and is merely mis-routed into mid-layer circuits. Copying h_ls into h_lt can also inject a cleaner/more linearly readable entity encoding into a stage that never natively received it, or exploit late-layer enrichment (late→mid cluster). Irrelevant-fact and Pmem→Pgen controls (Table 6, Fig. 6) rule out pure noise but do not uniquely establish native mid-layer usability. A native mid-layer readout, no-copy reweighting control, or early-only source ablation is needed before ‘misalignment of storage and computation’ is the uniquely supported mechanism rather than one interpretation of recovery.
  2. [§5.5, Table 7] §5.5 and Table 7: The fixed heuristic is presented as practical evidence that the diagnostic finding can be exploited, yet it still requires intervening on residual streams at predetermined layers at inference time. The manuscript should clarify whether this is intended as a deployable remedy, a diagnostic proxy, or a proof-of-concept for future training-time alignment, and should discuss cost, applicability outside residual-stream access, and whether training-time objectives could induce the same mid-layer alignment without patching.
minor comments (5)
  1. [§4.1, Table 3] Figure 1 and Table 3: saturation window w and exact epoch definitions for T_mem / T_gen should be stated once in the main text (they appear mainly in §4.1 and Appendix F) so the temporal-lag numbers are self-contained.
  2. [Figure 4] Figure 4 heatmaps: color scale and ‘full recovery / partial / no effect’ thresholds should be explicit in the caption; several cells are hard to interpret without a numeric colorbar.
  3. [§6, Appendix H] Appendix H limitations are thorough but should be briefly mirrored in the main conclusion so readers see underestimation of headroom, lack of early predictive signal, and layer-level (not head/MLP) granularity without opening the appendix.
  4. [§5.1] Notation: residual-stream states are written both as h^l_t(P) and with tilde for patched runs; a short glossary of Ps/Pt, E_head, and I(·,y*) near Algorithm 1 would help.
  5. [§2] Related work: the distinction from grokking is useful; a sentence contrasting self-patching more sharply with ROME/MQuAKE-style editing evaluations would further situate the contribution.

Circularity Check

0 steps flagged

No load-bearing circularity: gap metrics and self-patching recoveries are measured on held-out compositional queries and causal interventions, not forced by definition or self-citation.

full rationale

The Knowing–Using Gap is defined via Amem(t) vs Agen(t;T) on disjoint single-hop memorization queries versus multi-hop chaining/intersection templates never seen in training (Eqs. 1–2, §3.2–4.1, Table 3). Self-patching (Alg. 1, §5.1) is an independent residual-stream intervention that measures ΔI after relocating an entity representation; positive cells and Table 4 oracle lifts are empirical outcomes, not algebraic identities of the training loss. The fixed heuristic (§5.5) merely freezes two layer pairs observed in the same maps (Fig. 5) and reports the measured fraction of oracle headroom (Table 7); that fraction is not fitted to force a target value, nor is any uniqueness theorem or prior self-result invoked to declare the mechanism inevitable. Controls (irrelevant patching, cross-context Pmem→Pgen, CoT, token-position ablations) further separate the measurements from pure construction. Interpretation debates about whether patching creates versus relocates a signal are correctness concerns, not circular reductions of the reported numbers to their inputs. Hence only a negligible residual (post-hoc choice of the two fixed pairs from the diagnostic maps) and score 1.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The paper is empirical-interventional. Free parameters are experimental design choices (layer fractions, lag window, PEFT hyperparameters), not physical constants fitted to force a theory. Invented entities are named phenomena and tools with experimental handles; the misalignment hypothesis itself is the least independently evidenced item and remains interpretive.

free parameters (3)
  • fixed heuristic layer fractions = ~0.8L→0.5L and ~0.1L→0.5L
    Predetermined pairs ~0.8L→0.5L and ~0.1L→0.5L chosen from the two-cluster pattern in Figure 5; data-driven from the same experiments that measure headroom, not per-instance optimized.
  • saturation window w for temporal lag = w=2 (default)
    Defines T_gen as earliest epoch with stable perfect accuracy for w consecutive epochs; default w=2 with appendix sensitivity, still a free definitional choice that affects ΔT.
  • LoRA rank and alpha (main multi-fact runs) = r=16, α=32
    r=16, α=32 and related PEFT choices affect how knowledge is written into weights; full fine-tuning is also reported but main tables use these settings.
axioms (3)
  • domain assumption Residual-stream activations at entity tokens can be meaningfully copied across layers and contexts without destroying the rest of the forward pass.
    Inherited from activation-patching literature and used as the operational definition of self-patching in Algorithm 1 and §5.1.
  • domain assumption After filtering base-model-correct instances, the STaRK-derived facts are novel to the pretrained models.
    Supported by Table 2 zero-shot accuracies ≤6% and explicit filtering in §5; still an empirical assumption about pretraining contamination.
  • domain assumption Mid-layer residual-stream positions are the primary computation-effective sites for multi-hop reasoning.
    Drawn from prior three-step / late-hopping literature and corroborated by the empty late-target region in Figure 5; load-bearing for the misalignment interpretation.
invented entities (3)
  • Knowing–Using Gap independent evidence
    purpose: Name and quantify the accuracy gap plus temporal lag between memorization and multi-hop use of injected facts.
    Operationalized with Amem, Agen, ΔA, ΔT on held-out compositional queries; measurable on any fine-tuning run.
  • self-patching independent evidence
    purpose: Causal intervention that relocates entity representations across layers to test whether knowledge is present but unused.
    Defined in Algorithm 1; produces layer-pair maps and recovers accuracy on failed cases; falsifiable by null effects under controls.
  • knowledge–circuit misalignment hypothesis no independent evidence
    purpose: Explain the gap as spatial storage-vs-computation mismatch rather than missing capacity or missing circuits.
    Supported by self-patching dynamics and recovery experiments but remains an interpretive hypothesis; no independent external measurement of native circuits beyond the intervention itself.

pith-pipeline@v1.1.0-grok45 · 27164 in / 3212 out tokens · 67413 ms · 2026-07-10T08:09:49.033908+00:00 · methodology

0 comments
read the original abstract

Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the \textit{\textbf{Knowing--Using Gap}}, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58--75\% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding.

Figures

Figures reproduced from arXiv: 2607.08393 by Hanqing Wang, Hao Liu, Hui Xiong, Lu Dai, Yili Wang, Ziyang Rao.

Figure 1
Figure 1. Figure 1: Illustration of the Knowing–Using Gap. To address these questions, we conduct a fine-grained analysis of the training dynamics during knowledge injection. We construct datasets from two real-world domain knowledge bases and eliminate overlap with pretraining. We define two types of reasoning QA tasks to evaluate how LLMs generalize the learned knowledge: chaining task requires resolving a bridge entity in … view at source ↗
Figure 2
Figure 2. Figure 2: Results of know-use gap on different data sizes and model sizes. Model and data scale. We also test the knowing-using gap across different model scales and data scales [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Method Comparison. Causal tracing corrupts a run to identify causally relevant locations. Patchscope interprets a hidden state by decoding it into natural language. Self-patching tests the effect position by swapping its internal representation across layers and contexts. 5.1 Self-Patching Algorithm 1: Self-patching scan Input: model M, prompts Ps, Pt, anchor E, an￾swer y ∗ , score I. For each layer pair (… view at source ↗
Figure 4
Figure 4. Figure 4: Permeation dynamics of know–use transition, aligned with fine-tuning process. Cell (lsrc, ltgt) measures whether patching the head-entity representation from lsrc to ltgt increases general￾ization accuracy, with red indicating full recovery, blue for no effect, and white for partial gain. Top: a natural failed instance. The red region indicates the position of knowledge storage, which permeates but halts b… view at source ↗
Figure 5
Figure 5. Figure 5: Effective patch locations concentrate into two clusters. Location of most effective patching. We further draw the spatial positions with most effective patching in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cross-context consistency. Detect knowledge existence. To rule out the chance that gains are driven by prompt artifacts or noises, we further cross-context patch the Ehead representation from Pmem which contains knowledge representation to Pgen [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: shows epoch-by-epoch memorization and generation accuracy for the Qwen MAG training runs. Chaining exhibits the starkest decoupling: memorization reaches >0.95 by epoch 15 while generation remains near zero throughout—a pattern that holds across Qwen model scales and mirrors the biomedical domain. Intersection shows memorization and generation rising in tandem, reaching similarly high final values—consiste… view at source ↗
Figure 8
Figure 8. Figure 8: Patchscope views of memorization￾related layers at the head-entity position. We use PatchScope to interpret the hidden state in the memorization prompt. We use the in￾context prompt shown in [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: More permeation dynamics cases on LLaMA-3.1-8B. H.3 Assets, Licenses, and Released Artifacts The study adapts the STARK benchmark Wu et al. [2024], specifically STARK-PRIME and STARK￾MAG; the public STaRK repository is released under the MIT License.1 We use open-weight Qwen2.5 models under their public licenses: Qwen2.5-1.5B and Qwen2.5-7B under Apache 2.0, and 1https://github.com/snap-stanford/stark 18 … view at source ↗
Figure 10
Figure 10. Figure 10: More permeation dynamics cases on Qwen-2.5-7B. Qwen2.5-3B under the Qwen Research/Tongyi Qianwen Research terms.2 We use LLaMA-3.1 and LLaMA-3.2 models under the Meta Llama Community License terms.3 We release the derived memorization-to-generalization QA data and accompanying code as new artifacts under the MIT License in the anonymous repository. The released data are generated from knowledge-graph fact… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 4 internal anchors

  1. [1]

    Alan Agresti and Brent A. Coull. Approximate is better than ``exact'' for interval estimation of binomial proportions. The American Statistician, 52 0 (2): 0 119--126, 1998

    work page 1998

  2. [2]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=HJ4-rAVtl

    work page 2017

  3. [3]

    Physics of language models: Part 3.3, knowledge capacity scaling laws

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. In The Thirteenth International Conference on Learning Representations

    work page

  4. [4]

    Probing classifiers: Promises, shortcomings, and advances

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics, 48 0 (1): 0 207--219, 2022. doi:10.1162/COLI\_A\_00422. URL https://doi.org/10.1162/coli\_a\_00422

    work page doi:10.1162/coli 2022

  5. [5]

    a is b” fail to learn “b is a

    Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on “a is b” fail to learn “b is a”. In The Twelfth International Conference on Learning Representations

    work page

  6. [6]

    Hopping too late: Exploring the limitations of large language models on multi-hop queries

    Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113--14130, 2024

    work page 2024

  7. [7]

    Evaluating the ripple effects of knowledge editing in language models

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 11: 0 283--298, 2024

    work page 2024

  8. [8]

    Knowledge neurons in pretrained transformers

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 8493--...

    work page doi:10.18653/v1/2022.acl-long.581 2022

  9. [9]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https://openreview.net/forum?id=tcsZt9ZNKD

    work page 2025

  10. [10]

    Key-value memory in the brain

    Samuel J. Gershman, Ila Fiete, and Kazuki Irie. Key-value memory in the brain. CoRR, abs/2501.02950, 2025. doi:10.48550/ARXIV.2501.02950. URL https://doi.org/10.48550/arXiv.2501.02950

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.02950 2025

  11. [11]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216--12235, 2023

    work page 2023

  12. [12]

    Patchscopes: A unifying framework for inspecting hidden representations of language models

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In International Conference on Machine Learning, pages 15466--15490. PMLR, 2024

    work page 2024

  13. [13]

    Model editing at scale leads to gradual and catastrophic forgetting

    Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. In Findings of the Association for Computational Linguistics ACL 2024, pages 15202--15232, 2024

    work page 2024

  14. [14]

    Sparse autoencoders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=F76bwRSLeK

    work page 2024

  15. [15]

    Where to find grokking in LLM pretraining? monitor memorization-to-generalization without test

    Ziyue Li, Chenrui Fan, and Tianyi Zhou. Where to find grokking in LLM pretraining? monitor memorization-to-generalization without test. CoRR, abs/2506.21551, 2025. doi:10.48550/ARXIV.2506.21551. URL https://doi.org/10.48550/arXiv.2506.21551

    work page doi:10.48550/arxiv.2506.21551 2025

  16. [16]

    Michaud, Max Tegmark, and Mike Williams

    Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Proce...

    work page 2022

  17. [17]

    Mass-editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations

    work page

  18. [18]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Nov...

    work page 2022

  19. [19]

    How much do language models memorize?

    John X Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G Edward Suh, Alexander M Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize? arXiv preprint arXiv:2505.24832, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW

    work page 2023

  21. [21]

    interpreting GPT : the logit lens, August 2020

    nostalgebraist. interpreting GPT : the logit lens, August 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Accessed: 2025-12-29

    work page 2020

  22. [22]

    In-context Learning and Induction Heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield - Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish,...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2209.11895 2022

  23. [23]

    Fine-tuning or retrieval? comparing knowledge injection in llms

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 237--250, 2024

    work page 2024

  24. [24]

    Towards vision-language mechanistic interpretability: A causal tracing tool for BLIP

    Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for BLIP . In IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Workshops, Paris, France, October 2-6, 2023 , pages 2848--2853. IEEE , 2023. doi:10.1109/ICCVW60793.2023.00307. URL https://doi.org/10.1109...

    work page doi:10.1109/iccvw60793.2023.00307 2023

  25. [25]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=UGpGkLzwpP

    work page 2024

  26. [26]

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  27. [27]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=8sKcAWOf2D

    work page 2024

  28. [28]

    Fine tuning vs

    Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. Fine tuning vs. retrieval augmented generation for less popular knowledge. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 12--22, 2024

    work page 2024

  29. [29]

    Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization

    Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization. Advances in Neural Information Processing Systems, 37: 0 95238--95265, 2024

    work page 2024

  30. [30]

    Interpretability in the wild: a circuit for indirect object identification in GPT-2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul

    work page 2023

  31. [31]

    Do llamas work in english? on the latent language of multilingual transformers

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in english? on the latent language of multilingual transformers. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, Augus...

    work page doi:10.18653/v1/2024.acl-long.820 2024

  32. [32]

    Ioannidis, Karthik Subbian, James Y

    Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Y. Zou, and Jure Leskovec. Stark: Benchmarking LLM retrieval on textual and relational knowledge bases. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Ad...

    work page 2024

  33. [33]

    Knowledge circuits in pretrained transformers

    Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits in pretrained transformers. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Pr...

    work page 2024

  34. [34]

    Cake: Circuit-aware editing enables generalizable knowledge learners

    Yunzhi Yao, Jizhan Fang, Jia-Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, and Nanyun Peng. Cake: Circuit-aware editing enables generalizable knowledge learners. arXiv preprint arXiv:2503.16356, 2025

    work page arXiv 2025

  35. [35]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations

    work page

  36. [36]

    Mquake: Assessing knowledge editing in language models via multi-hop questions

    Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15686--15702, 2023

    work page 2023