Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Shared parameters—those updated by both fact-storing and fact-recalling examples—explain why mixed training beats two-stage training on unseen questions.

desk verdict Worth engaging: the cross-task gradient trace is a real contribution, but the mixed-vs-two-stage comparison is confounded by training budget and QA exposure, and the abstract overclaims what is measured. read the letter →

arxiv 2505.16178 v2 pith:GW3OUI7H submitted 2025-05-22 cs.CL

Ying Zhang , Benjamin Heinzerling , Dongyuan Li , Kentaro Inui This is my paper · ORCID
classification cs.CL
keywords factrecallmixedtrainingtwo-stagesharedparameterscross-taskgradienttraceattributionattentionheadsmechanisticinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why the same factual content behaves differently depending on how it is fine-tuned: training first on biographical statements (BIO) and then on question–answer pairs (QA) produces rote memorization, while interleaving the two formats produces recall that generalizes to unseen questions. Its central claim is that mixed training creates a larger and more central set of shared parameters—parameters strongly influenced by both the BIO and QA tasks—and that these parameters establish a format-invariant retrieval process. The evidence comes from a new cross-task gradient trace applied to two 2.8–3B parameter language models, with ablation and grafting experiments showing that the shared set, not random or single-task parameters, carries the ability to answer out-of-distribution questions. The paper further finds that shared parameters concentrate in a small number of critical attention heads, where they act as switches that activate fact recall. If this is right, the choice between memorization and knowledge is not about more data but about whether the optimizer jointly shapes the same parameters for storage and retrieval.

What carries the argument

The central object is the cross-task gradient trace, a measure of per-parameter influence defined by summing $lr(d_n) \times grad^{(n)}_i$ over examples of each task and ranking parameters for BIO and QA separately. A parameter is shared if it appears in both top-k sets, and the paper studies the union of two such intersections, one using BIO in-distribution individuals and one using BIO out-of-distribution individuals. This object carries the entire argument: the paper uses it to define the set whose ablation destroys generalization, to show that this set is over-represented in grafting-recovered fact recall parameters, and to prioritize attention heads for circuit intervention. The supporting machinery is grafting with a relaxed binary mask, which locates the sparse set of parameters sufficient to recover the fine-tuned model's recall, allowing the paper to test whether shared parameters are functionally central rather than merely updated.

What would settle it

Train a two-stage model with the same total number of gradient steps and the same number of QA examples as the mixed model; if the gap on unseen questions persists even when the shared-parameter sets have the same size, the gradient-trace explanation fails. Alternatively, ablate the top-k single-task parameters by summed update magnitude; if that ablation reproduces the shared-parameter ablation drop, the intersection is not carrying the causal weight.

Watch

Extended reading notes

Core claim

The paper's discovery is that the memorization-versus-knowledge gap between two-stage and mixed training is explained by the parameters that both tasks influence together. Using the cross-task gradient trace, which ranks parameters by summed learning-rate-weighted gradients from BIO and QA examples and takes the intersection of the top-k sets, the paper finds that mixed training yields a shared parameter set about 1.2 times larger in Llama and 1.5 times larger in Pythia at $k = 10^8$. Ablating these shared parameters drops the mixed-trained Llama's out-of-distribution QA accuracy from 74.7 to 8.7, whereas the stage-trained model drops only from 35.9 to 29.2; ablating the same number of random parameters has almost no effect. Grafting experiments show that shared parameters are over-represented in the sparse subsets that recover fact recall, and attention-head interventions show that over 60% of shared parameters are concentrated in the top 10% of heads, whose ablation removes over 90% of recall accuracy. The paper concludes that mixed training teaches knowledge because it produces more numerous and more centrally organized shared parameters, concentrated in critical attention heads that support reusable, format-invariant retrieval.

Load-bearing premise

The load-bearing premise is that the cross-task gradient trace, computed from 20 individuals' gradients, identifies the parameters that actually cause fact recall rather than merely the parameters with the largest update magnitudes.

Editorial extensions

If this is right

  • Training pipelines that separate fact storage from query-format adaptation will systematically produce fewer shared parameters, so knowledge injection should interleave statement and question–answer examples.
  • Shared parameters are the causal lever behind the generalization gap: resetting them to pretrained values is enough to cancel a mixed-trained model's advantage on unseen questions.
  • Because shared parameters concentrate in a small fraction of attention heads, monitoring this subset can predict whether a model has learned facts in a retrievable form.
  • The cross-task gradient trace is model-agnostic, so the same analysis can be applied to other paired storage/retrieval formats without retraining the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit is that the same gradient-consistency mechanism should show up whenever two task formats share the same underlying knowledge, so the trace could be used to predict which paired formats will benefit from mixing.
  • If the causal story is right, constraining fine-tuning to the shared-parameter subspace—through masking or low-rank adapters—should reproduce mixed-training generalization with fewer update steps; the paper does not test this.
  • This account suggests that some failures on paraphrased or reversed queries may be training-artifact effects rather than fixed model limits, which could be checked by applying mixed training to real knowledge bases with paraphrased QA.
  • A caveat I would add: the trace is estimated from only 20 individuals, so its reliability on larger, noisier datasets is an open question that the paper also acknowledges.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper investigates why mixed training (jointly optimizing on fact-storing statements and question-answer pairs) produces better generalized fact recall than two-stage training in language models. The authors introduce a 'cross-task gradient trace' that sums per-step gradient updates to identify shared parameters influenced by both tasks. Across Llama-3.2B and Pythia-2.8B on synthetic biography data, they report that mixed training yields a larger set of shared parameters, that ablating these parameters causes a larger accuracy drop, and that shared parameters concentrate in critical attention heads. The paper concludes that mixed training teaches knowledge by inducing gradient and representation consistency that establishes a format-invariant retrieval process.

Significance. The question addressed is timely and practically relevant, and the synthetic-data design plus the random-parameter ablation control (Appendix A.4.2) are useful strengths. If the central claim were established, the cross-task gradient trace would be a valuable tool for mechanistic interpretability of fine-tuning. However, the main quantitative comparison is currently confounded by unequal training budgets, and the claimed gradient/representation consistency is not directly measured, so the paper's significance is conditional on substantial revision.

major comments (5)
  1. [§3.1, Eq. (2); Table 2] The influence measure Δ_task_i is an unnormalized sum of learning-rate-scaled gradients over all traced examples. Because mixed training performs many more training steps than the two-stage pipeline (Llama: 10,571 vs. 7,220; Pythia: 19,437 vs. 14,781; Table 2), the raw sums are systematically larger under mixed training. This mechanically inflates the size of the top-k intersection S and the magnitude of the ablation drop (Figure 2b), so the central evidence for 'more numerous and functionally efficient shared parameters' is confounded with total update count and QA exposure. The authors should normalize the influence by the number of updates or per-example exposure, or run a matched-budget comparison.
  2. [Abstract; end of §3.2] The abstract states that mixed training 'induces gradient consistency' and 'drives the representation consistency between the two formats,' but no experiment in the paper measures gradient consistency (e.g., cosine similarity between BIO and QA gradients) or representation consistency (e.g., alignment of hidden states for BIO and QA). The only quantities measured are parameter-overlap counts and their ablation/grafting effects. The causal mechanism in the abstract is therefore not supported by the reported evidence; the authors should either measure these quantities or reframe the claims to what is actually tested.
  3. [§3.2, Experimental Setup] The shared set S is defined using an arbitrary threshold k=10^8, which the authors justify only as 'a simple default.' This k is then used for all downstream ablation (Figure 2b), grafting (Table 1), and circuit analyses (Figure 4). The paper does not show that the qualitative conclusions—larger ablation drop for mixed training, higher share in critical heads—are robust to the choice of k. A sensitivity analysis over k (e.g., 10^5, 10^6, 10^7, 10^8) is needed to ensure the conclusions are not an artifact of the chosen threshold.
  4. [§3.1, Eqs. (1)-(2); Appendix A.3] Equation (1) defines the parameter update as θ_i ← θ_i + lr(d_n) × grad, but the actual optimizer is AdamW, whose update normalizes gradients by their second moments. Consequently, Δ_task_i is a raw gradient-attribution score, not a measure of 'how much θ_i changes' as stated in Definition 3.1. The paper should either compute the actual parameter movement (θ_after − θ_before) or explicitly justify why raw gradient sums are the appropriate causal attribution, given the mismatch with the optimizer.
  5. [§3.1, Experimental Setup] The gradient trace uses only 20 individuals (10 for SI, 10 for SII). The stability of the top-k intersection across different random subsets of individuals is not reported. If the shared-set ranking is highly sensitive to the choice of the 20 traced individuals, the comparison between mixed and two-stage training may not be reliable. The authors should report variance or bootstrap confidence intervals over traced subsets.
minor comments (4)
  1. [A.7.1] In A.7.1, the text refers to 'Figure 4 (a)' when describing Pythia results; this should be Figure 17 (a).
  2. [Table 3] Table 3 contains a typo in the column header '|S∩γ|0| |' — there is an extra pipe in the subscript.
  3. [Figure 6] In Figure 6, the 'N/A' entries for the vanilla models on QA in-distribution/out-of-distribution are not explained; specifying that these models were not evaluated would improve clarity.
  4. [§4.1] Appendix A.7 explains that K/V projection counts are multiplied by 3 to balance projection sizes in Llama, but this reweighting is not mentioned in the main text; adding a sentence in §4.1 would prevent confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: shared-parameter conclusions are empirically grounded by ablation and grafting, and self-citations are not load-bearing.

full rationale

The paper's central derivation chain is not circular. Shared parameters are defined in Definition 3.1 via summed per-example gradient influence (Eq. 2), but the paper does not stop at the definition: it tests their functional importance by ablating them to pretrained values, by grafting masks with and without S, and by comparing against random-parameter ablations (Appendix A.4.2). These are independent empirical checks. The claim that mixed training yields more shared parameters is an empirical measurement, not a consequence of the definition: two-stage training separates phases by construction, but whether the top-k BIO and QA gradient sets intersect more under mixed training is not forced, and the comparison is not merely a step-count artifact (Llama Mix: 10,571 updates at batch 32 vs Stage 6,820 at batch 32 plus 400 at batch 256, so per-example BIO/QA presentations differ by only about 10%). The paper's self-citations [25,26] are background references on factual knowledge and are not load-bearing; the mixed-versus-two-stage premise comes from external work [3] and the new ablation/grafting experiments. The appendix limitation (A.1) correctly concedes the absence of a detailed causal analysis, but that is a scope limitation, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central contribution rests on a small number of hand-chosen thresholds (k, alpha, the K/V reweighting) rather than on many fitted constants. The more serious obligation is conceptual: the shared parameter set is a construct defined by the paper's own attribution method, and the "representation consistency" mechanism in the abstract is never directly observed. The list above separates the arbitrary analysis thresholds from the domain assumptions (gradient attribution validity, MLP-as-knowledge-base, synthetic data representativeness) that the conclusions inherit from prior work.

free parameters (4)
  • k (top-k threshold for shared parameters) = 10^8 (100 million)
    Used to define the shared parameter set S in Definition 3.1; chosen as a "simple default" in §3.1 rather than selected by a principled criterion. The qualitative comparisons such as "1.2x larger" and the ablation drops depend on this threshold.
  • alpha (grafting initial sparsity) = Per-model values; Llama: 0.10, 0.135, 0.18, 0.50; Pythia: 0.055, 0.095, 0.30, 0.70
    Tuned over [0,100%] in Appendix A.5.1 to make grafted models recover 92-97% of full accuracy; different models are compared at different alpha values, which weakens the parameter-efficiency comparison.
  • K/V projection reweighting factor for Llama = 3
    Applied in Appendix A.7 to balance query/output vs key/value projection parameter counts when computing shared parameter concentration in attention heads; the concentration percentages in §4.1 change with this factor.
  • Gradient-estimation subset size = 20 individuals (10 BIO_in plus 10 BIO_out/QA_in)
    Used to estimate per-task influence values for shared parameters in §3.1, far smaller than the 10,000 individual dataset; the influence ranking is an estimate.
assumptions (5)
  • domain assumption Gradient attribution (sum of lr * grad over examples) measures the task influence of a parameter.
    Definition 3.1 and Eq. (2) assume top-k by summed gradient updates identifies parameters functionally important for a task; this is a standard attribution assumption imported from [5] and is not independently validated here.
  • domain assumption MLP neurons serve as a factual knowledge base, so attention heads can be analyzed in isolation for fact recall.
    Invoked in §4.1 to justify intervening only on attention heads; supported by prior work and Appendix A.6.1 continuity metrics, but not by a controlled test that separates storage and recall.
  • domain assumption Synthetic biographical statements and question templates adequately instantiate fact-storing and fact-recalling formats for this study.
    All experiments use synthetically generated facts following [3]; the authors acknowledge in Limitations that real-world noise, imbalance, and conflict are not covered.
  • domain assumption Exact-match accuracy on held-out questions distinguishes memorization from generalized knowledge.
    The in-distribution vs out-of-distribution QA split operationalizes knowledge; the paper does not check alternate metrics such as partial credit or semantic equivalence.
  • domain assumption A circuit subgraph over attention heads and MLP neurons can capture the mechanism behind fact recall.
    Section 4 adopts circuit theory from prior work [13,44,49] and assumes node-level ablation and grafting preserve the relevant computation.
invented entities (2)
  • Shared parameter set S
    purpose: Explains why mixed training generalizes: parameters jointly influenced by BIO and QA are claimed to enable format-invariant fact recall.
    S is defined entirely by the paper's own gradient-attribution procedure (Definition 3.1). It has an in-paper ablation handle, but no independent external measurement; its existence and size depend on the chosen threshold k.
  • Representation consistency
    purpose: Abstract claims mixed training drives representation consistency between storage and query formats, establishing a format-invariant retrieval process.
    No representation-similarity metric is computed anywhere in the paper; the only supporting evidence is a qualitative one-head case study in §4.2. This is a postulated latent mechanism rather than a measured quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge." pith.science (2026). https://pith.science/paper/GW3OUI7H

@misc{pith2026250516178,
  author       = {Pith},
  title        = {Pith review of: Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GW3OUI7H}},
  note         = {Machine review of arXiv:2505.16178}
}
abstract

While fine-tuning is the standard for injecting factual knowledge into large language models (LLMs), the mechanisms enabling reliable fact recall via unseen queries remain poorly understood. Common two-stage training strategies, which sequentially train on fact storage and query formats, often cause rote memorization. In contrast, mixed training jointly optimizes both formats and exhibits superior generalized recall. We investigate this success by comparing the two paradigms across 2.8$\sim$4B LLMs and identify the core mechanism: the joint optimization objective in mixed training induces gradient consistency across storage and query formats. This in turn drives the representation consistency between the two formats, establishing a format-invariant retrieval process that maps unseen queries to stored facts. In contrast, the lack of such an objective in two-stage training results in inconsistent representations and failed recall. The consistency further localizes to the parameters updated by both formats, a set that is substantially larger under mixed training than under two-stage training. At the input level, the consistency leaves an interpretable signature: mixed training encodes facts in storage format from subject-relation tokens, the same components available in queries, while two-stage training relies on the full context. Our findings characterize the mechanisms of fact recall and offer mechanistic foundation for optimizing knowledge injection in LLMs.

Figures

Figures reproduced from arXiv: 2505.16178 by the authors.

Figure 1
Figure 1. (a) Performance of fine-tuned Llama and Pythia on QA out-of-distribution set. The Mix-tuned model substantially outperforms the Stage-tuned model, demonstrating superior generalization in the fact recall task. (b) Overview of the proposed tool. The fine-tuned model (θ task) first performs a forward pass to compute task-specific loss. During the backward pass (i.e., backpropagation), we track gradients for each param… view at source ↗
Figure 3
Figure 3. (a) Grafting procedure. (b) Number of fact recall-related parameters (|γ|0) in grafted models. Mix-tuned models include fewer fact recall-related parameters than Stage-tuned models. (b) Accuracy before and after ablating shared parameters, indicating their impact. (a) Proportion across top-k influenced parameters [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Shared parameters (S) in Llama: distribu￾tion and impact. (a) Mixed training yields more shared parameters than two-stage training. (b) Mix-tuned mod￾els show a larger accuracy drop after ablation, demon￾strating their impact. Results. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Intervention results on fine-grained attention heads. The Shared Size metric most effectively identifies minimal sufficient heads in (a) and most critical heads in (b) for fact recall circuits. (c) shows the corresponding fraction of intervened parameters. on the QA ou…
Figure 5
Figure 5. Figure 5: Attention pattern of grafted Mix-tuned Llama in Layer 21, Head 17. Left: attention pattern and output logits for the BIO input: Alexandra Leblanc was welcomed into life on April 11, 1982... Right: attention pattern and logits for the QA input from the same individual, …
Figure 6
Figure 6. Figure 6: shows the performance of fine-tuned models. Both the QA-tuned and Mix-tuned models perform well on data seen during training. Specifically, they retain biographical facts (BIO Accuracy ≥ 88.7%) and adapt well to the question-answer format (QA in_dist accuracy ≥ 98.1%).…
Figure 7
Figure 7. Figure 7: Shared parameters (S) in Pythia: distribution and impact. A.4 Details on shared parameters A.4.1 Distribution and impact of shared parameters on Pythia models [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 9
Figure 9. Figure 9: Accuracy after ablating random parameters S ′ . Values in parentheses (e.g., -0.0) denote accuracy differences computed as ablated minus original. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Tuning curves of 0.07% 100 Majority Voting [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Robustness of grafting with different random seeds. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Consistency and Effectiveness of using grafting to locate [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Number of fact recall-related parameters ( [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Case studies showing attribute prediction steps of grafted Mix-tuned models. Each marker [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Forward and backward intervention results on MLP neurons. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Fraction of intervened parameters in MLP neurons when using the [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Intervention results on fine-grained attention heads of Pythia models. The Shared Size metric most effectively identifies minimal sufficient heads in (a) and most critical heads in (b) for fact recall circuits. (c) shows the corresponding fraction of intervened parame…
Figure 18
Figure 18. Figure 18: Attention pattern of grafted Mix-tuned Llama in Layer 21, Head 17. Left: attention pattern and output logits for the BIO input: Alexandra Leblanc was welcomed into life on April 11, 1982. He ’s birthplace is Deborahview, NC. He benefited from the world-class education…
Figure 19
Figure 19. Figure 19: Attention pattern of grafted Stage-tuned Llama in Layer 21, Head 17. Left: attention pattern and output logits for the BIO input: Alexandra Leblanc was welcomed into life on April 11, 1982. He ’s birthplace is Deborahview, NC. He benefited from the world-class educati…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LMs as Task-Specific Knowledge Bases: An Interpretability Analysis

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    LMs store facts in task-specific parameter subsets, shown by inconsistent emergence across tasks during training and distinct localized parameters for the same fact.

  2. Reverse Convolution and Its Applications to Image Restoration

    cs.CV 2025-08 reject novelty 4.0 of 10

    The abstract and body of this submission are two unrelated papers; the reverse-convolution claims appear nowhere in the full text.

Reference graph

Works this paper leans on

54 extracted references · 45 canonical work pages · cited by 2 Pith papers

  1. [1]

    Evaluating correctness and faithfulness of instruction-following models for question answering

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12:681–699, 2024

    work page 2024

  2. [2]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024

    2024

  3. [3]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In International Conference on Machine Learning, pages 1067–1077. PMLR, 2024

    work page 2024

  4. [4]

    Physics of language models: Part 3.2, knowledge manipula- tion

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipula- tion. In International Conference on Learning Representations, 2025

    work page 2025

  5. [5]

    Towards better understanding of gradient-based attribution methods for deep neural networks

    Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018

    work page 2018

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine L...

    work page 2023

  8. [8]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392–18402. IEEE, 2023

    work page 2023

Show all 54 references
  1. [9]

    Causal scrubbing: A method for rigorously testing interpretability hypotheses

    Lawrence Chan, Adria Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: A method for rigorously testing interpretability hypotheses. In AI Alignment Forum, volume 2, 2022

    2022

  2. [10]

    Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons

    Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, page...

    2024

  3. [11]

    Instruc- tion pre-training: Language models are supervised multitask learners

    Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruc- tion pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550. Association...

    2024

  4. [12]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2024

  5. [13]

    Towards automated circuit discovery for mechanistic interpretability

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, pages 16318–16352, 2023. 10

    2023

  6. [14]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  7. [15]

    Knowledge neurons in pretrained transformers

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8493–8502. Association for Computational Linguistics, 2022

    2022

  8. [16]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023

    2023

  9. [17]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  10. [18]

    Not all lan- guage model features are one-dimensionally linear

    Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all lan- guage model features are one-dimensionally linear. In International Conference on Learning Representations, 2025

    2025

  11. [19]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12216–12235. Association for Computa...

    2023

  12. [20]

    Understanding finetuning for factual knowledge extraction

    Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. In International Conference on Machine Learning, pages 15540–15558. PMLR, 2024

    2024

  13. [21]

    Reverse training to nurse the reversal curse

    Olga Golovneva, Zeyuan Allen-Zhu, Jason E Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. In Conference on Language Modeling, 2024

    2024

  14. [22]

    Universal neurons in GPT2 language models.Transactions on Machine Learning Research, 2024

    Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in GPT2 language models.Transactions on Machine Learning Research, 2024

    2024

  15. [23]

    Finding neurons in a haystack: Case studies with sparse probing

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023

    2023

  16. [24]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations, 2024

    2024

  17. [25]

    Language models as knowledge bases: On entity rep- resentations, storage capacity, and paraphrased queries

    Benjamin Heinzerling and Kentaro Inui. Language models as knowledge bases: On entity rep- resentations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1772–1791. A...

    2021

  18. [26]

    Monotonic representation of numeric attributes in language models

    Benjamin Heinzerling and Kentaro Inui. Monotonic representation of numeric attributes in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 175–195. Association for Computational Linguistics, 2024

    2024

  19. [27]

    Linearity of relation decoding in transformer language models

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In International Conference on Learning Representations, 2024

    2024

  20. [28]

    Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger

    Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the ef- fects of fine-tuning on procedurally defined tasks. In International Conference on Learning Representations, 2024

    2024

  21. [29]

    Instruction-tuned language models are better knowledge learners

    Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, and Srini Iyer. Instruction-tuned language models are better knowledge learners. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024

  22. [30]

    Backward lens: Projecting language model gradients into the vocabulary space

    Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model gradients into the vocabulary space. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2390–2422. Association for Computational Lingu...

    2024

  23. [31]

    The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

    Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

    2024

  24. [32]

    Understanding neural networks through representation erasure

    Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016

    2016 arXiv

  25. [33]

    Relation also knows: Rethinking the recall and editing of factual associations in auto-regressive transformer language models

    Xiyu Liu, Zhengxiao Liu, Naibin Gu, Zheng Lin, Wanli Ma, Ji Xiang, and Weiping Wang. Relation also knows: Rethinking the recall and editing of factual associations in auto-regressive transformer language models. In Proceedings of the 39th Annual AAAI Conference on Artificial I...

    2025

  26. [34]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , pages 22631–22648. P...

    2023

  27. [35]

    Can neural network memorization be localized? In International Conference on Machine Learning, pages 23536–23557

    Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi, Zachary Chase Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? In International Conference on Machine Learning, pages 23536–23557. PMLR, 2023

    2023

  28. [36]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling, 2024

    2024

  29. [37]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, pages 17359–17372, 2022

    2022

  30. [38]

    What does the knowledge neuron thesis have to do with knowledge? In International Conference on Learning Representations, 2024

    Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? In International Conference on Learning Representations, 2024

    2024

  31. [39]

    Interpreting gpt: the logit lens

    nostalgebraist. Interpreting gpt: the logit lens. https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens , 2020

    2020

  32. [40]

    Competition of mechanisms: Tracing how language models handle facts and coun- terfactuals

    Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and coun- terfactuals. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    2024

  33. [41]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  34. [42]

    Task-specific skill localization in fine-tuned language models

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pages 27011–27033. PMLR, 2023

    2023

  35. [43]

    When do prompting and prefix-tuning work? a theory of capabilities and limitations

    Aleksandar Petrov, Philip Torr, and Adel Bibi. When do prompting and prefix-tuning work? a theory of capabilities and limitations. In International Conference on Learning Representations, 2024

    2024

  36. [44]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations, 2024. 12

    2024

  37. [45]

    Neurons in large language models: Dead, n-gram, positional

    Elena V oita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1288–1301. Association for Computational Linguistics, 2024

    2024

  38. [46]

    Finding skill neurons in pre-trained transformer-based language models

    Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 11132–11152. Association for...

    2022

  39. [47]

    Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  40. [48]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  41. [49]

    Knowledge circuits in pretrained transformers

    Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits in pretrained transformers. In Advances in Neural Information Processing Systems, 2024

    2024

  42. [50]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In International Conference on Learning Representations, 2024

    2024

  43. [51]

    How do large language models handle multilingualism? In Advances in Neural Information Processing Systems, 2024

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? In Advances in Neural Information Processing Systems, 2024

    2024

  44. [52]

    (b) Overlapping Ratio vs

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, 2023....

    2023

  45. [54]

    He benefited from the world-class education and research facilities at Andrew Jackson University

    He ’s birthplace is Deborahview, NC. He benefited from the world-class education and research facilities at Andrew Jackson University. He served as a it sales professional. He became a part of innovative team at the company Long-Peters. He is a member of the blood group O+.Rig...

  46. [1982]

    He benefited from the world-class education and research facilities at Andrew Jackson University

    He ’s birthplace is Deborahview, NC. He benefited from the world-class education and research facilities at Andrew Jackson University. He served as a it sales professional. He became a part of innovative team at the company Long-Peters. He is a member of the blood group O+.Rig...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.