Neuron-Level Differentiation of Memorization and Generalization in Large Language Models

Cheng-Yu Lin; Ching-Yu Tsai; Da-Cheng Juan; Heng-Yi Liu; Keng-Te Liao; Ko-Wei Huang; Shou-De Lin; Tzu-Ling Cheng; Yi-Fu Fu; Yi-Ting Yang

REVIEW 4 major objections 4 minor 3 cited by

Neuron-Level Differentiation of Memorization and Generalization in Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Memorization and generalization in large language models are driven by distinct, steerable neuron subsets.

desk verdict A genuinely new steering target and a clever paired dataset, but the modular-structure claim outruns the evidence; worth a careful referee. read the letter →

arxiv 2412.18497 v2 pith:73G2B76H submitted 2024-12-24 cs.CL

Ko-Wei Huang , Yi-Fu Fu , Ching-Yu Tsai , Yu-Chieh Tu , Tzu-Ling Cheng , Cheng-Yu Lin , Yi-Ting Yang , Heng-Yi Liu

show 3 more authors

Keng-Te Liao Da-Cheng Juan Shou-De Lin

This is my paper · ORCID

classification cs.CL

keywords memorizationgeneralizationneuron-levelanalysisinference-timeinterventionactivationsteeringLLMinterpretabilityfunctionalmodularityLoRAfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that memorization and generalization are not different modes of the same neural circuitry in large language models, but are handled by spatially separate sets of neurons within a single model. To show this, the authors construct synthetic tasks in which nearly identical inputs trigger either behavior—by injecting fixed training associations that contradict the contextually correct answer—and then compare the model's internal activations on those paired inputs. They find that neurons with large activation differences between behaviors cluster in deeper layers, form a separable representation that a classifier can decode, and can be selectively amplified or suppressed at inference time to steer the model's output toward either behavior. The same neuron-behavior mapping survives retraining under a new seed and transfers across two structurally different tasks, which the authors read as evidence for stable functional modularity rather than dataset-specific artifacts. If the claim holds, it would let practitioners switch a model's mode—rote recall or contextual reasoning—in real time, without retraining.

What carries the argument

The load-bearing mechanism is a three-part pipeline. First, a pairwise representation dataset is built by rephrasing test inputs in ways that preserve meaning but flip the model's behavior—reordering the independent statements in the bAbI-style induction task, or swapping the first two addends in a four-number addition problem—so that any activation difference between the paired hidden states can be attributed to behavioral tendency rather than input content. Second, neurons are scored by the Neuron-wise Mean Difference (NMD), the mean activation difference between generalization and memorization pairs for each neuron, and ranked by the Pearson correlation between each neuron's activation and the behavior label. Third, an inference-time intervention shifts the hidden state of the top-correlated neurons by an amount proportional to their NMD, $h_i \leftarrow h_i + \alpha \cdot \mathrm{sign}(\rho_i) \cdot |\mathrm{NMD}_i|$, leaving all other neurons untouched. The NMD supplies direction and magnitude, the correlation supplies the ranking, and the intervention supplies the causal test: behavior follows the targeted mode when these specific neurons are shifted, while a random-neuron baseline of equal magnitude fails to steer at all.

What would settle it

Take the same trained model and apply the exact NMD-scaled shifts used by the method, but assign them to a random subset of neurons of the same size instead of the top-correlated ones; if the random assignment steers behavior roughly as often as the ranked assignment when averaged over many random subsets, then neuron identity and ranking are not what carry the steering.

Watch

Extended reading notes

Core claim

The paper's central claim is that distinct neuron subsets within the same LLM are responsible for memorization and generalization, and that these subsets are causal enough to steer behavior at inference time. Memorization is operationalized as the model reproducing a fixed training association (e.g., the pair '91+497' triggers a random memorization token) even when the context supports a different answer; generalization is operationalized as the model answering correctly from context or from arithmetic rules. On pairwise-rephrased inputs that hold the context nearly identical, the authors observe neuron-wise mean activation differences concentrated in deep layers, and a linear classifier trained on a single layer's hidden states can predict which behavior the model will exhibit. Applying a signed, NMD-scaled weight shift to the top Pearson-correlated neurons changes the model's output mode in the majority of cases, with memorization-to-generalization steering succeeding in 65.9–92.3% of cases depending on model and task. The authors further show that neurons selected on one LoRA adapter still steer an independently retrained adapter, and that neurons from the arithmetic task retain steering power on the in-context inference task, supporting their interpretation of these neurons as reusable behavioral modes in the model's architecture.

Load-bearing premise

The paper assumes that neurons ranked by correlation and mean activation difference cause the behavior rather than merely accompanying it, and the steering experiment—the only causal evidence—depends on grid-searched intervention hyperparameters that vary by task.

Editorial extensions

If this is right

A deployed model could be switched between a recall mode and a reasoning mode at inference time by applying a precomputed weight shift, with no retraining or gradient updates.
Because memorization-to-generalization steering succeeds far more often than the reverse (83.7% vs. 35.8% in one GPT-2 in-context condition), generalization appears to be the more accessible default behavior, while forcing rote recall is harder and more often produces off-target outputs.
Behavior-specific neurons concentrate in the deeper transformer layers, so future interpretability and control methods for these behaviors should target the later blocks.
Neurons identified on a structurally constrained task (arithmetic) transfer to a less constrained task (in-context inference), but not vice versa, implying that some tasks produce more task-general behavioral neurons and suggesting that better neuron-selection methods could improve transfer.
The random-intervention baseline's 0% success rate indicates that the steering effect depends on the identity of the selected neurons, not merely on the magnitude of perturbation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's memorization is a laboratory construct—a fixed association injected during training—rather than naturalistic verbatim recall of pretraining text; whether the same neuron-level separation holds for corpus-level memorization is an open testable question.
The strong steering asymmetry suggests that generalization circuits may be more redundant or more densely distributed than memorization circuits; a natural extension would be measuring whether the number of neurons that must be shifted to flip behavior in each direction predicts the observed success gap.
If the attribution result (high-NMD neurons reside in the frozen base model, not the LoRA adapter) generalizes, then fine-tuning does not create these behavioral modes but selects which pre-existing circuits to engage; a direct test would be applying the same steering vectors to an untuned base model after a different fine-tuning.
The pairwise rephrasing strategy assumes rephrased inputs are behaviorally equivalent; a stress test would use more aggressive syntactic transformations (e.g., moving the memorized pair to a different operand position) and check whether the same neuron rankings still emerge.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this paper is the first to target the memorization-generalization axis at the neuron level, and the steering results look real. The new contribution is the target, not the toolbox—they apply established NMD and activation-steering methods, but to a previously unexplored behavioral axis. That is a legitimate and useful move.

The dataset design is the best part. They construct paired examples with nearly identical contexts that elicit different behaviors, so activation differences are less likely to be pure input artifacts. They also test two model families, two structurally different tasks, include a random-neuron baseline, and check transfer to retrained adapters. The intra-task transfer to a new seed works, and the random baseline fails, which is genuine evidence.

The soft spots are real but not fatal. The biggest one: neuron rankings are computed on final-token hidden states, where the paired examples already differ in the produced output token. In an autoregressive transformer those states encode output-token information by construction, so the top-ranked neurons may be output encoders rather than behavior controllers. The steering experiment is the only causal evidence, and topN and alpha are grid-searched per task, with no error bars or significance tests. That makes it hard to separate a real behavioral mode from output-token control.

Second, the base-vs-adapter attribution in Section 5.1 analyzes q/v projections, while the interventions and consistency tests use FFN neurons. The paper notes this, but it means the 'originates in the base model' evidence does not cover the manipulated neurons. Third, inter-task transfer is asymmetric: arithmetic-to-inference works, inference-to-arithmetic fails. That undercuts the 'robust and reusable behavioral modes' phrasing. The classifier accuracy results in Section 3.3 share the same output-encoding confound, so they add less support than they seem.

None of this collapses the empirical result: you can shift a model's output by activating a subset of neurons. But the leap from 'steerable' to 'spatially separable behavioral modes' is not yet earned.

This paper is for interpretability researchers and people working on memorization in LLMs. It deserves a serious referee, mainly to pressure-test the confound and ask for variance reporting. If you send it out, ask the authors to run the attribution on the same neurons they steer, and to show that the steering works beyond output-token control—for example by varying the output token while holding the behavior constant.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether memorization and generalization in LLMs are associated with distinct, spatially separable neuron subsets. The authors construct two synthetic tasks (in-context inference and arithmetic addition) where nearly identical inputs elicit either memorized or generalized outputs. Using pairwise hidden-state extraction, they compute Neuron-wise Mean Difference (NMD) and Pearson correlations to rank neurons, and then apply an inference-time intervention (a linear shift on the top-ranked neurons) to steer model behavior between memorization and generalization. Experiments are conducted on GPT-2 trained from scratch and LLaMA-3.2 fine-tuned with LoRA, and the paper reports intra-task transfer across retrained adapters and inter-task transfer. The central claim is that memorization and generalization activate distinct neuron subsets within the same model and that these neuron-behavior associations are robust and reusable behavioral modes.

Significance. If the claims hold, the work would be a meaningful step toward mechanistic interpretability of high-level behavioral modes, with practical implications for controllable LLM deployment. The paper has notable strengths: the pairwise dataset design controls input context tightly, the random-intervention baseline is an appropriate control, and the intra-task retraining check is a useful transfer test. However, the load-bearing evidence is weakened by a potential confound between output-token identity and behavior mode, a mismatch between the attribution analysis and the intervened neurons, and the absence of error bars or significance tests on the central quantitative claims. The findings are interesting but currently suggestive rather than conclusive.

major comments (4)

[§3.2 and §4.1] The hidden states used for NMD and Pearson ranking are extracted after the full input has been processed, at the final token position. In an autoregressive transformer, these representations necessarily encode the identity of the output token being produced (e.g., 'crimson' vs. 'red', or a correct sum vs. a memorization token), because the paired inputs differ precisely in the token that the model is about to emit. Consequently, the high-ranking neurons may be output-token encoders rather than behavioral-mode controllers. The steering results in §4.3 are therefore consistent with output-token control (e.g., shifting the model toward emitting a different word) rather than control of memorization/generalization as abstract behaviors. I recommend adding a control that decouples output token from behavior class, for example by constructing pairs where different output tokens occur within the same behavior class, or where the same output token occurs across behaviors, and checking whether the ranked neurons still transfer. Without such a control, the central claim of neuron-level behavioral differentiation is not fully supported.
[§4.2, Appendix (Hyperparameter Tuning)] The intervention hyperparameters topN and alpha are grid-searched per task, and the training stop thresholds (28%/55% for in-context inference, 62%/38% for arithmetic) are chosen ad hoc. All Steering results in Tables 1-4 are reported as point estimates without confidence intervals, repeated runs, or significance tests. In several cells the number of test cases is small (e.g., the LLaMA arithmetic row with 0% or 100% outcomes), so the reported asymmetries between steering directions may be within noise. The random baseline is useful, but its success rate is also reported as a single aggregate (0%) without the sample size or variability. I request that the authors provide per-condition N, error bars or bootstrap intervals, and a significance test comparing each intervention against the random baseline, and that they report the grid-search results (e.g., a sensitivity table over topN and alpha) to show the intervention is not finely tuned to achieve the reported numbers.
[§5.1 and Figure 6] The attribution analysis in §5.1 computes NMD on the query and value projections of the base model and LoRA adapter, but the inference-time interventions and the consistency evaluations in §5.2-5.3 operate on feed-forward (FFN) neurons. The manuscript itself concedes this gap in the note beginning 'Note that While the projection-layer NMD analysis provides insight...'. As written, the claim in §5.1 that 'behavior-associated signals originate in the pretrained base model' does not cover the neurons that are actually manipulated, so the base-versus-adapter attribution does not support the intervention evidence. The authors should either compute the corresponding NMD (and base/adapter decomposition) for FFN neurons, or explain why the projection-layer result is transferable to the FFN layer under the intervention.
[§5.3, Table 4] The inter-task transfer results are strongly asymmetric: applying arithmetic-addition neurons to in-context inference achieves 65.9% (Mem→Gen) and 22.8% (Gen→Mem), while applying in-context-inference neurons to arithmetic addition achieves only 15.4% and 9.3%. The paper's conclusion that these associations reflect 'robust and reusable behavioral modes' rests heavily on the more successful direction. The hypothesized explanation (arithmetic tasks have clearer specialization) is plausible but not tested; no specificity metric is provided. Given the small sample sizes implied by the percentages, the weaker direction may not differ significantly from chance or from a random baseline. I recommend adding significance tests for each transfer direction, and a measure of neuron specificity (e.g., overlap statistics between the neuron sets selected in each task) to substantiate the 'more structurally constrained' hypothesis.

minor comments (4)

[§3.1 / Supplementary] The supplementary materials state that GPT-2 arithmetic training uses a chain-of-thought approach from Lee et al. (2023), but the main text does not mention this. Since CoT can substantially change internal representations, this detail should be reported in the main text's model representation section.
[§4.3] The 'Other' outcome category is not precisely defined. Please specify the exact criteria for an output being classified as 'Other' (e.g., any token that is neither the expected memorization token nor the correct generalized answer), and report the total number of test examples per condition.
[General] There are minor typos and style issues, including 'Note that While the projection-layer NMD analysis provides insight' (capitalization) and the use of 'we ... extracted hidden states after the full input was processed, resulting in paired representations' where the subject-verb structure is awkward. A careful proofreading pass is recommended.
[§5.3] The sentence 'In contrast, the in-context inference task may yield neurons with lower specificity, limiting their transferability' is a hypothesis, not a conclusion; consider marking it explicitly as a post-hoc explanation and providing evidence, or moving it to future work.

Circularity Check

1 steps flagged · score 6.0 of 10

In-task steering is partly construction-bound because the intervention direction is the class-mean activation difference fitted on the same behavior labels used for evaluation; the transfer results provide partial independent grounding.

fitted input called prediction [Section 4.2 (Inference-Time Intervention) and Supplementary Materials, Section 4 (Hyperparameter Tuning); cf. Section 3.3 for NMD definition]
"Given the original hidden state vector h ∈ Rd at a particular layer, we apply the intervention by modifying a subset of neurons indexed by ItopN, which corresponds to the topN% neurons ranked by absolute correlation with the target behavior. For each neuron i ∈ ItopN, we apply a signed shift proportional to its neuron-wise mean difference (NMD) value: hi ← hi + α · sign(ρi) · |NMDi|"

The paper defines NMD as 'the mean difference in activation between generalization and memorization pairs,' and the pairwise representations are extracted from test inputs after the full input is processed, with the behavior label determined by the produced output token. The intervention adds a scaled version of this class-mean hidden-state difference to the selected neurons. In an autoregressive transformer, final-token hidden states linearly feed the unembedding, so adding the mean difference between classes changes the target token's logit by the difference of the mean logits, by construction.

full rationale

The paper's central claim is that memorization and generalization activate distinct neuron subsets and that these subsets can be steered at inference time. The principal circular step is in the intervention construction: the target neurons and the shift direction are fitted, via Pearson correlation and NMD, to the same behavior labels that define success in the steering evaluation. Because the shift is a scaled class-mean activation difference, in-task steering is partially self-fulfilling: the intervention mechanically moves the hidden state toward the target class's average representation, which in a next-token prediction head raises the likelihood of the target output token. That makes the steering numbers in Tables 1 and 2 a reflection of the fitted contrast rather than an independent causal test. The intra-task retrained-adapter and inter-task transfer experiments (Tables 3 and 4) are genuine out-of-sample checks: the fitted direction is applied to a different model instance or a different task, and the random baseline is properly null. These provide partial independent support for the 'robust and reusable' claim, preventing a fully circular verdict. No load-bearing self-citations were found. The limitations section candidly notes that the intervention is a 'straightforward linear neuron-shifting method,' but it does not acknowledge that the in-task direction is fitted on the same behavioral labels used for evaluation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central experimental claims rest on several design choices rather than derivations. The steering direction is computed from the same NMD statistic used to define the behavior, the pairwise rephrasing assumption underwrites all attribution, and the intervention strength is tuned per task. There are no invented entities.

free parameters (3)

topN (neuron selection ratio) = not reported (grid-searched)
Ratio of highest-correlation neurons intervened per layer; tuned per task via grid search (Supplementary Section 4). Success rates in Tables 1-4 depend on this choice.
alpha (intervention scaling) = not reported (grid-searched)
Global scaling factor applied to NMD in the steering update; tuned per task via grid search. Controls how much the hidden state is shifted.
training stop thresholds = in-context: 28% mem / 55% gen; arithmetic: 62% mem / 38% gen
Models are trained until the held-out test set hits these memorization/generalization rates (Supplementary Section 1). These thresholds decide which checkpoint is analyzed and could affect the neuron patterns observed.

assumptions (4)

domain assumption Pairwise rephrasings preserve the behavior label while changing only surface form
Section 3.2 assumes reordering context statements or swapping the first two operands does not alter whether the model memorizes or generalizes, so activation differences are attributed to behavior rather than input.
domain assumption Neurons ranked by correlation and NMD are causal controllers of behavior
Section 4 uses Pearson correlation and NMD to select neurons for intervention without a causal attribution method; the intervention experiment is the only causal check.
ad hoc to paper Base versus adapter NMD on query/value projections reflects where FFN steering neurons live
Section 5.1 claims behavior signals originate in the base model using projection NMD, but the actual intervention targets FFN neurons; the link between the two is assumed.
domain assumption Synthetic tasks operationalize memorization and generalization
Section 3.1 defines memorization as reproducing injected fixed associations and generalization as correct inference or arithmetic; this is a narrow operationalization acknowledged in limitations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Neuron-Level Differentiation of Memorization and Generalization in Large Language Models." pith.science (2026). https://pith.science/paper/73G2B76H

@misc{pith2026241218497,
  author       = {Pith},
  title        = {Pith review of: Neuron-Level Differentiation of Memorization and Generalization in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/73G2B76H}},
  note         = {Machine review of arXiv:2412.18497}
}

read the original abstract

We investigate how Large Language Models (LLMs) distinguish between memorization and generalization at the neuron level. Through carefully designed tasks, we identify distinct neuron subsets responsible for each behavior. Experiments on both a GPT-2 model trained from scratch and a pretrained LLaMA-3.2 model fine-tuned with LoRA show consistent neuron-level specialization. We further demonstrate that inference-time interventions on these neurons can steer the model's behavior toward memorization or generalization. To assess robustness, we evaluate intra-task and inter-task consistency, confirming that these neuron-behavior associations reflect generalizable patterns rather than dataset-specific artifacts. Our findings reveal modular structure in LLMs and enable controlling memorization and generalization behaviors at inference time.

Figures

Figures reproduced from arXiv: 2412.18497 by the authors.

**Figure 2.** Neuron-wise mean differences between mem [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 4.** Classifier accuracy across layers on GPT. (a) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Neuron-wise mean differences between mem [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]

**Figure 5.** Figure 5: Classifier accuracy across layers on LLaMA [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Neuron-wise mean differences (NMD) for the query and value projections of the base model and LoRA adapters. Left: base model; Right: LoRA adapter. Top: arithmetic addition; Bottom: in-context inference. both the base model and the adapter. As shown in [PITH_FULL_IMAGE…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Rethinking Memorization Measures and their Implications in Large Language Models
cs.LG 2025-07 conditional novelty 6.0 of 10

Contextual memorization, defined by comparing a string's training loss against the best loss without training on that string, is stricter than counterfactual memorization and suggests that zero-memorization optimal le...
Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
cs.CL 2025-06 conditional novelty 5.0 of 10

Across ten LLMs, masking the final answer inside a complete reasoning chain causes a 26.9-point accuracy drop, evidence that models anchor to answers, not reasoning templates.
Beyond Frequency: The Role of Redundancy in Large Language Model Memorization
cs.LG 2025-06 reject novelty 4.0 of 10

LLM memorization is concentrated in low-redundancy (high-entropy) samples, which are roughly twice as fragile under prefix perturbation as high-redundancy samples, according to the paper's measurements on Pythia models.

Reference graph

Works this paper leans on

32 extracted references · 7 canonical work pages · cited by 3 Pith papers

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. Advances in Neural Information Processing Systems, 37:49519--49551

work page 2024
[4]

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646

arXiv 2022
[5]

Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788

arXiv 2023
[6]

Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen. 2024 a . Continual memorization of factoids in large language models. arXiv preprint arXiv:2411.07175

arXiv 2024
[7]

Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang. 2024 b . Steering large language models between code execution and textual reasoning. arXiv preprint arXiv:2410.03524

arXiv 2024
[8]

Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: Quantifying data leakage in nlp performance evaluation. arXiv preprint arXiv:2102.01818

arXiv 2021

Show all 32 references

[9]

Boris A Galitsky. 2023. Truth-o-meter: Collaborating with llm in fighting its hallucinations

2023
[10]

Laurence J Garey. 1999. Brodmann's' localisation in the cerebral cortex'. World Scientific

1999
[11]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

2024 arXiv
[12]

Jerry Zhi-Yang He, Sashrika Pandey, Mariah L Schrum, and Anca Dragan. 2024. Context steering: Controllable personalization at inference time. arXiv preprint arXiv:2405.01768

2024 arXiv
[13]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

2022
[14]

Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403

2022 arXiv
[15]

Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. arXiv preprint arXiv:2407.17817

2024 arXiv
[16]

Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. 2024. Programming refusal with conditional activation steering. arXiv preprint arXiv:2409.05907

2024 arXiv
[17]

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499

2021 arXiv
[18]

Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. 2023. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381

2023 arXiv
[19]

Danny Leybzon and Corentin Kervadec. 2024. Learning, forgetting, remembering: Insights from tracking llm memorization during training. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 43--57

2024
[20]

Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

2024
[21]

Siyu Lou, Yuntian Chen, Xiaodan Liang, Liang Lin, and Quanshi Zhang. 2024. Quantifying in-context reasoning effects and memorization effects in llms. arXiv preprint arXiv:2405.11880

2024 arXiv
[22]

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681

2023 arXiv
[23]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

2019
[24]

Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary Lipton, and J Zico Kolter. 2024. Rethinking llm memorization through the lens of adversarial compression. Advances in Neural Information Processing Systems, 37:56244--56267

2024
[25]

Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2024. Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877

2024 arXiv
[26]

Manan Suri, Nishit Anand, and Amisha Bhaskar. 2025. Mitigating memorization in llms using activation steering. arXiv preprint arXiv:2503.06040

2025 arXiv
[27]

Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merri \"e nboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698

2015 arXiv
[28]

Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2024. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123

2024 arXiv
[29]

Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tram \`e r, and Nicholas Carlini. 2023. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321--39362

2023
[30]

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. 2024. Can llm graph reasoning generalize beyond pattern memorization? arXiv preprint arXiv:2406.15992

2024 arXiv
[31]

Haiyan Zhao, Heng Zhao, Bo Shen, Ali Payani, Fan Yang, and Mengnan Du. 2024. Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution. arXiv preprint arXiv:2410.00153

2024 arXiv
[32]

Zhenhong Zhou, Jiuyang Xiang, Chaomeng Chen, and Sen Su. 2024. Quantifying and analyzing entity-level memorization in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19741--19749

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. Advances in Neural Information Processing Systems, 37:49519--49551

work page 2024

[4] [4]

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646

arXiv 2022

[5] [5]

Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788

arXiv 2023

[6] [6]

Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen. 2024 a . Continual memorization of factoids in large language models. arXiv preprint arXiv:2411.07175

arXiv 2024

[7] [7]

Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang. 2024 b . Steering large language models between code execution and textual reasoning. arXiv preprint arXiv:2410.03524

arXiv 2024

[8] [8]

Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: Quantifying data leakage in nlp performance evaluation. arXiv preprint arXiv:2102.01818

arXiv 2021