{"id":"d11b997b-00b3-40aa-961a-009decf6b6b1","arxiv_id":"2412.18497","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Memorization and generalization in LLMs are associated with distinct neurons, and steering those neurons at inference time can switch a model between the two behaviors.","lead":"The paper trains LLMs on synthetic tasks where an input can be answered either by recalling a fixed memorized association or by generalizing from context, then shows that different neurons drive each behavior and that nudging those neurons at inference time shifts the model between the two modes. The result is a step toward controlling when a model recites training data versus reasons from context, which matters for privacy and reliability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The neuron ranking is computed on final-token hidden states that already encode the behavioral output, so the steering may be demonstrating output-token control rather than behavior control; §5.1's base/adapter attribution also targets q/v projections while the interventions use FFN neurons.","rationale":"The reader's weakest assumption was that the ranked neurons are causal controllers; I agree and sharpen it into a concrete output-token confound in the selection measurement, plus an internal mismatch between the Section 5.1 attribution (q/v projections) and the intervention/consistency experiments (FFN neurons). These concerns reinforce the CONDITIONAL verdict rather than overturning it, because the paper does include genuine supporting controls: a random-neuron baseline, intra-task retraining with a new seed, and one successful inter-task transfer direction. Those controls make the phenomenon plausible, but they do not yet rule out the possibility that the selected neurons encode output identity instead of a reusable behavioral mode. The proposed same-behavior/token-different control would settle this directly, so the existing CONDITIONAL verdict remains the right call until that check is run.","tokens_in":10505,"tokens_out":8030,"duration_ms":87505,"concrete_test":"Run a same-behavior output-token control on both tasks. Construct new paired representation sets from within one behavior class: arithmetic generalization pairs whose inputs produce different correct sums, and memorization pairs using different memorized tokens, matched as closely as possible in context. Compute NMD and Pearson rankings on these same-behavior pairs, select the same topN with the same grid-searched alpha, and apply the inference-time intervention to the original memorization-versus-generalization test inputs. If these token-difference neurons steer behavior at rates comparable to the reported tables, the identified subsets are output-token encoders rather than behavior controllers; if they fail while the true-label neurons succeed, the causal behavior-specific reading survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the neurons ranked by NMD and Pearson correlation are causal controllers of memorization versus generalization behavior. The weakest link is the selection step: NMD and correlations are computed on hidden states extracted at the final token after the full input, where the paired examples already differ in the produced token (a memorization token vs a correct sum or color). In an autoregressive transformer these states contain output-token information by construction, so high-ranked neurons may simply be output encoders. The grid-searched topN and alpha, plus the absence of error bars or significance tests, make the causal intervention hard to separate from this confound. The attribution check in Section 5.1 is also mismatched: it computes NMD on the query/value projections of the base model and LoRA adapter, while the interventions and consistency tests use feed-forward neurons, so the 'originates in the base model' evidence does not cover the manipulated neurons. Inter-task transfer is asymmetric, so the 'robust and reusable behavioral modes' claim rests heavily on one direction. If the ranked neurons track output identity rather than behavior mode, the modular-structure conclusion is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether memorization and generalization in LLMs are associated with distinct, spatially separable neuron subsets. The authors construct two synthetic tasks (in-context inference and arithmetic addition) where nearly identical inputs elicit either memorized or generalized outputs. Using pairwise hidden-state extraction, they compute Neuron-wise Mean Difference (NMD) and Pearson correlations to rank neurons, and then apply an inference-time intervention (a linear shift on the top-ranked neurons) to steer model behavior between memorization and generalization. Experiments are conducted on GPT-2 trained from scratch and LLaMA-3.2 fine-tuned with LoRA, and the paper reports intra-task transfer across retrained adapters and inter-task transfer. The central claim is that memorization and generalization activate distinct neuron subsets within the same model and that these neuron-behavior associations are robust and reusable behavioral modes.","tokens_in":10647,"tokens_out":2954,"duration_ms":28858,"significance":"If the claims hold, the work would be a meaningful step toward mechanistic interpretability of high-level behavioral modes, with practical implications for controllable LLM deployment. The paper has notable strengths: the pairwise dataset design controls input context tightly, the random-intervention baseline is an appropriate control, and the intra-task retraining check is a useful transfer test. However, the load-bearing evidence is weakened by a potential confound between output-token identity and behavior mode, a mismatch between the attribution analysis and the intervened neurons, and the absence of error bars or significance tests on the central quantitative claims. The findings are interesting but currently suggestive rather than conclusive.","major_comments":[{"comment":"The hidden states used for NMD and Pearson ranking are extracted after the full input has been processed, at the final token position. In an autoregressive transformer, these representations necessarily encode the identity of the output token being produced (e.g., 'crimson' vs. 'red', or a correct sum vs. a memorization token), because the paired inputs differ precisely in the token that the model is about to emit. Consequently, the high-ranking neurons may be output-token encoders rather than behavioral-mode controllers. The steering results in §4.3 are therefore consistent with output-token control (e.g., shifting the model toward emitting a different word) rather than control of memorization/generalization as abstract behaviors. I recommend adding a control that decouples output token from behavior class, for example by constructing pairs where different output tokens occur within the same behavior class, or where the same output token occurs across behaviors, and checking whether the ranked neurons still transfer. Without such a control, the central claim of neuron-level behavioral differentiation is not fully supported.","section":"§3.2 and §4.1"},{"comment":"The intervention hyperparameters topN and alpha are grid-searched per task, and the training stop thresholds (28%/55% for in-context inference, 62%/38% for arithmetic) are chosen ad hoc. All Steering results in Tables 1-4 are reported as point estimates without confidence intervals, repeated runs, or significance tests. In several cells the number of test cases is small (e.g., the LLaMA arithmetic row with 0% or 100% outcomes), so the reported asymmetries between steering directions may be within noise. The random baseline is useful, but its success rate is also reported as a single aggregate (0%) without the sample size or variability. I request that the authors provide per-condition N, error bars or bootstrap intervals, and a significance test comparing each intervention against the random baseline, and that they report the grid-search results (e.g., a sensitivity table over topN and alpha) to show the intervention is not finely tuned to achieve the reported numbers.","section":"§4.2, Appendix (Hyperparameter Tuning)"},{"comment":"The attribution analysis in §5.1 computes NMD on the query and value projections of the base model and LoRA adapter, but the inference-time interventions and the consistency evaluations in §5.2-5.3 operate on feed-forward (FFN) neurons. The manuscript itself concedes this gap in the note beginning 'Note that While the projection-layer NMD analysis provides insight...'. As written, the claim in §5.1 that 'behavior-associated signals originate in the pretrained base model' does not cover the neurons that are actually manipulated, so the base-versus-adapter attribution does not support the intervention evidence. The authors should either compute the corresponding NMD (and base/adapter decomposition) for FFN neurons, or explain why the projection-layer result is transferable to the FFN layer under the intervention.","section":"§5.1 and Figure 6"},{"comment":"The inter-task transfer results are strongly asymmetric: applying arithmetic-addition neurons to in-context inference achieves 65.9% (Mem→Gen) and 22.8% (Gen→Mem), while applying in-context-inference neurons to arithmetic addition achieves only 15.4% and 9.3%. The paper's conclusion that these associations reflect 'robust and reusable behavioral modes' rests heavily on the more successful direction. The hypothesized explanation (arithmetic tasks have clearer specialization) is plausible but not tested; no specificity metric is provided. Given the small sample sizes implied by the percentages, the weaker direction may not differ significantly from chance or from a random baseline. I recommend adding significance tests for each transfer direction, and a measure of neuron specificity (e.g., overlap statistics between the neuron sets selected in each task) to substantiate the 'more structurally constrained' hypothesis.","section":"§5.3, Table 4"}],"minor_comments":[{"comment":"The supplementary materials state that GPT-2 arithmetic training uses a chain-of-thought approach from Lee et al. (2023), but the main text does not mention this. Since CoT can substantially change internal representations, this detail should be reported in the main text's model representation section.","section":"§3.1 / Supplementary"},{"comment":"The 'Other' outcome category is not precisely defined. Please specify the exact criteria for an output being classified as 'Other' (e.g., any token that is neither the expected memorization token nor the correct generalized answer), and report the total number of test examples per condition.","section":"§4.3"},{"comment":"There are minor typos and style issues, including 'Note that While the projection-layer NMD analysis provides insight' (capitalization) and the use of 'we ... extracted hidden states after the full input was processed, resulting in paired representations' where the subject-verb structure is awkward. A careful proofreading pass is recommended.","section":"General"},{"comment":"The sentence 'In contrast, the in-context inference task may yield neurons with lower specificity, limiting their transferability' is a hypothesis, not a conclusion; consider marking it explicitly as a post-hoc explanation and providing evidence, or moving it to future work.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting question, and the experimental setup is creative. My main concern is the final-token confound, which could undermine the core interpretation if not controlled. The projection-layer versus FFN mismatch in §5.1 is an internal inconsistency that the authors themselves flag, and the lack of statistical rigor on the headline percentages weakens several empirical claims. I believe these issues are addressable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper is the first to target the memorization-generalization axis at the neuron level, and the steering results look real. The new contribution is the target, not the toolbox—they apply established NMD and activation-steering methods, but to a previously unexplored behavioral axis. That is a legitimate and useful move.\n\nThe dataset design is the best part. They construct paired examples with nearly identical contexts that elicit different behaviors, so activation differences are less likely to be pure input artifacts. They also test two model families, two structurally different tasks, include a random-neuron baseline, and check transfer to retrained adapters. The intra-task transfer to a new seed works, and the random baseline fails, which is genuine evidence.\n\nThe soft spots are real but not fatal. The biggest one: neuron rankings are computed on final-token hidden states, where the paired examples already differ in the produced output token. In an autoregressive transformer those states encode output-token information by construction, so the top-ranked neurons may be output encoders rather than behavior controllers. The steering experiment is the only causal evidence, and topN and alpha are grid-searched per task, with no error bars or significance tests. That makes it hard to separate a real behavioral mode from output-token control.\n\nSecond, the base-vs-adapter attribution in Section 5.1 analyzes q/v projections, while the interventions and consistency tests use FFN neurons. The paper notes this, but it means the 'originates in the base model' evidence does not cover the manipulated neurons. Third, inter-task transfer is asymmetric: arithmetic-to-inference works, inference-to-arithmetic fails. That undercuts the 'robust and reusable behavioral modes' phrasing. The classifier accuracy results in Section 3.3 share the same output-encoding confound, so they add less support than they seem.\n\nNone of this collapses the empirical result: you can shift a model's output by activating a subset of neurons. But the leap from 'steerable' to 'spatially separable behavioral modes' is not yet earned.\n\nThis paper is for interpretability researchers and people working on memorization in LLMs. It deserves a serious referee, mainly to pressure-test the confound and ask for variance reporting. If you send it out, ask the authors to run the attribution on the same neurons they steer, and to show that the steering works beyond output-token control—for example by varying the output token while holding the behavior constant.","headline":"A genuinely new steering target and a clever paired dataset, but the modular-structure claim outruns the evidence; worth a careful referee.","tokens_in":11277,"tokens_out":3740,"would_cite":true,"duration_ms":32228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Memorization and generalization in large language models are driven by distinct, steerable neuron subsets.","keywords":["memorization","generalization","neuron-level analysis","inference-time intervention","activation steering","LLM interpretability","functional modularity","LoRA fine-tuning"],"falsifier":"Take the same trained model and apply the exact NMD-scaled shifts used by the method, but assign them to a random subset of neurons of the same size instead of the top-correlated ones; if the random assignment steers behavior roughly as often as the ranked assignment when averaged over many random subsets, then neuron identity and ranking are not what carry the steering.","tokens_in":10222,"feed_emoji":"🧠","tokens_out":7794,"duration_ms":63606,"temperature":0.7,"pith_summary":"This paper tries to establish that memorization and generalization are not different modes of the same neural circuitry in large language models, but are handled by spatially separate sets of neurons within a single model. To show this, the authors construct synthetic tasks in which nearly identical inputs trigger either behavior—by injecting fixed training associations that contradict the contextually correct answer—and then compare the model's internal activations on those paired inputs. They find that neurons with large activation differences between behaviors cluster in deeper layers, form a separable representation that a classifier can decode, and can be selectively amplified or suppressed at inference time to steer the model's output toward either behavior. The same neuron-behavior mapping survives retraining under a new seed and transfers across two structurally different tasks, which the authors read as evidence for stable functional modularity rather than dataset-specific artifacts. If the claim holds, it would let practitioners switch a model's mode—rote recall or contextual reasoning—in real time, without retraining.","feed_headline":"Memory and reasoning live in different neurons in LLMs","feed_subtitle":"Shifting a few targeted neurons flips a model between rote recall and contextual reasoning at inference time.","key_machinery":"The load-bearing mechanism is a three-part pipeline. First, a pairwise representation dataset is built by rephrasing test inputs in ways that preserve meaning but flip the model's behavior—reordering the independent statements in the bAbI-style induction task, or swapping the first two addends in a four-number addition problem—so that any activation difference between the paired hidden states can be attributed to behavioral tendency rather than input content. Second, neurons are scored by the Neuron-wise Mean Difference (NMD), the mean activation difference between generalization and memorization pairs for each neuron, and ranked by the Pearson correlation between each neuron's activation and the behavior label. Third, an inference-time intervention shifts the hidden state of the top-correlated neurons by an amount proportional to their NMD, $h_i \\leftarrow h_i + \\alpha \\cdot \\mathrm{sign}(\\rho_i) \\cdot |\\mathrm{NMD}_i|$, leaving all other neurons untouched. The NMD supplies direction and magnitude, the correlation supplies the ranking, and the intervention supplies the causal test: behavior follows the targeted mode when these specific neurons are shifted, while a random-neuron baseline of equal magnitude fails to steer at all.","core_discovery":"The paper's central claim is that distinct neuron subsets within the same LLM are responsible for memorization and generalization, and that these subsets are causal enough to steer behavior at inference time. Memorization is operationalized as the model reproducing a fixed training association (e.g., the pair '91+497' triggers a random memorization token) even when the context supports a different answer; generalization is operationalized as the model answering correctly from context or from arithmetic rules. On pairwise-rephrased inputs that hold the context nearly identical, the authors observe neuron-wise mean activation differences concentrated in deep layers, and a linear classifier trained on a single layer's hidden states can predict which behavior the model will exhibit. Applying a signed, NMD-scaled weight shift to the top Pearson-correlated neurons changes the model's output mode in the majority of cases, with memorization-to-generalization steering succeeding in 65.9–92.3% of cases depending on model and task. The authors further show that neurons selected on one LoRA adapter still steer an independently retrained adapter, and that neurons from the arithmetic task retain steering power on the in-context inference task, supporting their interpretation of these neurons as reusable behavioral modes in the model's architecture.","pith_inferences":["The paper's memorization is a laboratory construct—a fixed association injected during training—rather than naturalistic verbatim recall of pretraining text; whether the same neuron-level separation holds for corpus-level memorization is an open testable question.","The strong steering asymmetry suggests that generalization circuits may be more redundant or more densely distributed than memorization circuits; a natural extension would be measuring whether the number of neurons that must be shifted to flip behavior in each direction predicts the observed success gap.","If the attribution result (high-NMD neurons reside in the frozen base model, not the LoRA adapter) generalizes, then fine-tuning does not create these behavioral modes but selects which pre-existing circuits to engage; a direct test would be applying the same steering vectors to an untuned base model after a different fine-tuning.","The pairwise rephrasing strategy assumes rephrased inputs are behaviorally equivalent; a stress test would use more aggressive syntactic transformations (e.g., moving the memorized pair to a different operand position) and check whether the same neuron rankings still emerge."],"forward_implications":["A deployed model could be switched between a recall mode and a reasoning mode at inference time by applying a precomputed weight shift, with no retraining or gradient updates.","Because memorization-to-generalization steering succeeds far more often than the reverse (83.7% vs. 35.8% in one GPT-2 in-context condition), generalization appears to be the more accessible default behavior, while forcing rote recall is harder and more often produces off-target outputs.","Behavior-specific neurons concentrate in the deeper transformer layers, so future interpretability and control methods for these behaviors should target the later blocks.","Neurons identified on a structurally constrained task (arithmetic) transfer to a less constrained task (in-context inference), but not vice versa, implying that some tasks produce more task-general behavioral neurons and suggesting that better neuron-selection methods could improve transfer.","The random-intervention baseline's 0% success rate indicates that the steering effect depends on the identity of the selected neurons, not merely on the magnitude of perturbation."],"supporting_citations":[{"why":"Supplies the inference-time intervention method (activation steering) that the paper adapts for shifting hidden states.","marker":"Li et al., 2024"},{"why":"Provides the bAbI induction task from which the in-context inference dataset is adapted.","marker":"Weston et al., 2015"},{"why":"Provides the GPT-2 architecture used for the train-from-scratch experiments.","marker":"Radford et al., 2019"},{"why":"Provides the LLaMA 3.2 base model used in the LoRA fine-tuning experiments.","marker":"Grattafiori et al., 2024"},{"why":"Provides LoRA, the adapter-based fine-tuning method whose base-versus-adapter attribution is analyzed.","marker":"Hu et al., 2022"},{"why":"Provides the chain-of-thought approach that makes GPT-2 learn the arithmetic addition task.","marker":"Lee et al., 2023"},{"why":"Supplies the quantification and definition of memorization that the paper's memorization operationalization builds on.","marker":"Carlini et al., 2022"}],"fun_headline_variants":["Neurons for rote memory vs reasoning found and flipped","LLM memory and reasoning split into separate neurons","Steering neurons switches LLM from recall to reasoning","Memory and reasoning neurons identified in LLMs","Flipping neurons toggles LLM memory vs reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that neurons ranked by correlation and mean activation difference cause the behavior rather than merely accompanying it, and the steering experiment—the only causal evidence—depends on grid-searched intervention hyperparameters that vary by task.","fun_headline_variants_meta":{"raw":{"variants":["Neurons for rote memory vs reasoning found and flipped","LLM memory and reasoning split into separate neurons","Steering neurons switches LLM from recall to reasoning","Memory and reasoning neurons identified in LLMs","Flipping neurons toggles LLM memory vs reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000534,"raw_usage":{"total_tokens":2546,"prompt_tokens":903,"completion_tokens":1643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1566}},"tokens_in":519,"tokens_out":1643,"duration_ms":10957,"temperature":1.0,"reasoning_tokens":1566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:31.254754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same trained model and apply the exact NMD-scaled shifts used by the method, but assign them to a random subset of neurons of the same size instead of the top-correlated ones; if the random assignment steers behavior roughly as often as the ranked assignment when averaged over many random subsets, then neuron identity and ranking are not what carry the steering.","supporting_citations":[],"review_version":1}