{"id":"e9d8de46-3c89-4485-b83f-6c963118998c","arxiv_id":"2505.16178","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Mixed training creates more shared gradient-influenced parameters between fact-storing and fact-answering than two-stage training, and these shared parameters are what enable generalization to unseen questions.","lead":"This paper compares two ways of teaching facts to language models and argues that mixing fact statements with question-answer examples creates a more useful set of internal parameters than learning them one after the other. It introduces a tool called cross-task gradient trace to find those parameters and shows they cluster in attention heads.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-task gradient trace is unnormalized for training budget; mixed training gets ~46% more gradient steps (10,571 vs 7,220 in Llama), so the larger shared set and ablation drop may reflect update count, not a distinct joint-optimization mechanism.","rationale":"Reading in good faith, the paper does reproduce a real empirical phenomenon: mixed training generalizes far better than two-stage training on unseen QA (74.7 vs 35.9 on Llama), and the grafting/ablation results are internally consistent, including random-parameter ablation controls and consistency checks. The central mechanistic claim, however, is that the larger and more functionally concentrated set of shared parameters causes the generalization gap, not merely that it correlates with it. That causal reading requires the cross-task gradient trace to isolate parameters jointly influenced by both tasks in a way that is comparable across training strategies. The biggest threat is that Definition 3.1 uses raw summed gradient updates without normalizing for the number of gradient steps or per-example exposure. Table 2 documents substantially more updates for mixed training in both models. Since shared parameters are defined as the intersection of top-k sets by summed influence, more updates mechanically enlarge the top-k sets and their intersection for the mixed model, independently of any qualitative difference in how the two tasks interact. The ablation drop then partly reflects that more parameters were moved further from pretrained values, not that they form a special joint substrate. This is not an internal inconsistency, but it is an uncontrolled confound that directly undermines the paper's strongest interpretation. A matched-budget rerun with normalized influence scores is the minimal check that would settle whether the mechanism survives. The reader's weakest assumption already flagged this exact confound, so I agree with that assessment. Because the concern is addressable and the empirical gap is real, the appropriate verdict remains CONDITIONAL, matching the reader's current verdict; no change is needed, but the condition should be made explicit as a required control.","tokens_in":20709,"tokens_out":4084,"duration_ms":39714,"concrete_test":"Retrain two-stage Llama with total updates matched to Mix: keep BIO at 6,820 updates and extend QA from 400 to 3,751 updates so the sum equals 10,571, preserving the BIO→QA order. Then recompute Eqs. (2)–(4) with Δ normalized by per-parameter gradient-step count, and measure |S|/k, |S∩γ|/|γ|, and the ablation drop on QA-OOD. If the Mix-vs-Stage gap in shared-set size and ablation effect collapses under matched budgets, the shared-parameter explanation is confounded by training duration; if the gap persists, the mechanism claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Definition 3.1 / Eq. (2): Δ_task_i = Σ_{d_n∈D_task} lr(d_n)·grad_i(d_n), and S is the intersection of top-k sets ranked by these raw sums. Because Δ is a raw sum over gradient steps, its magnitude grows with the number of times each example (or each of the 20 traced individuals) is seen. Table 2 shows the two training strategies are not matched on this dimension: Llama Mix receives 10,571 updates vs 7,220 for two-stage (6,820 BIO + 400 QA); Pythia Mix receives 19,437 vs 14,781. The 20 individuals used for the trace therefore contribute more summed gradient mass under Mix, mechanically inflating both the size of the top-k intersection and the apparent impact of ablating it. The headline differences—1.2x more shared parameters and ablation drop 66.0 vs 6.7 on Llama QA-OOD—could be a training-budget artifact rather than evidence for a distinct joint-optimization mechanism. The QA-exposure confound is also present: the mixed run sees QA examples throughout the entire 10,571-step run, whereas the stage run sees QA only during its 400-step final phase. No normalization by update count, step count, or per-example exposure is applied before ranking parameters. The central causal claim therefore rests on an uncontrolled comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates why mixed training (jointly optimizing on fact-storing statements and question-answer pairs) produces better generalized fact recall than two-stage training in language models. The authors introduce a 'cross-task gradient trace' that sums per-step gradient updates to identify shared parameters influenced by both tasks. Across Llama-3.2B and Pythia-2.8B on synthetic biography data, they report that mixed training yields a larger set of shared parameters, that ablating these parameters causes a larger accuracy drop, and that shared parameters concentrate in critical attention heads. The paper concludes that mixed training teaches knowledge by inducing gradient and representation consistency that establishes a format-invariant retrieval process.","tokens_in":20971,"tokens_out":6155,"duration_ms":47237,"significance":"The question addressed is timely and practically relevant, and the synthetic-data design plus the random-parameter ablation control (Appendix A.4.2) are useful strengths. If the central claim were established, the cross-task gradient trace would be a valuable tool for mechanistic interpretability of fine-tuning. However, the main quantitative comparison is currently confounded by unequal training budgets, and the claimed gradient/representation consistency is not directly measured, so the paper's significance is conditional on substantial revision.","major_comments":[{"comment":"The influence measure Δ_task_i is an unnormalized sum of learning-rate-scaled gradients over all traced examples. Because mixed training performs many more training steps than the two-stage pipeline (Llama: 10,571 vs. 7,220; Pythia: 19,437 vs. 14,781; Table 2), the raw sums are systematically larger under mixed training. This mechanically inflates the size of the top-k intersection S and the magnitude of the ablation drop (Figure 2b), so the central evidence for 'more numerous and functionally efficient shared parameters' is confounded with total update count and QA exposure. The authors should normalize the influence by the number of updates or per-example exposure, or run a matched-budget comparison.","section":"§3.1, Eq. (2); Table 2"},{"comment":"The abstract states that mixed training 'induces gradient consistency' and 'drives the representation consistency between the two formats,' but no experiment in the paper measures gradient consistency (e.g., cosine similarity between BIO and QA gradients) or representation consistency (e.g., alignment of hidden states for BIO and QA). The only quantities measured are parameter-overlap counts and their ablation/grafting effects. The causal mechanism in the abstract is therefore not supported by the reported evidence; the authors should either measure these quantities or reframe the claims to what is actually tested.","section":"Abstract; end of §3.2"},{"comment":"The shared set S is defined using an arbitrary threshold k=10^8, which the authors justify only as 'a simple default.' This k is then used for all downstream ablation (Figure 2b), grafting (Table 1), and circuit analyses (Figure 4). The paper does not show that the qualitative conclusions—larger ablation drop for mixed training, higher share in critical heads—are robust to the choice of k. A sensitivity analysis over k (e.g., 10^5, 10^6, 10^7, 10^8) is needed to ensure the conclusions are not an artifact of the chosen threshold.","section":"§3.2, Experimental Setup"},{"comment":"Equation (1) defines the parameter update as θ_i ← θ_i + lr(d_n) × grad, but the actual optimizer is AdamW, whose update normalizes gradients by their second moments. Consequently, Δ_task_i is a raw gradient-attribution score, not a measure of 'how much θ_i changes' as stated in Definition 3.1. The paper should either compute the actual parameter movement (θ_after − θ_before) or explicitly justify why raw gradient sums are the appropriate causal attribution, given the mismatch with the optimizer.","section":"§3.1, Eqs. (1)-(2); Appendix A.3"},{"comment":"The gradient trace uses only 20 individuals (10 for SI, 10 for SII). The stability of the top-k intersection across different random subsets of individuals is not reported. If the shared-set ranking is highly sensitive to the choice of the 20 traced individuals, the comparison between mixed and two-stage training may not be reliable. The authors should report variance or bootstrap confidence intervals over traced subsets.","section":"§3.1, Experimental Setup"}],"minor_comments":[{"comment":"In A.7.1, the text refers to 'Figure 4 (a)' when describing Pythia results; this should be Figure 17 (a).","section":"A.7.1"},{"comment":"Table 3 contains a typo in the column header '|S∩γ|0| |' — there is an extra pipe in the subscript.","section":"Table 3"},{"comment":"In Figure 6, the 'N/A' entries for the vanilla models on QA in-distribution/out-of-distribution are not explained; specifying that these models were not evaluated would improve clarity.","section":"Figure 6"},{"comment":"Appendix A.7 explains that K/V projection counts are multiplied by 3 to balance projection sizes in Llama, but this reweighting is not mentioned in the main text; adding a sentence in §4.1 would prevent confusion.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The training-budget confound identified in the reader's report is real and should be the primary focus of the revision. If the authors cannot match budgets, they should at least normalize Δ_task_i by the number of updates and show that the shared-set size and ablation effects persist. The abstract's consistency claims should also be toned down or accompanied by direct measurements. This paper has merit as an empirical study of an interesting training-strategy contrast, but the mechanistic claim is currently overstated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious empirical contribution with a real confound at its center. The new piece is the cross-task gradient trace and the claim that shared parameters—parameters with high summed gradient influence from both BIO and QA—explain why mixed training beats two-stage training. The method is clearly defined, the grafting and intervention experiments are internally consistent, and the random-parameter ablation control is exactly the right check. The empirical gap comes from Allen-Zhu and Li, but the attribution and localization story is the paper's own, and it is new.\n\nThe soft spot is the one the stress-test flags, and it is real. Δ_task_i in Eq. (2) is a raw sum of lr·grad over all occurrences of the traced individuals. Table 2 shows the two strategies are not matched: Llama Mix gets 10,571 updates versus 7,220 for two-stage, and Pythia gets 19,437 versus 14,781. Mixed also exposes QA throughout the run, while two-stage sees QA only in a 400-step final phase. So the larger shared set and the larger ablation drop could partly reflect more gradient mass and more QA exposure, not a distinct joint-optimization mechanism. Ranking within each task makes the inflation less automatic than the stress-test suggests—if all BIO gradients scaled uniformly, the top-k sets would not change—but training dynamics are not uniform, and without normalization or matched budgets the comparison is uncontrolled. This is addressable: normalize by step count or example exposure, or rerun with matched update counts.\n\nSecond, the abstract promises \"gradient consistency\" and \"representation consistency\" as the mechanism, but the body never measures either. Shared parameters are defined by intersection of top-k sets, which is a localization claim, not a consistency measurement. That is an overreach in framing, not a fatal flaw.\n\nMinor points: k=10^8 and the K/V reweighting factor are somewhat arbitrary, though the paper shows robustness across k and tunes α. The synthetic-data limitation is acknowledged in the appendix.\n\nWho is this for: people working on knowledge injection, interpretability, and circuit analysis of LLMs. It deserves a serious referee. Bottom line: yes, send it to review; the method is worth publishing, and the mechanism will be credible once the budget confound is addressed.","headline":"Worth engaging: the cross-task gradient trace is a real contribution, but the mixed-vs-two-stage comparison is confounded by training budget and QA exposure, and the abstract overclaims what is measured.","tokens_in":21531,"tokens_out":3381,"would_cite":true,"duration_ms":32263,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shared parameters—those updated by both fact-storing and fact-recalling examples—explain why mixed training beats two-stage training on unseen questions.","keywords":["fact recall","mixed training","two-stage training","shared parameters","cross-task gradient trace","gradient attribution","attention heads","mechanistic interpretability"],"falsifier":"Train a two-stage model with the same total number of gradient steps and the same number of QA examples as the mixed model; if the gap on unseen questions persists even when the shared-parameter sets have the same size, the gradient-trace explanation fails. Alternatively, ablate the top-k single-task parameters by summed update magnitude; if that ablation reproduces the shared-parameter ablation drop, the intersection is not carrying the causal weight.","tokens_in":20465,"feed_emoji":"🧠","tokens_out":10157,"duration_ms":83468,"temperature":0.7,"pith_summary":"The paper asks why the same factual content behaves differently depending on how it is fine-tuned: training first on biographical statements (BIO) and then on question–answer pairs (QA) produces rote memorization, while interleaving the two formats produces recall that generalizes to unseen questions. Its central claim is that mixed training creates a larger and more central set of shared parameters—parameters strongly influenced by both the BIO and QA tasks—and that these parameters establish a format-invariant retrieval process. The evidence comes from a new cross-task gradient trace applied to two 2.8–3B parameter language models, with ablation and grafting experiments showing that the shared set, not random or single-task parameters, carries the ability to answer out-of-distribution questions. The paper further finds that shared parameters concentrate in a small number of critical attention heads, where they act as switches that activate fact recall. If this is right, the choice between memorization and knowledge is not about more data but about whether the optimizer jointly shapes the same parameters for storage and retrieval.","feed_headline":"Mixed training makes fact recall generalize via shared parameters","feed_subtitle":"Interleaved training builds more shared parameters in critical attention heads, enabling recall on unseen questions.","key_machinery":"The central object is the cross-task gradient trace, a measure of per-parameter influence defined by summing $lr(d_n) \\times grad^{(n)}_i$ over examples of each task and ranking parameters for BIO and QA separately. A parameter is shared if it appears in both top-k sets, and the paper studies the union of two such intersections, one using BIO in-distribution individuals and one using BIO out-of-distribution individuals. This object carries the entire argument: the paper uses it to define the set whose ablation destroys generalization, to show that this set is over-represented in grafting-recovered fact recall parameters, and to prioritize attention heads for circuit intervention. The supporting machinery is grafting with a relaxed binary mask, which locates the sparse set of parameters sufficient to recover the fine-tuned model's recall, allowing the paper to test whether shared parameters are functionally central rather than merely updated.","core_discovery":"The paper's discovery is that the memorization-versus-knowledge gap between two-stage and mixed training is explained by the parameters that both tasks influence together. Using the cross-task gradient trace, which ranks parameters by summed learning-rate-weighted gradients from BIO and QA examples and takes the intersection of the top-k sets, the paper finds that mixed training yields a shared parameter set about 1.2 times larger in Llama and 1.5 times larger in Pythia at $k = 10^8$. Ablating these shared parameters drops the mixed-trained Llama's out-of-distribution QA accuracy from 74.7 to 8.7, whereas the stage-trained model drops only from 35.9 to 29.2; ablating the same number of random parameters has almost no effect. Grafting experiments show that shared parameters are over-represented in the sparse subsets that recover fact recall, and attention-head interventions show that over 60% of shared parameters are concentrated in the top 10% of heads, whose ablation removes over 90% of recall accuracy. The paper concludes that mixed training teaches knowledge because it produces more numerous and more centrally organized shared parameters, concentrated in critical attention heads that support reusable, format-invariant retrieval.","pith_inferences":["A natural extension the authors leave implicit is that the same gradient-consistency mechanism should show up whenever two task formats share the same underlying knowledge, so the trace could be used to predict which paired formats will benefit from mixing.","If the causal story is right, constraining fine-tuning to the shared-parameter subspace—through masking or low-rank adapters—should reproduce mixed-training generalization with fewer update steps; the paper does not test this.","This account suggests that some failures on paraphrased or reversed queries may be training-artifact effects rather than fixed model limits, which could be checked by applying mixed training to real knowledge bases with paraphrased QA.","A caveat I would add: the trace is estimated from only 20 individuals, so its reliability on larger, noisier datasets is an open question that the paper also acknowledges."],"forward_implications":["Training pipelines that separate fact storage from query-format adaptation will systematically produce fewer shared parameters, so knowledge injection should interleave statement and question–answer examples.","Shared parameters are the causal lever behind the generalization gap: resetting them to pretrained values is enough to cancel a mixed-trained model's advantage on unseen questions.","Because shared parameters concentrate in a small fraction of attention heads, monitoring this subset can predict whether a model has learned facts in a retrievable form.","The cross-task gradient trace is model-agnostic, so the same analysis can be applied to other paired storage/retrieval formats without retraining the model."],"supporting_citations":[{"why":"It supplies the original two-stage versus mixed training result, the performance gap, and the synthetic fact dataset recipe that this paper builds on.","marker":"[3]"},{"why":"It provides the gradient-attribution foundation used by the cross-task gradient trace to quantify parameter influence.","marker":"[5]"},{"why":"It contributes the grafting method with a relaxed binary mask used to locate fact recall-related parameters.","marker":"[42]"},{"why":"It identifies attention heads and MLP blocks involved in factual association recall, motivating the focus on critical components.","marker":"[19]"},{"why":"It provides the knowledge-circuit framing and mover/extract head findings that the intervention analysis builds on.","marker":"[49]"},{"why":"It shows that fact recall generalization correlates with how facts are stored, grounding the paper's storage-retrieval analysis.","marker":"[20]"}],"fun_headline_variants":["Mixed training builds shared parameters for fact recall","Why mixed training generalizes: shared attention heads","Shared parameters explain knowledge vs memorization","Fact recall improves when training shares parameters","Two-stage memorizes, mixed teaches via shared parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cross-task gradient trace, computed from 20 individuals' gradients, identifies the parameters that actually cause fact recall rather than merely the parameters with the largest update magnitudes.","fun_headline_variants_meta":{"raw":{"variants":["Mixed training builds shared parameters for fact recall","Why mixed training generalizes: shared attention heads","Shared parameters explain knowledge vs memorization","Fact recall improves when training shares parameters","Two-stage memorizes, mixed teaches via shared parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001073,"raw_usage":{"total_tokens":4532,"prompt_tokens":1020,"completion_tokens":3512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":3445}},"tokens_in":636,"tokens_out":3512,"duration_ms":24550,"temperature":1.0,"reasoning_tokens":3445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:06:15.444269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-stage model with the same total number of gradient steps and the same number of QA examples as the mixed model; if the gap on unseen questions persists even when the shared-parameter sets have the same size, the gradient-trace explanation fails. Alternatively, ablate the top-k single-task parameters by summed update magnitude; if that ablation reproduces the shared-parameter ablation drop, the intersection is not carrying the causal weight.","supporting_citations":[{"cited_title":"Physics of language models: Part 3.1, knowledge storage and extraction","cited_arxiv_id":null,"evidence_quote":"It supplies the original two-stage versus mixed training result, the performance gap, and the synthetic fact dataset recipe that this paper builds on."},{"cited_title":"Towards better understanding of gradient-based attribution methods for deep neural networks","cited_arxiv_id":null,"evidence_quote":"It provides the gradient-attribution foundation used by the cross-task gradient trace to quantify parameter influence."},{"cited_title":"Task-specific skill localization in fine-tuned language models","cited_arxiv_id":null,"evidence_quote":"It contributes the grafting method with a relaxed binary mask used to locate fact recall-related parameters."},{"cited_title":"Dissecting recall of factual associations in auto-regressive language models","cited_arxiv_id":null,"evidence_quote":"It identifies attention heads and MLP blocks involved in factual association recall, motivating the focus on critical components."},{"cited_title":"Knowledge circuits in pretrained transformers","cited_arxiv_id":null,"evidence_quote":"It provides the knowledge-circuit framing and mover/extract head findings that the intervention analysis builds on."},{"cited_title":"Understanding finetuning for factual knowledge extraction","cited_arxiv_id":null,"evidence_quote":"It shows that fact recall generalization correlates with how facts are stored, grounding the paper's storage-retrieval analysis."}],"review_version":1}