{"id":"d615797b-035a-4b9f-935d-e404ba69bb4c","arxiv_id":"2602.23197","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In a linear-attention model, full fine-tuning toward zero-shot loss eliminates the query-key mechanism that enables in-context learning, while value-only fine-tuning preserves it; an auxiliary few-shot loss trades out-of-distribution few-shot accuracy for target-task accuracy.","lead":"This paper analyzes a simplified linear-attention model and shows mathematically that fine-tuning all attention weights to improve zero-shot accuracy destroys the model's ability to learn from few-shot examples, while updating only the value matrix keeps few-shot ability. The result gives a concrete mechanism for a widely observed LLM trade-off and suggests which parameters to fine-tune.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global-minimizer assumption underdetermines the prediction: Theorem 4.3 leaves w (and other flat directions) free, and Fig. 4(b) fits w=0.52 post hoc, so the theory does not uniquely predict the ICL loss from full fine-tuning.","rationale":"The paper's strongest theoretical insight—that full fine-tuning to a zero-shot global optimum forces Q11=0 and thereby removes the query-key pathway—is internally coherent and appears correct for the constructed family. The proof in A.8 correctly verifies that the family in (9) attains the irreducible error σ², and the qualitative claim that few-shot performance is worse than zero-shot holds for all w in that family. However, the load-bearing step for the paper's practical message is the jump from 'there exists a bad global minimizer' to 'fine-tuning will reach such a minimizer.' The objective has flat directions: w is not identified by the zero-shot loss, and in the full-FT case other blocks (v22, V11, v12, q12, q21) are similarly free. Corollary 4.4's finite-n curves depend on w, so Figure 4(b) is not a parameter-free prediction; w=0.52 is chosen to match the trained model. The same underdetermination affects Theorem 4.6, where value-matrix FT's zero-shot loss does not select w; Proposition 4.7 invokes a minimal-update convention, which is reasonable but is an extra assumption, not a consequence of the objective. This does not warrant rejection: the theoretical statements are conditional and the paper's own Limitations section is appropriately transparent about the linear-attention setting. But it does justify the reader's CONDITIONAL verdict. My concrete test would settle whether the empirical match is genuine or post hoc: if gradient flow from the stated pretrained initialization naturally converges to Q11=0 and to w≈0.52, the concern is resolved; if not, the paper's quantitative predictions are not derived.","tokens_in":41099,"tokens_out":18891,"duration_ms":179603,"concrete_test":"Re-run the Section 5.1 full-FT experiment (d=5, σ²=0.1, Σ=I, m=n=20) from the same pretrained init using full-batch gradient descent, and log Q11(t), q0(t), v21(t), and v22(t). Compare the converged implied w (e.g., w = v22/q0 or the scale of v21 relative to θ0) to w=0.52. Then repeat with L2 weight decay λ∈{1e-4,1e-3} on all fine-tuned parameters. If the converged Q11 is not numerically zero in the unregularized run, or if the implied w deviates from 0.52 beyond the fitting tolerance, the Figure 4(b) match is post hoc; if Q11 remains nonzero for any λ>0, the practical applicability of Theorem 4.3 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result is conditional on fine-tuning actually reaching a global minimizer of the unregularized zero-shot loss, and even that condition is underdetermined. Theorem 4.3 (proof A.8) shows only that the displayed family attains the Bayes lower bound σ²; it does not prove every global minimizer has the form (9), nor does it analyze which member of the family gradient flow from the pretrained initialization selects. The zero-shot objective leaves w free (and in the full-FT case also v22, V11, v12, q12, q21), and Corollary 4.4's finite-n few-shot error depends on w (Figure 2). The empirical match in Figure 4(b) is obtained by taking w=0.52; the paper gives no derivation of this value from the pretrained init or optimizer. The same issue is sharper for value-matrix FT: Theorem 4.6's minimizer set contains arbitrary V11, v12, and w; Proposition 4.7 selects w by a minimal-update heuristic that is not part of the fine-tuning objective. If real fine-tuning (SGD with early stopping, weight decay, or LoRA) stops at a point with Q11≠0 or with a different w, the predicted ICL degradation—and the claimed preservation under value-only updates—need not occur. The toy experiments make the global-minimizer assumption plausible but do not test it because the free w is fit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a linear attention model for linear regression, with pretraining on a task distribution followed by fine-tuning on a target task. It derives closed-form expressions for global minimizers of the zero-shot fine-tuning loss under three regimes: full fine-tuning, value-matrix fine-tuning, and value-matrix fine-tuning with an auxiliary few-shot loss. The central qualitative claim is that full fine-tuning can destroy the query-key mechanism (Q11=0 at the zero-shot optimum), causing few-shot error to exceed zero-shot error even on the target task, while value-matrix fine-tuning preserves few-shot performance. The theoretical results are compared with linear-regression experiments and with LoRA fine-tuning of Qwen2.5-3B on MMLU.","tokens_in":41356,"tokens_out":11884,"duration_ms":112804,"significance":"The paper gives a clean, detailed mathematical analysis of a practically important phenomenon: task-specific fine-tuning can degrade in-context learning. The moment computations (Isserlis/Wishart identities) are careful, and the core mechanism—that the zero-shot optimum forces Q11=0, removing the preconditioned-gradient-descent mechanism for ICL—is simple and convincing. The value-matrix fine-tuning result is a useful, non-obvious qualitative prediction. If the gaps identified below are addressed, the paper would be a solid theoretical contribution. The MMLU experiments, while small-scale and using LoRA rather than full fine-tuning, provide suggestive support.","major_comments":[{"comment":"The theorem shows only that the displayed family (9) attains the Bayes error σ²; it does not characterize all global minimizers of the zero-shot loss. The section's wording ('characterize the optimal parameters') and the conclusion of §4.2 imply that full fine-tuning generically degrades ICL. But the zero-shot objective is invariant to several blocks (V11, v12, v22, q12, q21, q22) and to the rescaling q·v21=θ0, so other global minimizers outside family (9) exist and may have different few-shot behavior. The proof in Appendix A.8 does not analyze these directions or which member gradient flow from the pretrained initialization would select. Please either characterize the full set of global minimizers and prove the few-shot property for all of them, or explicitly restrict the claim to the displayed family and to 'can' rather than 'does'.","section":"§4.2, Theorem 4.3"},{"comment":"The theoretical curve for full fine-tuning uses w=0.52, a free parameter in the optimum family of Theorem 4.3. No procedure is given for deriving this value from the pretrained initialization or the optimizer, so the empirical agreement is not a parameter-free validation of the theory. The same point affects the quantitative prediction of the ICL loss after full fine-tuning. The authors should either derive w from the initialization (e.g., by analyzing gradient flow in the flat directions of the zero-shot loss) or report w as a fitted parameter and discuss the sensitivity of the comparison.","section":"§5.1, Figure 4(b)"},{"comment":"The claim that value-matrix fine-tuning preserves ICL depends on selecting w by a minimal-Frobenius-distance rule that is not part of the zero-shot fine-tuning objective. Theorem 4.6 leaves w free; Proposition 4.7 is a heuristic. Corollary 4.8 shows that the minimal-update w approximates the task-averaged optimal w* when m=n and n+d is large, but this is an external justification. A more natural and direct argument is available: because the zero-shot loss is independent of v22, gradient flow from the pretrained v22 keeps it at m/(m+1+d), so the minimal-update choice is exactly what an unregularized gradient method would produce. Please make this explicit or otherwise justify the selection; without it, the 'preserves ICL' conclusion is conditional on a heuristic.","section":"§4.3, Proposition 4.7"},{"comment":"All main theorems concern global minimizers of the expected population loss, with no regularization, early stopping, or optimization noise. Real fine-tuning—including the paper's own MMLU experiments, which use LoRA and select the best zero-shot checkpoint—will generally stop at a point that is not a global minimizer of the unregularized objective. The Limitations section acknowledges this in general terms, but the main-text conclusions (abstract, Table 1, §6) are easy to misread as statements about actual fine-tuning dynamics. Please state the exact idealization (population loss, global optimum, no regularization) in the statements of the main results and soften the causal wording where appropriate.","section":"General assumptions, §4 and §5"}],"minor_comments":[{"comment":"Please explain in the main text how w=0.52 is chosen in panel (b); the caption merely states the value.","section":"Figure 4 caption"},{"comment":"Consider adding a sentence that (9) is a subfamily of the zero-shot optima, not necessarily the full set, to avoid overclaiming.","section":"§4.2, family (9)"},{"comment":"The MMLU experiments cite Achiam et al. (2023) for gpt-4.1-mini; this reference is the GPT-4 technical report and likely does not describe gpt-4.1-mini. Please use the appropriate model documentation.","section":"References"},{"comment":"The notation 'ZS fine-tuning on Q/K/V' is a bit cryptic when first used; a one-line definition would improve readability.","section":"Table 2 / §5.2"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism is interesting and the derivations appear sound, but the paper currently overclaims the universality of the full fine-tuning result: Theorem 4.3 only exhibits a family of optima, and the free parameter w in Figure 4(b) is fitted post hoc. These issues are fixable within the manuscript's scope—e.g., by analyzing the flat directions/gradient flow or by explicitly framing the results as existential and parameter-free only in the qualitative direction. I would support acceptance after the authors address the characterization and the empirical fitting issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful theory paper, and the central result is not derailed by the free-parameter issue the stress-test note flags. The qualitative prediction — full fine-tuning can kill in-context learning in linear attention, value-matrix fine-tuning preserves it — holds across the whole family of optima. Send it to review.\n\nWhat's new: Ahn et al. (2023) gave the pretrained solution; this paper goes further and characterizes what task-specific fine-tuning does. The clean observation is that minimizing the zero-shot loss on a fixed task admits global minimizers with Q11=0, which by construction dismantles the query-key pathway that enables ICL. Value-matrix-only fine-tuning keeps Q11 intact and pays only a O(1/d) zero-shot penalty, and the auxiliary few-shot loss trades off target-task few-shot gains against OOD degradation with explicit excess-error formulas. The proofs are detailed, use standard Wishart/Isserlis machinery, and look checkable.\n\nSoft spots: Figure 4(b) matches the full-fine-tune theory using w=0.52, but the theory says w is free. Nothing derives that value from the initialization or the optimizer, so the quantitative agreement is partly a fit. The qualitative conclusion is robust — few-shot error degrades for every w — but the empirical validation would be much stronger with a principled tie-breaking rule or a sensitivity plot. The same issue appears in value-matrix fine-tuning (V11, v12, and w free at the optimum); Proposition 4.7's minimal-update choice is reasonable but heuristic, and the preservation claim holds across the family asymptotically, so the mechanism is not threatened. The MMLU section supports 'degrades much less' better than 'preserves': value-matrix LoRA still drops 7-shot accuracy by a couple of points, just far less than full FT. No code or configs are released, which matters for a paper claiming empirical validation. The global-minimizer assumption is a limitation but a standard one in this literature, and the qualitative story is insensitive to it.\n\nBottom line: the paper deserves a serious referee. Clarify how w is chosen, report sensitivity, release code, and soften 'preserves' — then it's a clear accept.","headline":"Solid theory paper: the full-FT kills ICL result is robust to the free-parameter w, but Fig 4(b) fits w=0.52 without derivation — still deserves peer review.","tokens_in":41947,"tokens_out":2887,"would_cite":true,"duration_ms":32070,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning all attention parameters to minimize zero-shot loss collapses the query–key mechanism that powers in-context learning; fine-tuning only the value matrix avoids this collapse.","keywords":["in-context learning","fine-tuning","linear attention","linear regression","zero-shot vs few-shot","query-key matrix","value-matrix fine-tuning","fine-tuning without forgetting"],"falsifier":"Train the paper's linear attention model (d=5, Σ=I₅, σ²=0.1, m=n=20) and fully fine-tune on a fixed θ0 but with L2 regularization or early stopping before the zero-shot loss reaches its global minimum. If the resulting model keeps n-shot error below zero-shot error on θ0 for large n, or keeps Q11 noticeably nonzero, the claim that every zero-shot-optimal full fine-tune destroys in-context learning is refuted. A second check on a real language model: fine-tune with adapters restricted to value projections and compare 7-shot accuracy on a held-out category; the paper predicts near-zero degradati","tokens_in":40881,"feed_emoji":"🧠","tokens_out":8925,"duration_ms":72372,"temperature":0.7,"pith_summary":"This paper argues that, in a tractable linear-attention model of linear regression, the way you fine-tune decides whether a pretrained model keeps its in-context learning ability. Its central result is that fine-tuning all attention parameters to minimize zero-shot loss destroys the query–key block that makes few-shot prompting work, so the resulting model's few-shot predictions are strictly worse than its own zero-shot predictions, even on the exact task it was fine-tuned on. The paper also proves a safer route: freeze the query–key block and update only the value matrix; zero-shot error approaches the irreducible noise level as the input dimension grows, while few-shot error stays close to the pretrained level. Finally, adding an auxiliary few-shot loss during value-matrix fine-tuning improves few-shot accuracy on the fine-tuning task but degrades it on dissimilar tasks, with the excess error growing with task dissimilarity. If correct, these results give a parameter-level explanation for the observed fine-tuning/ICL trade-off and a concrete prescription: preserve the query–key channel to keep few-shot ability.","feed_headline":"Full fine-tuning erases few-shot ability, even on the target task","feed_subtitle":"Theory shows fine-tuning all attention weights kills the query-key mechanism; freezing it preserves few-shot learning.","key_machinery":"The merged query–key matrix Q11 and the value row v21 inside the linear attention layer. The pretrained in-context learning mechanism is the condition Q11=Σ^{-1}: it is what makes the n examples in a prompt act like a step of preconditioned gradient descent on the task. Full fine-tuning's global minimizers (Theorem 4.3) set Q11=0 and v21=wθ0, so the predictor becomes a fixed zero-shot estimator plus a negligible prompt term—this is the 'erase' step. Value-matrix fine-tuning freezes Q11 at Σ^{-1} and only sets v21=θ0/(d+4), keeping the in-context channel open. The remaining scalar w parameterizes all zero-shot-optimal value matrices; the paper resolves it by minimal Frobenius update from the","core_discovery":"Central claim: in-context learning lives in one parameter block, and fine-tuning either preserves or destroys it. In the linear attention model, pretraining gives Q11≈Σ^{-1}, the query–key block that makes demonstrations work like a gradient step. Full fine-tuning's global minimizers set Q11=0 and the value row to wθ0, so prompts stop mattering: few-shot error exceeds zero-shot error for every n and tends to σ²+θ0^TΣθ0. Value-matrix fine-tuning keeps Q11=Σ^{-1} and sets the value row to θ0/(d+4), giving zero-shot error σ²+2/(d+4)θ0^TΣθ0 and asymptotic few-shot error σ²+1/(d+4)^2 θ0^TΣθ0. An auxiliary few-shot loss tunes a residual parameter w toward the target task, improving its few-shot er","pith_inferences":["Inference: if the same collapse happens with softmax attention, parameter-efficient methods that leave query/key projections untouched (value-only adapters) should be a safer default than methods that also alter query/key projections; this is directly testable with controlled comparisons.","Inference: the global-minimizer analysis implies that regularization, early stopping, or optimization noise that keeps Q11 away from zero may retain some in-context learning; Q11's distance from Σ^{-1} after fine-tuning could serve as a diagnostic for few-shot degradation.","Inference: the paper's annealing observation suggests a practical rule—schedule any auxiliary few-shot loss to zero during fine-tuning—to avoid the zero-shot penalty a fixed combination would incur.","Inference: in high-dimensional settings the 2/(d+4) zero-shot penalty of value-matrix fine-tuning becomes negligible, so the cost of preserving in-context learning shrinks for wide models; value-only fine-tuning is most attractive there."],"forward_implications":["Full fine-tuning on a target task can make the fine-tuned model's few-shot performance worse than its zero-shot performance, even on that same task (Corollary 4.4).","Freezing the query–key side and fine-tuning only the value matrix keeps few-shot error near the pretrained level while zero-shot error drops to σ²+2/(d+4)θ0^TΣθ0, which approaches the noise floor as d grows (Theorem 4.6, Corollary 4.8).","Adding an auxiliary few-shot loss to value-matrix fine-tuning further reduces few-shot error on the fine-tuning task but increases few-shot error on other tasks, by an amount proportional to (1−ρ)² where ρ is the Σ-inner-product cosine similarity between tasks (Proposition 4.10).","The free parameter w can be chosen at test time to trade off zero-shot and few-shot behavior; the minimal-update choice w=m/(m+1+d) approximates the task-averaged few-shot optimum (Proposition 4.7, Corollary 4.8).","The degradation from full fine-tuning is structural, not a small-sample artifact: even with infinitely many shots, few-shot error converges to σ²+θ0^TΣθ0, strictly above the zero-shot error σ² (Corollary 4.4)."],"fun_headline_variants":["Full fine-tuning erases few-shot learning, even on target","Value-matrix tuning preserves few-shot, improves zero-shot","Fine-tune all? Few-shot dead, zero-shot no better","Freeze QK block to keep in-context learning intact","Auxiliary few-shot loss boosts target but kills transfer"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The theory assumes fine-tuning reaches a global minimizer of the expected zero-shot loss, with no regularization, early stopping, or optimization noise; if real optimization stops short of that global optimum, the predicted collapse of in-context learning need not occur.","fun_headline_variants_meta":{"raw":{"variants":["Full fine-tuning erases few-shot learning, even on target","Value-matrix tuning preserves few-shot, improves zero-shot","Fine-tune all? Few-shot dead, zero-shot no better","Freeze QK block to keep in-context learning intact","Auxiliary few-shot loss boosts target but kills transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1304,"prompt_tokens":760,"completion_tokens":544,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":459}},"tokens_in":504,"tokens_out":544,"duration_ms":5296,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:25:47.836896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the paper's linear attention model (d=5, Σ=I₅, σ²=0.1, m=n=20) and fully fine-tune on a fixed θ0 but with L2 regularization or early stopping before the zero-shot loss reaches its global minimum. If the resulting model keeps n-shot error below zero-shot error on θ0 for large n, or keeps Q11 noticeably nonzero, the claim that every zero-shot-optimal full fine-tune destroys in-context learning is refuted. A second check on a real language model: fine-tune with adapters restricted to value projections and compare 7-shot accuracy on a held-out category; the paper predicts near-zero degradati","supporting_citations":[],"review_version":1}