{"id":"d1350060-d57b-42a5-8ca4-8e93a14ef4ae","arxiv_id":"2602.20062","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In diagonal linear networks, fine-tuning generalization is governed by a tunable per-dimension penalty whose sparsity and pretraining dependence define four regimes and a trade-off between feature reuse and new-feature learning.","lead":"This paper derives an analytical theory for how initialization scale and task overlap control feature reuse versus new-feature learning during pretraining-fine-tuning in a simplified linear network. It identifies four learning regimes and shows that the relative scale across layers is a practically relevant lever for fine-tuning generalization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1's k_d formula conflicts with its own proof: under k_d=4c_FT^2, the γ_FT term should enter as 2γ_FT^2, not γ_FT^2; all downstream γ_FT predictions inherit this factor-of-2 discrepancy.","rationale":"The reader's weakest_assumption focused on the non-rigorous replica method and the α_PT≥1 restriction. Those are legitimate limitations, but I find a more direct and more load-bearing problem: the paper's own proof of Theorem 4.1 does not yield the stated formula. The derivation in B.1 computes c_FT = c_PT(1+λ_PT)(1+√(1+(β_PT/c_PT)^2)) + γ_FT^2 and then states k_d = 4c_FT^2 = (2c_PT(1+λ_PT)(1+√(...)) + γ_FT^2)^2. Simple algebra gives (2X + 2γ_FT^2)^2, not (2X + γ_FT^2)^2. This is not a convention difference: the paper defines v±_FT(0)=γ_FT in Eq. (5), so the v-v product is γ_FT^2. The gamma coefficient in k_d is off by a factor of two unless an unstated redefinition is assumed. Because k_d is the input to every phase-portrait calculation and every replica fixed-point curve for γ_FT, the paper's quantitative claims about readout initialization are built on an internally inconsistent formula. This is a stronger concern than the replica non-rigor: even before trusting the replica step, the object being fed into it is derived incorrectly as written. I still do not think this warrants outright rejection: the qualitative story — negative λ_PT moves toward a rich, pretraining-dependent regime and γ_FT pushes toward lazy independence — may survive a corrected factor, and the error is localized and fixable. It does, however, reinforce the conditional verdict: the exact expressions need to be re-derived and re-checked, and the γ_FT figures should be regenerated with the corrected formula. I therefore keep the reader's CONDITIONAL verdict unchanged, while flagging a different root concern. Secondary issues such as the abstract mismatching Section 6 and the missing code remain, but they are less central than the algebraic inconsistency.","tokens_in":25097,"tokens_out":15109,"duration_ms":115603,"concrete_test":"Evaluate the inconsistency at a single point: set c_PT=1, λ_PT=0, β_PT=0, γ_FT=1. Appendix B.1 gives c_FT = 1*(1+0)*(1+√1) + 1 = 3, so k_d=4c_FT^2=36 (√k_d=6). Theorem 4.1 Eq. (14) gives k_d=(2*1*1*(1+√1)+1)^2=(4+1)^2=25 (√k_d=5). These differ by a factor 1.44 in k. The decisive check is to recompute the Fig. 3 γ_FT=1 replica curve (and the γ_FT=10 curve) using the k_d that follows from B.1. If the predicted generalization error shifts by more than the reported Monte Carlo error bars, the exact-formula claim is unsupported; if it does not, the discrepancy is numerically negligible and the paper should still state which convention is correct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central implicit-bias formula, Theorem 4.1 Eq. (14), states k_d = (2c_PT(1+λ_PT)(1+√(1+(β̂_PT,d/c_PT)^2)) + γ_FT^2)^2. However, Appendix B.1 derives c_FT = c_PT(1+λ_PT)(1+√(1+(β_PT/c_PT)^2)) + γ_FT^2 and explicitly uses k_d = 4c_FT^2. Substitution gives k_d = (2c_PT(1+λ_PT)(1+√(...)) + 2γ_FT^2)^2. The readout reinitialization scale γ_FT enters with an extra factor of 2 in the very quantity that defines the penalty Q_k. This is an internal algebraic inconsistency, not a matter of replica rigor or asymptotic ansatz. Since Section 5.2's phase portraits and Section 5.3's generalization curves (Fig. 3) all use this k_d, every γ_FT-dependent prediction in the claimed exact theory is computed from a formula that does not follow from the paper's own proof. If the theorem is intended to be read with v±_FT(0)=γ_FT/√2, that convention is never stated and contradicts the setup in Eq. (5). If the proof is intended, the theorem is unproven as written. Either way, the central claim that the paper derives exact generalization expressions as a function of γ_FT is not currently supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops an analytical theory of pretraining followed by fine-tuning (PT+FT) in diagonal linear networks. Theorem 4.1 characterizes the implicit bias of fine-tuning as a coordinate-wise minimization with penalty Q_k, whose coefficient k_d is claimed to depend on the pretraining initialization (c_PT, λ_PT) and the fine-tuning readout reinitialization scale γ_FT. Proposition 4.2 then uses replica theory to turn this implicit bias into fixed-point equations for the generalization error. The paper identifies four learning regimes—rich/pretraining-independent, lazy/pretraining-dependent, lazy/pretraining-independent, and an intermediate rich/pretraining-dependent regime—and argues that a smaller early-layer scale (negative λ_PT) places the network in the intermediate regime, improving generalization when fine-tuning relies on a subset of pretrained features. The theoretical predictions are compared with simulations on diagonal networks and with ResNet experiments on CIFAR-100.","tokens_in":25446,"tokens_out":10578,"duration_ms":95827,"significance":"If the central derivation were correct, this would be a useful and fairly complete treatment of an important problem. It extends prior work by Lippl & Lindsey from the infinitesimal-scale limit to a full two-parameter initialization plane, and it is one of the few papers to connect an implicit-bias characterization of PT+FT to replica-theoretic generalization curves. The paper is also commendably transparent about the numerical replica solver: Appendix E documents damping, forward/backward continuation, Monte Carlo error bars, and branch-selection diagnostics, and the diagonal-network simulations are compared with the predictions without fitting constants. The four-regime taxonomy and the proposal that relative layer scale controls feature reuse versus refinement are conceptually appealing. However, the manuscript contains internal algebraic inconsistencies in the equations that drive all downstream quantitative claims. These must be resolved before the paper can be judged on its merits.","major_comments":[{"comment":"Theorem 4.1 as stated does not follow from the proof. Appendix B.1 derives c_FT=(λ̃_PT+c_PT)(1+√(1+(β̂_PT,d/c_PT)^2))+γ_FT^2 and then invokes k_d=4c_FT^2. Substitution gives k_d=[2c_PT(1+λ_PT)(1+√(1+(β̂_PT,d/c_PT)^2))+2γ_FT^2]^2, not the expression in Eq. (14), where the γ_FT^2 term appears without the factor of 2. The only way to reconcile Eq. (14) with k=4c^2 would be to set v_FT(0)=γ_FT/√2, which contradicts Eq. (5). Since Eq. (14) is used for the phase portraits (Fig. 2b) and all replica generalization curves (Fig. 3), every γ_FT-dependent quantitative prediction in the paper would need to be recomputed. This is not a presentational issue but an internal algebraic inconsistency in the central theorem.","section":"Section 4.1, Eq. (14); Appendix B.1"},{"comment":"The fixed-point equation for χ is written with a prefactor θ in Eq. (20), but with prefactor θ_0 in Appendix C, Eq. (39), and in the numerical solver described in Appendix E.2.1. These are not equivalent under the closure relations θ=(χ+λ)/α and θ_0=(σ_0^2+p)/α. The reader cannot determine whether Fig. 3 was generated with the equation stated in the main text or with the equation stated in the appendix. Please reconcile the displayed theorem with the derivation and with the code, and state which form was used for the reported curves.","section":"Section 4.2, Eq. (20); Appendix C, Eq. (39); Appendix E.2.1"},{"comment":"The abstract and Section 4 describe the generalization error as 'exact', but Proposition 4.2 is conditional on the replica-symmetric ansatz, which the paper itself calls non-rigorous (Section 2 and Appendix C.1), and on the assumption α_PT≥1, i.e. perfect pretraining recovery. The Appendix also lists explicit reservations about the ResNet mapping. These qualifications should be reflected in the abstract and in any statement of exactness; as written, the claims overstate what is proved. This is partly a framing issue, but it is load-bearing because the paper's central advertised contribution is an 'exact expression for the generalization error'.","section":"Abstract and Section 4; Appendix D.5"}],"minor_comments":[{"comment":"The abstract claims experiments on ResNets trained on CIFAR-100 and SVHN as well as Transformers trained on modular arithmetic tasks, but the full text contains only ResNet experiments on CIFAR-100. Either the missing experiments should be added or the abstract should be corrected.","section":"Abstract (arXiv metadata)"},{"comment":"The text appears to swap the roles of Fig. 4a and Fig. 4b: the text says 'In Fig. 4a, we observe that a ResNet initialized with a non-standard small value of κ', but the caption assigns κ/λ_PT to Fig. 4b and cP_T to Fig. 4a. Please align the text with the figure.","section":"Section 6 and Figure 4"},{"comment":"There are typographical errors ('discribed') and inconsistent use of symbols (c_FT vs. c_PT in D.4) that should be cleaned up.","section":"Appendix D.3, D.4"},{"comment":"The phase portrait in Fig. 2b is evaluated at a single representative point β_FT,d=1/√ρ_FT. Since ℓ-order and PD depend on β_FT,d, the portrait may depend on this choice. A brief sensitivity analysis or an explicit statement of this limitation would help.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The algebraic discrepancy in Theorem 4.1 and the θ/θ_0 mismatch in Proposition 4.2 are the key obstacles; both are fixable in principle but require recomputing most quantitative figures. The abstract also advertises experiments that are not in the manuscript, so please check the submitted version carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Sam, here's my read of 2602.20062. The paper delivers a real extension: an analytic replica-theoretic characterization of PT+FT generalization in diagonal linear networks over a broad initialization range, the four-regime taxonomy (rich/lazy × pretraining-dependent/independent), and the identification of relative layer scale λ_PT as the lever that moves you between regimes. The diagonal-network simulations match the replica curves, and the derivation is honest about the non-rigorous replica assumption. The ResNet results are a useful sanity check, not a proof.\n\nNow the soft spots, in order of severity. First, the central theorem has an internal algebraic inconsistency. Theorem 4.1 states k_d = (2c_PT(1+λ_PT)(1+√(1+(β̂_PT,d/c_PT)^2)) + γ_FT²)^2. But Appendix B.1 derives c_FT = c_PT(1+λ_PT)(1+√(...)) + γ_FT² and explicitly uses k_d = 4c_FT². Substituting gives k_d = (2c_PT(...) + 2γ_FT²)^2. The γ_FT term is off by a factor of 2. This isn't a rigor complaint; it's basic algebra. It propagates into the κ formula and into every γ_FT-dependent curve in Fig. 3. Either the theorem has a typo or the proof does. The paper as written doesn't give exact expressions as claimed.\n\nSecond, the abstract in the arXiv metadata promises SVHN and Transformer experiments that the full text does not contain—only CIFAR-100 ResNets appear. That mismatch needs to be reconciled. Third, no code or commit hash is provided, though the appendix names scripts and gives enough detail to reimplement. Fourth, the replica results are asymptotic and unproven; the paper should state that more prominently than 'exact expressions.'\n\nThe qualitative story—negative λ_PT gives a rich pretraining-dependent regime that helps when fine-tuning relies on a subset of pretrained features—probably survives the factor-of-2 fix, because the limits are unchanged. But the quantitative claim of exactness does not.\n\nWho is this for? Anyone working on theory of transfer learning or implicit bias. It deserves a serious referee, not a desk reject. I'd send it to review with a request to fix the algebra, align the abstract, and release code. Once that's done, it's a solid contribution.","headline":"Strong extension of Lippl & Lindsey with a useful regime taxonomy, but the central k_d formula doesn't match its own proof—worth refereeing once that's fixed.","tokens_in":25930,"tokens_out":4808,"would_cite":false,"duration_ms":42773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining sets a per-feature penalty that decides whether fine-tuning reuses or rewrites features.","keywords":["pretraining","fine-tuning","inductive bias","diagonal linear networks","feature reuse","generalization error","replica method","initialization scale"],"falsifier":"Train a diagonal linear network at D=5000 with the paper's pretrain-then-finetune protocol, sweep α_FT and the three initialization parameters, and compare the measured generalization error to the fixed-point prediction of Proposition 4.2. Any systematic mismatch that does not shrink with D, or a measured (ℓ-order, PD) pair outside the claimed triangle ℓ-order∈[1,2], PD∈[-1,0], would refute the central claim.","tokens_in":24954,"feed_emoji":"🎛️","tokens_out":8785,"duration_ms":70067,"temperature":0.7,"pith_summary":"The paper claims that after pretraining, fine-tuning a diagonal linear network inherits a coordinate-wise penalty written by the pretraining stage. The size of the penalty on each coordinate is controlled by three initialization knobs — overall weight scale, relative scale between first and second layer, and readout re-initialization scale — together with whether pretraining activated that coordinate. From this penalty the authors prove that every initialization produces an inductive bias on a universal line segment: sparsity preference between ℓ1 and ℓ2, and pretraining dependence between zero and inverse proportionality to the pretrained feature. These extremes define four learning regimes, one of which is new. The paper's practical claim is that a smaller early-layer scale selects the intermediate regime that reuses pretrained features and still learns sparse new ones, which generalizes best when the fine-tuning task draws its features from a subset of pretrained features.","feed_headline":"Layer-scale balance sets fine-tuning's reuse-vs-relearn regime","feed_subtitle":"Pretraining writes a per-feature penalty; making early layers small yields the regime that best handles subset tasks.","key_machinery":"The central object is the coordinate-wise penalty Q_k(β)=Σ_d q_{k_d}(β_d), where q_k(z)=(√k/4)(1−√(1+4z²/k)+(2z/√k) arcsinh(2z/√k)). Each coordinate's stiffness k_d carries all of the pretraining information through the identity in Eq. (14); it interpolates between ℓ1-like and ℓ2-like behavior, with the two conserved quantities of the network (absolute scale c_PT and relative scale λ_PT) determining the interpolation. The logarithmic derivative of q_k with respect to |β_FT,d| defines the paper's ℓ-order (sparsity preference), and its logarithmic derivative with respect to |β_PT,d| defines pretraining dependence PD. The theorem works because c_PT and λ_PT are conserved during pretraining, so","core_discovery":"The paper's central claim, Theorem 4.1, is that gradient flow through pretraining and fine-tuning converges to the minimizer of a separable penalty Q_k(β)=Σ_d q_{k_d}(β_d) subject to interpolation on the fine-tuning set. The per-coordinate coefficient k_d depends on c_PT, λ_PT, γ_FT and on the pretrained function value β̂_PT,d, so the fine-tuning solution genuinely depends on what was learned. From this formula the paper derives universal ranges ℓ-order∈[1,2], pretraining dependence PD∈[-1,0], and ℓ-order+PD∈[1,2], four regimes including the new intermediate rich/pretraining-dependent one. The practical claim: making the early-layer initialization relatively small (negative λ_PT) while keepi","pith_inferences":["Inference: the theory implies an optimal λ_PT that moves with the overlap fraction ρ_shared/ρ_new; practitioners could tune only the early-layer scale and expect the crossovers shown in the paper's generalization curves, but the paper does not turn this into a tuning recipe.","Inference: because k_d depends on |β̂_PT,d/c_PT|, coordinates that were weakly represented during pretraining should behave like new coordinates even if technically shared; that predicts a smooth degradation in reuse benefit with pretraining signal strength, which the paper does not analyze because it assumes α_PT≥1.","Inference: the per-coordinate penalty view suggests that pretraining data quality and initialization scale are interchangeable to some degree — a poorer pretraining fit could be partially compensated by a more negative λ_PT — an exchange the paper leaves untested.","Inference: if the effect survives nonlinearity, it gives a principled reason to use depth-wise scaled initialization in transfer learning rather than global scaling, and suggests that the common choice to reinitialize only the final layer leaves the reuse/refinement balance unexplored."],"forward_implications":["Fine-tuning inherits an implicit regularization written by pretraining, so the same fine-tuning data can lead to different solutions — and different generalization — depending only on how the network was initialized before pretraining.","The relative scale between layers (λ_PT) is a genuine control knob: negative values push toward the rich pretraining-dependent regime, positive values toward the lazy pretraining-dependent regime.","The four regimes have distinct task preferences: full task overlap favors lazy reuse, no overlap favors learning new sparse features, and subset overlap favors the intermediate rich pretraining-dependent regime.","Very large absolute scale c_PT or readout scale γ_FT erases the benefit of pretraining, putting the network in a lazy pretraining-independent regime no matter what λ_PT was.","The same qualitative signatures appear in ResNets: down-scaling early blocks improves fine-tuning accuracy on CIFAR-100, up-scaling all weights hurts, and smaller readout re-initialization helps at intermediate sample sizes."],"fun_headline_variants":["Small early-layer scale unlocks fine-tuning feature reuse","Init scale sets four fine-tuning regimes, theory shows","Pretraining shapes fine-tuning: early-layer size matters","Layer-scale balance governs feature reuse in fine-tuning","Theoretical regimes for fine-tuning from initialization"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The quantitative predictions rest on the replica-symmetric calculation being exact in the high-dimensional limit — the paper itself flags the replica method as non-rigorous — and on pretraining perfectly recovering the pretraining teacher (α_PT≥1), leaving the finite-pretraining regime, where most real fine-tuning happens, outside the theory.","fun_headline_variants_meta":{"raw":{"variants":["Small early-layer scale unlocks fine-tuning feature reuse","Init scale sets four fine-tuning regimes, theory shows","Pretraining shapes fine-tuning: early-layer size matters","Layer-scale balance governs feature reuse in fine-tuning","Theoretical regimes for fine-tuning from initialization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000428,"raw_usage":{"total_tokens":2056,"prompt_tokens":801,"completion_tokens":1255,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1193}},"tokens_in":545,"tokens_out":1255,"duration_ms":8942,"temperature":1.0,"reasoning_tokens":1193,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:24:07.298118+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a diagonal linear network at D=5000 with the paper's pretrain-then-finetune protocol, sweep α_FT and the three initialization parameters, and compare the measured generalization error to the fixed-point prediction of Proposition 4.2. Any systematic mismatch that does not shrink with D, or a measured (ℓ-order, PD) pair outside the claimed triangle ℓ-order∈[1,2], PD∈[-1,0], would refute the central claim.","supporting_citations":[],"review_version":1}