{"id":"3ff99dfa-1d1d-4ce2-b495-04c39eabb532","arxiv_id":"2506.04288","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Selecting backbone pre-training data with an influence-based score and adding it to the adaptation set can improve LoRA and DreamBooth training when adaptation data are scarce.","lead":"This paper proposes Backbone Augmented Training (BAT), a method that adds selected backbone pre-training data to the adaptation dataset when fine-tuning large models with very few examples. If the method works, it could make personalization and language adaptation more reliable in data-scarce settings, but the theoretical and experimental support is incomplete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.2 depends on the unproved identity Hbat|A − Hbat = HA, which is false under the paper's own definitions, so the central guarantee ρbat|A ≤ ρA is not established.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing fault: the unproved identity Hbat|A − Hbat = HA in Supp. A.7. I reviewed the definitions in Sec. 3.1–3.2 and the statement of Proposition 4.2, and the identity is not merely unproved but generically false under the paper's own mixture construction. The combined empirical risk on Dbat|A = DB′ ∪ DA has a Hessian that is a weighted average of the backbone and adaptation Hessians; subtracting the backbone Hessian leaves a term proportional to HA − HB′, not HA. Since the right-hand side of the proposition's condition is meant to reproduce the ordinary adaptation error, the inequality in Proposition 4.2 does not establish the claimed comparison. This is a correctness risk in the central claim, not a disagreement with prior consensus. I did not find a separate, stronger objection; the Newton-step mismatch and experimental issues noted by the reader are secondary but compound the problem. Because the reader already recommended REJECT and my analysis supports that conclusion, the verdict is unchanged.","tokens_in":22212,"tokens_out":5473,"duration_ms":51724,"concrete_test":"Re-derive the Hessian algebra for a minimal linear model: let X_A and X_B be feature matrices for DA and DB′ with distinct empirical covariances Σ_A and Σ_B, loss ||y − Xθ||^2, and no regularizer. Compute Hbat|A = (nΣ_A + mΣ_B)/(n+m), Hbat = Σ_B, so Hbat|A − Hbat = n/(n+m)(Σ_A − Σ_B). Then substitute this into Eq. 25 of Supp. A.7 and check whether the right-hand side equals the standard adaptation term (HA)−1 Σ_DA ∇θLA. Symbolically verify whether the inequality is equivalent to ρbat|A ≤ ρA; if it is not, Proposition 4.2's condition is not sufficient. This check requires only a few lines of algebra, no training runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim, Proposition 4.2, asserts that if γ||(Hbat|A)−1 Σ_Dbat ∇θLbat|A|| ≤ ||(Hbat|A − Hbat)−1 Σ_DA ∇θLbat|A|| + oP(1), then BAT's asymptotic error coefficient is no worse than standard adaptation. The proof in Supp. A.7 inserts, without derivation, the identity Hbat|A − Hbat = HA, and then uses it to replace the right-hand side by the standard adaptation term. This identity does not follow from the preceding sentence that Lbat|A converges to LA*: convergence of risks does not imply equality of Hessians. Under the paper's own Def. 3.1, Rbat|A is an average over DA ∪ DB′, so for a twice-differentiable loss, Hbat|A = (|DA|/k)HA + (|DB′|/k)HB′ (up to the regularizer), whereas Hbat is the Hessian on DB′. Therefore Hbat|A − Hbat = (|DA|/k)(HA − HB′), which equals HA only in degenerate cases (e.g., HB′ = 0 and |DA|/k = 1). Since |DA|/k → γ ∈ (0,1), the identity is generically false under the paper's own assumptions. The condition in Proposition 4.2 therefore does not compare the BAT estimator with the ordinary adaptation estimator; the reduction that 'HA vanishes' and the subsequent O(D) complexity argument inherit this defect. Even if the Hessian identity were repaired, Eqs. 27–28 model the estimator as a single Newton step, and no argument connects these equations to the AdamW/SGD dynamics used in the experiments. The central improvement guarantee is thus unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Backbone Augmented Training (BAT), which augments a small adaptation dataset with selected samples from the backbone's pre-training data, and claims two theoretical results: Proposition 4.1 (validity: the BAT estimator's asymptotic error coefficient exists) and Proposition 4.2 (a condition under which BAT is no worse than standard adaptation). It then introduces ALBAT, a practical selection algorithm based on a biased data-selection score and a DataInf-style complexity reduction to O(nDL), and reports experiments on DreamBooth/LoCon for personalized image generation and on LLAMA-2/3 with LoRA/DoRA for commonsense reasoning. The main theoretical contribution is the claimed proof that, under a stated condition on the Hessian and gradients, BAT's asymptotic error coefficient is no larger than that of ordinary adaptation.","tokens_in":22707,"tokens_out":5252,"duration_ms":48867,"significance":"If the theoretical claims were correct, the paper would provide a principled justification for using backbone data in adaptation, a topic of practical interest. The paper also makes a useful engineering contribution by adapting influence-function-style data selection to large models with a claimed O(nDL) complexity, and it ships a substantial experimental study. However, the central theoretical result is not established: the proof of Proposition 4.2 relies on an unproved and generically false Hessian identity, and the proof of Proposition 4.1 uses an undefined auxiliary function. The experimental section does not directly test the proposition's condition and tunes the augmentation ratio per benchmark. Thus, the paper's main claimed guarantee is unsupported.","major_comments":[{"comment":"The proof of Proposition 4.2 asserts the identity Hbat|A − Hbat = HA without derivation. Under the paper's own definitions, this is generically false. From Definition 3.1, Rbat|A is the average over DA ∪ DB′, so for twice-differentiable losses Hbat|A = (|DA|/k)HA + (|DB′|/k)HB′, while Hbat is the Hessian on DB′ only, so Hbat = HB′. Therefore Hbat|A − Hbat = (|DA|/k)(HA − HB′). By Definition 3.2, |DA|/k → 1−γ (if γ is the backbone proportion) or more generally lies in (0,1), so the difference equals HA only in degenerate cases such as HB′ = 0 and |DA|/k = 1. The claimed reduction in Eq. (26) and the subsequent inequality ρbat|A ≤ ρA therefore do not follow. This is a load-bearing step in the central theorem.","section":"Supp. A.7, Eqs. (25)–(26); Def. 3.1 and Def. 3.2"},{"comment":"The proof models the estimators through a single Newton step: θ̂bat|A_k − θA* = (1/k)(Hbat|A)^{-1} Σ ∇Lbat|A and θ̂A_n − θA* = (1/NA)(HA)^{-1} Σ ∇LA. No argument connects these Newton updates to the AdamW/SGD dynamics actually used in the experiments (Sections 6.2 and Appendix B). Without such a connection, the asymptotic error coefficients derived here do not describe the optimizers that produce the reported results, so the theoretical guarantee is not relevant to the empirical evaluation.","section":"Supp. A.7, Eqs. (27)–(28)"},{"comment":"The proof of Proposition 4.1 introduces a function fB\\A satisfying fB(x; θB*) = fA(x, θB*) − fB\\A(x, θB*\\θA), but fB\\A is never defined, its existence is not established, and the notation θB*\\θA is not explained. The subsequent lower-semicontinuity argument for fA(xB; θA) depends critically on this entity. As a result, the proof of Proposition 4.1 is incomplete, and the convergence guarantee it claims is not rigorously supported.","section":"Supp. A.6, Eq. (24)"},{"comment":"The selection scheme is imported from [22] and claimed to satisfy the paper's propositions, but no derivation shows that the thresholded score Z(x; S) yields data satisfying the condition in Proposition 4.2. The statement \"According to [22], this setting yields an asymptotic error coefficient ρ\" and the appeal to Proposition 4.1 do not bridge the gap: [22] analyzes a different objective, and Proposition 4.1 only asserts existence of ρbat|A, not that the selected DB′ satisfies the inequality of Proposition 4.2. The link between the selection algorithm and the theoretical condition is therefore not established.","section":"Sec. 5.1, Eqs. (3)–(5)"},{"comment":"The experimental validation does not test the condition of Proposition 4.2 directly; instead, the augmentation ratio γ is tuned per benchmark (e.g., γ=0.9 for LoRA/DoRA, γ=0.992/0.9993 for HellaSwag, γ=0.95 for DreamBooth), and the ablation in Sec. 6.4 is interpreted through [22] rather than through the paper's own proposition. Consequently, the reported improvements are partly fitted to the benchmarks and do not serve as a parameter-free confirmation of the theory. Reporting error bars or significance tests would also be needed to support the empirical claims.","section":"Sec. 6.2 and Sec. 6.4"}],"minor_comments":[{"comment":"The definition 'n/k → γ' does not specify what n denotes; earlier the paper uses N_A and N_B for dataset sizes, and Definition 3.1 uses k = |Dbat|A|. Please clarify whether n = |DA|, n = |DB′|, or something else, and state the relation between γ and the proportions of DA and DB′ in Dbat|A.","section":"Def. 3.2"},{"comment":"The condition uses 'Σ_Dbat ∇θLbat|A' but Dbat is not defined; presumably Dbat = Dbat|A. Also, 'for any θ∈(θA∩θB)' is unclear since θA and θB are parameter vectors, not sets. Please rewrite the quantifier precisely.","section":"Prop. 4.2"},{"comment":"There is a typo: 'acoross' should be 'across'.","section":"Table 1 caption"},{"comment":"The title 'V alidity of BAT' contains an extra space; should be 'Validity of BAT'.","section":"Prop. 4.1 title"},{"comment":"The text says 'if γ = 0.95, the sample ratio is 0.5, and the adaptation dataset contains 57 images, we first sample 50% of the images from the backbone dataset and then select 3 images for Dbat.' This conflates the sample ratio with γ and with the selection threshold; please clarify the precise relationship among these quantities.","section":"Sec. 6.1"},{"comment":"The Stanford Alpaca dataset is cited twice, as [45] and [48], with identical URLs; please consolidate or differentiate the citations.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central theoretical claim, Proposition 4.2, is not supported: the proof relies on an identity that is false under the paper's own definitions, and the proof of Proposition 4.1 is incomplete. These are not local presentation issues but fundamental gaps in the main contribution. The experimental study is extensive and may have practical value, but it is presented as evidence for a theory that is not established, and the per-benchmark tuning of γ weakens the claim of validation. If the authors can repair the theory—for example, by deriving a correct condition based on the actual composition of the BAT risk and by connecting the analysis to the real optimizer—the manuscript could be resubmitted as a major revision. As it stands, the load-bearing errors cannot be fixed within the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: the practical idea here is reasonable—augment a scarce adaptation set with data from the backbone's pre-training distribution—and the empirical results are suggestive. But the theoretical centerpiece that supposedly makes the method principled does not hold up. Proposition 4.2's proof depends on the identity Hbat|A − Hbat = HA, stated without derivation. Using the paper's own Definition 3.1, Hbat|A is the Hessian of the mixed risk over DA ∪ DB′, namely (|DA|/k)HA + (|DB′|/k)HB′ up to the regularizer, while Hbat is the Hessian on DB′ alone. The difference is (|DA|/k)(HA − HB′), which equals HA only if HB′ = 0 and |DA|/k = 1. Neither is true under the paper's assumptions (γ ∈ (0,1)). So the identity is generically false, and the claimed inequality ρbat|A ≤ ρA does not follow. The subsequent remark that 'HA vanishes' inherits the error.\n\nGive credit where due: the framing is clear, and converting DataInf-style approximations into a tractable selection algorithm (ALBAT) is a sensible engineering contribution. The qualitative selection examples are illustrative, and the experiments span personalization and language tasks, with most benchmarks showing gains. The authors are also honest about the scope of their theoretical claims, explicitly flagging Sec. 6.5 as lacking formal treatment.\n\nSoft spots in proportion: Proposition 4.1's proof uses an undefined function f^{B\\A} and a continuity argument that is hand-waving. The proof of Prop 4.2 models the estimator with a single Newton step (Eqs. 27–28), yet the experiments use SGD/AdamW; no argument connects the two. The selection scheme is imported from Kolossov et al. [22] and assumed to satisfy the propositions—there is no demonstration that the data chosen by ALBAT actually satisfy Prop 4.2's condition. The experiments have no error bars, γ is tuned per benchmark (special values for HellaSwag), and the language 'backbone dataset' is Alpaca fine-tuning data, not the true pre-training corpus.\n\nBottom line: useful idea, broken proof. The empirical trend is worth a second look if the theory is repaired and the experiments are run with fixed hyperparameters and uncertainty estimates. As is, the central guarantee is not established, and I would not cite this version. I would still send it to a serious referee, because the flaw is instructive and the underlying method may be salvageable; but I would not spend a reading group on it.","headline":"The practical idea is sensible, but the central theory is broken: Prop 4.2 rests on a Hessian identity that contradicts the paper's own definitions.","tokens_in":23171,"tokens_out":5581,"would_cite":false,"duration_ms":46245,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F12","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding carefully selected backbone pre-training data to a small adaptation dataset provably improves the adaptation's asymptotic convergence, and gives an algorithm, ALBAT, for choosing that data.","keywords":["backbone augmented training","adaptation","data selection","asymptotic error coefficient","ALBAT","diffusion models","LoRA","DreamBooth"],"falsifier":"Train a small model with a synthetic backbone dataset and a disjoint adaptation dataset, compute the three curvature matrices exactly at the trained parameters, and check whether $H_{\\mathrm{bat}|A}-H_{\\mathrm{bat}}-H_A$ is zero; any nonzero entry in a direction used by Proposition 4.2 breaks the proof. A complementary experiment would select backbone data that violate the Proposition 4.2 inequality, run BAT with AdamW, and measure whether it still matches or beats standard adaptation on held-out loss; if it does, the asymptotic condition is not the operative mechanism for real optimizers.","tokens_in":21981,"feed_emoji":"🎯","tokens_out":9036,"duration_ms":86862,"temperature":0.7,"pith_summary":"Adaptation fine-tunes a large frozen backbone on a small user-supplied dataset, so scarce data makes training unstable or overfit. This paper tries to show that mixing in a carefully chosen slice of the backbone's own pre-training data can make the adaptation converge to the same optimum, and to do so no slower, and typically faster, than standard adaptation. Two propositions are the load-bearing claims: Proposition 4.1 states that the BAT estimator converges to the adaptation optimum, and Proposition 4.2 gives an explicit inequality involving the selected backbone data, the Hessian of the combined risk, and the backbone augmentation ratio $\\gamma$ under which BAT's asymptotic error coefficient is no worse than the unaugmented one. The paper then derives a biased selection score from that condition and packages it as ALBAT, reducing the selection cost to $O(nDL)$. The practical upshot is that if the propositions are correct, users with very few adaptation examples can choose backbone data with a computable criterion instead of guessing, and the reported DreamBooth, LoCon, LoRA, and DoRA experiments support that this improves personalization and language generation.","feed_headline":"Backbone data can make model adaptations converge faster","feed_subtitle":"A new condition on selected pre-training data guarantees adapted models are no worse than standard tuning, and often better","key_machinery":"The argument is carried by three linked objects. The first is the asymptotic error coefficient $\\rho(S)=p\\text{-}\\lim_{k\\to\\infty} k\\|\\hat\\theta-\\theta^*\\|_S^2$, a weighted measure of how quickly an estimator approaches the optimum; comparing $\\rho_{\\mathrm{bat}|A}$ and $\\rho_A$ is how the paper decides whether BAT helps. The second is the BAT risk $R_{\\mathrm{bat}|A}$ on the union of selected backbone data and adaptation data, with the requirement that the adapted model initialized from the backbone is the same model used in standard adaptation. The third is the biased selection score $Z(x;S)$, a curvature-adjusted influence-type score; thresholding it as $S(x)=\\mathbf{1}\\{Z(x;S)>\\eta\\}$ turns Proposition 4.2's condition into an algorithmic selection rule. Theorem 5.1 then brings the cost down to $O(nDL)$ by applying Bartlett's second identity, a damped Hessian, and the Sherman\\textendash Morrison update, so the selection rule is feasible for modern diffusion and transformer adapters.","core_discovery":"Formally, the paper's central claim is that Backbone Augmented Training is valid and conditionally better than ordinary adaptation. Let $R_{\\mathrm{bat}|A}$ be the empirical risk on $D_{\\mathrm{bat}|A}=D_{B'}\\cup D_A$, let $\\hat\\theta_{\\mathrm{bat}|A}$ be its minimizer, and let $\\rho_{\\mathrm{bat}|A}(S)=p\\text{-}\\lim_{k\\to\\infty} k\\|\\hat\\theta_{\\mathrm{bat}|A}-\\theta_A^*\\|_S^2$ be the asymptotic error coefficient. Proposition 4.1 asserts that $\\rho_{\\mathrm{bat}|A}$ exists under assumptions A1\\textendash A5, so BAT converges to the adaptation optimum. Proposition 4.2 asserts that if the selected backbone data satisfy $\\gamma\\|(H_{\\mathrm{bat}|A})^{-1}\\sum_{D_{\\mathrm{bat}}}\\nabla_\\theta L_{\\mathrm{bat}|A}\\| \\le \\|(H_{\\mathrm{bat}|A}-H_{\\mathrm{bat}})^{-1}\\sum_{D_A}\\nabla_\\theta L_{\\mathrm{bat}|A}\\|+o_P(1)$, then $\\rho_{\\mathrm{bat}|A}\\le \\rho_A$, and strictly so unless $\\gamma\\to 1$. The condition says, roughly, that the curvature-adjusted signal from the added backbone data must not exceed the signal from the original adaptation data. ALBAT implements the resulting biased selection rule with a score function $Z(x;S)$ and a threshold $\\eta$ chosen so that the expected selection rate is $\\gamma$, reducing Hessian computation from cubic to $O(nDL)$ via a damped Hessian and the Sherman\\textendash Morrison formula.","pith_inferences":["A natural testable extension is to apply the same score $Z$ to rank the adaptation data itself, not just backbone data: the structure of the condition suggests it measures which points contribute usable curvature to the adaptation optimum, which could inform dataset pruning in scarce-data regimes.","The paper's theory uses the Hessian identity $H_{\\mathrm{bat}|A}-H_{\\mathrm{bat}}=H_A$; a direct finite-model check of that identity would show whether the sufficient condition is practically identifiable, since the identity is exactly what lets ALBAT avoid computing the adaptation Hessian.","The external-data experiment hints that distributionally similar substitute data can work when the true backbone data are inaccessible, so extending Proposition 4.2 to cover surrogate backbone distributions would materially widen the method's applicability.","Because the error-coefficient comparison is asymptotic, the ranking given by $Z$ should be tested on final held-out performance under AdamW or SGD; if the ranking holds under the actual optimizer, the condition is a practical selection rule rather than only an asymptotic guarantee."],"forward_implications":["For any adaptation method whose risks satisfy the paper's regularity assumptions, adding backbone data that meets the Proposition 4.2 condition gives an asymptotic error coefficient no larger than that of standard adaptation, and strictly smaller unless the backbone fraction approaches the whole training set.","The $O(nDL)$ selection score makes the criterion computable for log-loss models, so the same ALBAT pipeline can be applied to other diffusion-based and transformer-based adapters, not only the four tested.","In the reported data-scarce settings, ALBAT improves DreamBooth and LoCon personalization metrics and raises most commonsense reasoning scores of LoRA and DoRA on LLaMA models, with the improvement generally growing as more backbone data are used for selection.","A partially trained surrogate model still produces gains over no augmentation, which means the selection overhead can be reduced by shortening surrogate training."],"supporting_citations":[{"why":"Supplies the general asymptotic regularity assumptions under which the BAT estimator is shown to converge to the adaptation optimum.","marker":"[21]"},{"why":"Supplies the weak-supervision biased data-selection framework and the asymptotic error coefficient that the paper uses to state and prove Proposition 4.2.","marker":"[22]"},{"why":"Supplies the Sherman-Morrison based Hessian approximation used to reduce the selection score computation to O(nDL), the basis of Theorem 5.1.","marker":"[23]"},{"why":"Defines LoRA, the low-rank adaptation method used in the language experiments and as one instantiation for Proposition 4.1.","marker":"[6]"},{"why":"Defines DreamBooth, the personalization method whose loss and training setup are used in the image-generation experiments and in the Proposition 4.1 instantiation.","marker":"[9]"},{"why":"Representative arbitrary backbone-data augmentation without validation, the baseline practice that BAT's theoretical condition is meant to replace.","marker":"[20]"}],"fun_headline_variants":["Backbone data provably speeds adaptation","Proven boost: add backbone data for adaptations","Adaptation training: backbone data yields proof","Why backbone data makes adaptations better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speedup guarantee stands on an unproved equality among three curvature matrices, that the BAT risk's curvature is exactly the sum of the backbone and adaptation curvatures, together with the simplifying model of the estimator as one Newton step; if either is wrong, the claimed inequality between convergence coefficients is not established.","fun_headline_variants_meta":{"raw":{"variants":["Backbone data provably speeds adaptation","Proven boost: add backbone data for adaptations","Adaptation training: backbone data yields proof","Why backbone data makes adaptations better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1711,"prompt_tokens":1031,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":647,"tokens_out":680,"duration_ms":7231,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:55:21.842011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small model with a synthetic backbone dataset and a disjoint adaptation dataset, compute the three curvature matrices exactly at the trained parameters, and check whether $H_{\\mathrm{bat}|A}-H_{\\mathrm{bat}}-H_A$ is zero; any nonzero entry in a direction used by Proposition 4.2 breaks the proof. A complementary experiment would select backbone data that violate the Proposition 4.2 inequality, run BAT with AdamW, and measure whether it still matches or beats standard adaptation on held-out loss; if it does, the asymptotic condition is not the operative mechanism for real optimizers.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the general asymptotic regularity assumptions under which the BAT estimator is shown to converge to the adaptation optimum."},{"cited_title":"Towards a statistical theory of data selection under weak supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the weak-supervision biased data-selection framework and the asymptotic error coefficient that the paper uses to state and prove Proposition 4.2."},{"cited_title":"Sdxl photorealistic lora tips: Reflections on training and releasing 10 different models, 2024","cited_arxiv_id":null,"evidence_quote":"Representative arbitrary backbone-data augmentation without validation, the baseline practice that BAT's theoretical condition is meant to replace."}],"review_version":1}