{"id":"ebf7869f-e05f-468e-b015-9eb4bf08372a","arxiv_id":"2505.12871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LoRA fine-tuning is more robust than full fine-tuning against backdoor attacks but more vulnerable to untargeted label-flipping poisoning, based on BERT/GLUE experiments and NTK-information-geometry analysis.","lead":"This paper finds that LoRA, a popular cheap way to fine-tune language models, resists backdoor triggers better than full fine-tuning but suffers more when training labels are randomly poisoned. The trade-off is explained through low-rank structure, neural tangent kernels, and information geometry, with BERT experiments across GLUE tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The double-edged-sword conclusion is not derived: Theorem 3.6 only bounds the magnitude of LoRA's parameter updates; the UPA/BPA sign flip rests on an unproven orthogonality heuristic (Eqs. 16-17).","rationale":"The reader's weakest_assumption identifies exactly the same gap: Theorem 3.6 does not imply the BPA/UPA asymmetry. My stress-test pass confirms that the formal results stop at a Loewner-type ordering of Fisher information, and the attack-specific conclusions are bolted on via an informal orthogonality heuristic. This is the load-bearing joint of the paper because the empirical pattern alone does not answer the title question; the 'due to over-simplified information geometry' clause in the abstract is the paper's central explanatory contribution. If the heuristic is wrong, the theoretical explanation collapses even though the experiments may remain valid. That said, I do not find a reason to move the verdict beyond CONDITIONAL: the empirical results across GLUE datasets, rank sweeps, variance sweeps, alternative initializations, extra attacks, and generative models provide real support for the headline pattern, and the theoretical gap is in principle addressable by additional derivation or by testing the bridge explicitly. I therefore agree with the reader and recommend no change to the verdict. I also note the paper's own caveats (footnote 3 and the weak variance effect in Section 4.4.2) as corroborating that the theoretical mechanism is not fully nailed down.","tokens_in":31934,"tokens_out":10492,"duration_ms":106290,"concrete_test":"In a two-layer NTK model satisfying the OOLD setup of Assumption 3.2, compute IB and Hα from the Fisher matrix for LoRA and FF across r ≤ n and σ^2 ≤ 1/n, and directly evaluate the BPA and UPA objectives in Eqs. (16)-(17) on clean/poisoned pairs (trigger appended for BPA, label flip for UPA, as in Section 4.2). Check whether the LoRA-vs-FF ordering of E|∇L_c^T∇L_t| and E|∇L_c^T∇L_u| follows the claimed double edge. If pairwise alignment does not move monotonically with IB/Hα, or if the sign flip reverses in any regime satisfying Theorem 3.6's conditions, the double-edged-sword claim is not a consequence of the theory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper—LoRA is more robust to backdoor attacks but more vulnerable to untargeted poisoning (Abstract, Section 4.5)—depends on the 'double-edged sword' argument in Section 3.3. Theorem 3.6 only establishes IBLoRA ≤ IBFF and HαLoRA ≤ HαFF under r ≤ n_{l−1} and σ^2 ≤ 1/n_{l−1}, i.e., a Loewner-type ordering of Fisher information. But this ordering bounds the magnitude of the expected parameter-update difference (M' in Section 2.3); it says nothing about the sign of the effect on attack success. Indeed Section 2.3 equates larger K_ntk with lower TTR, so Theorem 3.6 would predict LoRA is more robust to any perturbation, including UPA. The subsequent flip for UPA is introduced via Eqs. (16)-(17), which assume without proof that BPA benefits from decoupled gradients while UPA benefits from aligned gradients, and that smoother geometry (lower IB/Hα) implies smaller angles between clean and poisoned gradients. No theorem connects these pairwise, data-dependent, attack-specific inner products to the global spectral measures IB/Hα. The paper's own caveats weaken the claim: footnote 3 says UPA susceptibility may vanish for over-parameterized models, and Section 4.4.2 reports that initialization variance has minimal effect on poisoning, contradicting the theory's predicted trade-off. Since the theoretical framework is the paper's stated main contribution, an unsupported bridge leaves the central explanatory claim unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an analytical framework, combining neural tangent kernels and information geometry, to compare the training-time robustness of LoRA fine-tuning with full fine-tuning against untargeted poisoning attacks (UPA) and backdoor attacks (BPA). It claims theoretically, and supports experimentally on GLUE tasks with BERT-large plus additional generative-model experiments, that LoRA is more robust than full fine-tuning against backdoor attacks but more vulnerable to untargeted poisoning, and that LoRA rank and initialization variance are the key control parameters. The stated main contribution is the theoretical framework, with the experiments presented as corroboration.","tokens_in":32323,"tokens_out":7750,"duration_ms":84123,"significance":"If the central claim is correct, the paper addresses a genuinely underexplored and practically important question: whether the efficiency gains of LoRA come with systematic changes in vulnerability to training-time attacks. The empirical comparison is broad for this literature: four GLUE tasks, two attack families, rank and initialization ablations, four additional backdoor variants, and a generative-language-model extension. The authors also ship source code and give concrete design guidance (low rank for backdoor defense, small initialization variance for robustness). These are real strengths. However, the significance hinges on the theoretical bridge between kernel/information-geometric ordering and attack-specific behavior, which is where the manuscript's load-bearing defects lie.","major_comments":[{"comment":"The sign of the proposed TTR metric appears reversed. Kntk(xc,x~c) is defined in Eq. (6) as the gradient inner product; a large inner product means the clean and poisoned samples induce nearly aligned gradient updates, so the parameter-update difference in Eq. (5) is small. That is high robustness, not low robustness. The text states the opposite ('a large value of Kntk ... reflects lower training-time robustness'). With the correct sign, Theorem 3.6 (K_LoRA <= K_FF under the stated conditions) would predict that LoRA has lower robustness for every perturbation type, including backdoor attacks, which contradicts the paper's central backdoor claim. The definition or the interpretive sentence must be corrected, and the direction of all subsequent conclusions re-examined.","section":"Section 2.3, Eq. (7)"},{"comment":"The definition of information bits is mathematically wrong: 1/2 log det I Theta equals 1/2 sum_i log lambda_i, not 1/2 sum_i lambda_i. As written, Eq. (9) defines half the trace, not a log-determinant. The proof of Theorem 3.6 in Appendix A.5 actually establishes an inequality for the sum of eigenvalues (trace), so the proof and the definition are inconsistent even under a charitable reading. Since IB is one of the two central quantities connecting geometry to attack robustness, this must be fixed.","section":"Section 2.4, Eq. (9)"},{"comment":"The stated eigenvalue distribution of M_Delta^(l) = A^(l)T A^(l) - I is incorrect. Because rank(A^(l)T A^(l)) <= r, the matrix A^(l)T A^(l) has n_{l-1}-r zero eigenvalues and r positive eigenvalues; consequently A^(l)T A^(l) - I has n_{l-1}-r eigenvalues equal to -1 and r eigenvalues approximately equal to sigma_a^2 n_{l-1} - 1. The theorem's claim of 'r eigenvalues equal to sigma_a^2 n_{l-1} and n-r eigenvalues equal to 0' describes A^(l)T A^(l), not the shifted matrix. The negative-semi-definiteness conclusion survives, but the subsequent heuristic in Section 3.3 explicitly appeals to '(n_{l-1}-r) zero eigenvalues' to argue for a smaller backdoor-trigger search space; that argument is invalid because the corresponding eigenvalues of M_Delta^(l) are -1, not 0. The eigenvalue claim must be corrected and the orthogonality argument rebuilt or removed.","section":"Section 3.2, Theorem 3.4"},{"comment":"The double-edged-sword conclusion—that lower IB/H_alpha improves backdoor robustness but harms poisoning robustness—is not derived. Theorem 3.6 only establishes an ordering of Fisher information under the stated conditions; combined with the (corrected) TTR metric it predicts the same robustness direction for all perturbations. The sign flip is introduced through an informal orthogonality heuristic involving inner products between clean and poisoned gradients, with no proof that the global spectral quantities IB/H_alpha control this pairwise, attack-specific inner product in opposite directions. The paper's own caveats weaken the claim further: footnote 3 says UPA susceptibility may disappear for over-parameterized models, and Section 4.4.2 reports that initialization variance 'slightly influences the poisoning' and 'deviates from our theoretical analysis'. Since the abstract and Section 4.5 state the double-edged effect as a headline theoretical finding, this unsupported bridge is load-bearing and must be either rigorously established or explicitly downgraded to an empirically motivated conjecture.","section":"Section 3.3, Eqs. (16)-(17)"},{"comment":"The generalized proof beyond the OOLD assumption contains a sign error. From Delta^(2,k)' <= 0 and Delta^(l)_r <= 0, the recursion Delta^(l,k)' = Delta^(l-1,k)' * Sigma_dot^(l) + Delta^(l)_r yields a non-positive quantity, not 'Delta^(l,k)' >= 0' as written. The subsequent sentence 'Delta^(l,k)' >= 0 holds for l=1,...,L' contradicts the inductive base and reverses the conclusion of the generalization. This needs correction for the generalized version of Theorem 3.6 in Section 3.4 to be valid.","section":"Appendix A.6.1"}],"minor_comments":[{"comment":"H_alpha is defined on the eigenvalues of the Fisher information matrix, but the visualization in Eq. (18) and Figure 3 uses eigenvalues of A^(l)T A^(l) instead; no theorem connects these two spectra, so the figure's supporting role is unclear.","section":"Section 2.4, Eqs. (9)-(10) vs Eq. (18) and Figure 3"},{"comment":"The statement 'for all lambda_LoRA and all lambda_FF, lambda_LoRA <= lambda_FF' is imprecise; the correct argument is Weyl's monotonicity for sorted eigenvalues (lambda_k(I_LoRA) <= lambda_k(I_FF) for each k), and the proof should state this explicitly.","section":"Appendix A.5, proof of Theorem 3.6"},{"comment":"The notation 'gradient_theta L^T gradient_theta L' treats the Fisher information as a scalar; for a parameter vector it should be the outer product or the quadratic form in Eq. (8). The intended meaning is clear from context, but the notation should be made consistent.","section":"Section 2.4, Theorem 2.2"},{"comment":"The learning rates differ between LoRA (3e-5) and FF (3e-6). Because attack resistance can be sensitive to step size, this hyperparameter difference should be justified or ablated; otherwise the UPA/BPA comparisons may partly reflect optimization dynamics rather than the low-rank structure.","section":"Section 4.1 and Figure 1/2"},{"comment":"The Section 3 title contains a repeated word ('Does LoRA Lead to LoRA'), and several appendix figures (e.g., Figure 11, 12) omit dataset names or axis labels in some panels. These should be cleaned up.","section":"Section 3 title and several figure captions"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is potentially useful and the authors are transparent about divergent results (e.g., the variance ablation in Section 4.4.2). However, the theoretical core, which the paper frames as its main contribution, contains several correctable but currently load-bearing errors. I would suggest the editor consider whether a revised version that fixes the sign conventions, corrects the eigenvalue and log-determinant statements, and substantially weakens or properly derives the double-edged claim would be within the scope of a major revision. If the authors prefer to reposition the paper as a primarily empirical study with the NTK/IG material as a heuristic discussion, that would also be a viable path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper has a genuinely interesting empirical finding—LoRA resists backdoor attacks better than full fine-tuning but suffers more under untargeted label-flipping—and the experiments back it up across several GLUE tasks, rank sweeps, and additional backdoor variants. That alone makes it worth a serious look.\n\nThe direct head-to-head comparison between LoRA and FF under both attack types appears to be new; prior work mostly used LoRA as an attack vehicle. The authors also deserve credit for shipping code, testing four additional backdoor attacks, and—to their credit—openly reporting when the variance experiment undercuts their own theory (Section 4.4.2).\n\nThe soft spots are real, though. The theory section has mechanical errors. Equation 9 defines IB as 1/2 log det but writes it as 1/2 sum of eigenvalues; those are not the same. Theorem 3.4 says A^T A - I has n-r zero eigenvalues when rank(A^T A) <= r, but if A^T A has n-r zero eigenvalues, then A^T A - I has n-r eigenvalues equal to -1, not 0. That matters because the \"zero eigenvalues\" are later used to argue for a smaller backdoor trigger search space.\n\nMore importantly, the central double-edged-sword conclusion is not derived. Theorem 3.6 only establishes IB_LoRA <= IB_FF and H_alpha_LoRA <= H_alpha_FF under the stated conditions. By the paper's own metric, larger kernel values mean lower training-time robustness, so that ordering would predict LoRA is more robust to any perturbation, including UPA. The flip for UPA is introduced via equations 16–17, an unproven orthogonality heuristic connecting smooth geometry to gradient alignment. No theorem bridges that gap. The paper's own caveats weaken the claim further: footnote 3 says UPA susceptibility may vanish for overparameterized models, and Section 4.4.2 reports that initialization variance has minimal effect on poisoning, contradicting the theory's predicted trade-off.\n\nThere is also a straightforward empirical confound: LoRA uses a learning rate of 3e-5 while FF uses 3e-6. Different learning rates can change poisoning resistance, so the headline comparison is not fully controlled.\n\nWho is this for? People working on PEFT security, data poisoning, and backdoor defenses. They will get a useful empirical datapoint and a theory worth arguing against. It deserves a serious referee—the empirical phenomenon is important enough—but not acceptance as-is. I would recommend peer review with a strong request to separate the empirical finding from the overclaimed theory, fix the mechanical errors, and run a learning-rate-controlled comparison.","headline":"The empirical headline that LoRA is more robust than full fine-tuning against backdoor attacks but less robust against untargeted poisoning is new and mostly supported by the experiments; the theoretical framework, however, overreaches and does not actually derive the double-edged-sword conclusion.","tokens_in":32834,"tokens_out":2535,"would_cite":true,"duration_ms":27106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62B10","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"LoRA is harder to backdoor than full fine-tuning but more vulnerable to untargeted data poisoning, with the trade-off governed by its rank and initialization variance.","keywords":["LoRA","low-rank adaptation","training-time attacks","data poisoning","backdoor attacks","neural tangent kernel","information geometry","Fisher information"],"falsifier":"On a fixed task and model, sweep LoRA rank from 4 to 512 and initialization variance scale $k$ from 0.1 to 2.0 under fixed poisoning rates; the claimed trade-off predicts that backdoor attack success rate falls as rank or variance decreases while untargeted-poisoning accuracy loss rises. A configuration where lower rank or lower variance improves both metrics, or degrades both, would contradict the proposed mechanism.","tokens_in":31715,"feed_emoji":"🛡️","tokens_out":7303,"duration_ms":76597,"temperature":0.7,"pith_summary":"The paper argues that LoRA's low-rank update is not security-neutral: relative to full fine-tuning, LoRA is harder to poison with backdoor triggers but easier to derail with untargeted label flipping. The proposed reason is geometric: LoRA's training trajectory has a smoother information surface, which shrinks the space of effective backdoor triggers while also flattening the decision surface so poisoned labels push the model further. The claim matters because LoRA has become a default way to fine-tune large language models, so knowing this structural vulnerability trade-off tells practitioners which attacks to fear and how rank and initialization variance affect those risks.","feed_headline":"LoRA is backdoor-safe but poison-prone","feed_subtitle":"Low rank smooths the model's geometry, hiding backdoor triggers yet exposing it to untargeted poisoning.","key_machinery":"The load-bearing object is the kernel-matrix difference $M_\\Delta^{(l)} = A^{(l)T}A^{(l)} - I$, which is shown to be negative semi-definite under the rank and variance conditions. This single comparison converts the LoRA-versus-FF question into an eigenvalue comparison: negative semi-definiteness makes LoRA's NTK, Fisher information, information bits, and R\\'enyi entropy all no larger than FF's. The attack-specific conclusions are then carried by the orthogonality of gradient inner products, expressed in Eqs. (16)-(17), which is where the two attack objectives diverge.","core_discovery":"The central claim is that, when LoRA's matrix $A$ has rank $r \\leq n_{l-1}$ and initialization variance $\\sigma^2 \\leq 1/n_{l-1}$, the neural tangent kernel of LoRA is no larger than that of full fine-tuning, so LoRA's Fisher information, information bits, and R\\'enyi entropy are all no larger than FF's. That smoother information geometry is a double-edged sword: it gives backdoor triggers a smaller search space, improving backdoor resistance, while making the model more sensitive to untargeted poisoning because poisoned and clean gradient directions align more easily on an oversimplified manifold. Experiments on BERT-large and GLUE tasks corroborate both halves of the trade-off, and rank and initialization variance are identified as the controlling factors.","pith_inferences":["If the geometric mechanism is right, any intervention that smooths the fine-tuning manifold, such as weight decay or spectral-norm constraints, should shift the same backdoor-versus-poisoning trade-off rather than only low rank doing so.","A cheap security audit could be derived: compute the information bits of the adapter's Fisher matrix before deployment, since low IB would flag both poisoning vulnerability and backdoor resistance in one number.","The same NTK comparison could be extended to other parameter-efficient fine-tuning methods by writing their kernel difference as a semi-definite correction, which would test whether the trade-off is specific to LoRA or generic to constrained fine-tuning.","The theory predicts that the trade-off should appear even without explicit attacks, simply from measuring how much label noise changes the model's predictions; this could be checked as a direct probe of the proposed mechanism."],"forward_implications":["LoRA-fine-tuned models should be more resistant than full fine-tuning to backdoor attacks, with the paper reporting up to roughly 30% accuracy gains on some GLUE tasks under trigger-injected attacks.","LoRA-fine-tuned models should suffer larger accuracy drops than full fine-tuning under untargeted label-flipping poisoning, with the gap widening as the poisoning rate increases.","Raising LoRA's rank improves robustness against untargeted poisoning but weakens backdoor resistance, so rank defines a security trade-off rather than a single monotone improvement.","Lowering the initialization variance of the $A$ matrix improves backdoor resistance with little effect on clean performance, supporting the paper's recommendation to use small initialization variance.","Practical guidance follows directly: set the rank as low as performance allows and keep the initialization variance small when backdoor attacks are the main threat."],"supporting_citations":[{"why":"Defines the LoRA low-rank update $\\Delta W = BA$ that is the object of the security analysis.","marker":"Hu et al., 2021"},{"why":"Supplies the neural tangent kernel convergence and constancy properties used in Theorem 2.1 to simplify training dynamics.","marker":"Jacot et al., 2021"},{"why":"Provides the Gaussian-process covariance recursion used to derive the NTK expressions for full fine-tuning and LoRA.","marker":"Lee et al., 2018"},{"why":"Supplies the information-geometry framework connecting the Fisher metric to model robustness.","marker":"Amari, 2016"},{"why":"Defines the Fisher information matrix whose eigenvalues form the information bits measure.","marker":"Fisher, 1922"},{"why":"Defines the R\\'enyi entropy $H_\\alpha$ used to quantify the complexity or curvature of the fine-tuning manifold.","marker":"Rényi, 1961"},{"why":"Defines the untargeted label-flipping poisoning attack used in the UPA experiments and interpretation.","marker":"Fan et al., 2022"},{"why":"Provides the backdoor poisoning attack protocol with trigger injection and label modification used for BPA evaluation.","marker":"Wan et al., 2023"},{"why":"Supports the claim that LoRA and other PEFT methods operate in the NTK regime, justifying the paper's analytic simplification.","marker":"Malladi et al., 2023"},{"why":"States the earlier expressivity-equivalence result that the paper contrasts with its information-geometry finding.","marker":"Zeng & Lee, 2024"}],"fun_headline_variants":["LoRA: backdoor-resistant, poison-sensitive","LoRA trade-off: backdoor shield, poison exposure","LoRA's smooth geometry: backdoor-safe, poison-vulnerable","Low rank LoRA: tougher vs backdoors, weaker vs poisoning","LoRA backdoor robustness vs poisoning vulnerability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that smoother information geometry (lower information bits and lower $H_\\alpha$) is what makes LoRA more robust to backdoor triggers yet more susceptible to untargeted poisoning, a sign flip the theorems do not prove and which rests on the orthogonality heuristic of Eqs. (16)-(17).","fun_headline_variants_meta":{"raw":{"variants":["LoRA: backdoor-resistant, poison-sensitive","LoRA trade-off: backdoor shield, poison exposure","LoRA's smooth geometry: backdoor-safe, poison-vulnerable","Low rank LoRA: tougher vs backdoors, weaker vs poisoning","LoRA backdoor robustness vs poisoning vulnerability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1784,"prompt_tokens":871,"completion_tokens":913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":487,"tokens_out":913,"duration_ms":9095,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:22.297576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed task and model, sweep LoRA rank from 4 to 512 and initialization variance scale $k$ from 0.1 to 2.0 under fixed poisoning rates; the claimed trade-off predicts that backdoor attack success rate falls as rank or variance decreases while untargeted-poisoning accuracy loss rises. A configuration where lower rank or lower variance improves both metrics, or degrades both, would contradict the proposed mechanism.","supporting_citations":[{"cited_title":"Neural tangent kernel: convergence and generalization in neural networks (invited paper)","cited_arxiv_id":null,"evidence_quote":"Supplies the neural tangent kernel convergence and constancy properties used in Theorem 2.1 to simplify training dynamics."},{"cited_title":"S., Pennington, J., and Sohl - Dickstein, J","cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian-process covariance recursion used to derive the NTK expressions for full fine-tuning and LoRA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Fisher information matrix whose eigenvalues form the information bits measure."},{"cited_title":"Poisoning language models during instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the backdoor poisoning attack protocol with trigger injection and label modification used for BPA evaluation."},{"cited_title":"A kernel-based view of language model fine-tuning","cited_arxiv_id":null,"evidence_quote":"Supports the claim that LoRA and other PEFT methods operate in the NTK regime, justifying the paper's analytic simplification."}],"review_version":1}