{"id":"5051ebd4-0baa-4dbb-9414-13b4d250252e","arxiv_id":"2509.09097","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.","lead":"DP-FedLoRA combines low-rank adaptation (LoRA) with differential privacy so edge devices can fine-tune a shared language model while keeping raw data local. It reports competitive benchmark scores, but its privacy accounting has a sensitivity error and its privacy budget is weak, so the headline claims are not yet supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clipped-matrix sensitivity is 2C not C, so Eqs. (10)-(11) understate privacy loss; B/A and round composition also unaccounted for.","rationale":"The reader's weakest assumption correctly identifies the privacy accounting as the load-bearing point. The single most direct error is the sensitivity factor: Eq. (9) requires the worst-case L2 difference of the function, but Eqs. (10)-(11) set that difference to the clip norm C. For a projection onto the Frobenius ball of radius C, two neighboring datasets can be mapped to opposite points on the sphere, so the difference is 2C. This is not a conservative choice; it halves the effective noise and doubles the privacy loss. The paper also ignores composition between the B and A releases and across training rounds, but those are additional gaps. The variance bound in Eq. (23) is dimensionally wrong (Var[Bα] scales with n, not m), which further indicates the theoretical analysis is not reliable, but it is secondary to the privacy claim. The proposed test—recomputing the per-matrix ε with sensitivity 2C—directly falsifies the calibration in Eqs. (10)-(11) for the reported parameters. Therefore the reader's REJECT verdict stands unchanged.","tokens_in":13494,"tokens_out":12467,"duration_ms":144684,"concrete_test":"Recompute the achieved privacy parameter for a single released matrix: set C=0.1, δ=1e-5, ε=25; calculate σ from Eq. (10) with S=C. Then compute the actual ε of the Gaussian mechanism with sensitivity S=2C and this same σ (using the analytic Gaussian mechanism or the formula ε = sqrt(2 log(1.25/δ))·(2C)/σ). If the resulting ε is ≈50 (i.e., double the claimed 25), the sensitivity assumption in Eqs. (10)-(11) is contradicted and the per-matrix privacy claim fails. This numerical check settles the concern without retraining.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central DP claim rests on the sensitivity assignment S_B = C_B and S_A = C_A in Eqs. (10)-(11). Under the replacement-neighbor definition in Section IV, the clipped output can take any matrix with Frobenius norm ≤ C. Two neighboring datasets can therefore produce clipped matrices C·u and -C·u, making the L2 sensitivity 2C, not C. With noise σ calibrated for sensitivity C, the Gaussian mechanism delivers only ε' ≈ 2ε per matrix (or equivalently a δ larger than claimed). Since a client releases both B and A and participates in many rounds, the total privacy loss is further inflated by composition, which the paper never accounts for. Thus the assertion that each client satisfies (ε, δ)-DP at the reported ε is unsupported; the privacy guarantee is weaker by at least a factor of two per matrix, before round composition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DP-FedLoRA, a method for privacy-preserving federated fine-tuning of large language models with LoRA. Each client trains low-rank matrices B_k and A_k locally, clips them to Frobenius norms C_B and C_A, and adds isotropic Gaussian noise; the server aggregates the noisy matrices by horizontal/vertical stacking into global matrices \\tilde{B}, \\tilde{A}. The paper claims that this provides (epsilon, delta)-differential privacy per client via the Gaussian mechanism (Eqs. 9-11), that the noisy update is unbiased (Eq. 20), and that its variance is bounded by Eq. (26). Experiments on LLaMA-2-7B/13B with Alpaca-GPT-4 data compare seven federated optimizers with and without DP on MMLU, BBH, and CRASS, and include ablations over epsilon, clipping norm, LoRA rank, and model size.","tokens_in":13733,"tokens_out":6593,"duration_ms":80754,"significance":"The application is timely and the experimental effort is substantial: the paper releases code, covers seven federated optimization algorithms, and empirically studies the effect of LoRA rank and model size on update variance. The unbiasedness result, while standard, is stated cleanly. However, the central privacy claim is not supported by the analysis. The Gaussian mechanism is calibrated with sensitivity C after clipping to a Frobenius ball of radius C, whereas the true L2 sensitivity of the clipped release is 2C. Furthermore, the paper does not compose the B and A releases within a round, nor does it account for clients participating in multiple communication rounds. The variance bounds in Eqs. (23)-(26) also contain dimension errors. These issues are load-bearing: they undermine the advertised (epsilon, delta)-DP guarantee and the analytical variance bound.","major_comments":[{"comment":"The Gaussian mechanism guarantee in Eq. (9) depends on the L2 sensitivity S. After clipping to ||B||_F <= C_B, the set of possible outputs is the Frobenius ball of radius C_B. Two neighboring datasets can yield clipped updates C_B u and -C_B u, so the sensitivity is sup_{D~D'} ||clip(B(D)) - clip(B(D'))||_F = 2C_B, not C_B. With sigma_B calibrated using S_B = C_B, the noise is half what it should be for the stated (epsilon, delta); the achieved per-release privacy loss is at least approximately 2 epsilon (or a much larger delta) for each matrix. The same issue applies to A. The paper's claim that DP-FedLoRA satisfies (epsilon, delta)-DP at the reported epsilon is therefore not delivered. The fix is to use S = 2C (equivalently, double sigma or halve epsilon) throughout the calibration.","section":"Section V, Eqs. (9)-(11)"},{"comment":"Even after correcting the sensitivity, the paper provides no composition analysis. A client emits both \\tilde{B}_k and \\tilde{A}_k in the same round; under basic composition the combined release has privacy budget at least epsilon_B + epsilon_A, not two independent guarantees. More importantly, Section VII-A runs 200 communication rounds with 2 of 20 clients sampled per round; an individual client may appear in many rounds. The manuscript never defines the unit of the DP guarantee (per update, per round, or over the whole training run) and never accounts for repeated participation. Without this accounting, the abstract's 'strong privacy guarantees' claim is unsupported. The paper must either provide a full composition analysis over the training process or explicitly restrict and justify a per-round-only claim.","section":"Section V / Algorithm 1; Section VII-A"},{"comment":"The variance bounds contain dimension errors. For B in R^{m x r} and alpha in R^{r x n}, the total variance of B alpha (summed over all entries) is n sigma_alpha^2 ||B||_F^2, not m sigma_alpha^2 ||B||_F^2. Similarly, for beta in R^{m x r} and A in R^{r x n}, Var[beta A] = m sigma_beta^2 ||A||_F^2, not n sigma_beta^2 ||A||_F^2. Consequently Eq. (26) has the coefficients of the first two terms swapped. The third term, sigma_beta^2 sigma_alpha^2 m n r, is correct and matches the linear growth in rank reported in Section VII-D, but the stated bounds are not correct as written. This does not invalidate the unbiasedness result but weakens the claimed analytical variance characterization.","section":"Section VI, Eqs. (23)-(26)"}],"minor_comments":[{"comment":"The text says 'stricter privacy constraints (i.e., a larger value of epsilon)'; this is backwards. A larger epsilon corresponds to a weaker privacy guarantee. This terminology should be corrected throughout the ablation discussion.","section":"Section VII-B"},{"comment":"The claimed 'practical guidance for privacy-budget calibration' is not used to set the privacy budget. The experiments select epsilon = 25.0 based on the ablation in Fig. 1, not from the variance bounds. Either connect the analysis to the budget choice or temper the contribution claim.","section":"Sections VI and VII"},{"comment":"The sentence 'This factorization drastically reduces the number of trainable parameters from m x r to (rd + rk)' contains typos; it should presumably be r(m + n) or similar. Please clarify.","section":"Section III-A"},{"comment":"The stacking operation is introduced as 'symbolized by L' but the displayed equation uses a different symbol. Please unify the notation and define the dimensions of the stacked matrices explicitly.","section":"Section V, Eq. (3)"}],"recommendation":"reject","confidential_remarks":"The central DP guarantee of the manuscript is not supported because of a clear sensitivity miscalibration and the complete absence of composition accounting for the two released matrices and for repeated client participation across the 200 communication rounds used in the experiments. The variance-bound dimension errors reinforce that the theoretical core needs substantial reworking. The empirical study may be salvageable, but as submitted the main claim is not deliverable. I recommend rejection; a resubmission that corrects the sensitivity, provides a proper privacy accountant, and recalibrates the experimental privacy parameters could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a straightforward combination of LoRA, federated stacking aggregation, and clip-and-perturb Gaussian DP. The only new piece is an expectation/variance analysis of noise in the product of LoRA matrices, but that analysis is elementary and contains dimension errors. The empirical evaluation covers seven FL algorithms and shows the expected trade-offs, so the engineering effort is real. The central privacy claim, however, does not hold.\n\nThe main problem is the sensitivity calibration. The authors clip each LoRA matrix to Frobenius norm C and then claim sensitivity C in Eqs. (10)-(11). Under the replace-one adjacency defined in Section IV, two neighboring datasets can produce clipped matrices of norm C in opposite directions, so the l2-sensitivity is 2C. The noise is under-scaled by a factor of 2 per matrix. They also release both B and A without composing the two mechanisms, and a client participates in many rounds without any accounting for repeated releases. The stated (epsilon, delta) guarantee is therefore unsupported.\n\nThe variance analysis has the same carelessness: Eq. (23) should have n, not m, and Eq. (24) should have m, not n. Fortunately Eq. (25) is correct, so the qualitative rank-scaling result survives, but the quantitative bounds are wrong.\n\nOn the positive side, the paper is clearly written, the code link is provided, and the comparison across seven optimizer variants is thorough. The observation that noise variance grows linearly with LoRA rank is useful and consistent with a correct calculation. Still, the absence of DP baselines, the weak privacy budget (epsilon=25), and the large CRASS accuracy drops make the empirical claims shaky.\n\nGiven the load-bearing privacy error, I would not cite this in its current form. It could serve as a workshop note on common DP pitfalls, but it needs a corrected sensitivity analysis, explicit composition, and stronger evaluation before it can support its claims. A serious referee could help the authors fix these issues, so I would not desk-reject, but the right outcome in review is reject unless the analysis is substantially revised.","headline":"A competent but forgettable combination of LoRA and DP-FL that gets the sensitivity wrong, undercuts its own privacy claim, and would only be citable as a cautionary tale.","tokens_in":14222,"tokens_out":6069,"would_cite":false,"duration_ms":62442,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DP-FedLoRA claims that adding calibrated Gaussian noise to clipped LoRA updates makes federated fine-tuning of on-device LLMs differentially private with modest accuracy loss.","keywords":["federated learning","differential privacy","LoRA","large language models","on-device fine-tuning","Gaussian mechanism","membership inference","privacy budget"],"falsifier":"Compute the l2 sensitivity of the clip-and-noise mechanism for a pair of neighboring clients whose clipped updates are B and -B with ||B||_F=C; the output distributions are separated by 2C, so the noise scale must double to meet the same (ε,δ). A practical test is to run the released code with the paper's default ε=25 and measure membership-inference success on the noisy updates: an attacker success rate at chance level would support the claim, while near-perfect success would falsify it.","tokens_in":13371,"feed_emoji":"🔒","tokens_out":6811,"duration_ms":61421,"temperature":0.7,"pith_summary":"The paper aims to show that federated fine-tuning of large language models on edge devices can be made differentially private without destroying utility, by clipping each client's LoRA adaptation matrices to a Frobenius norm bound and adding Gaussian noise calibrated to that bound. If right, this gives a per-update (ε, δ)-DP guarantee against a semi-honest server performing membership inference, while preserving the communication efficiency and low-rank structure of LoRA. The paper further claims that the noisy update is unbiased in expectation, and its variance is bounded by a formula that grows with model dimensions and LoRA rank, which can be used to choose privacy parameters. Experiments with LLaMA-2-7B/13B across seven federated optimizers show modest drops on MMLU and BBH but a larger drop on CRASS counterfactual reasoning, which the paper attributes to noise sensitivity.","feed_headline":"Noisy LoRA fine-tuning keeps federated on-device LLMs private","feed_subtitle":"Adds a formal differential-privacy bound to clipped LoRA updates with only modest accuracy loss on MMLU and BBH.","key_machinery":"The load-bearing object is the clip-then-perturb mechanism applied independently to the two LoRA factors: each B_k and A_k is scaled down to Frobenius norm C_{Bk} or C_{Ak}, then isotropic Gaussian noise with standard deviation σ ≥ C √(2 log(1.25/δ))/ε is added. The server aggregates by horizontally stacking the noisy B's and vertically stacking the noisy A's, so the global adaptation remains a low-rank product. The theoretical results hinge on the independence and zero-mean of the noise matrices, giving unbiasedness Eq. (20), and on the variance bound Eq. (26), which is the tool for privacy-budget calibration.","core_discovery":"The paper proposes DP-FedLoRA, a federated fine-tuning framework for on-device LLMs. Each client locally fine-tunes low-rank matrices B_k and A_k, clips each to a Frobenius norm bound, and adds isotropic Gaussian noise with scale set by the standard Gaussian-mechanism formula. The server aggregates the noisy matrices via a stacking operation that preserves the LoRA product structure. The paper's central claim is that this mechanism enforces (ε,δ)-differential privacy per client update, that the expected noisy adaptation equals the clean adaptation, and that the added variance is bounded by m σ_α² ||B||_F² + n σ_β² ||A||_F² + σ_β² σ_α² mn r. Experiments on LLaMA-2-7B and 13B with seven federa","pith_inferences":["The paper's sensitivity step likely undercounts the true noise needed: for two neighboring datasets whose clipped updates are B and -B, both with norm C, the release distributions are 2C apart, so the claimed (ε,δ) guarantee would require doubling σ or halving ε.","The analysis never composes the budgets spent on B and A, nor the budgets spent across the training rounds in which a client participates; a proper composition accounting would raise the effective privacy cost, and a DP-FedLoRA variant with a composition-aware accountant would be a natural extension.","The large CRASS drop suggests counterfactual-reasoning benchmarks are particularly sensitive to update noise; a testable extension would be per-task clip thresholds or rank-adaptive noise schedules to protect those tasks.","The variance formula implies a hard trade-off between LoRA rank and model scale under a fixed privacy budget, which could be measured empirically as a contour map of accuracy versus rank and noise scale."],"forward_implications":["If a client's update is clipped to norm C and noise with σ ≥ C sqrt(2 log(1.25/δ))/ε is added, the released matrices satisfy (ε,δ)-DP, so the server cannot confidently perform membership inference on the update.","The noise-injected global update is unbiased: in expectation the noisy adaptation equals the clean adaptation, so the federated optimization trajectory is preserved on average.","The variance bound gives a calibration rule: for fixed clip bounds and privacy budget, the variance grows linearly with LoRA rank r and with model dimensions m,n, so smaller ranks and smaller models are preferable when noise is present.","Across seven federated optimizers, DP-FedLoRA yields only a ~4–5% average drop on MMLU and BBH, indicating that strong per-update DP can be compatible with competitive fine-tuning performance.","The stacking aggregation accommodates clients with heterogeneous ranks without zero-padding or averaging artifacts."],"fun_headline_variants":["DP-FedLoRA: Noisy LoRA adds privacy to on-device federated fine-tuning","Clipped LoRA updates plus Gaussian noise make federated on-device LLMs private","Private on-device LLM fine-tuning via DP-FedLoRA's noise-injected LoRA","DP-FedLoRA: Privacy-preserving LoRA for federated on-device models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee assumes a clipped LoRA matrix has l2-sensitivity equal to its clip bound C; if neighboring datasets can produce opposite updates of norm C, the true sensitivity is 2C and the claimed (ε,δ) bound is off by a factor of two.","fun_headline_variants_meta":{"raw":{"variants":["DP-FedLoRA: Noisy LoRA adds privacy to on-device federated fine-tuning","Clipped LoRA updates plus Gaussian noise make federated on-device LLMs private","Private on-device LLM fine-tuning via DP-FedLoRA's noise-injected LoRA","DP-FedLoRA: Privacy-preserving LoRA for federated on-device models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1537,"prompt_tokens":738,"completion_tokens":799,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":701}},"tokens_in":482,"tokens_out":799,"duration_ms":7793,"temperature":1.0,"reasoning_tokens":701,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:41:29.397448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the l2 sensitivity of the clip-and-noise mechanism for a pair of neighboring clients whose clipped updates are B and -B with ||B||_F=C; the output distributions are separated by 2C, so the noise scale must double to meet the same (ε,δ). A practical test is to run the released code with the paper's default ε=25 and measure membership-inference success on the noisy updates: an attacker success rate at chance level would support the claim, while near-perfect success would falsify it.","supporting_citations":[],"review_version":1}