Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-stage sketching lets federated LoRA update both adapters under differential privacy without the usual quadratic noise.

desk verdict Clever double-sketching aggregation, but the DP proof divides instead of sums per-client RDP, so the central privacy guarantee is unsupported as written. read the letter →

arxiv 2507.09990 v1 pith:RV5JH5WX submitted 2025-07-14 cs.CR cs.AI

Ming Wen , Jiaqi Zhu , Yuedong Xu , Yipeng Zhou , Dingding Han This is my paper
classification cs.CRcs.AI
keywords differentialprivacyfederatedlearninglow-rankadaptationLoRArandomizedSVDsketchinglargelanguagemodelsamplification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FedASK, a protocol for federated fine-tuning of large language models that applies differential privacy to LoRA while updating both low-rank adapters, not just one. Its central claim is that a two-stage sketching pipeline lets the server reconstruct the average of local LoRA updates exactly, and that adding calibrated noise only to the local B updates still gives an end-to-end (epsilon, delta)-differential-privacy guarantee. This would resolve the known dilemma in which noise on both adapters amplifies quadratically in the model update, while freezing one adapter reduces what the model can learn. On Llama-2 7B and 13B benchmarks, the paper reports that FedASK consistently beats fixed-matrix and basic federated baselines across privacy budgets and data distributions.

What carries the argument

The central mechanism is a two-stage projection pipeline in the spirit of randomized SVD. Stage one compresses every client's LoRA product into B_k(A_k Ω), where Ω is a shared n × (r+p) Gaussian matrix; the server's QR decomposition of the aggregated compressed sketches produces an orthonormal basis Q that captures the global update's subspace. Stage two aligns each client with that basis by sending (A_k)^T((B_k)^T Q), and the server's SVD of the aggregated second sketch produces the global factors. The machinery does two jobs at once: it keeps communication proportional to the sketching dimension r+p rather than the full adapter product, and it funnels privately learned information into both global matrices through the SVD, so the quadratic noise term from perturbing two adapters independently never appears.

What would settle it

Take two datasets that differ in one record, run the two-stage protocol many times, and measure the maximum divergence between the distributions of the second-stage sketches; if the observed divergence exceeds the bound claimed in the proof, the (epsilon, delta) guarantee fails.

Watch

Extended reading notes

Core claim

FedASK's discovery is that the server does not need the full adapter products to aggregate them privately. In the first sketching stage, each client sends B_k(A_k Ω) for a shared Gaussian projection matrix Ω, and the server aggregates these sketches and performs a QR decomposition to obtain an orthonormal basis Q that spans the global update subspace. In the second stage, each client sends (A_k)^T((B_k)^T Q), and the server's SVD of the aggregated second sketch yields global matrices B = Q U $Σ^{{1/2}}$ and A = $Σ^{{1/2}}$ V^T. The paper proves this reconstruction is exact: with over-sketching p ≥ dB − r + 2, the reconstructed global update equals (1/K) Σ_k B_k A_k in Frobenius norm, and the final matrices are (epsilon, delta)-differentially private when local B updates use DP-SGD noise. This is the claimed first combination of both-adapter learning, differential privacy, and exact aggregation in federated LoRA.

Load-bearing premise

The load-bearing premise is that the server-computed direction reused in the second projection is public information independent of each client's private data, even though it is actually constructed from the clients' first private projections.

Editorial extensions

If this is right

  • Federated LoRA can be trained under differential privacy without sacrificing the learnability that comes from updating both adapters.
  • Quadratic noise amplification is avoided because local DP noise is added to only one adapter matrix, while the server-side SVD still refreshes both.
  • When the over-sketching condition is met, the server's reconstructed global update exactly equals the average of the local LoRA updates.
  • Per-client communication remains proportional to the sketched dimension, keeping the protocol resource-comparable to single-adapter methods.
  • The privacy accounting composes DP-SGD noise over local steps, communication rounds, and client subsampling into an (epsilon, delta) guarantee for the final matrices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the proof's treatment of the broadcast basis as an external input needs a joint privacy-composition argument for the two sketches, because the basis is built from clients' first sketches and therefore carries information about their private data.
  • Editorial inference: a corrected composition would likely require more noise or a different protocol to reach the same epsilon, so the reported utility gains may need to be rechecked under that revised budget.
  • Editorial inference: the same two-stage sketching idea could transfer to other product-form parametrizations, such as factored adapters in vision transformers, whenever the global update is a sum of low-rank products.
  • Editorial inference: Theorem 2's exact equality is an ideal statement about low-rank products; under real clipping and gradient noise the local products are only approximately low-rank, so empirical near-exactness is the practical claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes FedASK, a federated LoRA framework that uses a two-stage sketching pipeline: clients first send B_k(A_k Omega), the server QR-decomposes the aggregate to obtain a basis Q, clients then send A_k^T B_k^T Q, and the server reconstructs global A and B via SVD. Local training is made differentially private by DP-SGD on B with A fixed. The paper claims two theoretical guarantees: an end-to-end (epsilon, delta)-DP guarantee for the final global matrices (Theorem 1) and an exact-aggregation guarantee (Theorem 2) under an over-sketching condition. Experiments on Llama-2-7B and Llama-2-13B report consistent gains over baselines across privacy budgets and heterogeneity settings.

Significance. If the two central theorems were correct, FedASK would be a valuable contribution: it would update both LoRA adapters under DP while avoiding the quadratic noise amplification of naive DP-LoRA, and it would provide a communication-efficient exact aggregation mechanism. The paper also ships a public code repository and includes error-bar experiments, which are strengths. However, both theoretical pillars have serious gaps as written. The privacy proof contains an unjustified division of the per-client RDP by the number of selected clients and relies on a false claim that the broadcast basis is external to the client's data; the aggregation theorem is stated for the full SVD while the algorithm truncates to rank r, making the claimed exactness impossible in general. Because the experiments' privacy budgets are tied to Theorem 1, the empirical privacy-utility claims are not supported.

major comments (4)
  1. [Appendix A.2, Eq. (21)] The proof divides the per-client RDP bound R_k by K_s to obtain R_agg for the aggregate, but no composition or amplification lemma is cited or proved for this step. In Algorithm 1 (lines 8-11 and 14-17) the server receives each client's individual sketches, so the mechanism's output includes the individual messages; under the standard parallel composition for disjoint client datasets the RDP of the joint mechanism is at most R_k, not R_k/K_s. If the authors intended a trusted aggregator or secure aggregation in which only the sums are released, the protocol must be modified and an additional analysis must show that the second-stage aggregate, which uses Q that depends on the first-stage aggregate, still enjoys the claimed amplification. This error propagates to the noise formula in Eq. (31), so the required sigma^2 is underestimated and the privacy budgets reported in Tables 2-4 are not justified.
  2. [Appendix A.2, Step 1] The statement that Q_t is 'an external input provided by the server' for the second sketch is inaccurate: Q_t is computed from the aggregate of the first sketches, including client k's own Yproj_k, so it depends on D_k. The proof does not apply Lemma 5 to compose the two releases (Yproj_k, tildeYproj_k); it simply assigns the same RDP bound R_k to both. A correct argument would need to show that the second sketch is post-processing of the first sketch or otherwise account for the dependence, but the text does not provide such an argument. As written, the paragraph does not establish the RDP guarantee for the pair of sketches released by a client.
  3. [Theorem 2 and Appendix A.3] The proof of Theorem 2 ignores the rank-r truncation in Algorithm 1 (lines 18-19). The final matrices satisfy rank(B_t A_t) <= r, while the target average Delta Wbar can have rank up to d_B > r; exact equality in Frobenius norm is then impossible. The proof analyzes the full SVD Q U Sigma V^T and never accounts for the selection of only U_r, Sigma_r, V_r. The qualifier 'before truncation' in the theorem statement does not describe any quantity actually produced by Algorithm 1. The claim should be restricted to settings where rank(Delta Wbar) <= r (e.g., the DP mode where A_k is fixed globally) or replaced by an approximation bound with a non-vanishing truncation error; as stated, Theorem 2 is false for the general algorithm.
  4. [Appendix A.3, Eq. (33)] The proof defines Yproj = (1/K) sum_k B_k(A_k Omega), but Algorithm 1 (line 10) aggregates the sum sum_k Yproj_k. The same discrepancy appears in the second-stage aggregate. Therefore the chain of equalities leading to Delta W^t = Q Q^T Delta Wbar does not correspond to the algorithm's actual aggregation, and a factor of K (the number of clients) is dropped. The theorem and the algorithm need to be made consistent with respect to averaging versus summation.
minor comments (4)
  1. [Abstract] The phrase 'amplifies synthetic noise on the model' appears to be a typo; it should probably read 'amplifies noise on the model'.
  2. [Appendix A.2, Lemma 7] Lemma 7 is invoked without verifying its assumptions (q_D = o(1) and the high-privacy regime) for the experimental batch sizes and dataset sizes; the proof should state the parameter regime in which the bound is valid.
  3. [Section 5.3] The empirical finding that p=0 achieves near-perfect aggregation fidelity is not explained in light of Theorem 2's over-sketching condition; if the effective rank of the average update is at most r in the DP setting, the paper should say so explicitly.
  4. [Table 1] The communication cost for FedASK is listed as O(K d_l r), but the protocol has two client-server exchanges per round; please clarify whether both sketches are counted in the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; FedASK's aggregation and privacy claims rest on external randomized-SVD and RDP results rather than reductions to fitted inputs or self-citations.

full rationale

FedASK's two claimed theoretical results are not circular. Theorem 2's exact-aggregation guarantee is derived by plugging the construction Yproj = ΔWbar Ω and Ytilde = ΔWbarᵀ Q into an external randomized-SVD error bound (Halko et al. Lemma 8 / [14]); the equality ΔWt = QQᵀ ΔWbar is algebraic by the SVD definition, and the condition p ≥ dB − r + 2 ensures the projection has no error, so the conclusion does not presuppose the target equality. Theorem 1's DP guarantee is built from standard RDP composition and subsampling lemmas (Mironov [28], Wang et al. [33], Noble et al. [29]) with the noise variance solved from the RDP conversion; no fitted constant is later renamed as a prediction. The manuscript's Appendix A.2 Step 1 does contain a serious gap: it treats Qt as an 'external input' for the second sketch, although Qt is computed from the aggregate of first sketches, so the per-client pair (Yproj_k, Ytilde_k) requires adaptive sequential composition; however, this is an unsupported step in the proof, not a circular reduction, and it is therefore a correctness risk rather than a circularity. There are no load-bearing self-citations, and no known result is merely renamed: the double-sketching pipeline is explicitly modeled on randomized SVD with attribution. Appendix B also acknowledges the local-A restriction as a limitation, not as an input-output identity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The main theoretical results rely on standard randomized SVD bounds, standard RDP composition lemmas, and the assumption that the broadcast basis Q is external to each client's private data. The privacy proof also implicitly assumes disjoint client datasets. No new physical or algorithmic entities are postulated. The proof's averaging error means the stated axiom set is not applied correctly in the privacy argument.

free parameters (4)
  • Over-sketching parameter p = 0 to 128 per experiment; 0 in Figure 3(b)
    Controls the sketch dimension r+p. Theorem 2 exactness requires p >= dB-r+2, yet the paper claims near-perfect fidelity at p=0, which is only plausible if the aggregated update has rank at most r in the tested setting.
  • LoRA rank r = 64 for Llama-2-7B, 128 for Llama-2-13B
    Set by hand; controls the subspace dimension and noise sensitivity, and interacts with p in the aggregation guarantee.
  • LoRA scaling factor alpha = 2r
    Chosen as twice the rank, a standard LoRA heuristic, and not derived from first principles.
  • Gradient clipping constant C = 1.0
    Assumed as the sensitivity bound in Theorem 1 and used as the clipping threshold in experiments.
assumptions (4)
  • standard math The randomized SVD tail bound (Lemma 8) applies to the aggregated update Delta_W_bar and gives zero error when the sketch dimension exceeds the rank.
    Imported from Halko et al. [14] and used in the proof of Theorem 2 in Appendix A.3.
  • domain assumption Client datasets are disjoint, so the Renyi privacy loss of the aggregate can be obtained from the per-client losses.
    Used in Appendix A.2 Step 2. The proof applies it incorrectly by averaging instead of summing, but disjointness itself is standard in federated learning.
  • ad hoc to paper The server-computed basis Q can be treated as an external and public input for the second sketch, requiring no additional privacy composition for client k.
    Appendix A.2 Step 1 states that Qt is an external input. This is load-bearing for the privacy proof and is questionable because Q depends on the client's own first sketch.
  • domain assumption Gradient sensitivity is normalized to C=1 for the local DP-SGD updates.
    Assumed at the start of Theorem 1 and enforced in experiments by clipping to 1.0.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix." pith.science (2026). https://pith.science/paper/RV5JH5WX

@misc{pith2026250709990,
  author       = {Pith},
  title        = {Pith review of: Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RV5JH5WX}},
  note         = {Machine review of arXiv:2507.09990}
}
read the original abstract

Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers a computationally efficient approach by training low-rank adapters. LoRA is also communication-efficient for federated LLMs when multiple users collaboratively fine-tune a global LLM model without sharing their proprietary raw data. However, even the transmission of local adapters between a server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adapters amplifies synthetic noise on the model, while fixing one adapter impairs the learnability of fine-tuning. In this paper, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double Sketching) , a novel federated LoRA framework to enable effective updating of both low-rank adapters with robust differential privacy. Inspired by randomized SVD, our key idea is a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy-preserving local updates, and then reconstructs the global matrices on the server to facilitate effective updating of both adapters. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions.

Figures

Figures reproduced from arXiv: 2507.09990 by the authors.

Figure 1
Figure 1. The dilemma of differential privacy with Federated LoRA: standard federated LoRA amplifies model noise, while fixing one adapter causes insufficient learnability. billions of parameters, renders full-parameter fine-tuning prohibitive for local clients in FL due to severe memory, computation, and communication constraints [42]. To overcome these limitations, Parameter-Efficient Fine-Tuning (PEFT) methods, particularl… view at source ↗
Figure 2
Figure 2. FedASK Pipeline. DP theoretically ensures that this algorithm’s output is nearly unaffected by the presence or absence of any single individual’s data in the dataset. In the context of federated learning, DP is commonly achieved through DP-Stochastic Gradient Descent (DP-SGD)[1, 22]. This method adds calibrated noise to the gradients computed at each client before aggregation. Applying DP-SGD directly to the standar… view at source ↗
Figure 3
Figure 3. Impact of over-sketching p on FedASK. (a) MMLU score of FedASK versus p. (b) Aggregation fidelity of FedASK (with p = 0) and (c) FedAvg, measured by cosine similarity with the ideal mean of local updates (1.0 indicates perfect fidelity). Subplots (b) and (c) vary selected clients (Ks) and Non-IID degrees. The empirical results, presented in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Performance of Llama 2-7B on IID data across LoRA ranks and differential privacy (DP) [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Performance of Llama 2-7B on Non-IID data ( [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Performance of Llama 2-13B on IID data across LoRA ranks and differential privacy (DP) [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Performance of Llama 2-13B on Non-IID data ( [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Impact of sketching dimension (x-axis) and non-IID degree (y-axis, Dirichlet α / IID) on FedASK’s aggregation fidelity (cosine similarity) for (a) 5, (b) 10, (c) 15, and (d) 20 clients, showing robust near-unity performance. fidelity evaluations, differential privacy m…
Figure 9
Figure 9. Figure 9: System resource utilization for five federated learning algorithms when training Llama 2-7B (left) and Llama 2-13B (right) models using 4-bit precision. The metrics, shown for a single client in a 5-client federated setup, include uplink and downlink communication volu…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Probing Memorization of Tabular In-Context Learning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    A new probing framework detects moderate parametric memorization signals in tabular in-context learning models under single-task fine-tuning, strongest on low-cardinality tasks, but signals largely disappear under rea...

  2. FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA

    cs.CR 2026-08 conditional novelty 5.0 of 10

    FedGSA aggregates differentially private federated LoRA updates as subspaces on a Grassmann manifold, improving GLUE and E2E NLG accuracy over Euclidean averaging baselines.

Reference graph

Works this paper leans on

43 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    Abadi, A

    M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Com- puter and Communications Security , CCS ’16, page 308–318, New York, NY , USA, 2016. Asso- ciation for Computing Machinery. ISBN 9781450341394. doi: 10.1145/2976749.2978318. ...

    arXiv 2016

  2. [2]

    Aghajanyan, L

    A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. URL https://arxiv.org/abs/2012.13255

    arXiv 2020

  3. [3]

    Babakniya, A

    S. Babakniya, A. R. Elkordy, Y . H. Ezzeldin, Q. Liu, K.-B. Song, M. El-Khamy, and S. Avestimehr. Slora: Federated parameter efficient fine-tuning of language models, 2023. URL https://arxiv.org/abs/ 2308.06522

    arXiv 2023

  4. [4]

    Balle, G

    B. Balle, G. Barthe, M. Gaboardi, J. Hsu, and T. Sato. Hypothesis testing interpretations and renyi differential privacy. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research, pages 2496–2506. PMLR, 26–28 Aug 2020....

    work page 2020

  5. [5]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, et al. Evaluating Large Language Models Trained on Code. arXiv e-prints, art. arXiv:2107.03374, July 2021. doi: 10. 48550/arXiv.2107.03374

    arXiv doi:10.48550/arxiv.2107.03374 2021

  6. [6]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training Verifiers to Solve Math Word Problems. arXiv e-prints, art. arXiv:2110.14168, Oct. 2021. doi: 10.48550/arXiv.2110.14168

    arXiv doi:10.48550/arxiv.2110.14168

  7. [7]

    D. Dua, Y . Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. arXiv e-prints, art. arXiv:1903.00161, Mar

    arXiv 1903

  8. [8]

    C. Dwork. Differential privacy. In M. Bugliesi, B. Preneel, V . Sassone, and I. Wegener, editors,Automata, Languages and Programming, pages 1–12, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-35908-1

    work page 2006

Show all 43 references
  1. [9]

    Dwork and A

    C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014. ISSN 1551-305X. doi: 10.1561/0400000042. URL http://dx.doi.org/10.1561/0400000042

    2014 doi

  2. [10]

    Dwork, G

    C. Dwork, G. N. Rothblum, and S. Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 51–60, 2010. doi: 10.1109/FOCS.2010.12

    2010 doi

  3. [11]

    J. Fu, Y . Hong, X. Ling, L. Wang, X. Ran, Z. Sun, W. H. Wang, Z. Chen, and Y . Cao. Differentially private federated learning: A systematic review. arXiv preprint arXiv:2405.08299, 2024

    2024

  4. [12]

    P. Guo, S. Zeng, Y . Wang, H. Fan, F. Wang, and L. Qu. Selective aggregation for low-rank adaptation in federated learning, 2025. URL https://arxiv.org/abs/2410.01463. 10

    2025 arXiv

  5. [13]

    Z. Guo, Y . Zhang, Z. Zhang, Z. Xu, and I. King. FedLFC: Towards efficient federated multilingual modeling with LoRA-based language family clustering. In K. Duh, H. Gomez, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 151...

    2024 doi

  6. [14]

    Halko, P

    N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217–288, 2011. doi: 10.1137/ 090771806. URL https://doi.org/10.1137/090771806

    2011 doi

  7. [15]

    Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2403.14608

    2024 arXiv

  8. [16]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring Massive Multitask Language Understanding. arXiv e-prints, art. arXiv:2009.03300, Sept. 2020. doi: 10.48550/ arXiv.2009.03300

    2009 doi arXiv

  9. [17]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv e-prints, art. arXiv:2103.03874, Mar. 2021. doi: 10.48550/arXiv.2103.03874

    2021 doi arXiv

  10. [18]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. LoRA: Low- Rank Adaptation of Large Language Models. arXiv e-prints, art. arXiv:2106.09685, June 2021. doi: 10.48550/arXiv.2106.09685

    2021 doi arXiv

  11. [19]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  12. [20]

    T. Kang, Z. Wang, H. He, J. Zhang, S. Song, and K. B. Letaief. Federated low-rank adaptation with differential privacy over wireless networks, 2024. URL https://arxiv.org/abs/2411.07806

    2024 arXiv

  13. [22]

    Lee and D

    J. Lee and D. Kifer. Concentrated differentially private gradient descent with adaptive per-iteration privacy budget. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1656–1665, New York, NY , USA, 2018. Associa...

    2018

  14. [23]

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020. doi: 10.1109/MSP.2020.2975749

    2020

  15. [24]

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith. Federated optimization in heterogeneous networks. In I. Dhillon, D. Papailiopoulos, and V . Sze, editors,Proceedings of Machine Learning and Systems , volume 2, pages 429–450, 2020. URL https://proceedings....

    2020

  16. [25]

    X.-Y . Liu, R. Zhu, D. Zha, J. Gao, S. Zhong, M. White, and M. Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Trans. Manage. Inf. Syst., 16(2), Mar

  17. [26]

    Lukas, A

    N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363, 2023. doi: 10.1109/SP46215.2023.10179300

    2023

  18. [27]

    H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. arXiv e-prints, art. arXiv:1602.05629, Feb. 2016. doi: 10.48550/arXiv.1602.05629

    2016 doi arXiv

  19. [28]

    I. Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263–275, 2017. doi: 10.1109/CSF.2017.11. 11

    2017 doi

  20. [29]

    Noble, A

    M. Noble, A. Bellet, and A. Dieuleveut. Differentially private federated learning on heterogeneous data. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors,Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedin...

    2022

  21. [30]

    Praneeth Karimireddy, S

    S. Praneeth Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. Theertha Suresh. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. arXiv e-prints, art. arXiv:1910.06378, Oct. 2019. doi: 10.48550/arXiv.1910.06378

    1910 doi arXiv

  22. [31]

    Y . Sun, Z. Li, Y . Li, and B. Ding. Improving lora in privacy-preserving federated learning, 2024. URL https://arxiv.org/abs/2403.12313

    2024 arXiv

  23. [32]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv. org/abs/2307.09288

    2023 arXiv

  24. [33]

    Y .-X. Wang, B. Balle, and S. P. Kasiviswanathan. Subsampled renyi differential privacy and analytical moments accountant. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 o...

    2019

  25. [34]

    Z. Wang, Z. Shen, Y . He, G. Sun, H. Wang, L. Lyu, and A. Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations, 2024. URL https://arxiv.org/abs/2409.05976

    2024 arXiv

  26. [35]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models. arXiv e-prints, art. arXiv:2201.11903, Jan. 2022. doi: 10.48550/arXiv.2201.11903

    2022 doi arXiv

  27. [36]

    R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y . Xu, Y . Du, Y . Wang, and S. Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page...

    2024

  28. [37]

    X. Yin, Y . Zhu, and J. Hu. A comprehensive survey of privacy-preserving federated learning: A taxonomy, review, and future directions. ACM Comput. Surv., 54(6), July 2021. ISSN 0360-0300. doi: 10.1145/ 3460427. URL https://doi.org/10.1145/3460427

    2021 doi

  29. [38]

    L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y . Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Meta- Math: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv e-prints, art. arXiv:2309.12284, Sept. 2023. doi: 10.48550/arXiv.2309.12284

    2023 doi arXiv

  30. [39]

    Zhang, S

    J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y . Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915–6919, 2024. doi: 1...

    2024

  31. [40]

    Zhang, S

    J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, Y . Zhou, G. Wang, and Y . Chen. Towards building the federated gpt: Federated instruction tuning, 2024. URL https://arxiv.org/abs/2305.05644

    2024 arXiv

  32. [41]

    Zhang, L

    L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning, 2024. URL https://openreview.net/forum?id=RbKThNNFxr

    2024

  33. [42]

    Zhang, L

    S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang. Instruction tuning for large language models: A survey, 2024. URL https://arxiv.org/abs/2308. 10792. A Theoretical Proof A.1 Proof of Lemma 1 Proof. Let At ∈ Rr×dl and Bt ∈ Rdl×r b...

    2024

  34. [2019]

    doi: 10.48550/arXiv.1903.00161

    1903 doi arXiv

  35. [2025]

    doi: 10.1145/3682068

    ISSN 2158-656X. doi: 10.1145/3682068. URL https://doi.org/10.1145/3682068

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.