REVIEW 3 major objections 5 minor 15 references
FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read FedGSA replaces factor-wise Euclidean averaging of LoRA updates with basis-invariant subspace aggregation on the Grassmann manifold, and argues this lifts differentially private federated fine-tuning accuracy without extra privacy cost.
desk verdict Plausible method and decent experiments, but the privacy theorem has the inequality backwards and the convergence proof is missing; needs major revision before acceptance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the Grassmann manifold $Gr(r, d_{\mathrm{out}})$—the set of $r$-dimensional subspaces of $\mathbb{R}^{d_{\mathrm{out}}}$—with each client update represented by a rank-$r$ orthogonal projection matrix $P_k = U_k U_k^\top$. The server approximates the manifold's Fréchet mean by the extrinsic weighted average of these projections followed by a top-$r$ eigenvalue decomposition; the resulting eigenspace is the consensus update direction. A thin SVD of $U_g^\top \bar M$ then reconstructs LoRA factors inside that subspace. This replaces Euclidean factor averaging with subspace averaging, so the aggregation is invariant to the reparameterization $\Delta W = (BQ)(Q^{-1}A)$.
What would settle it
A concrete check has two parts. First, for a fixed setting of $T$, $q_K$, $q_D$, $m$, $\epsilon$, and $\delta$, compute the lower bound on $\sigma^2$ required by DP-SGD composition; if the stated $O(\cdot)$ condition can be met while $\sigma^2$ is below that lower bound, the privacy theorem as written cannot be sound. Second, on a synthetic heterogeneous split, compare the Grassmann geodesic distance from the top-$r$ eigenspace of $\bar P$ to the subspace of the true average update against the same distance for the Euclidean-averaged factors; if the geometric estimate is not the closer one, ba
Extended reading notes
Core claim
The central discovery, on the paper's own terms, is that the non-uniqueness of low-rank factorization, not just DP noise, is a primary source of aggregation error in federated LoRA, and that this non-uniqueness can be removed by changing the object being aggregated. Each client's privatized low-rank update $M_k = B_k A$ is summarized by the rank-$r$ column subspace $\mathrm{span}(U_k)$ from its truncated SVD, represented by the projection $P_k = U_k U_k^\top$, a point in the Grassmann manifold $Gr(r, d_{\mathrm{out}})$. The server forms the weighted average $\bar P = \sum w_k P_k$ and takes the top-$r$ eigenspace $U_g$ as the global output-direction subspace, then reconstructs global factors
Load-bearing premise
The load-bearing premise is that every server-side operation—averaging projection matrices, taking the top eigenspace, reconstructing factors—is information-free post-processing of already-privatized client messages, and that the injected noise is large enough in the direction DP actually requires; if either fails, the 'no additional privacy loss' claim collapses.
Editorial extensions
If this is right
- The privacy guarantee of federated LoRA does not have to be weakened when the server does geometric aggregation: if the client messages are DP, all later processing can be treated as post-processing.
- Freezing one LoRA factor no longer has to cost expressiveness; the subspace step recovers directional information that factor-wise Euclidean averaging loses.
- Under stronger non-IID splits, the method should hold up better than Euclidean baselines because it ignores coordinate misalignment.
- The reconstructed orthogonal factors can improve conditioning and stability of subsequent local updates.
- The reported gains carry over to generation tasks, not only classification.
Reading between the lines
- Because the server only aggregates column spaces, singular-value magnitudes and input-side directions are discarded in the consensus step; one testable extension is to weight each client's projection by its singular values so dominant clients influence the subspace more.
- The extrinsic projection average is only an approximation to the intrinsic Fréchet mean; under large DP noise or widely separated client subspaces, the top eigenspace of $\bar P$ could blend distinct subspaces, and a robust geodesic mean might behave differently.
- The same basis-invariant aggregation idea could apply to other low-rank adapters or to gradient subspaces in federated learning generally, although the paper only claims it for LoRA.
- An editorial check follows from the theorem as written: recalculating the claimed privacy guarantee with the required lower bound on $\sigma^2$, rather than the stated upper-bound form, would reveal whether the reported $\epsilon$ values are actually the ones being spent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FedGSA, a federated LoRA fine-tuning method with differential privacy. Clients locally run DP-SGD on the LoRA B-factor, transform their privatized low-rank update M_k = B_k A into an r-dimensional subspace, and send the orthogonal projection matrix P_k. The server computes a weighted Euclidean average of the P_k, takes its dominant r-dimensional eigenspace as a global update subspace, projects the matrix-level average of client updates onto that subspace, and reconstructs new global LoRA factors via a thin SVD of the projected matrix. The authors claim that all server-side operations are post-processing and therefore incur no additional privacy loss, and they state that convergence can be established under standard assumptions. Experiments on four GLUE tasks and the E2E NLG benchmark report consistent improvements over six baselines, with the headline gains of 2.17% and 2.27% average accuracy over the strongest baseline at epsilon=6 and epsilon=3.
Significance. The geometric idea of aggregating client LoRA updates as Grassmann subspaces rather than as coordinate-dependent factors is interesting and potentially useful for federated LoRA, where factorization ambiguity is a known aggregation problem. The empirical evaluation is reasonably broad: it includes DP and non-DP settings, a heterogeneity sweep, and an NLG benchmark, and the reported gains are consistent across tasks and budgets. The paper also correctly identifies that server-side processing of privatized client messages is post-processing. However, the paper's formal support is not at the level promised by the abstract: the privacy theorem is stated incorrectly (with an upper bound instead of a lower bound on the noise variance and a suspicious 1/K dependence), and the convergence claim is never instantiated as a theorem or proof. These issues affect the two main formal contributions, so the current version cannot be accepted as a rigorous privacy paper, regardless of the empirical promise.
major comments (3)
- [Theorem 1 / Eq. (18)] The stated privacy condition is backwards. Eq. (18) asserts that DP holds when σ² = O( q_D² m q_K T log(...) / (ε² K) ), i.e., an upper bound on the injected noise variance. A DP guarantee requires the noise to be sufficiently large, so the condition must be a lower bound, roughly σ² ≥ Ω(·). As written, σ²=0 satisfies the condition, so the theorem is vacuous and cannot certify privacy for any experiment. Additionally, the appearance of 1/K is suspicious: if q_K is a client sampling ratio, the standard federated DP-SGD analysis (amplification by client sampling plus composition over T rounds) does not yield a per-record privacy loss that improves by a factor of K. The authors need to provide a correct composition analysis and a correct inequality direction.
- [Abstract / Conclusion / full text] The abstract and conclusion state that FedGSA 'proves no additional privacy loss' and 'establishes convergence under standard assumptions,' but the manuscript contains no proof of Theorem 1 and no convergence theorem, assumption list, or proof at all. The text jumps from the theorem statement directly to experiments. If these formal claims remain part of the contribution, the authors must state and prove them, or else remove the claims and present the method as empirically supported.
- [Method / Eq. (10)] The aggregation weights w_k in Eq. (10) are never defined. The 'no additional privacy loss' argument depends on these weights being either public constants (e.g., 1/K) or functions of the already-privatized messages. If w_k depend on private information such as local dataset sizes, local losses, or data quality, the post-processing argument does not automatically apply. Please specify how w_k are chosen in the algorithm and in all experiments, and state what the privacy analysis assumes about them.
minor comments (5)
- [Experimental Setups] The setup is under-specified: no LoRA rank for NLU experiments, number of clients, local epochs/steps, batch size, learning rate, communication rounds, or DP-SGD noise schedule are reported. This hampers reproducibility and makes the comparison to baselines harder to interpret.
- [Figure 3] The x-axis labels in Figure 3 appear garbled ('/s945 /s32/s61/s32/...'). Please fix the rendering.
- [References] Several references are dated 2026 (e.g., Kim et al. 2026, Liu et al. 2026, Zhang et al. 2026). Please verify that these works exist and are cited with correct years/venues.
- [Eq. (18)] The notation 'q2_D' is ambiguous. Define q_D, q_K, T, K and state clearly whether q_D and q_K are sampling ratios or absolute counts.
- [Table 2] The caption says 'five GLUE tasks' but lists four datasets (MNLI has two splits). Clarify the counting.
Circularity Check
No material circularity: FedGSA's privacy claim is a standard post-processing argument and its utility gains are empirical; the flagged Theorem 1 issue is a formal correctness gap, not a circular derivation.
full rationale
I walked the claimed derivation chain: (i) the 'no additional privacy loss' claim rests on server-side subspace extraction, aggregation, and reconstruction being post-processing of already DP-privatized client messages; this is the standard post-processing property of DP, not an identity with the paper's own assumptions. (ii) The convergence claim is asserted as 'under standard assumptions' with no theorem or proof supplied, which is a missing-support issue, not circularity. (iii) The empirical gains are measured against external baselines on GLUE and E2E, with no fitted parameter that is later renamed a prediction. (iv) The only self-reference I can identify is a background citation to Zhang et al. 2023 (which happens to include two of the present authors) for the generic point that pretrained models often need task-specific fine-tuning; that citation is not load-bearing for any of the paper's technical claims. I also considered Theorem 1's privacy condition: Eq. (18) states the injected DP-SGD noise variance satisfies σ² = O(·), whereas a DP guarantee needs a lower bound; as written the condition is vacuous and cannot certify privacy. This is a serious correctness flaw in the paper's advertised formal privacy claim, but it is not circularity—the theorem does not define the conclusion in terms of itself, nor does it rename fitted inputs as predictions. Therefore the circularity score is 0; the vacuous-Theorem-1 concern belongs under correctness risk rather than under circularity.
Assumptions & free parameters
free parameters (4)
- LoRA rank r =
r = 4 for NLG; unspecified for GLUE
- aggregation weights w_k =
not specified in the paper
- clipping constant C =
1
- local steps m, client sampling ratio q_K, data sampling ratio q_D =
not specified in the text
assumptions (3)
- standard math DP-SGD provides (epsilon, delta)-DP when noise variance is set according to the moments accountant (Abadi et al. 2016)
- domain assumption The extrinsic mean of projection matrices (top-r eigenspace of the weighted average) is a good proxy for the Frechet mean on the Grassmann manifold
- domain assumption Convergence under standard assumptions (e.g., smoothness, bounded gradient variance)
Cite this review
Pith. "Pith review of FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA." pith.science (2026). https://pith.science/paper/Z2AXPVY6
@misc{pith2026260803267,
author = {Pith},
title = {Pith review of: FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z2AXPVY6}},
note = {Machine review of arXiv:2608.03267}
}
abstract
Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $\epsilon=6$ and $\epsilon=3$, respectively.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
Federated Low-Rank Adaptation with Differential Privacy over Wireless Networks
Federated Low-Rank Adaptation with Dif- ferential Privacy over Wireless Networks.arXiv preprint arXiv:2411.07806. Kim, M.; Park, S.; Han, Y.; and Ahn, J.-H
-
[8]
Differentially private low-rank adapta- tion of large language model using federated learning.ACM Transactions on Management Information Systems, 16(2): 1–24. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy,O.;Lewis,M.;Zettlemoyer,L.;andStoyanov,V.2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692...
arXiv 2019
-
[10]
Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S
Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S
-
[12]
Williams, A.; Nangia, N.; and Bowman, S
Dif- ferentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix.arXiv preprint arXiv:2507.09990. Williams, A.; Nangia, N.; and Bowman, S
-
[13]
A broad-coveragechallengecorpusforsentenceunderstanding through inference. InProceedings of the 2018 conference oftheNorthAmericanchapteroftheassociationforcompu- tationallinguistics:humanlanguagetechnologies,volume1 (long papers), 1112–1122. Yin,X.;Zhu,Y.;andHu,J.2021.Acomprehensivesurveyof privacy-preserving federated learning: A taxonomy, review, and f...
work page 2018
-
[14]
FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
FedRot- LoRA: Mitigating Rotational Misalignment in Federated LoRA.arXiv preprint arXiv:2602.23638. Zhang, J.; Vahidian, S.; Kuo, M.; Li, C.; Zhang, R.; Yu, T.; Wang, G.; and Chen, Y
-
[15]
Towards building the federatedgpt:Federatedinstructiontuning. InICASSP2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6915–6919. IEEE. Zhang, S.; Dong, L.; Li, X.; Zhang, S.; Sun, X.; Wang, S.; Li, J.; Hu, R.; Zhang, T.; Wang, G.; et al
work page 2024
-
[353]
Geng, J.; Mou, Y.; Li, Q.; Li, F.; Beyan, O.; Decker, S.; andRong,C.2023. Improvedgradientinversionattacksand defenses in federated learning.IEEE Transactions on Big Data, 10(6): 839–850. Golub, G. H.; and Van Loan, C. F. 2013.Matrix computa- tions. JHU press. Guo, Z.; Zhang, Y.; Zhang, Z.; Xu, Z.; and King, I
work page 2023
Show all 15 references
-
[2016]
InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, 308–318
Deep learn- ing with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, 308–318. Absil,P.-A.;Mahony,R.;andSepulchre,R.2008.Optimiza- tion algorithms on matrix manifolds. Princeton University Press. Boyd,S.;andVandenbe...
2016 arXiv
-
[2018]
In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, 353–
GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, 353–
2018
-
[2019]
Socher,R.;Perelygin,A.;Wu,J.;Chuang,J.;Manning,C.D.; Ng, A
Naturallanguageunderstandingwiththequoraquestionpairs dataset.arXiv preprint arXiv:1907.01041. Socher,R.;Perelygin,A.;Wu,J.;Chuang,J.;Manning,C.D.; Ng, A. Y.; and Potts, C
1907 arXiv
-
[2020]
IEEE signal processing magazine, 37(3): 50–60
Fed- erated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3): 50–60. Liu,J.;Miao,Y.;Xi,N.;andLiu,J.2026. RethinkingLoRA forPrivacy-PreservingFederatedLearninginLargeModels. arXiv preprint arXiv:2602.19926. Liu, X.-Y.; Zhu, R.; Zha, D...
2026 arXiv
-
[2024]
Parameter-efficient fine-tuning for large models: A compre- hensive survey.arXiv preprint arXiv:2403.14608. Horn, R. A.; and Johnson, C. R. 2012.Matrix analysis. Cambridge university press. Huang, Y.; Gupta, S.; Song, Z.; Li, K.; and Arora, S
2012 arXiv
-
[2025]
arXiv preprint arXiv:2505.12805
FedSVD: Adaptive Or- thogonalization for Private Federated Learning with LoRA. arXiv preprint arXiv:2505.12805. Li, T.; Sahu, A. K.; Talwalkar, A.; and Smith, V
-
[2026]
Lee,S.;Park,S.;Lee,D.B.;Wagner,D.;Seong,H.;Bocklet, T.; Lee, J.; and Hwang, S
Adap- tive Selection of LoRA Components in Privacy-Preserving Federated Learning.arXiv preprint arXiv:2605.05769. Lee,S.;Park,S.;Lee,D.B.;Wagner,D.;Seong,H.;Bocklet, T.; Lee, J.; and Hwang, S. J
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.