Pith. sign in

REVIEW 5 major objections 5 minor 55 references

Federated LoRA aggregation fails when one client's adapter fires on another's data; Dysco prevents this by pinning each client's update to a server-chosen subspace.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:17 UTC pith:XIYTL24L

load-bearing objection Clean theoretical core about LoRA interference, but the headline empirical gains look like they may come from per-client personalization rather than the server-side subspace merge, and the synthetic experiments use oracle subspaces. the 5 major comments →

arxiv 2607.14367 v1 pith:XIYTL24L submitted 2026-07-15 cs.LG

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Haobo Zhang , Jiankun Wang , Suraj Rajendran , Weishen Pan , Lam Tsoi , Yong Chen , Fei Wang , Jiayu Zhou This is my paper
classification cs.LG
keywords federated learningLoRAlow-rank adaptationsubspace allocationdata heterogeneitydata-parameter interferencesubspace boostingconvergence analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the instability of LoRA-based federated learning under heterogeneous clients has a geometric cause: after aggregation, an adapter trained on one client acts on other clients' representations, producing 'data-parameter interference' that scales with the alignment between LoRA update subspaces and client activations. The paper proposes to treat aggregation as subspace allocation, not just parameter averaging, and introduces Dysco, which fixes each client's right LoRA factor to a server-merged subspace that is insensitive to other clients' features. The paper proves that this fixes the A-drift by construction, collapsing a bilinear interference term into a linear one and tightening the per-round convergence bound. Empirically it reports up to a 9x lower final training loss on synthetic federated tasks and consistent accuracy gains across five FL algorithms on real clinical-note classification, with roughly 0.9% wall-clock overhead. A sympathetic reader cares because the mechanism is plug-in: any existing FL algorithm can adopt the subspace allocation without changing the optimization dynamics.

Core claim

The central discovery is a structural identity: in FedAvg-LoRA, the aggregation error between the averaged factors and the averaged updates is bilinear in the cross-client drifts of both LoRA factors, ΔA ΔB. By fixing the right factor A on a per-client basis to a server-merged subspace before local training and keeping it fixed during global application, Dysco makes the A-drift identically zero, leaving an interference term proportional to ρA ΔB—linear in the left-factor drift alone. The merged subspace is computed via a closed-form eigenvalue problem: each client sends a basis for the least-sensitive directions of its activations, and the server allocates to each client the top eigenvectors

What carries the argument

Data-parameter interference: the cross term B2 A2 h1 that appears when client 2's adapter is applied to client 1's activation after merging. The paper's mechanism is subspace allocation via activation-insensitive bases: client i computes an orthonormal basis A*_i spanning the bottom-r eigenvectors of its local activation Gram matrix H_i^T H_i (directions that respond least to its own data), sends only the basis; the server forms a merged subspace A_merge^i from the top-r eigenvectors of S_i = sum_{j≠i} A*_j^T A*_j, maximizing overlap with other clients' insensitive directions. The key identity is that A_merge^i is used both locally and globally within a round, so the cross-client drift in A

Load-bearing premise

The entire argument depends on the right LoRA factor being identical during local training and at aggregation; the paper itself relaxes the orthonormality constraint during local training, so the elimination of A-drift is approximate, and if that relaxation is large the claimed linear bound is no longer guaranteed.

What would settle it

Run a two-client federation where the clients' activation subspaces overlap substantially; after one round, compute the empirical interference term ||B_2 A_2 h_1|| on client 1's holdout activations for both FedAvg-LoRA and Dysco. If Dysco's interference does not drop to a level consistent with a fixed A (i.e., does not scale with B-drift only, once local training relaxes the constraint), the structural mechanism is not operating as claimed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • In FedAvg-LoRA the aggregation error is bilinear in both factor drifts; with Dysco's fixed right factor it becomes linear in the B drift alone, tightening the per-round stationarity bound by a factor Θ(ΔA/ρA).
  • The relative advantage grows as Θ(K^2) with the number of local SGD steps, so the method becomes more attractive when clients train longer locally.
  • Dysco is a plug-in: applying it on top of FedAvg, FedAvgM, FedProx, Scaffold, and FedNova improves average task accuracy on the clinical-note benchmark in the paper's experiments, by up to 4.3 percentage points.
  • In the synthetic setting with orthogonally partitioned client subspaces, the final-round training loss drops by up to about 9x compared with FedAvg-LoRA, and the per-round communication cost remains just the low-rank bases plus the adapter's B factor.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The unbounded model growth—one frozen adapter block per round, so the final model has R adapters—is not discussed in the paper; a real deployment with bounded final model size would need a pruning/merging step, and that step would change the reported overhead and possibly the convergence behavior.
  • Since the mechanism only requires controlling the alignment of update subspaces with activations, a natural extension is to apply the same subspace-allocation view to other parameter-efficient methods (prefix tuning, side adapters) or to heterogeneous architectures via distillation, though the paper only claims the LoRA case.
  • The theory identifies a strict-orthogonal regime (N·r ≤ d); when that fails, the paper's own experiments show partial degeneration toward FedAvg. An empirical test of the method's value in the overlapping-subspace regime would clarify how much of the gain is due to the orthogonal construction versus the linear-drift collapse.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This manuscript studies cross-client interference in federated LoRA fine-tuning. It observes that a merged model W+B1A1+B2A2 evaluated on client 1's activation h1 contains a cross term B2A2h1, and proposes to control this term by allocating the row space of each client's right LoRA factor to directions that are insensitive to other clients' activations. Dysco has three components: local SVD-based selection of an activation-insensitive subspace A_i^* (Eq. 2), server-side merging of these bases into A_i^{merge} (Eqs. 3-4), and multi-round accumulation of frozen LoRA blocks to handle representation drift (Eq. 5). The theoretical section (Appendix B) embeds an 'interference' term in a nonconvex stationarity bound and claims that Dysco's bound is tighter because A-drift is eliminated by construction, reducing a bilinear Delta_A Delta_B term to a linear rho_A Delta_B term. Experiments include synthetic federated tasks and MIMIC-IV/GLUE classification with Llama-3.2-1B/RoBERTa-large.

Significance. If correct, the paper would offer a new geometric view of federated LoRA aggregation as subspace allocation rather than parameter averaging, with a plug-in mechanism applicable to standard FL algorithms. The strengths are real: the polarization identity in Eq. (7) cleanly isolates the bilinear cross-client term in FedAvg-LoRA; the subspace selection and merging problems have closed-form solutions; the implementation is plug-in and code is released. However, as reported, the evidence is not yet sufficient to support the central attribution of the gains to server-side subspace merging. The oracle-subspace construction in the synthetic experiments and the ablation result in Table 6 are load-bearing, and the theoretical comparison is against different ideal models for the two algorithms. With additional experiments and a corrected comparison, the idea could be a useful contribution; in its current form the main narrative is not established.

major comments (5)
  1. [Appendix C.1] The synthetic experiments do not exercise the proposed subspace-selection and merging pipeline. The setup states that for Dysco, 'A_i is set to U_i^T' and 'Dysco's A_i is fixed across rounds,' so A_i is the oracle subspace used to generate client i's data, not the output of Eq. (2) or the merging step in Eqs. (3)-(4). Consequently Figs. 2-5 and the 'up to 9 times' claim in the abstract validate an idealized orthogonal partition, not the actual method. Please rerun the synthetic benchmark with the real local-SVD plus server-merge pipeline (and with a w/o-Merging ablation), or clearly label these experiments as an oracle upper bound.
  2. [Table 6 / Appendix D.2] The ablation contradicts the central mechanism. At round 10 on MIMIC-IV-note, full Dysco averages 82.5, w/o Merging averages 82.3, w/o Boosting averages 82.6, and baseline FedAvg averages 80.8. Removing server-side merging--the component advertised as reducing cross-client interference--hardly changes accuracy, and removing boosting is slightly better. This suggests the measurable improvement over FedAvg comes from per-client A personalization (local SVD), not from the subspace-allocation/merging mechanism. The authors should report final-round ablations and add a variant that removes per-client A personalization (e.g., a random A fixed per client) to separate the two effects.
  3. [Appendix B.2.1, Eqs. (6)-(8)] The theoretical comparison is not apples-to-apples. For FedAvg-LoRA the 'ideal' is a shared global model W0 + (1/N) sum_i B_i A_i, while for Dysco the ideal is client-specific W0 + B_i A_merge^i and the 'actual' is also client-specific W0 + bar{B} A_merge^i. Corollary B.1 therefore bounds stationarity with respect to different references, and Theorem B.1's global f(W) does not directly apply if the deployed Dysco model is per-client. A meaningful comparison would state a common objective, global or personalized, and compute each algorithm's interference relative to that single objective.
  4. [Sec. 3.2, Eq. (5); Algorithm 1] The final model definition is inconsistent with the aggregation step. Eq. (5) writes W = W0 + sum_t B_t^i A_merge^{i,t} with a client index i on the right-hand side, but a single global W cannot depend on i. Algorithm 1 line 18 aggregates B_i to bar{B}, but Eq. (5) never uses bar{B}. If inference uses client-specific A_merge^i with a shared bar{B}, the method is personalized and should be described and compared as such; if a single global model is intended, the correct equation and evaluation protocol must be given. This ambiguity also affects the convergence analysis.
  5. [Sec. 3.2 / Eq. (5); Table 2] The boosting mechanism grows the model without bound: each round freezes a LoRA block, so the final adapter contains R blocks and the parameter/inference cost increases linearly with the number of communication rounds. Table 2 reports only per-round wall-clock overhead (0.9%) and does not account for final model size. The paper should report final parameter counts and inference latency, and discuss whether a bounded-memory variant (e.g., merging blocks after each round) preserves the reported gains.
minor comments (5)
  1. [Appendix B, Remark 1] Because the practical implementation relaxes orthonormality during local training, A_train differs from A_merge; the claim that A-drift is 'identically zero by construction' holds only in the idealized implementation. Please qualify this in the main text and quantify the perturbation in the empirical setting.
  2. [Algorithm 1 / Sec. 3.2] The relationship between the local trained tilde{B}_i, the server-aggregated bar{B}, and the B_t^i appearing in Eq. (5) should be defined formally; a notation table would remove the ambiguity.
  3. [Table 6] w/o Boosting (82.6) slightly outperforms full Dysco (82.5) at round 10 on MIMIC-IV-note. This needs a comment, since it weakens the claim that boosting is beneficial on this benchmark.
  4. [Table 1 / Sec. 5.1] The Scaffold baseline is near 50% random chance on MIMIC-IV-note; the +0.5 point gain should be interpreted with that degenerate baseline in mind.
  5. [Abstract vs. Sec. 7] The abstract states 'up to 9 times' while the conclusion says 'roughly 5-9x'; make the numbers consistent.

Circularity Check

2 steps flagged

Synthetic 9x result is written into the experiment (oracle U_i^T plus exactly orthogonal input subspaces), so the headline synthetic 'prediction' is by construction; the external MIMIC/GLUE results and transparent theory keep the paper from being fully circular.

specific steps
  1. self definitional [Appendix C.1 (Synthetic setup), reported in Section 4.2 / Fig. 2]
    "For each client i we draw a column-orthonormal subspace U_i ... and sequence latents Z_i ..., so that the input tokens x = Z_i U_i^T ... lie entirely in U_i. ... for Dysco, A_i is set to U_i^T, so Dysco's A_i is fixed across rounds while B_i is trained locally and merged on the server. When N·r ≤ d_model the subspaces {U_i} form a mutually orthogonal partition of R^{d_model}."

    The synthetic data are generated to lie entirely in per-client subspaces U_i, and Dysco is handed A_i = U_i^T. For i ≠ j, A_j applied to a client-i input equals U_j^T (Z_i U_i^T) = 0 whenever the U_j are mutually orthogonal, which the protocol guarantees for N·r ≤ d_model. This is exactly the A_j h_i ≈ 0 condition that Eq. (1) identifies as eliminating the cross-client interference term. The reported order-of-magnitude loss reduction (0.11 vs 0.57 at L=1) is therefore forced by the experimental construction; the local SVD selection (Eq. 2) and server merge (Eqs. 3–4) are bypassed because the oracle subspace is given directly. The 'prediction' reduces to the definition of the data and the oracle assignment.

  2. self definitional [Section 3.3; Appendix B.2.1, Eq. (8) and Lemma B.1]
    "Dysco fixes the right factor on a per-client basis to the server-merged A^{merge,t+1}_i, both during local training and at global application, which eliminates the A-drift contribution by construction and collapses the interference to a linear function of the left-factor drift alone."

    The claimed elimination of A-drift is the algorithm's definition rather than a derived consequence: A_train^i and A_application^i are both set equal to A_merge^i, so Eq. (8) contains no A-difference term by construction. The subsequent Θ(Δ_A/ρ_A) tightening is a restatement of the design choice 'fix A to the merged subspace.' The paper is transparent about this ('by construction'), so this is a design property, but it means the theoretical bound cannot independently validate the subspace-selection mechanism; the same construction is later presented as the source of the synthetic empirical advantage.

full rationale

Most of the derivation chain is self-contained and non-circular: Eq. (1) is an algebraic identity; Eqs. (2)–(4) have closed-form solutions; Lemma B.1 follows from norm inequalities; and the MIMIC-IV and GLUE results are external benchmarks that do not depend on the paper's fitted values. The self-citations ([50], [51]) are not load-bearing: [50] is used as a proof template and [51] only for a practical orthonormality relaxation, so they do not by themselves force the conclusions. The theoretical 'elimination of A-drift' is transparently by construction, which is a design property rather than a hidden fit. The principal circularity is in the synthetic experiment: the data are constructed so that per-client input subspaces are mutually orthogonal, and Dysco is given the oracle right factor U_i^T, so the 9x loss reduction is a direct consequence of the experimental setup, not of the actual subspace-selection pipeline. Additionally, the MIMIC ablation (Table 6, round 10: w/o Merging 82.3 vs full Dysco 82.5) weakens the claim that server-side merging drives real-world gains, but this is an internal-validity issue rather than a circularity. Overall, partial circularity is present in the synthetic demonstration, but the paper has independent real-world content, so it does not warrant a higher score.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The method introduces no new physical entities. The central load-bearing premises are standard FL smoothness/boundedness assumptions plus two ad hoc premises: the orthonormality-relaxation perturbation bound, and the unstated acceptability of unbounded adapter accumulation. The free parameters are the user-chosen rank r and the number of boosting rounds R, both of which directly affect the claimed theoretical and practical advantages.

free parameters (2)
  • LoRA/subspace rank r = r=8 for real experiments; r in {4,8,16,32} in synthetic sweeps
    The method requires each client's LoRA rank to match the subspace rank. The theoretical tightening scales as Theta(r), and the chosen rank directly controls capacity and the feasibility of orthogonal partitions (N*r <= d), so it is a load-bearing hand-chosen parameter, not a fitted constant.
  • number of boosting rounds R = 30 (main real experiments), 100 (synthetic)
    Eq. (5) accumulates one frozen LoRA block per round, so the final model size is linear in R. The number of rounds is a hand-set hyperparameter that determines the practical model-growth overhead, but it is not fitted to data.
axioms (7)
  • domain assumption Global objective f is beta-smooth in the Frobenius norm (Assumption B.1)
    Required for Theorem B.1 and all subsequent bounds. Standard in FL theory but not verified for the actual transformer objectives used in experiments.
  • domain assumption Stochastic LoRA-factor gradients are unbiased and uniformly bounded (Assumption B.2)
    Assumed a.s. for all clients and rounds. Used to control trajectory drift in Theorem B.2. Not verified empirically.
  • domain assumption LoRA-factor magnitudes are bounded (Assumption B.3)
    Assumed for all clients and rounds (||B||_F <= tau_B, ||A||_F <= tau_A). Plausible in fine-tuning regimes but not proven for the actual trained adapters.
  • ad hoc to paper Relaxing the orthonormality constraint during local training perturbs rho_A only by (1+epsilon_ortho)^2 (Remark 1)
    The paper asserts the perturbation is small and controlled by the local optimizer step size, but gives no proof. This is needed to keep the Dysco bound tight in the practical implementation described after Eq. (5).
  • domain assumption The empirical activation Gram matrix H_i^T H_i accurately captures the population geometry of client representations
    Eq. (2) computes the insensitive subspace from a finite local sample. If the sample is unrepresentative, the transmitted basis and subsequent server merge inherit error.
  • domain assumption All clients share the same base model architecture and use synchronous aggregation
    Stated explicitly in the Limitations section. Required for the server to merge all clients' basis matrices in the same parameter space.
  • ad hoc to paper Accumulating a frozen LoRA block every round is an acceptable modeling choice
    Eq. (5) sums R adapters and the paper never analyzes the resulting linear growth in parameter count/inference cost. The reported 'negligible overhead' is per-round wall-clock, not final model size.

pith-pipeline@v1.3.0-alltime-deepseek · 4240 in / 4713 out tokens · 183922 ms · 2026-08-02T02:17:27.589317+00:00 · methodology

0 comments
read the original abstract

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.

Figures

Figures reproduced from arXiv: 2607.14367 by Fei Wang, Haobo Zhang, Jiankun Wang, Jiayu Zhou, Lam Tsoi, Suraj Rajendran, Weishen Pan, Yong Chen.

Figure 1
Figure 1. Figure 1: Overview of Dysco. (1) Local subspace selection: each client runs SVD on its latent features to obtain a basis A∗ i spanning directions insensitive to its own data, and uploads only this basis. (2) Server-side subspace merging: the server combines the other clients’ bases into a merged subspace Amerge for each client, allocating directions that are broadly insensitive to the other clients’ representations.… view at source ↗
Figure 2
Figure 2. Figure 2: Training loss (top) and test accuracy (bottom) across Transformer depths [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training loss (top) and test accuracy (bottom) when LoRA targets only [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss (top) and test accuracy (bottom) as the number of clients [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training loss (top) and test accuracy (bottom) for LoRA ranks [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average training loss vs. communication round on GLUE (a) and MIMIC-IV-note (b) [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training loss (top) and test accuracy (bottom) for [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training loss (top) and test accuracy (bottom) at the boundary case [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training loss (top) and test accuracy (bottom) at the small-capacity case [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Average accuracy across all GLUE tasks over communication rounds with and without [PITH_FULL_IMAGE:figures/full_fig_p030_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 8 linked inside Pith

  1. [1]

    Bai, J., Chen, D., Qian, B., Yao, L., and Li, Y. (2024). Federated fine-tuning of large language models under heterogeneous tasks and client resources. volume 37, pages 14457–14483

    2024

  2. [2]

    Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. pages 1–14

    2017

  3. [3]

    J., Liu, L., Xu, Z., Fahrezi, A., and Joshi, G

    Cho, Y. J., Liu, L., Xu, Z., Fahrezi, A., and Joshi, G. (2024). Heterogeneous lora for federated fine-tuning of on-device foundation models. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 12903–12913

    2024

  4. [4]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186

    2019

  5. [5]

    Dolan, W. B. and Brockett, C. (2005). Automatically constructing a corpus of sentential paraphrases. InProceedings of the third international workshop on paraphrasing (IWP2005)

    2005

  6. [6]

    Fallah, A., Mokhtari, A., and Ozdaglar, A. (2020). Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach.Advances in neural information processing systems, 33:3557–3568

    2020

  7. [7]

    Alphaedit: Null-space constrained model editing for language models

    Fang, J., Jiang, H., Wang, K., Ma, Y., Shi, J., Wang, X., He, X., andChua, T.-S.(2025). Alphaedit: Null-space constrained model editing for language models. InThe Thirteenth International Conference on Learning Representations. 15

    2025

  8. [8]

    Farajtabar, M., Azizan, N., Mott, A., and Li, A. (2020). Orthogonal gradient descent for continual learning. InInternational conference on artificial intelligence and statistics, pages 3762–3773. PMLR

    2020

  9. [9]

    Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. (2020). An efficient framework for clustered federated learning.Advances in neural information processing systems, 33:19586–19597

    2020

  10. [10]

    Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, W. B. (2007). The third pascal recognizing textual entailment challenge. InProceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9

    2007

  11. [11]

    L., Amaral, L

    Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. (2000). Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.circulation, 101(23):e215–e220

    2000

  12. [12]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models.arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  13. [13]

    Guo, P., Zeng, S., Wang, Y., Fan, H., Wang, F., and Qu, L. (2025a). Selective aggregation for low- rank adaptation in federated learning. InInternational Conference on Learning Representations, volume 2025, pages 99003–99027

    2025

  14. [14]

    Guo, P., Zeng, S., Wang, Y., Fan, H., Wang, F., and Qu, L. (2025b). Selective aggregation for low- rank adaptation in federated learning. InInternational Conference on Learning Representations, volume 2025, pages 99003–99027

    2025

  15. [15]

    H., Qi, H., and Brown, M

    Hsu, T.-M. H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification.arXiv preprint arXiv:1909.06335

    Pith/arXiv arXiv 2019

  16. [16]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022a). Lora: Low-rank adaptation of large language models.Iclr, 1(2):3

  17. [17]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022b). Lora: Low-rank adaptation of large language models. volume 1, page 3

  18. [18]

    Huang, Y., Chu, L., Zhou, Z., Wang, L., Liu, J., Pei, J., and Zhang, Y. (2021). Personalized cross-silo federated learning on non-iid data. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 7865–7873

    2021

  19. [19]

    A., and Mark, R

    Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L. A., and Mark, R. (2024). MIMIC-IV.PhysioNet. Version 3.1

    2024

  20. [20]

    A., and Mark, R

    Johnson, A., Pollard, T., Horng, S., Celi, L. A., and Mark, R. (2023a). MIMIC-IV-Note: Deidentified free-text clinical notes.PhysioNet. Version 2.2

  21. [21]

    E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T

    Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023b). Mimic-iv, a freely accessible electronic health record dataset.Scientific data, 10(1):1

  22. [22]

    P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A

    Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. InInternational conference on machine learning, pages 5132–5143. PMLR. 16

    2020

  23. [23]

    Koo, J., Jang, M., and Ok, J. (2025). Towards robust and efficient federated low-rank adaptation with heterogeneous clients. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 416–429

    2025

  24. [24]

    Li, Q., He, B., and Song, D. (2021a). Model-contrastive federated learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713–10722

  25. [25]

    Li, T., Hu, S., Beirami, A., and Smith, V. (2021b). Ditto: Fair and robust federated learning through personalization. InInternational conference on machine learning, pages 6357–6368. PMLR

  26. [26]

    K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V

    Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated optimization in heterogeneous networks.Proceedings of Machine learning and systems, 2:429–450

    2020

  27. [27]

    U., and Jaggi, M

    Lin, T., Kong, L., Stich, S. U., and Jaggi, M. (2020). Ensemble distillation for robust model fusion in federated learning.Advances in neural information processing systems, 33:2351–2363

    2020

  28. [28]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692

    Pith/arXiv arXiv 2019

  29. [29]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101

    Pith/arXiv arXiv 2017

  30. [30]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication- efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. Pmlr

    2017

  31. [31]

    Niu, Y., Xiao, H., Liu, D., Chen, N., and Li, J. (2025). Mitigating the safety alignment tax with null-space constrained policy optimization.arXiv preprint arXiv:2512.11391

    arXiv 2025

  32. [32]

    Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., and Qian, D. (2024). Fdlora: Personalized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925

    Pith/arXiv arXiv 2024

  33. [33]

    Qiu, Z., Wang, L., Cao, Y., Zhang, R., Su, B., Xu, Y., Meng, F., Xu, L., Wu, Q., and Li, H. (2025). Null-space filtering for data-free continual model merging: Preserving transparency, promoting fidelity.arXiv preprint arXiv:2509.21413

    arXiv 2025

  34. [34]

    Qiu, Z., Xu, Y., He, C., Meng, F., Xu, L., Wu, Q., and Li, H. (2026). Mingle: Mixture of null-space gated low-rank experts for test-time continual model merging.Advances in Neural Information Processing Systems, 38:143841–143878

    2026

  35. [35]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392

    2016

  36. [36]

    Sattler, F., Korjakow, T., Rischke, R., and Samek, W. (2021). Fedaux: Leveraging unlabeled auxiliary data in federated learning.IEEE Transactions on Neural Networks and Learning Systems, 34(9):5531–5543

    2021

  37. [37]

    Sheng, L., Shen, C., Zhao, W., Fang, J., Liu, X., Liang, Z., Wang, X., Zhang, A., and Chua, T.-S. (2025). Alphasteer: Learning refusal steering with principled null-space constraint.arXiv preprint arXiv:2506.07022. 17

    arXiv 2025

  38. [38]

    P., Singh, H., Mathur, N., Mandloi, A., and Kamath, S

    Singh, A. P., Singh, H., Mathur, N., Mandloi, A., and Kamath, S. (2026). Sonoedit: Null-space constrained knowledge editing for pronunciation correction in llm-based tts.arXiv preprint arXiv:2601.17086

    arXiv 2026

  39. [39]

    D., Ng, A

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642

    2013

  40. [40]

    Sun, Y., Li, Z., Li, Y., and Ding, B. (2024a). Improving lora in privacy-preserving federated learning

  41. [41]

    Sun, Y., Li, Z., Li, Y., and Ding, B. (2024b). Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313

    Pith/arXiv arXiv

  42. [42]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355

    2018

  43. [43]

    Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization.Advances in neural information processing systems, 33:7611–7623

    2020

  44. [44]

    Wang, Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu, L., and Li, A. (2024). Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. volume 37, pages 22513–22533

    2024

  45. [45]

    Warstadt, A., Singh, A., and Bowman, S. R. (2019). Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641

    2019

  46. [46]

    Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122

    2018

  47. [47]

    Wu, F., Guo, S., Wang, H., Zhang, H., Qu, Z., Zhang, J., and Liu, Z. (2023). From deterio- ration to acceleration: A calibration approach to rehabilitating step asynchronism in federated optimization.IEEE Transactions on Parallel and Distributed Systems, 34(5):1548–1559

    2023

  48. [48]

    Yi, L., Yu, H., Wang, G., Liu, X., and Li, X. (2023). pfedlora: Model-heterogeneous personalized federated learning with lora tuning.arXiv preprint arXiv:2310.13283

    Pith/arXiv arXiv 2023

  49. [49]

    Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. (2020). Gradient surgery for multi-task learning.Advances in neural information processing systems, 33:5824–5836

    2020

  50. [50]

    Zhang, H., Kim, D., Cha, S., and Vikalo, H. (2026). Fedrot-lora: Mitigating rotational misalignment in federated lora.arXiv preprint arXiv:2602.23638

    Pith/arXiv arXiv 2026

  51. [51]

    and Zhou, J

    Zhang, H. and Zhou, J. (2025). Unraveling lora interference: Orthogonal subspaces for robust model merging. pages 26459–26472

    2025

  52. [52]

    Zhang, L., Shen, L., Ding, L., Tao, D., and Duan, L.-Y. (2022). Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174–10183. 18

    2022

  53. [53]

    Zhang, Y., Kang, Y., Chen, H., Li, J., and Hu, J.-F. (2025). Null-lora: Low-rank adaptation on null space.arXiv preprint arXiv:2512.15233

    arXiv 2025

  54. [54]

    Zhu, Z., Hong, J., and Zhou, J. (2021). Data-free knowledge distillation for heterogeneous federated learning. InInternational conference on machine learning, pages 12878–12889. PMLR. 19 Appendix Contents A Broader Impact 20 B Comparison between Dysco and FedAvg Under Data Heterogeneity 20 C Additional Synthetic Experiments 25 D Additional Real-World Expe...

    2021

  55. [55]

    ∆t+1 A ∆t+1 B 4η2 # +O(η), Dysco: f(W 0)−f ⋆ ηT + 3βη 2 + 1 2T η X t E

    By Jensen’s inequality,∥¯Et+1,Dy∥2 F ≤ρ t+1 A · 1 2 ∆t+1 B . Proof.(1) Apply the triangle inequality and then Cauchy–Schwarz to (7): ∥Et+1,F A∥F ≤ 1 2N 2 X i,j ∥Bi −B j∥F ∥Ai −A j∥F ≤ 1 2N 2 P i,j ∥Bi −B j∥2 F 1/2 P i,j ∥Ai −A j∥2 F 1/2 = 1 2 q ∆t+1 B ∆t+1 A , using the identityP i,j ∥Xi −X j∥2 F =N 2∆t+1 X . Squaring yields the stated bound. (2) For each...