Pith. sign in

REVIEW 3 major objections 5 minor 79 references

The paper proposes FedSLM, a federated learning framework in which clients with SVD-compressed language models—running at about half the GPU memory of the full model—collectively fine-tune a full-scale server foundation model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:10 UTC pith:XJIAFQOM

load-bearing objection Clever combination of SVD compression, nested subspaces, and weak-to-strong distillation for federated foundation-model fine-tuning; the central aggregation guarantee rests on a condition the authors admit is atypical and that experiments never stress. the 3 major comments →

arxiv 2607.29071 v1 pith:XJIAFQOM submitted 2026-07-31 cs.LG cs.AI

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

Shengkun Zhu , Jinshan Zeng , Zhihua Allen-Zhao , Mayi Xu , Quanqing Xu , Wei Ren , Qiang Yang , Yang Liu This is my paper
classification cs.LG cs.AI
keywords federated learningfoundation modelsmodel compressionsingular value decompositionweak-to-strong generalizationheterogeneous clientsfine-tuninglow-rank adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FedSLM tries to dissolve the resource-asymmetry paradox in federated foundation-model fine-tuning: the institutions with the most valuable data are often the ones that cannot host a billion-parameter model. The framework derives compressed client models from the server model via SVD, keeps only a small trainable adapter per layer between fixed low-rank factors, aggregates adapters within same-rank groups, and then reconstructs full-rank layers and averages them across groups in the shared weight space. A final weak-to-strong step uses the aggregated model as a noisy supervisor to transfer knowledge into the full server model while suppressing compression artifacts with a confidence loss. If correct, this means organizations with roughly half the GPU memory can participate and the server model can still beat standard federated baselines on language and vision-language benchmarks.

Core claim

The central claim is that heterogeneous compressed clients can be made structurally compatible for parameter-level aggregation if all client models are derived from the same server weights by rank-truncated SVD. Because the left and right singular subspaces of different ranks are nested, the reconstruction U_i Sigma_i V_i of every group lives in the same ambient space and differs from the pretrained layer only by small perturbations; the weighted average of these reconstructions is therefore a meaningful consensus. The paper then shows that this aggregated weak model, used as a teacher with an auxiliary confidence loss, can elicit the full capacity of the server model, and that adapter-level

What carries the argument

The load-bearing object is the frozen SVD factorization W ≈ U_i Σ_i V_i of each selected layer: U_i and V_i are fixed low-rank factors of rank r_i and Σ_i is the only trainable part, a square r_i×r_i adapter initialized to identity. The nested structure of the singular subspaces (lower ranks contained in higher ranks) makes different compression groups compatible, and reconstruction into the common full-rank space turns cross-group fusion into a plain weighted average. The auxiliary confidence loss mixes imitation of the weak aggregated model with a self-anchor against the strong model's own confident predictions; its coefficient α controls the bias–variance trade-off.

Load-bearing premise

The load-bearing premise is that all group reconstructions, being fine-tuned from the same pretrained anchor, differ from it and from each other only by small perturbations, so a plain weighted average in the common weight space is a valid consensus; if client adapters drift far from that anchor, the averaged reconstruction becomes a poor supervisor and the pipeline degrades.

What would settle it

Train a federation where each client's local dataset is a very different domain or task, so that the reconstructed group layers move far from the shared pretrained anchor; measure the Frobenius distance between each group's reconstruction and the anchor, and check whether the cross-group average W_g still yields a server model that beats FedAvg+LoRA. If accuracy collapses when those distances are large, the anchor-proximity premise is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Institutions that cannot host a full billion-parameter model can still contribute domain data to federated fine-tuning, since client memory drops to roughly half.
  • Only the small square adapters are transmitted, so communication cost is tied to rank, not model size, and secure aggregation or differential privacy can be layered on the adapter exchange.
  • The server-side full model can recover and exceed the zero-shot accuracy of the uncompressed model after federated training, even from compressed teachers.
  • Client-side compressed models themselves improve through aggregation and can outperform their zero-shot baselines, making them useful deployed artifacts, not just teachers.
  • Adapter-level FedAvg converges at the standard O(T^{-1/2}) rate, with an explicit spectral penalty and heterogeneity floor.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to allocate per-client ranks based on the tail singular spectrum of each layer: the theory suggests rank matters most where the discarded singular energy is largest, so adaptive rank allocation should beat fixed ratios.
  • The same two-stage aggregation could generalize to pruning or quantization families that preserve a nested subspace structure, though current guarantees are stated for SVD-derived models.
  • The confidence-loss coefficient α could be tuned per layer or per sample rather than globally, since the artifact bound B_ξ varies by layer.
  • The weighted-average fusion rule implicitly assumes group reconstructions stay near the shared pretrained anchor; if clients specialize to distant task optima, a more robust fusion in a projected or aligned space might be needed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FedSLM, a federated fine-tuning framework in which resource-constrained clients train small square adapters on fixed SVD-derived low-rank factors of a server-side foundation model, while the server performs two-stage aggregation: intra-group FedAvg of adapters, followed by cross-group weighted averaging of reconstructed matrices in the common ambient space, and finally a weak-to-strong distillation step with an auxiliary confidence loss to transfer knowledge back to the full-scale model. Theoretical results are claimed for Stage-1 convergence (O(T^{-1/2})), for cross-group subspace alignment, and for artifact mitigation through a bias-variance trade-off. Experiments on LLaMA-2-7B/13B and LLaVA-NeXT-7B under IID and non-IID partitions report consistent gains over several federated baselines, with client GPU memory roughly halved relative to the full model.

Significance. If the claims hold, FedSLM addresses a practically relevant problem: enabling memory-constrained clients to participate in federated fine-tuning of large foundation models while preserving server-side performance. The framework is well motivated, the experimental scope is broad (two language model scales, a vision-language model, IID/non-IID partitions), and the efficiency profiling is useful. The theoretical sections are also unusually honest about their assumptions, and some parts (e.g., the adapter-level convergence bound) are standard but sound. However, the central cross-group fusion guarantee rests on a condition that the paper itself labels non-trivial and atypical, and the experimental comparison with standard LoRA baselines is not on equal footing. These issues are load-bearing for the paper's two headline claims — structural compatibility of compressed clients and improved performance over existing federated baselines — so the manuscript needs major revision before the claims can be considered supported.

major comments (3)
  1. [Section IV-B, Theorem 3 (Eqs. 30–31)] The cross-group fusion guarantee is proved only under the common-projection-target condition (28), which the paper itself describes immediately afterward as "a non-trivial geometric requirement." Proposition 5 is explicitly called "a clean limiting case rather than a typical operating point," and Proposition 6's perturbation bounds depend on the unmeasured quantities ε_G,i, ε_H,i, ρ_i (Eqs. 76–79) and restricted strong convexity. Because Theorem 3 feeds Lemma 1 and Propositions 3–4, the whole artifact-mitigation analysis inherits an unverified premise. Nested subspaces (Prop. 1) only give projector identities; they do not by themselves make the Euclidean average of points from different rank-r_i subspaces close to a useful reference. Moreover, every experiment includes 3–6 full-model clients (Sec. V-A, Tables II–III), which anchor W_g near the pretrained W; no all-compressed (zero full-m
  2. [Section V-A, Tables II–III] The comparison against FedAvg+LoRA, FFA-LoRA, HetLoRA, FlexLoRA, and Fed-RAC-LoRA is not on equal footing. The text states that these baselines "are evaluated on the subset of clients that natively host the full server model," i.e., 3 clients in the 10-client setup and 6 in the 20-client setup, while FedSLM uses all 10 or 20 clients including compressed ones. If the data partition is kept consistent over these different client pools, the baselines see strictly less client data; if the partition is adjusted, the comparison differs in yet another dimension. The reported 4–7 point gains over FedAvg+LoRA could therefore be partly an artifact of additional participating clients and data. A fair comparison should either give the LoRA baselines the same number of clients and the same total data (e.g., all full-model clients), or report a FedSLM variant using only the full-model clients.
  3. [Section IV-C, Proposition 4] The bias-variance analysis of the confidence loss is not empirically validated. The optimal mixture weight α* in Eq. (38) depends on E‖ξ‖² and V_self, but neither quantity is estimated in the experiments. The paper only compares α=0 and α=0.5 (Tables II–III and VI) and never reports whether 0.5 is close to α*. A simple α sweep on one benchmark, together with an estimate of Bξ from the singular-value spectrum, would test the predicted monotonicity of α* in compression. Without this, the claim that the auxiliary confidence loss provides a controllable "bias-variance trade-off" remains a theoretical statement only.
minor comments (5)
  1. [Abstract & Section III-C] The phrase "full-rank reconstructions" is misleading: W_i = U_i Σ_i V_i in Eq. (8) has rank at most r_i, and W_g has rank at most max_i r_i. These are full-dimensional or full-parameter matrices, not full-rank matrices. Consider "full-dimensional reconstructions" or "ambient-space reconstructions."
  2. [Section IV-C, Eq. (11)] The definition of the hardened self-prediction I[f_s(x;W_s)>τ] is unclear for vector-valued outputs. If it is applied elementwise, that should be stated; the thresholding rule "exactly half of the samples" also needs a precise definition for a C-class softmax output.
  3. [Section V-A and Appendix B] The source and role of the server-side unlabeled dataset D_server are not specified. If D_server is drawn from the same tasks/domains as the client data, the method assumes server access to the client data distribution in an unlabeled form; this should be stated explicitly and discussed as a limitation.
  4. [Figure 2/3 and Section V-B] The abstract and Section V claim "roughly 50%" GPU memory, but the reported footprints are 54–65% for SVD-0.6+Adapter at 7B and 43–65% at 13B. "Roughly half" is an overstatement; "roughly 50–65%" would be more accurate.
  5. [Tables II and III] Several entries in the FedSLM rows are typeset with missing spaces (e.g., "80.764.774.0" and "64.774.6"); this makes the tables hard to read and should be corrected.

Circularity Check

0 steps flagged

No circularity: the derivation chain is self-contained; the admitted geometric assumptions are limitations, not circular reasoning.

full rationale

I walked the paper's claimed derivation chain. The Stage-1 adapter aggregation theorem (Theorem 1, Eqs. 17-18) is a standard FedAvg/local-SGD convergence bound with an effective smoothness constant L_eff defined from the frozen SVD factors; it does not fit any parameter to the downstream benchmarks. The cross-group fusion rule W_g = sum_i (n_i/N) W_i (Eq. 9) is indeed the closed-form minimizer of the consensus objective (Eq. 4) by construction, but the paper presents this as the aggregation rule, not as a prediction of task accuracy. Theorem 2/3 is a triangle-inequality decomposition of W_g - cW into optimization, misalignment, and subspace residuals; the key common-projection-target condition (Eq. 28) is explicitly called 'a non-trivial geometric requirement,' and Proposition 5 is explicitly described as 'a clean limiting case rather than a typical operating point.' These are honest, testable assumptions and limitations, not circularly imported results. The weak-to-strong analysis (Propositions 3 and 4) is exact algebra of the explicitly defined loss: the gradient decomposition and the quadratic minimizer alpha* follow from the definition of L_conf, and alpha is not fitted to data (experiments use alpha=0 or 0.5). The paper does not rename a known result: it openly says the ambient weighted average 'recovering FedAvg semantics in a rank-heterogeneous setting.' Citations to weak-to-strong learning [30] and SVD compression [31-33] are external and non-load-bearing; no uniqueness theorem or central premise is justified only by the authors' own prior work. The experimental claims are externally benchmarked against multiple federated baselines. The main scientific risk is that the aggregation guarantee depends on the unmeasured and admittedly atypical common-projection-target condition and on anchor proximity, but that is a correctness/falsifiability concern, not circularity. I find no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claim rests on standard smoothness/heterogeneity assumptions, plus two strong structural assumptions that are acknowledged in the text (common-projection-target, uncorrelated artifact). The free parameters are hand-chosen hyperparameters, not fitted to the test benchmarks. No new physical or conceptual entities are introduced.

free parameters (3)
  • alpha (weak-to-strong mixing weight) = 0.5 in main experiments, 0 in ablation
    Chosen by hand rather than set via the paper's own alpha-star formula in Proposition 4; affects the balance between weak supervision and self-anchoring.
  • SVD compression ratios per group = 0.4 and 0.6 for LLaMA models; 0.6 and 0.9 for LLaVA-NeXT
    Hand-selected to represent client resource tiers; they directly determine the memory savings and the strength of compression residuals in the theory.
  • tau (self-anchor confidence threshold) = Adaptive per batch, keeping exactly half of samples
    Defines which predictions count as 'confident' in the auxiliary confidence loss; a heuristic choice that is not optimized or analyzed beyond the assumption in Proposition 4.
axioms (6)
  • domain assumption Assumption 1: L-smoothness, unbiased stochastic gradients with bounded variance, and bounded data heterogeneity for every client adapter loss
    Standard FL assumptions invoked for Theorem 1; they are stated but not verified on the actual benchmarks.
  • domain assumption Assumption 2: Lipschitz forward pass from weight-space error to output-space probability error (Eq. 33)
    Needed to transport the weight-space aggregation bound into the output-space artifact bound of Lemma 1.
  • domain assumption Polyak-Lojasiewicz inequality for each group adapter loss (Eq. 26)
    Used in Proposition 2(a) to convert Stage-1 gradient-norm convergence into an iterate-distance bound for the optimization residual.
  • ad hoc to paper Common-projection-target condition: the geometric projection adapter minimizes each group's adapter loss (Eq. 28)
    This strong condition makes the misalignment residual vanish in Theorem 3; the paper acknowledges it is non-trivial and typically violated in the federated fine-tuning regime.
  • ad hoc to paper Uncorrelatedness of the compression artifact and the self-anchoring residual in Proposition 4
    Required to separate the bias term from the variance term in the alpha trade-off; not checked empirically.
  • domain assumption Fine-tuned group models remain close to the shared pre-trained anchor so ambient-space weighted averaging is meaningful (Section III-C)
    If adapters drift far from the SVD anchor, Eq. (9) averages models that may have diverged, and the aggregated weak supervisor loses fidelity.

pith-pipeline@v1.3.0-daily-deepseek · 44968 in / 12576 out tokens · 132061 ms · 2026-08-03T14:10:58.784036+00:00 · methodology

0 comments
read the original abstract

Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

Figures

Figures reproduced from arXiv: 2607.29071 by Jinshan Zeng, Mayi Xu, Qiang Yang, Quanqing Xu, Shengkun Zhu, Wei Ren, Yang Liu, Zhihua Allen-Zhao.

Figure 1
Figure 1. Figure 1: Overview of the FedSLM framework. Step 1: the server derives a family of structurally compatible compressed models at heterogeneous compression [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Client-side efficiency of SVD-compressed LLaMA-2-7B models, with a Full+LoRA reference. Left: GPU memory footprint before and after attaching [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Client-side efficiency of SVD-compressed LLaMA-2-13B models, with a Full+LoRA reference. Left: GPU memory footprint before and after attaching [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss convergence of LLaMA-2-7B clients over communication rounds on all eight benchmarks under the non-IID ( [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training loss convergence of LLaMA-2-13B clients over communication rounds on all eight benchmarks under the non-IID ( [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 12 linked inside Pith

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Large language models in medicine,

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,”Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023

    1930

  5. [5]

    Foundation models for generalist medical artificial intelligence,

    M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,”Nature, vol. 616, no. 7956, pp. 259–265, 2023

    2023

  6. [6]

    Bloomberggpt: A large language model for finance,

    S. Wu, O. Irsoy, S. Lu, V . Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,”arXiv preprint arXiv:2303.17564, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inAISTATS. PMLR, 2017, pp. 1273–1282

    2017

  8. [8]

    Federated machine learning: Concept and applications,

    Q. Yang, Y . Liu, T. Chen, and Y . Tong, “Federated machine learning: Concept and applications,”ACM Transactions on Intelligent Systems and Technology, vol. 10, no. 2, pp. 1–19, 2019

    2019

  9. [9]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” inICLR, 2022

    2022

  10. [10]

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,

    Z. Zhang, Y . Yang, Y . Dai, Q. Wang, Y . Yu, L. Qu, and Z. Xu, “Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” inACL. Association for Computational Linguistics, 2023, pp. 9963–9977

    2023

  11. [11]

    Towards building the federatedgpt: Federated instruction tuning,

    J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y . Chen, “Towards building the federatedgpt: Federated instruction tuning,” inICASSP. IEEE, 2024, pp. 6915–6919

    2024

  12. [12]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources,

    J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li, “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” inNeurIPS, 2024

    2024

  13. [13]

    Heterogeneous lora for federated fine-tuning of on-device foundation models,

    Y . J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” inEMNLP, 2024, pp. 12 903–12 913

    2024

  14. [14]

    Flora: Federated fine-tuning large language models with heterogeneous low- rank adaptations,

    Z. Wang, Z. Shen, Y . He, G. Sun, H. Wang, L. Lyu, and A. Li, “Flora: Federated fine-tuning large language models with heterogeneous low- rank adaptations,” inNeurIPS, 2024

    2024

  15. [15]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Heterofl: Computation and communi- cation efficient federated learning for heterogeneous clients,

    E. Diao, J. Ding, and V . Tarokh, “Heterofl: Computation and communi- cation efficient federated learning for heterogeneous clients,” inICLR, 2021

    2021

  17. [17]

    Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout,

    S. Horv ´ath, S. Laskaridis, M. Almeida, I. Leontiadis, S. I. Venieris, and N. D. Lane, “Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout,” inNeurIPS, 2021, pp. 12 876–12 889

    2021

  18. [18]

    Fedrolex: Model- heterogeneous federated learning with rolling sub-model extraction,

    S. Alam, L. Liu, M. Yan, and M. Zhang, “Fedrolex: Model- heterogeneous federated learning with rolling sub-model extraction,” in NeurIPS, 2022

    2022

  19. [19]

    Depthfl : Depthwise federated learning for heterogeneous clients,

    M. Kim, S. Yu, S. Kim, and S. Moon, “Depthfl : Depthwise federated learning for heterogeneous clients,” inICLR, 2023

    2023

  20. [20]

    Scalefl: Resource-adaptive federated learning with heterogeneous clients,

    F. Ilhan, G. Su, and L. Liu, “Scalefl: Resource-adaptive federated learning with heterogeneous clients,” inCVPR. IEEE, 2023, pp. 24 532– 24 541

    2023

  21. [21]

    Communication-efficient federated learning via knowledge distillation,

    C. Wu, F. Wu, L. Lyu, Y . Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,”Nature communications, vol. 13, no. 1, p. 2032, 2022. PREPRINT 17

    2032

  22. [22]

    Fedmkt: Federated mutual knowledge transfer for large and small language models,

    T. Fan, G. Ma, Y . Kang, H. Gu, Y . Song, L. Fan, K. Chen, and Q. Yang, “Fedmkt: Federated mutual knowledge transfer for large and small language models,” inCOLING, 2025, pp. 243–255

    2025

  23. [23]

    PCEvolve: Private contrastive evolution for synthetic dataset generation via few- shot private data and generative APIs,

    J. Zhang, Y . Liu, J. Fu, Y . Hua, T. Zou, J. Cao, and Q. Yang, “PCEvolve: Private contrastive evolution for synthetic dataset generation via few- shot private data and generative APIs,” inICML, vol. 267, 2025, pp. 75 575–75 590

    2025

  24. [24]

    Explaining knowledge distillation by quantifying the knowledge,

    X. Cheng, Z. Rao, Y . Chen, and Q. Zhang, “Explaining knowledge distillation by quantifying the knowledge,” inCVPR, 2020, pp. 12 922– 12 932

    2020

  25. [25]

    Text representation distillation via information bottleneck principle,

    Y . Zhang, D. Long, Z. Li, and P. Xie, “Text representation distillation via information bottleneck principle,” inEMNLP, 2023, pp. 14 372–14 383

    2023

  26. [26]

    Dual-space knowledge distillation for large language models,

    S. Zhang, X. Zhang, Z. Sun, Y . Chen, and J. Xu, “Dual-space knowledge distillation for large language models,” inEMNLP, 2024, pp. 18 164– 18 181

    2024

  27. [27]

    Towards cross- tokenizer distillation: the universal logit distillation loss for llms,

    N. Boizard, K. E. Haddad, C. Hudelot, and P. Colombo, “Towards cross- tokenizer distillation: the universal logit distillation loss for llms,”arXiv preprint arXiv:2402.12030, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Universal cross-tokenizer distillation via approximate likelihood matching,

    B. Minixhofer, I. Vuli ´c, and E. M. Ponti, “Universal cross-tokenizer distillation via approximate likelihood matching,” inNeurIPS, vol. 38, 2025, pp. 79 297–79 326

    2025

  29. [29]

    Tokalign: Efficient vocabulary adaptation via token alignment,

    C. Li, J. Zhang, and C. Zong, “Tokalign: Efficient vocabulary adaptation via token alignment,” inACL, 2025, pp. 4109–4126

    2025

  30. [30]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision,

    C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbren- ner, Y . Chen, A. Ecoffet, M. Joglekar, J. Leike, I. Sutskever, and J. Wu, “Weak-to-strong generalization: Eliciting strong capabilities with weak supervision,” inICML, 2024

    2024

  31. [31]

    ASVD: Activation-aware singular value decomposition for compressing large language models,

    Z. Yuan, Y . Shang, Y . Song, D. Yang, Q. Wu, Y . Yan, and G. Sun, “ASVD: Activation-aware singular value decomposition for compressing large language models,”arXiv preprint arXiv:2312.05821, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    SVD-LLM: Truncation- aware singular value decomposition for large language model compres- sion,

    X. Wang, Y . Zheng, Z. Wan, and M. Zhang, “SVD-LLM: Truncation- aware singular value decomposition for large language model compres- sion,” inICLR, vol. 2025, 2025, pp. 19 299–19 319

    2025

  33. [33]

    Dobi-svd: Differentiable svd for llm compression and some new perspectives,

    Q. Wang, J. Ke, M. Tomizuka, K. Keutzer, and C. Xu, “Dobi-svd: Differentiable svd for llm compression and some new perspectives,” inICLR, 2025, pp. 12 561–12 590

    2025

  34. [34]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  35. [35]

    Fedgems: Federated learning of larger server models via selective knowledge fusion,

    S. Cheng, J. Wu, Y . Xiao, and Y . Liu, “Fedgems: Federated learning of larger server models via selective knowledge fusion,”arXiv preprint arXiv:2110.11027, 2021

    arXiv 2021

  36. [36]

    Group knowledge transfer: Federated learning of large cnns at the edge,

    C. He, M. Annavaram, and S. Avestimehr, “Group knowledge transfer: Federated learning of large cnns at the edge,” inNeurIPS, 2020

    2020

  37. [37]

    Crosslm: A data- free collaborative fine-tuning framework for large and small language models,

    Y . Deng, Z. Qiao, Y . Zhang, Z. Ma, Y . Liu, and J. Ren, “Crosslm: A data- free collaborative fine-tuning framework for large and small language models,” inMobiSys. ACM, 2025, pp. 124–137

    2025

  38. [38]

    Towards diverse device heterogeneous federated learning via task arithmetic knowledge integration,

    M. Morafah, V . Kungurtsev, H. Chang, C. Chen, and B. Lin, “Towards diverse device heterogeneous federated learning via task arithmetic knowledge integration,” inNeurIPS, 2024

    2024

  39. [39]

    Bild: Bi-directional logits difference loss for large language model distillation,

    M. Li, F. Zhou, and X. Song, “Bild: Bi-directional logits difference loss for large language model distillation,” inCOLING, 2025, pp. 1168–1182

    2025

  40. [40]

    Federated dropout - A simple ap- proach for enabling federated learning on resource constrained devices,

    D. Wen, K. J. Jeon, and K. Huang, “Federated dropout - A simple ap- proach for enabling federated learning on resource constrained devices,” IEEE Wirel. Commun. Lett., vol. 11, no. 5, pp. 923–927, 2022

    2022

  41. [41]

    Improving lora in privacy-preserving federated learning,

    Y . Sun, Z. Li, Y . Li, and B. Ding, “Improving lora in privacy-preserving federated learning,” inICLR, 2024

    2024

  42. [42]

    Fate- llm: A industrial grade federated learning framework for large language models,

    T. Fan, Y . Kang, G. Ma, W. Chen, W. Wei, L. Fan, and Q. Yang, “Fate- llm: A industrial grade federated learning framework for large language models,”arXiv preprint arXiv:2310.10049, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,

    W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y . Xie, Y . Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” inSIGKDD. ACM, 2024, pp. 5260–5271

    2024

  44. [44]

    Openfedllm: Training large language models on decentralized private data via federated learning,

    R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y . Xu, Y . Du, Y . Wang, and S. Chen, “Openfedllm: Training large language models on decentralized private data via federated learning,” inKDD. ACM, 2024, pp. 6137– 6147

    2024

  45. [45]

    SVD-LLM V2: optimizing singular value truncation for large language model compression,

    X. Wang, S. Alam, Z. Wan, H. Shen, and M. Zhang, “SVD-LLM V2: optimizing singular value truncation for large language model compression,” inNAACL, 2025, pp. 4287–4296

    2025

  46. [46]

    Qsvd: Efficient low-rank approxi- mation for unified query-key-value weight compression in low-precision vision-language models,

    Y . Wang, H. Wang, and S. Q. Zhang, “Qsvd: Efficient low-rank approxi- mation for unified query-key-value weight compression in low-precision vision-language models,” inNeurIPS, vol. 38, 2025, pp. 1789–1820

    2025

  47. [47]

    SCAFFOLD: Stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” inICML, 2020, pp. 5132–5143

    2020

  48. [48]

    Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition,

    H. Karimi, J. Nutini, and M. Schmidt, “Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition,” inECML PKDD. Springer, 2016, pp. 795–811

    2016

  49. [49]

    Loss landscapes and optimization in over-parameterized non-linear systems and neural networks,

    C. Liu, L. Zhu, and M. Belkin, “Loss landscapes and optimization in over-parameterized non-linear systems and neural networks,”Appl. Comput. Harmon. A., vol. 59, pp. 85–116, 2022

    2022

  50. [50]

    On the convergence of local descent methods in federated learning,

    F. Haddadpour and M. Mahdavi, “On the convergence of local descent methods in federated learning,”arXiv preprint arXiv:1910.14425, 2019

    Pith/arXiv arXiv 1910

  51. [51]

    Exact and linear convergence for federated learning under arbitrary client participation is attainable,

    B. Ying, Z. Li, and H. Yang, “Exact and linear convergence for federated learning under arbitrary client participation is attainable,” inNeurIPS, vol. 38, 2025, pp. 40 156–40 201

    2025

  52. [52]

    Decentralized federated averaging,

    T. Sun, D. Li, and B. Wang, “Decentralized federated averaging,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 4, pp. 4289–4301, 2023

    2023

  53. [53]

    Llava- next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava- next: Improved reasoning, ocr, and world knowledge,”arXiv preprint, 2024

    2024

  54. [54]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  55. [55]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “Piqa: Reasoning about physical commonsense in natural language,” inAAAI, vol. 34, no. 05, 2020, pp. 7432–7439

    2020

  56. [56]

    Winogrande: an adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: an adversarial winograd schema challenge at scale,”Commun. ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  57. [57]

    Social iqa: Commonsense reasoning about social interactions,

    M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y . Choi, “Social iqa: Commonsense reasoning about social interactions,” inEMNLP-IJCNLP, 2019, pp. 4462–4472

    2019

  58. [58]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” inACL, 2019, pp. 4791– 4800

    2019

  59. [59]

    Choice of plausible alternatives: An evaluation of commonsense causal reasoning,

    M. Roemmele, C. A. Bejan, and A. S. Gordon, “Choice of plausible alternatives: An evaluation of commonsense causal reasoning,” inAAAI, 2011

    2011

  60. [60]

    Medical-flashcards,

    “Medical-flashcards,” https://huggingface.co/datasets/medalpaca/ medical meadow medical flashcards, 2023

    2023

  61. [61]

    Pubmedqa: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” inEMNLP- IJCNLP, 2019, pp. 2567–2577

    2019

  62. [62]

    Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering,” inCHIL. PMLR, 2022, pp. 248–260

    2022

  63. [63]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” inNeurIPS, 2022

    2022

  64. [64]

    Vizwiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” inCVPR, 2018, pp. 3608–3617

    2018

  65. [65]

    Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adap- tation,

    G. Malinovsky, U. Michieli, H. A. A. K. Hammoud, T. Ceritli, H. Elesedy, M. Ozay, and P. Richt ´arik, “Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adap- tation,”arXiv preprint arXiv:2410.08305, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Fedproto: Federated prototype learning across heterogeneous clients,

    Y . Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” inAAAI, vol. 36, no. 8, 2022, pp. 8432–8440

    2022

  67. [67]

    Fedbiot: Llm local fine-tuning in federated learning without full model,

    F. Wu, Z. Li, Y . Li, B. Ding, and J. Gao, “Fedbiot: Llm local fine-tuning in federated learning without full model,” inKDD, 2024, pp. 3345–3355

    2024

  68. [68]

    Transformer feed-forward layers are key-value memories,

    M. Geva, R. Schuster, J. Berant, and O. Levy, “Transformer feed-forward layers are key-value memories,” inEMNLP, 2021, pp. 5484–5495

    2021

  69. [69]

    Locating and editing factual associations in GPT,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in GPT,” inNeurIPS, 2022

    2022

  70. [70]

    A unified theory of decentralized SGD with changing topology and local updates,

    A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. U. Stich, “A unified theory of decentralized SGD with changing topology and local updates,” inICML. PMLR, 2020, pp. 5381–5393

    2020

  71. [71]

    Optimizing neural networks with kronecker-factored approximate curvature,

    J. Martens and R. B. Grosse, “Optimizing neural networks with kronecker-factored approximate curvature,” inICML, vol. 37, 2015, pp. 2408–2417. PREPRINT 18 APPENDIXA PROOFS ANDSUPPORTINGLEMMAS Throughout the appendix we adopt the notation of the main text. In particular,U r ∈R m×r andV r ∈R r×n denote the (non-orthonormal) factors delivered to clients, as ...

    2015

  72. [72]

    WhenAis itself square and invertible,A † coincides with the ordinary inverseA −1

    Moore–Penrose pseudoinverse:For a column-full-rank matrixA∈R m×r withm≥r, the Moore–Penrose pseudoin- verse is defined by A† ≜(A ⊤A)−1A⊤ ∈R r×m,(39) whereA ⊤Ais invertible precisely becauseAhas full column rank. WhenAis itself square and invertible,A † coincides with the ordinary inverseA −1. For a row-full-rank matrix B∈R r×n withn≥r, the analogous formu...

  73. [73]

    Orthogonal projectors:We begin with the formal defi- nition. Definition 5(Orthogonal projector).A matrixP∈R m×m is anorthogonal projectoronto a subspaceM ⊆R m if it satisfies the following three equivalent properties: (a)P 2 =P,P ⊤ =P, andrange(P) =M; (b)Pacts as the identity onMand as zero on the orthogonal complementM ⊥; (c)for everyv∈R m,Pvis the uniqu...

  74. [74]

    A standard sufficient condition isf(x) =g(Ax)withgstrongly convex, whereAmay be rank-deficient [48]; compositions of this form arise naturally in overparameterized learning

    Polyak–Łojasiewicz condition and quadratic growth:A differentiable functionf:R d →Rwith nonempty minimizer setarg minfand infimumf ∗ = inffsatisfies thePolyak– Łojasiewicz (PŁ) inequalitywith constantµ >0if ∥∇f(x)∥ 2 ≥2µ f(x)−f ∗ ,∀x.(45) Inequality (45) is weaker than strong convexity: it does not require convexity offand permits non-unique minimizers. A...

  75. [75]

    , λr)≻0, andS∈R r×r invertible

    Supporting Lemmas: Lemma 6(SVD isometry with scaling).LetU r = U◦ rΛ1/2 r S−1 andV r =S Λ 1/2 r V◦ r withU ◦ ⊤ r U◦ r =I r, V◦ r V◦ ⊤ r =I r,Λ r = diag(λ 1, . . . , λr)≻0, andS∈R r×r invertible. For anyA∈R r×r, ∥Ur A Vr∥F =∥Λ 1/2 r S−1A S Λ1/2 r ∥F ,(47) and consequently λr κ(S)−1 ∥A∥F ≤ ∥Ur A Vr∥F ≤λ 1 κ(S)∥A∥ F .(48) In the balanced caseS=I r, the bound...

  76. [76]

    By Lemma 7, fi(Σ(t+1) i )≤f i(Σ(t) i ) +⟨∇f (t) i ,Σ (t+1) i −Σ (t) i ⟩ + Leff 2 ∥Σ(t+1) i −Σ (t) i ∥2 F

    Proof of Theorem 1: Proof.We begin with a one-round descent. By Lemma 7, fi(Σ(t+1) i )≤f i(Σ(t) i ) +⟨∇f (t) i ,Σ (t+1) i −Σ (t) i ⟩ + Leff 2 ∥Σ(t+1) i −Σ (t) i ∥2 F . (54) We next bound the inner-product term. The server update is Σ(t+1) i −Σ (t) i =−η X c∈Ci nc ni E−1X e=0 egc(Σ(t,e) i,c ).(55) Taking expectations and using Assumption 1 (b), E⟨∇f (t) i ...

  77. [77]

    Using the standard iden- tity∇ z ℓCE(p,softmax(z)) = softmax(z)−p, the gradient ofL conf with respect toz s is ∇zs Lconf = (1−α)(f s −f w) +α(f s − ˆfs)

    Gradient Decomposition:Letz s(x)∈R |Y| de- note the pre-softmax logits of the strong model, so that fstrong(x;W s) = softmax(z s(x)). Using the standard iden- tity∇ z ℓCE(p,softmax(z)) = softmax(z)−p, the gradient ofL conf with respect toz s is ∇zs Lconf = (1−α)(f s −f w) +α(f s − ˆfs). (108) Substitutingf w =f ∗ +ξ(x)from Definition 4 and rearranging yie...

  78. [78]

    Expanding Bξ =L f δrmin ( cW)yields (36)

    Proof of Proposition 3: Proof.The bound follows from Lemma 1: (1−α)ξ(x) 2 ≤(1−α)∥ξ(x)∥ 2 ≤(1−α)∥ξ(x)∥ 1 ≤(1−α)B ξ, (109) where the second inequality uses∥ · ∥2 ≤ ∥ · ∥1. Expanding Bξ =L f δrmin ( cW)yields (36)

  79. [79]

    Proof of Proposition 4: Proof.The logit-gradient error decomposes as(1−α)ξ(x)− α(fstrong − ˆfstrong). By the assumed uncorrelatedness ofξ(x) andf strong − ˆfstrong, the expected squared norm separates: the excess riskR(α)is a quadratic inα: R(α) = (1−α) 2A+α 2B(110) withA=E∥ξ(x)∥ 2 2 andB=V self. Setting dR dα =−2(1− α)A+2αB= 0givesα ∗ = A A+B , which is ...