Pith. sign in

REVIEW 3 major objections 6 minor 45 references

H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read H2Tune is a federated fine-tuning framework that handles model, layer, dimension, task, and resource heterogeneity at once, proving an $O(1/\sqrt{T})$ convergence rate and up to 15.4% accuracy gains over baselines.

desk verdict A genuinely new hybrid-heterogeneity setup with a clever layer-alignment matrix, but the resource-adaptation claim fails the paper's own equations and the O(1/sqrt(T)) theorem covers only a simplified abstraction. read the letter →

arxiv 2507.22633 v2 pith:OX3YBPWU submitted 2025-07-30 cs.LG cs.AI

Wei Guo , Siyuan Lu , Yiqi Tong , Zhaojun Hu , Fuzhen Zhuang , Xiao Zhang , Tao Fan , Jin Dong This is my paper
classification cs.LGcs.AI MSC 68T0790C26
keywords federatedlearningfoundationmodelfine-tuninghybridheterogeneityLoRAtaskknowledgedisentanglementparameter-efficientbi-leveloptimizationconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated fine-tuning of foundation models usually assumes that every client runs the same model, the same layer structure, the same hidden width, and the same kind of task. This paper argues that real deployments instead mix model families, model scales, layer depths, hidden dimensions, task objectives, and hardware budgets, and it proposes H2Tune, a framework meant to make knowledge transfer work under all of those differences at once. The central claim is that three mechanisms — sparsified triple matrix decomposition, relation-guided layer alignment, and alternating task-knowledge disentanglement — let clients share a common task-shared matrix while keeping task-specific and resource-specific structure local. If the claim is right, the long-standing assumption that federated fine-tuning requires matched models is removable, with a provable $O(1/\sqrt{T})$ convergence rate and consistent accuracy gains, up to 15.4%, across three heterogeneous-model scenarios.

What carries the argument

The object that carries the argument is TriLoRA, a sparsified triple matrix decomposition of each layer's update: $\Delta W = (A_k + A_k \cdot (\Phi_k \cdot R_k)) \cdot B_k$, with $A_k \in \mathbb{R}^{a \times r_g}$, $B_k \in \mathbb{R}^{r_g \times b}$, and $R_k, \Phi_k \in \mathbb{R}^{r_g \times r_g}$. $A_k$ and $B_k$ are dense private matrices that carry task-specific knowledge; $R_k$ is the public task-shared matrix that clients upload and the server averages; $\Phi_k$ is a client-specific sparse matrix whose sparsity ratio $\beta_k$ tunes the shared part to local resources. Two further mechanisms handle the other heterogeneity dimensions: the layer relation matrix $\Omega_k \in \mathbb{R}^{L_k \times L_g}$ linearly maps local shared matrices to a global layer count and back, and the alternating optimization (freeze $A_k$, $B_k$ while training $R_k$; then freeze $R_k$ while training $A_k$, $B_k$) is what separates task-shared from task-specific knowledge. The proof recasts the whole scheme as the bi-level problem $\min_R L(R, H^*(R)) + h(R)$ with $H^*(R) = \arg\min_H G(R, H)$, where a strongly convex regularizer makes the inner problem tractable and yields the $O(1/\sqrt{T})$ rate.

What would settle it

Measure a client's peak GPU memory and per-step FLOPs under the Table 4 configuration that declares a client with support max rank 16 while the global rank is set to 64. If the footprint tracks the rank-16 budget, the sparsity mechanism delivers the resource savings; if it scales with the global rank 64, the resource-heterogeneity claim fails, since the dense private matrices $A_k$ and $B_k$ are never shrunk by the sparsity mask. A second, independent check is to train with the task-shared matrix $R_k$ removed and test whether a client's accuracy on another client's task collapses, which would indicate whether task-specific knowledge had been leaking into the exchanged matrix.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that hybrid heterogeneous federated fine-tuning — clients that differ at once in model architecture, layer count, hidden dimension, rank budget, and downstream task — has a workable solution called H2Tune. Each client decomposes its layer-wise low-rank update as $\Delta W = (A_k + A_k \cdot (\Phi_k \cdot R_k)) \cdot B_k$, where $A_k$ and $B_k$ are private matrices that absorb task-specific knowledge, $R_k$ is a task-shared middle matrix held at one uniform global rank $r_g$ across all clients, and $\Phi_k$ is a client-specific sparsity mask that adapts the shared part to local resources. A trainable relation matrix $\Omega_k$ maps each client's shared matrices onto a common global layer count before server averaging and maps the global result back, and an alternating optimization updates $R_k$ while freezing $A_k$ and $B_k$, then updates $A_k$ and $B_k$ while freezing $R_k$, so that only task-shared knowledge enters the exchanged matrix. The paper proves an $O(1/\sqrt{T})$ convergence rate for this scheme by treating it as a bi-level optimization problem, and reports accuracy gains over the LOCAL, FLLM, HetLoRA, and FLTLA baselines across three scenarios built from seven foundation models, with the largest gain at 15.4%.

Load-bearing premise

The load-bearing premise is that a per-client sparsity mask over the shared middle matrix fully reconciles one uniform global rank with smaller client budgets, even though the dense private matrices $A_k$ and $B_k$ still scale with that global rank — so a client declared to support rank 16 in Table 4 is apparently storing rank-64 matrices when the global rank is 64 — and the paper never measures that client's memory or compute.

Editorial extensions

If this is right

  • Clients running different foundation model families and scales — for example an 8-billion-parameter model alongside a 1-billion-parameter model — can federate fine-tuning without agreeing on architecture, layer count, hidden width, or task.
  • Only the task-shared middle matrix is uploaded to the server, so the channel for cross-client knowledge transfer is explicit and the task-specific matrices never leave the client.
  • The proved $O(1/\sqrt{T})$ convergence rate means that handling this extra heterogeneity does not force a worse asymptotic convergence order than standard federated fine-tuning.
  • A client's resource budget is represented by a sparsity ratio rather than by its LoRA rank, so one global rank can serve clients of different capacities, and the hyper-parameter study indicates that the ratio can be tuned per task.
  • The same framework covers both the homogeneous case (identical models and tasks) and the heterogeneous case, with consistent gains across all three tested scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is a resource-aware rule for choosing the global rank and global layer count: the paper's own sweeps (Tables 4 and 6) show accuracy plateauing after rank 128 and layer count 32, so the sweet spot could be selected per federation instead of fixed by hand.
  • The proof appendix skips the verification of Lemma 2, the inner-problem loss being strongly convex, calling it trivial; the claim is standard given the regularization coefficient, but the convergence theorem depends on a short verification that is not written out.
  • Because only the task-shared matrix $R_k$ is exchanged while $A_k$ and $B_k$ stay local, the framework's communication pattern is compatible with adding secure aggregation or differential privacy over the uploaded matrices; the paper does not analyze privacy, but nothing in the design rules it out.
  • The paper does not report a direct measure of task-specific knowledge leaking into the shared matrix; an obvious test is to remove the KL term from the shared-loss objective and check whether cross-client transfer changes, which would show whether that term is what prevents interference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes H2Tune, a federated fine-tuning framework for foundation models that targets hybrid heterogeneity: clients differ in model architecture (layers, hidden dimensions, rank), downstream task, and computational resources. The method decomposes each layer's LoRA update as ΔW = (A + A(Φ R))B, where R is a task-shared middle matrix, A and B are task-specific private matrices, and Φ is a client-specific sparsification matrix. A trainable layer-relation matrix Ω aligns local layer counts to a global layer count for server aggregation. An alternating optimization step updates (R, Φ, Ω) and (A, B) separately to disentangle task-shared from task-specific knowledge. The authors prove a convergence theorem for the alternating scheme and report experiments on MATHInstruct and GLUE across Gemma, Llama, SmolLM, and Yi models, claiming up to 15.4% average accuracy improvement over baselines.

Significance. If the claims were fully supported, H2Tune would be a notable contribution: it is among the first FFT frameworks to jointly address layer, dimension, architecture, task, and resource heterogeneity, and it provides a convergence theorem plus experiments across seven modern LLM families. The paper's strengths include the clearly motivated problem formulation, the breadth of the empirical evaluation, the ablation study showing each component contributes, and the availability of code. However, two load-bearing pillars need scrutiny: the resource-heterogeneity mechanism does not actually shrink the dense per-client matrices, and the convergence guarantee is proved for a simplified proximal abstraction rather than for the deployed algorithm. The empirical headline figure also needs to be read against the fact that it is an average over all baselines, including a weak no-federation lower bound.

major comments (3)
  1. [§4.2, Eq. (3), and Table 4] The resource-heterogeneity claim is not supported by the method's own equations. Eq. (3) defines ΔW_l_k = (A_l_k + A_l_k(Φ_l_k R_l_k))B_l_k with A_l_k ∈ R^{a×r_g}, R_l_k, Φ_l_k ∈ R^{r_g×r_g}, and B_l_k ∈ R^{r_g×b}. The sparsification ratio β_k applies only to Φ_l_k; A_l_k and B_l_k remain dense matrices whose width is the global rank r_g. Therefore a client declared to 'support max rank 16' in Table 4 must still instantiate dense matrices of width r_g = 64, 128, or 192 when the global rank is set to those values, which is 4x, 8x, and 12x the parameter and activation memory of a rank-16 LoRA client. Sparsifying Φ_l_k does not reduce the size of A_l_k or B_l_k, so the mechanism described in §4.2 does not actually accommodate resource-heterogeneous clients within their stated budgets. The resource-heterogeneity branch of the central claim is therefore not established.
  2. [Appendix A, Appendix B, and Theorem 1] The convergence guarantee does not cover the deployed algorithm. Theorem 1 is restated in Appendix B as Theorem 9 and proved for Algorithm 2, a proximal abstraction that drops the sparsification matrix Φ_k, the layer-relation matrix Ω_k, the local sparsity constraints, and the layer-alignment mechanism. Algorithm 1, which is the algorithm actually evaluated in Section 6, includes all of these components. The theorem's variables are (R, H) only, so the O(1/√T) claim in the abstract and Section 5 does not govern the method as implemented. In addition, the proof relies on the assumption h(R_t^{τ-1}) ≤ h(R_t^0) for all t, stated in Theorem 1 and used in the derivation, but no justification or proof is provided for this trajectory condition. Finally, the abstract states a rate of O(1/√T), while Theorem 1's Eq. (11) bounds the average squared gradient norm by O(1/T); the relationship between the two rates should be clarified.
  3. [Table 3 and §6.2] The empirical claim of 'consistent improvements across all scenarios' and the headline 'up to 15.4% accuracy improvement' are based on the 'avg. Imp' row, which is the average improvement over all four baselines including the LOCAL no-federation lower bound. This metric is not the same as improvement over the strongest baseline. There are concrete counterexamples: in Scenario 1, heterogeneous GLUE NLI@128, H2Tune scores 86.0 while HetLoRA scores 86.8, so H2Tune is worse than the best baseline; several other cells (e.g., Scenario 1 SPC@64 and multiple Scenario 2 GLUE cells) are ties with HetLoRA. The paper should report per-column comparisons against the best state-of-the-art baseline, and the abstract and Section 6.2 should be reworded to reflect that the 15.4% figure is an average over baselines rather than a gain over the best competing method.
minor comments (6)
  1. [Abstract and Theorem 1] The abstract and Section 5 claim a convergence rate of O(1/√T), while Theorem 1's displayed bound is O(1/T) for the averaged squared gradient norm. Please state which quantity is being measured and make the rate statement consistent.
  2. [Figure 5 and Table 4] The text in Figure 5 refers to maximum ranks 'varying from 64 to 196', but Table 4 and the surrounding discussion use global rank values 64, 128, and 192. Please harmonize these numbers.
  3. [Algorithm 1] The update line 'update Φ^{t,j}_k R^{t,j}_k, = Φ^{t,j}_k R^{t,j−1}_k − η′∇_R L_share(x_i, y_i)' contains malformed notation and appears to conflate the matrix product Φ R with a single variable. Please rewrite this line and separate the updates for Φ_k and R_k.
  4. [Eqs. (6) and (7)] The definitions of y'_i and y''_i are not spelled out; in particular, Eq. (7) uses y''_i as 'the prediction based on optimized R_k' but it is unclear how this differs from y'_i and which parameters are frozen in each case.
  5. [Table 6] There are typographical errors in the table entries: '41,3' should be '41.3' and '42,9' should be '42.9'.
  6. [Table 3 caption] The caption does not define 'avg. Imp'. The text in Section 6.2 explains that it is an average improvement, but it should be stated explicitly in the caption that this is the average improvement over all baselines, not over the best baseline.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional circularity in the 'task-shared knowledge' label; core convergence proof and empirical results remain independent.

  1. self definitional [Section 4.4, Eqs. (6)-(7); Section 4.1 upload protocol]
    "we propose an alternating optimization approach that separates task-shared knowledge into matrix Rk and task-specific knowledge into matrices Ak and Bk. ... After local updating, all clients only upload their task-shared matrices Rk to server, maintaining privacy of their task-specific matrices."

    The label 'task-shared' is applied to Rk because Rk is the only matrix uploaded and aggregated, while Ak and Bk are kept private by protocol. Equation 6 trains Rk with KL(Rk, Rg->k), forcing Rk toward the global average, so 'sharedness' is imposed by construction. The claim that the alternating optimization 'separates task-shared knowledge into Rk' therefore restates the upload rule (only Rk is exchanged) rather than deriving a property of the learned parameters. Similarly, the negative KL term in Eq. 7 defines task-specificity as divergence from the shared prediction. This is definitional labeling rather than demonstrated disentanglement, though the TSM ablation provides independent empirical support, making the circularity minor.

full rationale

Aside from the definitional labeling issue in Section 4.4, the paper's central derivation is self-contained. The TriLoRA decomposition (Eq. 3) is a stated parameterization, not a prediction derived from fitted data; the layer alignment (Eqs. 4-5) is an explicit mapping; and the convergence rate is a mathematical theorem proved from standard Lipschitz/smoothness assumptions using external lemmas (e.g., Ghadimi and Wang, Ji et al., Ghadimi et al.), not from the experimental results. Accuracy gains are measured on external benchmarks (MATHInstruct, GLUE) against baselines, so the empirical claims are independently grounded. The resource-heterogeneity weakness in Section 4.2/Table 4 - dense Ak and Bk still scale with global rank rg even when the sparsity matrix Phi_k is sparsified - is a correctness gap in the resource-adaptation claim, not a circularity, so it does not increase the circularity score. No load-bearing self-citation chain or fitted-parameter-as-prediction pattern is present. Overall, one minor definitional circularity warrants a score of 2.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central mechanism requires three free hyperparameters (global rank, sparsity ratio, global layers) that are tuned on test accuracy (Tables 4, 6, 7), plus an unstated regularization V in the convergence condition. The proof relies on standard smoothness axioms and one unproven inequality (h(R^{tau-1}) <= h(R^0)), while the method itself assumes that linear layer mapping and middle-matrix sparsification achieve semantic and resource alignment.

free parameters (5)
  • global_rank_rg = 64, 128, 192
    Uniform rank of shared middle matrices; results reported as a function of rg (Table 4) and the best value is used in the main results, so it is selected on test accuracy.
  • sparsity_ratio_beta_k = 0.125, 0.500, 1.000
    Per-client sparsification ratio for Phi_k; Table 7 sweeps these and the best per task is used, i.e., selected on test accuracy.
  • global_layer_count_Lg = 18-48
    Number of global layers for Omega alignment; Table 6 sweeps L and the best per scenario is used.
  • regularization_coefficient_V = not stated
    Appears in Eq. 7 and controls alpha = -L + V required for strong convexity in Theorem 1; no value is given anywhere.
  • learning_rates_eta_eta_prime = 2e-7 to 2e-3
    Standard training hyperparameters, chosen by hand over a wide range.
assumptions (5)
  • standard math Smoothness, Lipschitz, bounded-domain assumptions (Assumptions 1-5)
    Standard conditions for bilevel optimization convergence proofs (cites 15, 22, 31).
  • ad hoc to paper h(R^{tau-1}_t) <= h(R^0_t) for all t
    Assumed in Theorem 1 without derivation; restricts the proximal iteration behavior and is not verified in the algorithm.
  • domain assumption A linear map Omega_k can align layers of different model families
    Eqs 4-5 assume a trainable linear transformation between local and global layer spaces is semantically meaningful across LLaMA, Gemma, Yi, SmolLM.
  • domain assumption Sparsifying Phi_k keeps clients within their resource budgets even though A_k and B_k have width r_g
    Section 4.2 claims resource adaptation via beta_k, but Eq. 3 sizes A_k and B_k by the global rank r_g.
  • domain assumption Alternating updates of R and A,B make R contain only task-shared knowledge
    Eqs 6-7 rely on the loss split to disentangle knowledge; no formal or measured guarantee is given.
invented entities (3)
  • task-shared middle matrix R_k independent evidence
    purpose: Uploaded to server for cross-client knowledge transfer
    Ablation w/o TSM degrades accuracy (Table 5), giving an empirical handle.
  • layer relation matrix Omega_k
    purpose: Aligns heterogeneous layer counts between clients and server
    No ablation isolates Omega_k; the w/o TSM ablation removes R, Omega, and the KL term together.
  • sparsification matrix Phi_k independent evidence
    purpose: Adjusts parameter cost per client resources
    Ablation w/o SM degrades MATHInstruct accuracy, but the effect is task-dependent (Table 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity." pith.science (2026). https://pith.science/paper/OX3YBPWU

@misc{pith2026250722633,
  author       = {Pith},
  title        = {Pith review of: H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OX3YBPWU}},
  note         = {Machine review of arXiv:2507.22633}
}
read the original abstract

Different from existing federated fine-tuning (FFT) methods for foundation models, hybrid heterogeneous federated fine-tuning (HHFFT) is an under-explored scenario where clients exhibit double heterogeneity in model architectures and downstream tasks. This hybrid heterogeneity introduces two significant challenges: 1) heterogeneous matrix aggregation, where clients adopt different large-scale foundation models based on their task requirements and resource limitations, leading to dimensional mismatches during LoRA parameter aggregation; and 2) multi-task knowledge interference, where local shared parameters, trained with both task-shared and task-specific knowledge, cannot ensure only task-shared knowledge is transferred between clients. To address these challenges, we propose H2Tune, a federated foundation model fine-tuning with hybrid heterogeneity. Our framework H2Tune consists of three key components: (i) sparsified triple matrix decomposition to align hidden dimensions across clients through constructing rank-consistent middle matrices, with adaptive sparsification based on client resources; (ii) relation-guided matrix layer alignment to handle heterogeneous layer structures and representation capabilities; and (iii) alternating task-knowledge disentanglement mechanism to decouple shared and specific knowledge of local model parameters through alternating optimization. Theoretical analysis proves a convergence rate of O(1/\sqrt{T}). Extensive experiments show our method achieves up to 15.4% accuracy improvement compared to state-of-the-art baselines. Our code is available at https://anonymous.4open.science/r/H2Tune-1407.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 29 canonical work pages

  1. [1]

    J.-H. Ahn, O. Simeone, and J. Kang. Wireless federated distillation for distributed edge learning with heterogeneous data. In 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pages 1–6. IEEE, 2019

    work page 2019

  2. [2]

    J.-H. Ahn, O. Simeone, and J. Kang. Cooperative learning via fed- erated distillation over fading channels. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8856–8860. IEEE, 2020

    work page 2020

  3. [3]

    S. Alam, L. Liu, M. Yan, and M. Zhang. Fedrolex: Model- heterogeneous federated learning with rolling sub-model extraction. Advances in neural information processing systems , 35:29677–29690, 2022

    work page 2022

  4. [4]

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Pro- cessing Systems, 2024

    work page 2024

  6. [6]

    Chang, V

    H. Chang, V . Shejwalkar, R. Shokri, and A. Houmansadr. Cronus: Ro- bust and heterogeneous collaborative learning with black-box knowl- edge transfer. arXiv preprint arXiv:1912.11279, 2019

    arXiv 1912

  7. [7]

    Chen and A

    J. Chen and A. Zhang. Fedmsplit: Correlation-adaptive federated multi- task learning across multimodal split networks. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data min- ing, pages 87–96, 2022

    work page 2022

  8. [8]

    J. Chen, R. Zhang, J. Guo, Y . Fan, and X. Cheng. Fedmatch: Federated learning over heterogeneous question answering data. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 181–190, 2021

    work page 2021

Show all 45 references
  1. [9]

    Y .-Q. Chen, T. Zhang, X.-L. Jiang, Q. Chen, C.-L. Gao, and W.-L. Huang. Fedbone: Towards large-scale federated multi-task learning. Journal of Computer Science and Technology, 39(5):1040–1057, 2024

    2024

  2. [10]

    Cheng, J

    S. Cheng, J. Wu, Y . Xiao, and Y . Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027, 2021

    2021

  3. [11]

    Y . J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12903–12913, 2024

    2024

  4. [12]

    E. Diao, J. Ding, and V . Tarokh. Heterofl: Computation and commu- nication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020

    2010 arXiv

  5. [13]

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.-M. Chan, W. Chen, et al. Parameter-efficient fine-tuning of large-scale pre- trained language models. Nature Machine Intelligence, 5(3):220–235, 2023

    2023

  6. [14]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Ghadimi and M

    S. Ghadimi and M. Wang. Approximation methods for bilevel program- ming. arXiv preprint arXiv:1802.02246, 2018

    2018 arXiv

  8. [16]

    Ghadimi, G

    S. Ghadimi, G. Lan, and H. Zhang. Mini-batch stochastic approxima- tion methods for nonconvex stochastic composite optimization. Mathe- matical Programming, 155(1):267–305, 2016

    2016

  9. [17]

    Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  10. [18]

    Horvath, S

    S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane. Fjord: Fair and accurate federated learning under heteroge- neous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876–12889, 2021

    2021

  11. [19]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [20]

    Huang, J

    F. Huang, J. Li, S. Gao, and H. Huang. Enhanced bilevel optimiza- tion via bregman distance. Advances in Neural Information Processing Systems, 35:28928–28939, 2022

    2022

  13. [21]

    J. Jang, H. Ha, D. Jung, and S. Yoon. Fedclassavg: Local representation learning for personalized federated learning on heterogeneous neural networks. In Proceedings of the 51st international conference on par- allel processing, pages 1–10, 2022

    2022

  14. [22]

    K. Ji, J. D. Lee, Y . Liang, and H. V . Poor. Convergence of meta-learning with task-specific adaptation over partial parameters. Advances in Neu- ral Information Processing Systems, 33:11490–11500, 2020

    2020

  15. [23]

    K. Ji, J. Yang, and Y . Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pages 4882–4892. PMLR, 2021

    2021

  16. [24]

    R. Jia, W. Xie, J. Lei, H. Qin, J. Ma, and L. Fang. Towards efficient model-heterogeneity federated learning for large models.arXiv preprint arXiv:2411.16796, 2024

    2024 arXiv

  17. [25]

    Y . Jia, X. Zhang, A. Beheshti, and W. Dou. Fedlps: heterogeneous fed- erated learning for multiple tasks with local parameter sharing. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12848–12856, 2024

    2024

  18. [26]

    Jiang, H

    J. Jiang, H. Jiang, Y . Ma, X. Liu, and C. Fan. Low-parameter federated learning with large language models. In International Conference on Web Information Systems and Applications , pages 319–330. Springer, 2024

    2024

  19. [27]

    Kuang, B

    W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y . Xie, Y . Li, B. Ding, and J. Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. InProceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data ...

    2024

  20. [28]

    X.-Y . Liu, R. Zhu, D. Zha, J. Gao, S. Zhong, M. White, and M. Qiu. Differentially private low-rank adaptation of large language model us- ing federated learning. ACM Transactions on Management Information Systems, 2023

    2023

  21. [29]

    Y . Liu, Y . Zhang, Q. Li, T. Liu, S. Feng, D. Wang, Y . Zhang, and H. Schütze. Hift: A hierarchical full parameter fine-tuning strategy. arXiv preprint arXiv:2401.15207, 2024

    2024 arXiv

  22. [30]

    X. Lu, Y . Liao, C. Liu, P. Lio, and P. Hui. Heterogeneous model fusion federated learning mechanism based on model mapping. IEEE Internet of Things Journal, 9(8):6058–6068, 2021

    2021

  23. [31]

    Rajput, A

    S. Rajput, A. Gupta, and D. Papailiopoulos. Closing the convergence gap of sgd without replacement. In International Conference on Ma- chine Learning, pages 7964–7973. PMLR, 2020

    2020

  24. [32]

    T. Shen, J. Zhang, X. Jia, F. Zhang, G. Huang, P. Zhou, K. Kuang, F. Wu, and C. Wu. Federated mutual learning. arXiv preprint arXiv:2006.16765, 2020

    2006 arXiv

  25. [33]

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  26. [34]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  27. [35]

    A. Wang. Glue: A multi-task benchmark and analysis platform for nat- ural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  28. [36]

    C. Wu, F. Wu, L. Lyu, Y . Huang, and X. Xie. Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022

    2022

  29. [37]

    Q. Yang, Y . Liu, T. Chen, and Y . Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

    2019

  30. [38]

    Z. Yang, S. Fu, W. Bao, D. Yuan, and A. Y . Zomaya. Fastslowmo: Fed- erated learning with combined worker and aggregator momenta. IEEE Transactions on Artificial Intelligence, 4(5):1041–1050, 2022

    2022

  31. [39]

    L. Yi, H. Yu, G. Wang, and X. Liu. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283, 2023

    2023 arXiv

  32. [40]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  33. [41]

    X. Yue, X. Qu, G. Zhang, Y . Fu, W. Huang, H. Sun, Y . Su, and W. Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  34. [42]

    Zhang, S

    J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y . Chen. Towards building the federatedgpt: Federated instruction tun- ing. In ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6915–6919. IEEE, 2024

    2024

  35. [43]

    Zhang, L

    L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y . Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learn- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174–10183, 2022

    2022

  36. [44]

    L2 1 − 2 L2 + α τ 1 α + 2 L2 + α L2L4 α + L3 ∆ 1 − 2 L2+α · α τ 1 − 2 L2+α · α − L2−α L2+α #2 + 4L2 2∆2. (36) Thus, we have: − ∥Gτ −1 t ∥2 ≤ −1 2 ∥ ˜Gτ −1 t ∥2 + 2(L2 + L2 2 α )2

    L. Zhang, D. Wu, and X. Yuan. Fedzkt: Zero-shot knowledge trans- fer towards resource-constrained federated learning with heterogeneous on-device models. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS), pages 928–938. IEEE, 2022. A Algorithm...

    2022

  37. [45]

    (43) Thus, we complete the proof of Theorem 1

    − inf R F (R) τ · T + 1 τ 5 9L0 + 2 9L0 (τ − 1) ( (L2 + L2 2 α )2 L2 − α L2 + α 2τ ∆2 + L2 1 L2(1 − 2 L2+α )τ α + 2 L2 + α L2L4 α + L3 ∆· (42) (1 − 2 L2+α · α)τ 1 − 2 L2+α · α − L2−α L2+α 2) + 1 τ 5 9L0 + 2 9L0 (τ − 2) L2 2∆2. (43) Thus, we complete the proof of Theorem 1

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.