Pith. sign in

REVIEW 5 major objections 4 minor 40 references

Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated fine-tuning can avoid catastrophic forgetting by adding trainable transformer blocks to a frozen backbone; the paper's FedBE method reports higher retention and faster convergence than existing FedFT methods without hurting…

desk verdict Plausible block-expansion method with an unverified forgetting claim: Table 3 never defines a before/after retention metric, so the headline numbers don't yet hold up. read the letter →

arxiv 2506.05977 v1 pith:B5QUVYAJ submitted 2025-06-06 cs.LG cs.DC

Yujia Huo , Jianchun Liu , Hongli Xu , Zhenguo Ma , Shilong Wang , Liusheng Huang This is my paper
classification cs.LGcs.DC
keywords FederatedlearningCatastrophicforgettingParameter-efficientfine-tuningTransformerblockexpansionLargelanguagemodelsDataheterogeneityKnowledgeretention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated fine-tuning of large language models has a specific failure mode: as clients adapt a shared pretrained model to their own tasks and data, the model loses general knowledge it had before fine-tuning, and non-IID data makes the loss worse. This paper proposes FedBE, a method that keeps the pretrained backbone frozen, appends trainable transformer blocks at selected layers, and assigns different subsets of those blocks to different clients according to their data distribution and computing power. The claim is that this structural separation of new task knowledge from old representations prevents overwriting, so the model retains more general ability while still adapting well to the downstream task. On RoBERTa-base and BERT-large, the paper reports 12-74% higher accuracy retention on GLUE tasks after fine-tuning on AG News or IMDB and a 1.9-3.1x faster convergence to target accuracy compared with FedNLP, FedAdapter, and FedLoRA.

What carries the argument

The load-bearing object is the expansion block: a trainable transformer block copied in architecture from a selected frozen layer, with all weights zero-initialized, inserted so its output adds into the residual stream, $x_{l+1}=x_l+\phi_l(x_l)+\phi_l^{\mathrm{expand}}(\phi_l(x_l))$. Because the new term is zero at initialization, the expanded model is exactly the original model until training; the gradient then flows preferentially into the new block, giving a structural separation of task-specific knowledge from pretrained knowledge. Selection of layers uses a composite score $S(l)=g(l)/\max_{l'} g(l')+\lambda\, d(l,E_k)/L$, where $g(l)$ is the $L^2$ gradient norm of layer $l$ and $d(l,E_k)$ is its distance to already selected expansion sites. The second machinery is the client-aware allocator, which ranks expanded blocks for each client by a score combining the Dirichlet heterogeneity measure $D_i=-\log(\alpha_i+\epsilon)$, historical training counts, and a resource capability score, then adjusts the number of blocks per client by the ratio of global target training time to the client's previous-round training time.

What would settle it

Run a retention test with one fixed classifier head attached to every model before and after fine-tuning, evaluating the same GLUE tasks with that identical head; if FedBE's advantage over the baselines and over ORI shrinks or disappears when heads and evaluation protocol are held fixed, the reported retention gains are an artifact of head adaptation rather than evidence of preserved pretrained knowledge.

Watch

Extended reading notes

Core claim

FedBE's central claim is that catastrophic forgetting in federated fine-tuning is avoidable without replay or regularization, by changing where new knowledge is stored rather than constraining how old weights move. The method expands the model by inserting zero-initialized transformer blocks in parallel with selected frozen layers; at initialization these blocks output zero, so they do not alter the original model, and during training gradients concentrate in the new blocks, leaving pretrained representations intact. Which layers get expanded is decided on the server by gradient norms measured on a small proxy dataset, with a spatial penalty to spread the blocks, and the number of blocks is capped by memory and FLOPs budgets. Clients then receive different subsets of the expanded blocks: clients with more uniform data get lower-level blocks, clients with skewed data get higher-level blocks, and resource-limited clients get fewer blocks based on previous-round training time. The paper's experiments on RoBERTa-base and BERT-large claim that this design retains 12-74% more general-task accuracy after downstream fine-tuning and reaches target accuracy 1.9-3.1x faster than three federated fine-tuning baselines, without reducing downstream accuracy.

Load-bearing premise

The evaluation assumes that a model's accuracy on GLUE tasks measured after fine-tuning on AG News or IMDB reflects how much pretrained knowledge it retained, but the paper does not specify how the GLUE classification heads are attached or what the ORI baseline represents, so some of FedBE's higher GLUE scores could reflect task transfer rather than retention.

Editorial extensions

If this is right

  • Federated fine-tuning no longer has to trade downstream performance against retention: a model can gain task-specific capacity in extra blocks while the original weights stay untouched.
  • The same gradient-norm expansion rule can be applied to any transformer model, since it only requires per-layer gradient magnitudes and a memory/FLOPs budget.
  • Heterogeneous clients can train with different sized workloads, so weak devices become a scheduling problem rather than a bottleneck; the paper reports 1.9-3.1x faster time-to-target accuracy.
  • After training, the original backbone remains fully intact, so the fine-tuned model is a superset of the pretrained model, potentially allowing rollback or reuse of the pretrained behavior without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ORI in the paper's Table 3 is a zero-shot pretrained model, then several FedBE GLUE scores being higher than ORI would show positive task transfer from AG News or IMDB, not knowledge retention; the retention claim needs a before/after comparison on identical heads to be fully established.
  • The expansion mechanism is architectural rather than federated-specific, so the same frozen-backbone-plus-trainable-blocks recipe could be tested in centralized continual learning, where it may also reduce forgetting without replay.
  • The client-aware allocator suggests a testable extension: use measured gradient alignment between each client's local task and each expansion layer to assign blocks, which could improve allocation under unknown data distributions.
  • Because each new task adds blocks, a deployment system would need block pruning or merging after a sequence of tasks to keep the footprint bounded; the paper's budget constraint covers one fine-tuning round but not unbounded task accumulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes FedBE, a federated fine-tuning framework that appends zero-initialized trainable transformer blocks to selected frozen layers of a pretrained LLM, selects expansion positions using gradient norms with a spread penalty, and dynamically allocates subsets of expanded blocks to clients based on Dirichlet concentration and device capability. Experiments on RoBERTa-base and BERT-large over AG News and IMDB with 80 Jetson devices report higher final accuracy, faster convergence, and 12–74% higher retention on GLUE tasks than FedNLP, FedLoRA, and FedAdapter.

Significance. If the retention and acceleration claims held, FedBE would be a useful contribution to federated fine-tuning for edge networks: it is a coherent architecture-level approach to knowledge isolation that avoids replay, it is validated on a real hardware prototype, and it includes ablations isolating block-position selection and dynamic allocation. The central idea of structurally separating new task-specific blocks from the frozen backbone is plausible and worth pursuing. However, the evidence for the headline forgetting claim is currently incomplete: the retention metric is not operationalized, the key hyperparameters are selected on the same evaluation benchmarks, and the reported numbers are single-run comparisons without variance. The method's strengths—real deployment, evaluation on two model sizes, and ablation design—are offset by these evaluation gaps.

major comments (5)
  1. [Section 4.2, Table 3] The paper's headline claim of 12–74% higher accuracy retention is not established because no retention metric is defined. Table 3 lists absolute GLUE accuracies after fine-tuning on AG News or IMDB and an 'ORI' column described only as 'original pretrained models,' but the text never states how GLUE task heads are attached to the fine-tuned backbone, whether those heads are frozen or retrained, or how the ORI scores are obtained (zero-shot, individually fine-tuned, or other). Since retention requires a before/after comparison, FedBE's higher Table 3 scores could be explained by positive transfer or by evaluation-protocol mismatch rather than by reduced forgetting; for example, the ORI CoLA value of 71.0% for RoBERTa-base is implausible as a zero-shot result and suggests an undocumented protocol. Note also that Section 2.2 uses individually fine-tuned models as the upper-bound reference ('Untrained'), which is inconsistent with the ORI baseline in Table 3. Please specify the exact evaluation pipeline and recompute the retention improvement relative to a clearly defined pre-fine-tuning reference.
  2. [Section 3.2 and Section 4.1] The number k of expanded blocks and the spread penalty λ are chosen using the same tasks later used to measure retention. Section 2.3 selects k from Fig. 3 using MRPC and evaluates on RTE, QNLI, and SST-2, and Section 4.1 fixes λ=0.5 without a sensitivity analysis; these same benchmarks appear in Table 3. The reported gains are therefore in-sample rather than out-of-sample predictions. Please either fix k and λ using a separate validation task not in Table 3, or provide a sensitivity analysis and state the selection procedure explicitly; otherwise the retention advantage may be an artifact of hyperparameter tuning.
  3. [Section 4.2, Tables 2–3 and Figs. 6–7] All experimental results are single-run comparisons with no seeds, error bars, or standard deviations. In Table 2, several differences are below one percentage point (e.g., 82.2% vs. 81.4% on IMDB with BERT-large), so the claimed improvements over baselines and the convergence acceleration ratio of 1.9–3.1x cannot be distinguished from training noise. Please report at least three seeds with means and standard deviations and, for the convergence comparison, confidence intervals or paired-run results.
  4. [Section 4.1, Baselines] The comparison set contains only FedNLP, FedLoRA, and FedAdapter. FedCurv is cited and discussed in Section 1 as a forgetting-aware federated method, but it is not included as a baseline; likewise, no regularization-based baseline (e.g., federated EWC or knowledge-distillation approaches) appears. Without a forgetting-aware control, the specific claim that FedBE 'mitigates catastrophic forgetting' relative to existing methods is not fully tested. Please add at least one such baseline or justify why it is inapplicable to LLM fine-tuning.
  5. [Section 4.1 and Fig. 7] The definition of the training-time metric is ambiguous: the text says the target accuracy is set to 'the lowest accuracy among the four methods,' but Fig. 7 is described as showing the time to reach 90% validation accuracy. These are different criteria, and a low target accuracy can make a method that converges quickly but ultimately performs worse look better. Please state the exact target, report the full accuracy-versus-time curves, and define the 1.9–3.1x acceleration ratio precisely with respect to that target.
minor comments (4)
  1. [References] Reference [22] cites a 2002 wireless routing paper as the source of MRPC; the correct source for the paraphrase corpus is Dolan and Brockett, 'Automatically Constructing a Corpus of Paraphrases,' ACL 2005.
  2. [Figures 6–9] The labels in Figures 6–9 appear garbled in the provided manuscript (e.g., sequences such as '/s48 /s49/s48'); please ensure the figures use readable text labels for method names and axes.
  3. [Section 3.2, Eq. (5) and Algorithm 1] Equation (5) and Algorithm 1 use slightly different formulations of the composite score: Eq. (5) writes g(l)/max g(l') plus λ·d(l,E_k)/L, while Algorithm 1 normalizes g_l to s_grad and then adds λ times the distance penalty without a separate normalization; please align the notation and clarify the role of λ.
  4. [Section 4.1, Evaluation Metrics] The term 'accuracy retention' is used in the abstract and Section 4.2 but is not defined in the Evaluation Metrics list; please define it explicitly there, including the reference model and the head-attachment protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; FedBE's reported gains are empirical comparisons, not conclusions forced by the paper's own definitions or self-citations.

full rationale

FedBE is an empirical systems paper, not a derivation from first principles: the central claims (12-74% retention advantage and 1.9-3.1x convergence acceleration) are asserted from Tables 2-3 and Figs. 6-7, which compare FedBE with FedNLP, FedLoRA, and FedAdapter. The block-expansion position rule (Eqs. 4-5) and client-allocation rule (Eqs. 8-11) are algorithmic choices; neither equation encodes the claimed retention margin, so no result is equal to its input by construction. The self-citations [27], [38], and [39] support only background claims such as Dirichlet data partitioning and prior FL methods; they are not load-bearing. The preliminary choice of k=3 and lambda=0.5 on MRPC/RTE/QNLI/SST-2 is an in-sample hyperparameter-selection weakness, and the Table 3 'ORI' protocol is under-specified, but these are experimental-validity concerns rather than circularity: the configuration choice does not force the reported margin over the baselines, and no fitted parameter is renamed as the retention prediction. Accordingly, no circular step can be exhibited under the quoted-reduction standard.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The ledger has no novel physical entities; the paper's assumptions are architectural and experimental. The main cost is paid in unstated or hand-set hyperparameters (k, lambda, allocation weights, tau, alpha) and in the unproven assumption that gradient norms and zero-initialized blocks deliver knowledge isolation. None of these are derived from first principles; all are tuned or asserted.

free parameters (5)
  • number of expanded blocks k = 3 (suggested by Fig. 3, not stated for main runs)
    Section 2.3 treats k as a system choice balancing MRPC accuracy and retention drops; the main experiments do not report which k was used.
  • lambda (spread penalty in Eq. 5) = 0.5
    Hand-set in Section 4.1 without sensitivity analysis; directly controls which layers are expanded.
  • allocation weights w_d, w_t, w_r (Eq. 9) = 0.5, 0.3, 0.2
    Chosen by hand; no ablation demonstrates robustness to these values.
  • target training time tau (Eq. 11) = not reported
    Controls the dynamic number of blocks per client; without a value the allocation dynamics cannot be reproduced.
  • Dirichlet concentration alpha for non-IID partitioning = not reported for the 80-client main experiments
    Section 2.2 uses alpha = 10, 1, 0.1 to motivate the problem, but Section 4.1 does not state which alpha is used in the reported FedBE results.
assumptions (3)
  • domain assumption Layer-wise gradient norms indicate which transformer blocks matter for a new task and should be expanded.
    Invoked in Section 2.3 and Eq. 5 to select expansion positions; no proof is given for the federated non-IID setting.
  • domain assumption Zero-initialized expansion blocks preserve pretrained behavior and structurally isolate task-specific knowledge.
    Zero output at initialization is true, but the stronger claims about gradient confinement and knowledge separation are asserted in Section 3.2; gradients still flow into later layers.
  • domain assumption The Dirichlet concentration parameter is a sufficient statistic of client data heterogeneity for block assignment.
    Used in Eq. 8 to compute D_i; the paper does not verify that this proxy correlates with actual forgetting in the 80-client setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning." pith.science (2026). https://pith.science/paper/B5QUVYAJ

@misc{pith2026250605977,
  author       = {Pith},
  title        = {Pith review of: Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5QUVYAJ}},
  note         = {Machine review of arXiv:2506.05977}
}
read the original abstract

Federated fine-tuning (FedFT) of large language models (LLMs) has emerged as a promising solution for adapting models to distributed data environments while ensuring data privacy. Existing FedFT methods predominantly utilize parameter-efficient fine-tuning (PEFT) techniques to reduce communication and computation overhead. However, they often fail to adequately address the catastrophic forgetting, a critical challenge arising from continual adaptation in distributed environments. The traditional centralized fine-tuning methods, which are not designed for the heterogeneous and privacy-constrained nature of federated environments, struggle to mitigate this issue effectively. Moreover, the challenge is further exacerbated by significant variation in data distributions and device capabilities across clients, which leads to intensified forgetting and degraded model generalization. To tackle these issues, we propose FedBE, a novel FedFT framework that integrates an adaptive transformer block expansion mechanism with a dynamic trainable-block allocation strategy. Specifically, FedBE expands trainable blocks within the model architecture, structurally separating newly learned task-specific knowledge from the original pre-trained representations. Additionally, FedBE dynamically assigns these trainable blocks to clients based on their data distributions and computational capabilities. This enables the framework to better accommodate heterogeneous federated environments and enhances the generalization ability of the model.Extensive experiments show that compared with existing federated fine-tuning methods, FedBE achieves 12-74% higher accuracy retention on general tasks after fine-tuning and a model convergence acceleration ratio of 1.9-3.1x without degrading the accuracy of downstream tasks.

Figures

Figures reproduced from arXiv: 2506.05977 by the authors.

Figure 1
Figure 1. The illustration of transformer block expansion and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Impact of different settings and methods on catas [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Impact of expanded block placement on downstream [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Workflow of FedBE. generally positive correlation between overall performance and the number of inserted expansion modules within a reasonable range. To ensure deployment efficiency, we consider both the additional computational cost and the potential inference latency…
Figure 6
Figure 6. Figure 6: Accuracy trends of different methods on AG News [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of training time across different methods [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Accuracy comparison under different ablation set [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Training time comparison under different ablation [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 19 canonical work pages

  1. [1]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  2. [2]

    Lan- guage models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhari- wal, A. Neelakantan, P . Shyam, G. Sastry, A. Askellet al., “Lan- guage models are few-shot learners,”Advances in neural informa- tion processing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  3. [3]

    Privacy-preserving deep learning,

    R. Shokri and V . Shmatikov, “Privacy-preserving deep learning,” inProceedings of the 22nd ACM SIGSAC conference on computer and communications security, 2015, pp. 1310–1321

    work page 2015

  4. [4]

    Fednlp: Benchmarking federated learning methods for natural language processing tasks,

    B. Y. Lin, C. He, Z. Zeng, H. Wang, Y. Huang, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, and S. Avestimehr, “Fednlp: Benchmarking federated learning methods for natural language processing tasks,”arXiv preprint arXiv:2104.08815, 2021

    arXiv 2021

  5. [5]

    Parameter- efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Larous- silhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter- efficient transfer learning for nlp,” inInternational conference on machine learning. PMLR, 2019, pp. 2790–2799

    2019

  6. [6]

    Fedadapter: Efficient federated learning for modern nlp,

    D. Cai, Y. Wu, S. Wang, F. X. Lin, and M. Xu, “Fedadapter: Efficient federated learning for modern nlp,”arXiv preprint arXiv:2205.10162, 2022

    arXiv 2022

  7. [7]

    Fedpetuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,

    Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “Fedpetuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,” in Annual Meeting of the Association of Computational Linguistics 2023. Association for Computational Linguistics (ACL), 2023, pp. 9963– 9977

    2023

  8. [8]

    A continual learn- ing survey: Defying forgetting in classification tasks,

    M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learn- ing survey: Defying forgetting in classification tasks,”IEEE trans- actions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3366–3385, 2021

    work page 2021

Show all 40 references
  1. [9]

    Remind your neural network to prevent catastrophic forgetting,

    T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan, “Remind your neural network to prevent catastrophic forgetting,” inEuropean conference on computer vision. Springer, 2020, pp. 466– 483

    2020

  2. [10]

    Rotate your networks: Better weight consoli- dation and less catastrophic forgetting,

    X. Liu, M. Masana, L. Herranz, J. Van de Weijer, A. M. Lopez, and A. D. Bagdanov, “Rotate your networks: Better weight consoli- dation and less catastrophic forgetting,” in2018 24th international conference on pattern recognition (ICPR). IEEE, 2018, pp. 2262–2268

    2018

  3. [11]

    Universal statistics of fisher information in deep neural networks: Mean field approach,

    R. Karakida, S. Akaho, and S.-i. Amari, “Universal statistics of fisher information in deep neural networks: Mean field approach,” inThe 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019, pp. 1032–1041

    2019

  4. [12]

    Overcoming forgetting in federated learning on non-iid data,

    N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor- Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,”arXiv preprint arXiv:1910.07796, 2019

    1910 arXiv

  5. [13]

    Feder- ated learning with non-iid data,

    Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Feder- ated learning with non-iid data,”arXiv preprint arXiv:1806.00582, 2018

    2018 arXiv

  6. [14]

    Advances and open problems in federated learning,

    P . Kairouz, H. B. McMahan, B. Aventet al., “Advances and open problems in federated learning,”arXiv preprint arXiv:1912.04977, 2019

    1912 arXiv

  7. [15]

    Fed- erated multi-task learning under a mixture of distributions,

    O. Marfoq, G. Neglia, A. Bellet, L. Kameni, and R. Vidal, “Fed- erated multi-task learning under a mixture of distributions,”Ad- vances in Neural Information Processing Systems, vol. 34, pp. 15 434– 15 447, 2021

    2021

  8. [16]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Robertset al., “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, 2020. IEEE TRANSACTIONS ON MOBILE COMPUTING 13

    2020

  9. [17]

    Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,

    J. Huang, J.-X. Bai, X. Zhang, Z. Liu, Y. Feng, J. Liu, X. Sun, M. Dong, and M. Li, “Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,”IEEE Transactions on Information Forensics and Security, 2024

    2024

  10. [18]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Ar- cas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273–1282

    2017

  11. [19]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.”ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  12. [20]

    Feder- ated continual learning via knowledge fusion: A survey,

    X. Yang, H. Yu, X. Gao, H. Wang, J. Zhang, and T. Li, “Feder- ated continual learning via knowledge fusion: A survey,”IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 8, pp. 3832–3850, 2024

    2024

  13. [21]

    Roberta: A ro- bustly optimized bert pretraining approach,

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A ro- bustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  14. [22]

    Mrpc: Maximizing network lifetime for reliable routing in wireless environments,

    A. Misra and S. Banerjee, “Mrpc: Maximizing network lifetime for reliable routing in wireless environments,” in2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No. 02TH8609), vol. 2. IEEE, 2002, pp. 800–806

    2002

  15. [23]

    The pascal recognising textual entailment challenge,

    I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” inMachine learning challenges work- shop. Springer, 2005, pp. 177–190

    2005

  16. [24]

    Recursive deep models for semantic compo- sitionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compo- sitionality over a sentiment treebank,” inProceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631–1642

    2013

  17. [25]

    Squad: 100,000+ questions for machine comprehension of text,

    P . Rajpurkar, J. Zhang, K. Lopyrev, and P . Liang, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  18. [26]

    Measuring the effects of non- identical data distribution for federated visual classification,

    T. Hsu, H. Qi, and M. Brown, “Measuring the effects of non- identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335, 2019

    1909 arXiv

  19. [27]

    Finch: Enhancing federated learning with hierarchical neural architecture search,

    J. Liu, J. Yan, H. Xu, Z. Wang, J. Huang, and Y. Xu, “Finch: Enhancing federated learning with hierarchical neural architecture search,”IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 6012–6026, 2023

    2023

  20. [28]

    A continual learning survey: Defy- ing forgetting in classification tasks,

    M. de Lange, R. Aljundi, M. Masana, S. Parisot, A. Davison, T. Tuytelaars, and L. V . Gool, “A continual learning survey: Defy- ing forgetting in classification tasks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3366–3385, 2022

    2022

  21. [29]

    Gradient flow in sparse neural networks and how lottery tickets win,

    U. Evci, T. Gale, E. Elsen, and J. Uszkoreit, “Gradient flow in sparse neural networks and how lottery tickets win,”arXiv preprint arXiv:2007.04686, 2022

    2007 arXiv

  22. [30]

    Docker: lightweight linux containers for consis- tent development and deployment,

    D. Merkelet al., “Docker: lightweight linux containers for consis- tent development and deployment,”Linux j, vol. 239, no. 2, p. 2, 2014

    2014

  23. [31]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, “Pytorch: An imperative style, high-performance deep learning library,”arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  24. [32]

    Gropp, E

    W. Gropp, E. Lusk, and A. Skjellum,Using MPI: portable parallel programming with the message-passing interface. MIT press, 1999, vol. 1

    1999

  25. [33]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  26. [34]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  27. [35]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  28. [36]

    Eu general data protection regulation,

    P . Regulation, “Eu general data protection regulation,”available at, available at: https://eur-lex. europa. eu/eli/reg/2016/679/oj/(accessed 10 September 2023), 2016

    2016

  29. [37]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,”arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  30. [38]

    Adaptive local update and neural composition for accelerating federated learning in heterogeneous edge networks,

    J. Liu, J. Yan, J. Qi, H. Xu, S. Wang, C. Qiao, and L. Huang, “Adaptive local update and neural composition for accelerating federated learning in heterogeneous edge networks,”IEEE Trans- actions on Networking, 2025

    2025

  31. [39]

    Enhancing semi-supervised federated learning with progressive training in heterogeneous edge computing,

    J. Liu, J. Liu, H. Xu, Y. Liao, Z. Yao, M. Chen, and C. Qian, “Enhancing semi-supervised federated learning with progressive training in heterogeneous edge computing,”IEEE Transactions on Mobile Computing, 2024

    2024

  32. [40]

    Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment,

    S. Dou, E. Zhou, Y. Liu, S. Gao, J. Zhao, W. Shen, Y. Zhou, Z. Xi, X. Wang, X. Fanet al., “Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment,”arXiv preprint arXiv:2312.09979, vol. 4, no. 7, 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.