Pith. sign in

REVIEW 4 major objections 3 minor 61 references

Update surgery beats joint training in federated continual learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:14 UTC pith:LJZZPWWX

load-bearing objection Clean server-side projection idea for FCIL with large reported gains, but the theory is overclaimed and the inference-time task-selection protocol is under-specified; worth a careful referee, not worth believing yet. the 4 major comments →

arxiv 2607.19384 v1 pith:LJZZPWWX submitted 2026-07-01 cs.LG

SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning

Jaeik Kim , Jaeyoung Do This is my paper
classification cs.LG
keywords federated learningclass-incremental learningcatastrophic forgettingadaptation vectorsmodel merginggradient surgeryserver-side aggregationmulti-task learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the two intertwined sources of forgetting in Federated Class Incremental Learning (FCIL)—client heterogeneity (spatial) and sequential tasks (temporal)—can be handled together by a single server-side operation: projecting each update vector to remove components aligned with other updates before merging. The authors argue that standard aggregation naively sums updates whose directional interference causes redundant and destructive parameter changes, and that surgically stripping those coupled components yields a cleaner global model. They claim this improves final accuracy by up to 22% over prior FCIL methods across vision and language benchmarks, and even exceeds centralized joint training on several datasets, all without modifying client training. If true, this would make interference-aware aggregation a cheap, drop-in replacement for plain averaging in continual federated systems.

Core claim

The paper's central discovery is that spatio-temporal catastrophic forgetting can be reduced by treating client and task updates as adaptation vectors and removing, at aggregation time, each vector's components that are aligned with the others (Eqs. 5 and 9). Spatial SUM applies this within a round to client updates; causal online temporal SUM applies it sequentially across task vectors anchored to a common base model. The refined vectors are then merged, and compact inference modules are built via sparsification, sign consensus, and task-specific masks. Empirically, SUM reports the highest Final Averaged Accuracy on six vision and two language benchmarks, outperforming all baselines and som

What carries the argument

The load-bearing identity is the projection-based surgery operator: for each adaptation vector v_i, compute \hat{v}_i = v_i - Σ_{j≠i} (v_i·v_j / ||v_j||²) v_j, then aggregate with \hat{v}_i instead of v_i. This operator is applied to client updates within a round (Spatial SUM) and recursively to task vectors that share a common pretrained base (Temporal SUM), making the temporal version a sequential orthogonalization. The same refinement is supported by Theorem 1, which states that under convexity, Lipschitz-smooth losses and a common base, the refined merge has loss no higher than the naive sum. The inference-ready module then compresses the refined task vectors into a unified direction plu

Load-bearing premise

The inference-ready module assumes that the correct task (or set of tasks) can be identified for each test input; if task identity is unavailable at test time, the reported accuracy gains may not materialize because the evaluation protocol becomes task-incremental rather than class-incremental.

What would settle it

Run the same FCIL protocol but evaluate inference with a single shared model (e.g., the spatially aggregated global model without per-task module selection) on test samples whose task labels are concealed; if the accuracy gain over baselines largely disappears, then the improvements hinge on task-selection rather than on the surgery operator itself.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the claim holds, any federated continual system can adopt interference-aware aggregation without changing client code, replay buffers, or extra communication.
  • SUM's improvement over the centralized joint baseline suggests that directional conflict resolution during aggregation can compensate for the loss of central data access.
  • The reported O(TD) server memory can be compressed with top-k/low-precision storage, so the method remains tractable for long task sequences.
  • The method's robustness to corrupted clients suggests that projection surgery also filters malicious update directions, not just heterogeneity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The strongest practical gains depend on the assumption that the correct task-specific module can be selected at inference; the paper does not report results under a fully class-incremental protocol where task identity is unknown, so the headline numbers may overstate the benefit in truly task-agnostic deployment.
  • The core surgery operator could be adapted to other settings that merge parameter updates, such as model merging of independently trained models or federated personalization, where interfering directions are known to hurt.
  • Because the refinement is purely server-side, one could test it as a drop-in addition to existing FL and FCIL pipelines by simply replacing the aggregation rule, making the claimed improvement directly reproducible.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes SUM (Surgery & Merge), a server-side aggregation framework for Federated Class Incremental Learning (FCIL). Updates from clients and tasks are treated as adaptation vectors in parameter space; spatial SUM projects each client update against the others within a round, and temporal SUM projects the current task vector against previously stored task vectors. The refined vectors are then merged, and a compact inference module is built via sparsification, sign-consensus, and element-wise masking. The authors claim up to 22% FAA improvement over prior FCIL methods, robustness to unreliable clients, and no additional client-side overhead. A theorem is stated asserting that SUM merging reduces the loss relative to standard merging under smoothness and convexity assumptions, with a proof built on a transfer of the PCGrad guarantee.

Significance. If the empirical claims hold under a fair FCIL protocol, the work is significant: it addresses a real problem (spatio-temporal interference in FCIL) with a purely server-side mechanism, preserves standard client communication, and reports consistent gains across vision and language benchmarks. The ablation study and robustness analyses are useful. However, the paper currently lacks code and error bars, and the evaluation protocol for inference-time task selection is not clearly specified. The theoretical guarantee, as stated, is not sound because the proof transfers a PCGrad result that applies only to conditional projection (negative inner products) to the unconditional projection in Eq. (5). These issues must be resolved before the central claims can be accepted.

major comments (4)
  1. [§5.1, Tab. 2, Appendix A.3] The evaluation protocol for FAA is not specified with respect to task identity. Appendix A.3 states that the inference-ready module 'assumes that the relevant task adaptation vectors can be selected at inference time,' and the main text does not describe how this selection is performed in Tables 2–3. If oracle task IDs are used to choose the correct module for each test sample, SUM is evaluated in a task-incremental/multi-head setting while baselines such as EWC, LwF, CCVR, and LoRM are single-model class-incremental learners. This would make the reported >Joint results and the 22% improvement an artifact of the protocol rather than of the geometric surgery. The authors must report exactly how inference was conducted, and provide results under strict class-incremental evaluation (or with the proposed retrieval mechanism actually implemented).
  2. [§4.1, Theorem 1, Appendix B.4] Theorem 1 as stated claims L(θSum) ≤ L(θstandard) unconditionally for convex losses with L-Lipschitz gradients. The proof maps Eq. (5) to a PCGrad update, but PCGrad's guarantee in Yu et al. (2020) holds only for the conditional projection used by PCGrad, which subtracts pairwise components only when the inner product is negative. SUM's Eq. (5) subtracts projection components for all nonzero inner products, including positive ones. Appendix B.5 implicitly acknowledges this by downgrading the result to a 'local descent-bound interpretation' under a signal–interference condition, but the main theorem still states an unconditional inequality. This is a load-bearing theoretical claim and must be corrected: either state and prove a conditional theorem with explicit assumptions, or remove the theorem and present the analysis as a heuristic. Note also that for two identical vectors, Eq. (5) can
  3. [§5.1, Tables 2–3] No error bars or standard deviations are reported for the main FAA results. Given the large claimed improvements and the fact that SUM has several per-dataset hyperparameters (z_thr, λ_S, k_pct in Tables 12–19) that appear to be tuned per distribution level, it is important to establish that the gains are not due to hyperparameter selection or run-to-run variance. At minimum, report means and standard deviations over multiple seeds, and state the number of runs. This is especially important because Table 7, which does include errors, shows a large variance (±9.6) for SUM under unreliable-client settings.
  4. [§4.1, Eq. (5)] The projection in Eq. (5) is applied to all pairwise interactions, including positive (aligned) ones. In federated learning, clients working on the same task are expected to share a common signal direction; removing that shared direction from every client's update may discard useful task information and can even reverse the aggregate update when multiple clients are aligned (for n equal vectors, each refined vector becomes (2−n)v). The paper's own ablation in Fig. 3 shows that positive-only projection underperforms the no-surgery baseline. This does not invalidate the empirical results, but it means the method's success cannot be attributed to the projection alone without a more careful analysis of when aligning versus decorrelating helps. The authors should clarify what property of the update distribution makes full orthogonalization beneficial, and ideally provide an experiment isolati
minor comments (3)
  1. [§5.1] The term 'Final Averaged Accuracy (FAA)' is used throughout but never formally defined. Please provide the exact formula (e.g., average accuracy over all tasks after the last task, using which inference rule).
  2. [Appendix B.3] The last displayed equation before Eq. (30) states the stochastic deviation is 'O(η_eff,i) in norm', while the text says O(η_eff,i^2) in mean-square. This is confusing; please be consistent about the quantity being bounded.
  3. [Fig. 3] The figure appears garbled in the PDF (many literal '/uni0000...' sequences), making the ablation curves unreadable. The source figure must be embedded properly.

Circularity Check

0 steps flagged

No significant circularity: the derivation chain is a deterministic projection method with self-contained algebraic/probabilistic supporting results, and the main theory reuses an external PCGrad guarantee rather than the paper's own outputs.

full rationale

The paper's central claim is that removing components of adaptation vectors aligned with other adaptation vectors before merging reduces spatial–temporal interference. This is implemented by deterministic equations: Eq. (5) defines the refined vector as v_i minus projections onto other vectors; Eq. (7) and Eq. (9) instantiate the same projection for client and task vectors. The supporting theory is not circular: Proposition 2 (Appendix B.2) derives the gradient-like interpretation of adaptation vectors from Lipschitz smoothness and local-step assumptions, with a written proof; Theorem 1 (Appendix B.4) transfers PCGrad's published guarantee to the model-merging setting by substituting the Proposition 2 relation, which is an external result, not a self-citation or a restatement of the paper's fitted values; Proposition 1 (Appendix B.6) is an algebraic identity showing projection on accumulated task vectors is equivalent to projection on sequential increments. No parameter is fitted to test labels or to the FAA numbers: the three hyperparameters (zthr, lambda_S, kpct) are validation-tuned and reported transparently, and the reported gains are empirical outcomes rather than consequences of the projection definition. The main limitation noted in Appendix A.3 — that inference-ready modules assume task-relevant adaptation vectors can be selected at inference time — is an evaluation-protocol assumption that could affect the fairness of comparisons, but it is not a circular derivation: the surgery equations themselves do not assume task identity, and the limitation is admitted rather than hidden. There are no load-bearing self-citations, no uniqueness theorem imported from the authors' prior work, and no fitted quantity relabeled as a prediction. Accordingly, the paper's derivation chain is self-contained and no circular step is exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The method rests on standard first-order approximations and an unproven transfer of PCGrad's conditional guarantee to unconditional projection. Three of the method's own hyperparameters (z_thr, lambda_S, k_pct) are tuned per dataset. No new physical entities are introduced; the 'adaptation vector' is a representational device.

free parameters (4)
  • z_thr (Z-score trimming threshold) = 4.5
    Chosen to trim outlier parameter coordinates before projection. Sensitivity plot (Fig. 6) suggests a peak near 3.0, yet 4.5 is used for all main runs.
  • lambda_S (spatial aggregation scaling) = 0.1–1.0 per dataset
    Selected per dataset and Dirichlet beta (Tabs. 12–18); main text cites 0.4 for CIFAR-100. A free strength of the surgery step.
  • k_pct (top-k sparsification percent) = 0.05 or 0.1
    Sparsification ratio for inference-module construction, tuned per dataset; affects the compactness and performance of the final modules.
  • Sum learning rate = 1e-5 to 1e-4 (vision), 2e-3 to 5e-3 (language)
    Standard training hyperparameter, but varies per dataset in the appendix tables; included for completeness.
axioms (6)
  • domain assumption FCIL can be modeled as multi-task learning with client and task updates as adaptation vectors (Sec. 3.1–3.2).
    Framing assumption that motivates the method and the projection operation; not mathematically required for the algorithm to run.
  • domain assumption Adaptation vectors approximate negative scaled gradients at the anchor model (Prop. 2, Appendix B.2).
    Requires small effective step sizes, bounded gradients, and a local neighborhood assumption; standard first-order approximation.
  • domain assumption All adaptation vectors share a common base model, enforced by re-anchoring task vectors to theta_pre (Eq. 8).
    Required for the geometric projection interpretation and for applying the PCGrad-based descent bound.
  • ad hoc to paper Removing directional components of adaptation vectors does not destroy task-specific information (core design premise).
    The method subtracts projections for all nonzero couplings. The paper's own ablation (Fig. 3) shows positive-only suppression harms accuracy, so the unconditional removal premise is not generally valid.
  • ad hoc to paper The PCGrad guarantee (Yu et al. 2020) extends to unconditional projection (Appendix B.4).
    PCGrad only projects conflicting (negative-cosine) gradients; SUM projects all nonzero couplings. The proof does not establish the transfer, and a convex quadratic with identical vectors refutes the unconditional inequality.
  • domain assumption Inference-time task selection is feasible (Appendix A.3).
    The inference-ready modules require knowing which task is being queried; the paper lists a retrieval idea but does not validate it in the main experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 33193 in / 17891 out tokens · 158637 ms · 2026-08-02T09:14:13.678892+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning." pith.science (2026). https://pith.science/paper/LJZZPWWX

@misc{pith2026260719384,
  author       = {Pith},
  title        = {Pith review of: SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJZZPWWX}},
  note         = {Machine review of arXiv:2607.19384}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Real-world intelligent systems often require both distributed collaboration across data-isolated clients and continual adaptation to evolving tasks. This setting naturally gives rise to Federated Class Incremental Learning (FCIL), which combines Federated Learning (FL) and Continual Learning (CL). However, their combination introduces two coupled sources of interference: spatial interference from heterogeneous clients and temporal interference from sequential tasks, jointly leading to Spatial-Temporal Catastrophic Forgetting (ST-CF). Existing approaches typically address spatial and temporal interference with separate mechanisms, often incurring additional client-side computation or communication, while leaving directional interactions among updates during aggregation unregulated. In this paper, we reinterpret FCIL as a unified multi-task learning problem, where both client and task updates are represented as adaptation vectors in a shared parameter space. Based on this view, we propose Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors (SUM), a purely server-side framework that performs geometric surgery on adaptation vectors during aggregation. Spatial SUM mitigates client-level interference within each round, while causal online temporal SUM removes cross-task interference over time without additional client-side computation, communication, or memory beyond standard federated training. Empirically, SUM achieves up to 22% improvement over prior FCIL methods across diverse vision and language benchmarks while remaining robust to unreliable clients and maintaining computational efficiency.

Figures

Figures reproduced from arXiv: 2607.19384 by Jaeik Kim, Jaeyoung Do.

Figure 1
Figure 1. Figure 1: Left: Client or task updates during FCIL are represented as adaptation vectors in parameter space from the base model. Middle: Directional interactions among these vectors may lead to redundant or destructive updates. Right: Rather than resolving these interactions along a single axis (FL-only or CL-only), Sum unifies both axes and regulates them, producing representations comparable to joint training. How… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Surgery & Merge (Sum). After local training for task k over R communication rounds, clients upload their updated weights to the server. The server then applies spatial and temporal Sum to refine the adaptation vectors and construct an inference-ready module for task k. Notably, Sum introduces no additional client-side overhead beyond standard federated model training and communication. 4 Method… view at source ↗
Figure 4
Figure 4. Figure 4: Scaling behavior of Sum. (a) Number of clients. (b) ViT backbone sizes. architectures and varying levels of client heterogeneity. Notably, the LoRA-based Sum even surpasses the full-finetuning (Tab. 2), consistent with findings that parameter-efficient modules preserve pretrained representations better than full fine-tuning while reducing communication cost [8, 41]. Overall, these results in￾dicate that Su… view at source ↗
Figure 5
Figure 5. Figure 5: Learning curves of Sum. A shallower V-shape and stable performance in￾dicate fast adaptation and reduced forgetting. 0.0 2.5 10.0 zthr 40 60 80 100 Mean Accuracy (%) 0.2 0.4 0.6 0.8 1.0 S 70 80 90 100 Mean Accuracy (%) 0 50 100 kpct 70 80 90 100 FAA (%) (a) Z-score threshold 0.0 2.5 10.0 zthr 40 60 80 100 Mean Accuracy (%) 0.2 0.4 0.6 0.8 1.0 S 70 80 90 100 Mean Accuracy (%) 0 50 100 kpct 70 80 90 100 FAA … view at source ↗
Figure 6
Figure 6. Figure 6: Hyperparameter sensitivity analysis of Sum. a representative baseline sensitive to heterogeneity, and LoRM [41], a SOTA method. As the number of clients increases, EWC quickly degrades and LoRM plateaus below Joint, while Sum maintains and even surpasses Joint perfor￾mance. For model scaling, we include FedProto [48], the strongest full-finetuning approach (Tab. 2). As ViT size grows from Tiny to Large, Fe… view at source ↗
Figure 7
Figure 7. Figure 7: Memory footprint and mitigation for server-side task scalability of Sum [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cosine similarity between inter-client vectors before and after Spatial [PITH_FULL_IMAGE:figures/full_fig_p032_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Analysis across communication rounds and aggregation behavior. become perfect orthogonal. This aligns with findings in the model-merging liter￾ature, where orthogonality between adaptation vectors is known to be a desirable property for constructing effective merged MTL models [54]. In Fig. 9a, we further show how the average cosine similarity between client vectors and their corresponding performance evol… view at source ↗
Figure 10
Figure 10. Figure 10: Forgetting metrics in FCIL. FL-specific methods forget heavily, CL-based methods reduce forgetting, and Sum achieves near-zero or even negative forgetting. Without trimming, the parameter sign-flip ratio is roughly 20–30%, and prior work has shown that sign flips during the merging of adaptation vectors can significantly degrade performance [13, 53]. With Z-score trimming, however, we observe that sign fl… view at source ↗
Figure 11
Figure 11. Figure 11: Training loss comparison between Sum and FedAvg. Sum exhibits lower loss in early rounds and approaches convergence more quickly than standard aggregation [PITH_FULL_IMAGE:figures/full_fig_p034_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 8 linked inside Pith

  1. [1]

    Babakniya, S., Fabian, Z., He, C., Soltanolkotabi, M., Avestimehr, S.: A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. Proc. of Neural Information Processing Systems (NeurIPS)36, 66408–66425 (2023) 16 J. Kim and J. Do

    2023

  2. [2]

    arXiv preprint arXiv:2307.04869 (2023)

    Bagwe, G., Yuan, X., Pan, M., Zhang, L.: Fed-cprompt: Contrastive prompt for rehearsal-free federated continual learning. arXiv preprint arXiv:2307.04869 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    arXiv preprint arXiv:2309.01289 (2023)

    Bakman, Y.F., Yaldiz, D.N., Ezzeldin, Y.H., Avestimehr, S.: Federated orthogonal training: Mitigating global catastrophic forgetting in continual federated learning. arXiv preprint arXiv:2309.01289 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2104.05025 (2021)

    Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., Belilovsky, E.: New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025 (2021)

    Pith/arXiv arXiv 2021

  5. [5]

    arXiv preprint arXiv:2504.06768 (2025)

    Chen, S., Zhou, T., Long, G., Jiang, J., Zhang, C.: Fedmerge: Federated personal- ization via model merging. arXiv preprint arXiv:2504.06768 (2025)

    arXiv 2025

  6. [6]

    In: Proc

    Dong, J., Wang, L., Fang, Z., Sun, G., Xu, S., Wang, X., Zhu, Q.: Federated class-incremental learning. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 10164–10173 (2022)

    2022

  7. [7]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recogni- tion at scale. arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  8. [8]

    In: Proc

    Guo, H., Zhu, F., Liu, W., Zhang, X.Y., Liu, C.L.: Pilora: Prototype guided incre- mental lora for federated class-incremental learning. In: Proc. of European Conf. on Computer Vision (ECCV). pp. 141–159. Springer (2024)

    2024

  9. [9]

    IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing12(7), 2217– 2226 (2019)

    Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing12(7), 2217– 2226 (2019)

    2019

  10. [10]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.: The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8340–8349 (2021)

    2021

  11. [11]

    In: Proc

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 15262–15271 (2021)

    2021

  12. [12]

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Proc. of Int’l Conf. on Learning Representations (ICLR)1(2), 3 (2022)

    2022

  13. [13]

    Huang, C., Ye, P., Chen, T., He, T., Yue, X., Ouyang, W.: Emr-merging: Tuning- free high-performance model merging. Proc. of Neural Information Processing Sys- tems (NeurIPS) (2024)

    2024

  14. [14]

    In: Proc

    Huang, W., Ye, M., Du, B.: Learn from others and be yourself in heterogeneous federated learning. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 10143–10153 (2022)

    2022

  15. [15]

    Ilharco, G., Ribeiro, M.T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., Farhadi, A.: Editing models with task arithmetic. Proc. of Int’l Conf. on Learn- ing Representations (ICLR) (2023)

    2023

  16. [16]

    Jin, X., Ren, X., Preotiuc-Pietro, D., Cheng, P.: Dataless knowledge fusion by merging weights of language models. Proc. of Int’l Conf. on Learning Representa- tions (ICLR) (2023)

    2023

  17. [17]

    In: International conference on machine learning

    Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: International conference on machine learning. pp. 5132–5143. PMLR (2020)

    2020

  18. [18]

    Proceedings of the national academy of sciences114(13), 3521–3526 (2017) Surgery & Merge17

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences114(13), 3521–3526 (2017) Surgery & Merge17

    2017

  19. [19]

    In: Proceedings of the IEEE international conference on computer vision workshops

    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine- grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 554–561 (2013)

    2013

  20. [20]

    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images.(2009) (2009)

    2009

  21. [21]

    Lang,K.:Newsweeder:Learningtofilternetnews.In:Machinelearningproceedings 1995, pp. 331–339. Elsevier (1995)

    1995

  22. [22]

    arXiv preprint arXiv:1909.02027 (2019)

    Larson, S., Mahendran, A., Peper, J.J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J.K., Leach, K., Laurenzano, M.A., Tang, L., et al.: An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027 (2019)

    Pith/arXiv arXiv 1909

  23. [23]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Li, Y., Wang, H., Qi, Y., Liu, W., Li, R.: Re-fed+: A better replay strategy for fed- erated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  24. [24]

    Advances in Neural Information Processing Systems38, 104726–104744 (2026)

    Li, Y., Wang, X., Xu, W., Wang, H., Qi, Y., Dong, J., Li, R.: Feature distillation is the better choice for model-heterogeneous federated learning. Advances in Neural Information Processing Systems38, 104726–104744 (2026)

    2026

  25. [25]

    arXiv preprint arXiv:2412.13779 (2024)

    Li, Y., Wang, Y., Wang, H., Qi, Y., Xiao, T., Li, R.: Rehearsal-free con- tinual federated learning with synergistic synaptic intelligence. arXiv preprint arXiv:2412.13779 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    IEEE transactions on pattern anal- ysis and machine intelligence40(12), 2935–2947 (2017)

    Li, Z., Hoiem, D.: Learning without forgetting. IEEE transactions on pattern anal- ysis and machine intelligence40(12), 2935–2947 (2017)

    2017

  27. [27]

    Liu, B., Liu, X., Jin, X., Stone, P., Liu, Q.: Conflict-averse gradient descent for multi-task learning. Proc. of Neural Information Processing Systems (NeurIPS) 34, 18878–18890 (2021)

    2021

  28. [28]

    In: Proc

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 11976–11986 (2022)

    2022

  29. [29]

    Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. Proc. of Neural Information Processing Systems (NeurIPS)30(2017)

    2017

  30. [30]

    Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., Feng, J.: No fear of heterogene- ity: Classifier calibration for federated learning with non-iid data. Proc. of Neural Information Processing Systems (NeurIPS)34, 5972–5984 (2021)

    2021

  31. [31]

    In: Int’l Joint Conf

    Ma, Y., Xie, Z., Wang, J., Chen, K., Shou, L.: Continual federated learning based on knowledge distillation. In: Int’l Joint Conf. on Artificial Intelligence (IJCAI). pp. 2182–2188 (2022)

    2022

  32. [32]

    In: Proc

    Marczak, D., Twardowski, B., Trzciński, T., Cygert, S.: Magmax: Leveraging model merging for seamless continual learning. In: Proc. of European Conf. on Computer Vision (ECCV). pp. 379–395. Springer (2024)

    2024

  33. [33]

    Matena, M.S., Raffel, C.A.: Merging models with fisher-weighted averaging. Proc. of Neural Information Processing Systems (NeurIPS)35, 17703–17716 (2022)

    2022

  34. [34]

    McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: Thesequentiallearningproblem.In:Psychologyoflearningandmotivation,vol.24, pp. 109–165. Elsevier (1989)

    1989

  35. [35]

    In: Artificial intelligence and statistics

    McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)

    2017

  36. [36]

    Journal of Machine Learning Research25(418), 1–53 (2024) 18 J

    Mhanna, E., Assaad, M.: Countering the communication bottleneck in federated learning: A highly efficient zero-order optimization technique. Journal of Machine Learning Research25(418), 1–53 (2024) 18 J. Kim and J. Do

    2024

  37. [37]

    In: ICML 2025 Workshop on Collaborative and Federated Agentic Workflows (2025)

    Nguyen, D.M., Nguyen, L.T., Pham, Q.V.: Spatio-temporal gradient matching for federated continual learning. In: ICML 2025 Workshop on Collaborative and Federated Agentic Workflows (2025)

    2025

  38. [38]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Phan, H., Tran, L., Tran, Q., Tran, N., Truong, T., Lei, Q., Ho, N., Phung, D., Le, T.: Beyond losses reweighting: Empowering multi-task learning via the gener- alization perspective. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2440–2450 (2025)

    2025

  39. [39]

    Journal of machine learning research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)

    2020

  40. [40]

    arXiv preprint arXiv:2103.09762 (2021)

    Saha, G., Garg, I., Roy, K.: Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762 (2021)

    Pith/arXiv arXiv 2021

  41. [41]

    Salami, R., Buzzega, P., Mosconi, M., Bonato, J., Sabetta, L., Calderara, S.: Closed-form merging of parameter-efficient modules for federated continual learn- ing. Proc. of Int’l Conf. on Learning Representations (ICLR) (2024)

    2024

  42. [42]

    arXiv preprint arXiv:2406.02447 (2024)

    Salami, R., Buzzega, P., Mosconi, M., Verasani, M., Calderara, S.: Federated class-incremental learning with hierarchical generative prototypes. arXiv preprint arXiv:2406.02447 (2024)

    Pith/arXiv arXiv 2024

  43. [43]

    Sener, O., Koltun, V.: Multi-task learning as multi-objective optimization. Proc. of Neural Information Processing Systems (NeurIPS)31(2018)

    2018

  44. [44]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Senushkin, D., Patakin, N., Kuznetsov, A., Konushin, A.: Independent component alignment for multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20083–20093 (2023)

    2023

  45. [45]

    In: Proc

    Shi, C., Zhao, H., Zhang, B., Zhou, M., Guo, D., Chang, Y.: Fedawa: Adaptive optimization of aggregation weights in federated learning using client vectors. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 30651–30660 (2025)

    2025

  46. [46]

    Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. Proc. of Neural Information Processing Systems (NeurIPS)30(2017)

    2017

  47. [47]

    In: Proc

    Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., Kira, Z.: Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 11909–11919 (2023)

    2023

  48. [48]

    In: Proceedings of the AAAI conference on artificial intelligence

    Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., Zhang, C.: Fedproto: Federated prototype learning across heterogeneous clients. In: Proceedings of the AAAI conference on artificial intelligence. vol. 36, pp. 8432–8440 (2022)

    2022

  49. [49]

    14768027,https://doi.org/10.5281/zenodo.14768027

    Tziolas, G., Siniosoglou, I., Sarigiannidis, P., Argyriou, V.: A contemporary sur- vey of federated continual learning (2025).https://doi.org/10.5281/zenodo. 14768027,https://doi.org/10.5281/zenodo.14768027

    doi:10.5281/zenodo 2025

  50. [50]

    Wah,C.,Branson,S.,Welinder,P.,Perona,P.,Belongie,S.:Thecaltech-ucsdbirds- 200-2011 dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)

    2011

  51. [51]

    In: Proc

    Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: Proc. of Computer Vision and Pattern Recognition (CVPR). pp. 139–149 (2022)

    2022

  52. [52]

    Worsham, J., Kalita, J.: Multi-task learning for natural language processing in the 2020s: Where are we going? Pattern Recognition Letters136, 120–126 (2020)

    2020

  53. [53]

    Yadav, P., Tam, D., Choshen, L., Raffel, C.A., Bansal, M.: Ties-merging: Resolving interference when merging models. Proc. of Neural Information Processing Systems (NeurIPS)36(2023) Surgery & Merge19

    2023

  54. [54]

    Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., Tao, D.: Adamerging: Adaptive model merging for multi-task learning. Proc. of Int’l Conf. on Learning Representations (ICLR) (2024)

    2024

  55. [55]

    IEEE Transactions on Knowledge and Data Engineering36(8), 3832–3850 (2024)

    Yang, X., Yu, H., Gao, X., Wang, H., Zhang, J., Li, T.: Federated continual learn- ing via knowledge fusion: A survey. IEEE Transactions on Knowledge and Data Engineering36(8), 3832–3850 (2024)

    2024

  56. [56]

    Yang, Z., Zhang, Y., Zheng, Y., Tian, X., Peng, H., Liu, T., Han, B.: Fedfed: Fea- ture distillation against data heterogeneity in federated learning. Proc. of Neural Information Processing Systems (NeurIPS)36, 60397–60428 (2023)

    2023

  57. [57]

    In: Proc

    Yoon, J., Jeong, W., Lee, G., Yang, E., Hwang, S.J.: Federated continual learning with weighted inter-client transfer. In: Proc. of Int’l Conf. on Machine Learning (ICML). pp. 12073–12086. PMLR (2021)

    2021

  58. [58]

    In: Proceedings of the 32nd ACM International Conference on Multimedia

    Yu, H., Yang, X., Gao, X., Feng, Y., Wang, H., Kang, Y., Li, T.: Overcoming spatial-temporal catastrophic forgetting for federated class-incremental learning. In: Proceedings of the 32nd ACM International Conference on Multimedia. pp. 5280–5288 (2024)

    2024

  59. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yu, H., Yang, X., Zhang, L., Gu, H., Li, T., Fan, L., Yang, Q.: Handling spatial- temporal data heterogeneity for federated continual learning via tail anchor. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4874–4883 (2025)

    2025

  60. [60]

    Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., Finn, C.: Gradient surgery for multi-task learning. Proc. of Neural Information Processing Systems (NeurIPS) 33, 5824–5836 (2020)

    2020

  61. [61]

    In: Proc

    Zhang, J., Chen, C., Zhuang, W., Lyu, L.: Target: Federated class-continual learn- ing via exemplar-free distillation. In: Proc. of Int’l Conf. on Computer Vision (ICCV). pp. 4782–4793 (2023) Surgery & Merge1 Appendix In this supplementary material, we provide additional theoretical analyses, im- plementation details, and extended experimental results tha...

    2023