Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Redefining Clustered Federated Learning for System Identification: The Path of ClusterCraft

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IC-SYSID claims clustered federated learning can identify system dynamics with no prior knowledge of cluster count.

desk verdict Useful incremental contribution to federated SYSID with a realistic vehicle study, but the 'no prior knowledge' claim needs a held-out test and sensitivity analysis on the clustering thresholds. read the letter →

arxiv 2505.16857 v1 pith:AHK2SMMX submitted 2025-05-22 cs.LG

Ertuğrul Keçeci , Müjde Güzelkaya , Tufan Kumbasar This is my paper
classification cs.LG MSC 68T0593B30
keywords federatedlearningincrementalclusteringsystemidentificationclustermodelstabilityvehicledynamicspersonalizedmodelsdeepLTIsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that clustered federated learning for system identification can be made to work without prior knowledge of the data: no prescribed number of clusters, no warm initialization near the true dynamics. The proposed IC-SYSID method starts with one cluster model and, on the central server, grows new clusters only when a worker's fit has stagnated while still poor, a rule the authors call ClusterCraft. A stability regularizer plus scaled Glorot initialization keeps the learned state matrices from going unstable, and a mini-batch deep-learning update with early stopping handles large local datasets. On synthetic systems and on a fleet of vehicles learning lateral-dynamics models, IC-SYSID reports higher fit, far fewer unstable cluster models, and fewer communication rounds than the fixed-K C-SYSID baseline. If the paper is right, the result matters because it removes the need to know in advance how many dynamic regimes a fleet contains.

What carries the argument

The load-bearing mechanism is the cluster-identity flag maintained on the central server together with the incremental growth criterion of ClusterCraft. In each round the server scans every worker's normalized fit: a worker whose fit has stagnated relative to the previous round while still below the quality threshold is assigned to a brand-new cluster, so the number of clusters grows only on evidence of a distinct regime. ClusterMerge then averages any cluster-model parameters closer than a similarity threshold in Euclidean norm, and enhanced ClusterCraft (eCC) prevents near-duplicate clusters during training by moving every worker whose parameters lie within a similarity threshold of the first stalled worker into the new cluster together. The stability component is the regularized local loss $\mu\|\tilde{A}\|_F$ added to the prediction error, combined with scaled Glorot initialization, which together bias every local and aggregated state matrix toward eigenvalues of magnitude less than one.

What would settle it

Run IC-SYSID on a fresh system-identification benchmark with unknown cluster count, keeping the published thresholds for fit stagnation and fit quality and the same validation split, then score the discovered clusters on an independent held-out set; if the held-out fits are systematically lower than the reported validation fits, or if cluster count and unstable-model percentage worsen sharply without per-dataset threshold tuning, the no-prior-knowledge claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central assertion is that the failure modes of C-SYSID—unstable aggregate models, large-data training cost, and sensitivity to the number of clusters and to initialization—can be overcome jointly by an incremental clustering formulation. IC-SYSID replaces the fixed cluster set of IFCA-style clustered federated learning with a growing cluster set: ClusterCraft moves a worker into a newly created cluster when its normalized fit has stalled while still below a quality threshold, and the enhanced version eCC additionally groups all workers whose parameters are close to the first stalled worker. Stability is attacked at both ends: the local loss gains a Frobenius-norm penalty on the state matrix, and new clusters are initialized from a scaled Glorot distribution whose support keeps initial eigenvalues inside the unit circle. On the synthetic benchmark this yields near-zero misassigned workers (eCC reaches 0 percent), and on the vehicle fleet it yields 0 percent unstable cluster models after ClusterMerge or eCC, compared with at least 10 percent for C-SYSID, while using fewer communication rounds. The paper also reports that the method discovers more clusters than the nominal four speed groups with CC-CM (6.8) or fewer with eCC (3.2), which it reads as evidence that velocity groups are not the only regime structure in the data.

Load-bearing premise

The load-bearing premise is that the stagnation-and-quality test in Equation (21) separates a worker that needs a new cluster from a worker that is still improving, using fixed thresholds that are not derived from the data; the paper supplies no procedure for choosing those thresholds, and its two experiments use different values.

Editorial extensions

If this is right

  • If IC-SYSID works as claimed, a fleet operator can learn one linear model per dynamic regime from vehicle trajectories without first counting regimes or initializing parameters near ground truth.
  • With eCC, the number of communication rounds drops by roughly half relative to CC-CM (77 vs 166 on the car fleet, 100 vs 191 on the synthetic set), lowering the main cost of federated training.
  • The stability penalty cuts the fraction of unstable aggregate models from at least 10 percent in C-SYSID to 0.36 percent with CC and 0 percent with ClusterMerge or eCC, so deployment can trust simulation-mode predictions.
  • ClusterMerge reduces the cluster count produced by CC by 89.1 percent on the synthetic benchmark without degrading fit, suggesting that merged models form a compact representation of the data.
  • The clusters found on the car fleet (6.8 for CC-CM, 3.2 for eCC) imply that the ground-truth structure is not identical to the four nominal speed groups, so the clustering is doing genuine regime discovery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the no-prior-knowledge claim survives on datasets where the fixed thresholds used in the stagnation test must be chosen without tuning; its own experiments use different threshold values for the synthetic and car datasets.
  • Because the same validation split drives cluster assignment, early stopping, and the reported performance numbers, an independent held-out set would be needed to confirm that the reported fits are not inflated by selection on the evaluation set.
  • The Frobenius-norm penalty discourages large entries in the state matrix but does not certify stability, so a follow-up that enforces spectral-radius or Lyapunov constraints could turn the empirical stability gain into a guarantee.
  • The fact that eCC forms fewer clusters than the nominal velocity groups suggests cluster granularity is controlled by the similarity threshold, making that threshold a practical dial for trading model count against fit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes IC-SYSID, a clustered federated learning algorithm for system identification that avoids the need for a known number of clusters by incrementally creating clusters (ClusterCraft, CC), merging similar clusters post-training (ClusterMerge, CM), and adding an enhanced version (eCC) to reduce cluster count and communication rounds. Stability is encouraged through a Frobenius-norm regularizer on the state matrix and scaled Glorot initialization, and local training uses mini-batched Adam with early stopping. The method is evaluated on a synthetic five-cluster LTI dataset and a Matlab Vehicle Dynamics Blockset fleet scenario, with prediction- and simulation-mode comparisons against C-SYSID and a state-space baseline. The claimed results are consistently better fit, lower instability rate, and fewer communication rounds for eCC.

Significance. IC-SYSID addresses a genuinely practical gap: C-SYSID requires the number of clusters and an initialization close to the ground truth, whereas IC-SYSID starts from one cluster and grows the cluster set online. The algorithmic pipeline (CC, CM, eCC) is clearly specified, the synthetic setup is reproducible, and the experiments use 10 seeds with small reported variance. The observation that unstable models can look good in prediction but fail in simulation, and that regularized federated training reduces instability in the tested scenarios, is useful. If the validation concerns below are resolved, the method would be a solid contribution to federated system identification. The paper's strong claim of removing prior knowledge currently overstates what the threshold-based heuristic delivers.

major comments (4)
  1. [§4.3.1, Eq. (21), Table 6] The central claim that IC-SYSID 'eliminates the dependency on prior knowledge of the dataset' is undermined by the threshold dependence of the cluster-splitting rule. CC and eCC require ϵ∆, ϵp, and ϵs, and CM requires ϵθ. Table 6 shows these are dataset-dependent (ϵp=0.500 vs. 0.750; ϵθ=0.500 vs. 0.250; ϵs=0.100 vs. 0.050 for Synthetic vs. Car), and the paper gives no procedure for choosing them and no sensitivity analysis. Without this, the method has not demonstrated that it works without dataset-specific prior information; a sensitivity study over these thresholds, or a principled selection rule, is needed before the no-prior-knowledge claim can be maintained.
  2. [§4.2, Algorithm 1, §5.2] The evaluation uses the same D_i^CC set for cluster assignment, early stopping, and final performance reporting. Algorithm 1 computes P(i) and Lbar(i) on D_i^CC (lines 12 and 21), and Tables 4-5 report fit values computed from the same quantity. The cluster count, stopping round, and all reported fit values are therefore selected on the same data used to evaluate them, creating a channel for optimistic bias. There is no description of a held-out test set or cross-validation procedure. Please report final metrics on a separate test split, or otherwise quantify how much of the reported advantage survives out-of-sample selection.
  3. [§4.1, Eq. (13), Table 4] The abstract and conclusion state that IC-SYSID 'prevents the learning of unstable clusters,' but the text explicitly notes that optimizing (13) does not strictly guarantee stability. The UC% results in Table 4 are empirical, dataset-specific (CC 0.36%, CC-CM 0%, eCC 0%), and depend on the chosen thresholds and regularization coefficient. The stability claim should be softened to 'reduces instability in the scenarios tested' unless a formal stability guarantee or a much broader empirical verification is provided.
  4. [§4.2, Eq. (19)] The printed formula for the normalized fit P_n^(i) is not well-defined for general data: it sums over t the square root of a ratio of per-sample deviations, whose argument can be negative, and it does not correspond to the standard normalized fit 1 - ||x - x_tilde|| / ||x - x_bar||. Since P(i) in Eq. (20) drives the CC/eCC decisions and all reported performance tables, please correct the equation and, if the implementation differs from the intended standard metric, recompute the affected numerical results.
minor comments (6)
  1. [Abstract] 'Real-world representing SYSID problem' should be rephrased as 'a realistic SYSID problem' or 'a representative real-world SYSID scenario.'
  2. [Tables 4-5 and §5.2] The table captions say 'Standard deviations' while the text refers to '±1 standard error'; please make the labeling consistent.
  3. [§4.2, Eq. (18)] The early-stopping description says training stops if there is no improvement of the moving average, but the exact comparison (e.g., vs. best previous value, patience length) is not specified; please clarify.
  4. [§4.1, Eq. (14)] The scaled Glorot interval uses a 0.1 factor, but the paper does not explain its derivation relative to the standard Glorot bounds; a one-sentence justification would help.
  5. [Algorithm 4] The first loop uses 'goto end for' after finding the first worker satisfying Eq. (21), so if multiple workers satisfy the condition only the first is used to seed the new cluster; this should be stated explicitly as a design choice and its consequences discussed.
  6. [Algorithm 3] The merge result depends on the order in which the first element of θ is selected, since iteration order is not specified; please state the ordering used or note that the output is order-dependent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: IC-SYSID's cluster and stability results are empirical and compared against external baselines; validation reuse and per-dataset thresholds are correctness risks, not construction-level circularity.

full rationale

The paper's central claims (that IC-SYSID clusters without prior knowledge of K and restricts unstable models) rest on algorithmic constructions in Eqs. (21)-(26) and on direct experimental comparisons with C-SYSID on shared synthetic and automotive datasets. No reported prediction is algebraically identical to a fitted input or to the metric used to define it. The only self-citation, [24], is an attribution of the preliminary IC-SYSID framework ('This work builds on our preliminary study in [24]'), but the quantitative results in Tables 3-5 are generated in this paper, so the citation is not load-bearing. The stability claim is explicitly hedged in the text ('optimizing (13) does not strictly guarantee the stability of the cluster model'), making the UC% results empirical rather than derived. Two concerns raised in the reader's take are real but do not constitute circularity under the strict criteria: (1) D_i_CC is used for cluster assignment, early stopping, and final performance reporting (Algorithm 1, lines 12 and 17), which is a validation-overfitting risk rather than a reduction of the derivation to its inputs; and (2) the convergence thresholds epsilon_p differ between datasets in Table 6, which weakens the 'no prior knowledge' claim but does not make any prediction equivalent to a fitted parameter. Because the paper's main results are benchmark comparisons rather than formal derivations, no circular step can be exhibited by quoting an equation in which output equals input by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several hand-set hyperparameters, especially the clustering thresholds in Eq. (21), which are adjusted per dataset. The stability claim rests on a regularization heuristic that the authors admit does not provide a strict guarantee. No new physical entities are introduced.

free parameters (6)
  • regularization coefficient µ = 0.001
    Added to loss Eq. (13) to penalize ||A||_F and encourage stable models. The value is chosen by hand with no sensitivity analysis.
  • CC stagnation threshold ϵ∆ = 0.0005
    Used in Eq. (21) to decide when a worker's performance has converged. Dataset-specific values appear in Table 6.
  • CC performance threshold ϵp = 0.500 (synthetic), 0.750 (car)
    Used in Eq. (21) to declare a worker's fit poor and trigger a new cluster. This choice directly controls the number of clusters and is the main load-bearing hyperparameter.
  • ClusterMerge similarity threshold ϵθ = 0.500 (synthetic), 0.250 (car)
    Used in Eq. (25) to decide when two cluster models are merged. Strongly affects the final number of clusters after merging.
  • eCC similarity threshold ϵs = 0.100 (synthetic), 0.050 (car)
    Used in Eq. (26) to decide which workers are grouped with the reference worker in the enhanced ClusterCraft.
  • scaled Glorot scaling factor = 0.1
    Used in Eq. (14) to initialize cluster models with small entries so the initial A is stable. The 0.1 factor is chosen ad hoc.
assumptions (5)
  • domain assumption The LTI model in Eq. (3) adequately represents the dynamics of all workers.
    The entire SYSID formulation reduces to estimating A and B matrices. The authors acknowledge this in the conclusion by stating future work will extend to nonlinear models.
  • standard math Stability of the learned model is equivalent to all eigenvalues of A lying inside the unit circle.
    Invoked in Section 3.1 to define stability of the cluster models.
  • domain assumption The normalized fit P(i) in Eq. (19) reliably separates workers that should belong to different clusters.
    ClusterCraft and eCC base all cluster identity decisions on whether P(i) stagnates below ϵp. No evidence is given that this metric is a sufficient statistic for cluster membership.
  • domain assumption The training/CC split Di = {Di_T, Di_CC} provides an unbiased evaluation signal for cluster assignment.
    The same Di_CC is used for cluster assignment, early stopping, and final performance reporting, so the evaluation signal is not fully independent of the decisions it drives.
  • ad hoc to paper The regularization term µ||A||_F implicitly moves the eigenvalues of A inside the unit circle.
    The paper itself states in Section 4.1 that optimizing Eq. (13) does not strictly guarantee stability, yet the stability claim of the method depends on this heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Redefining Clustered Federated Learning for System Identification: The Path of ClusterCraft." pith.science (2026). https://pith.science/paper/AHK2SMMX

@misc{pith2026250516857,
  author       = {Pith},
  title        = {Pith review of: Redefining Clustered Federated Learning for System Identification: The Path of ClusterCraft},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHK2SMMX}},
  note         = {Machine review of arXiv:2505.16857}
}
read the original abstract

This paper addresses the System Identification (SYSID) problem within the framework of federated learning. We introduce a novel algorithm, Incremental Clustering-based federated learning method for SYSID (IC-SYSID), designed to tackle SYSID challenges across multiple data sources without prior knowledge. IC-SYSID utilizes an incremental clustering method, ClusterCraft (CC), to eliminate the dependency on the prior knowledge of the dataset. CC starts with a single cluster model and assigns similar local workers to the same clusters by dynamically increasing the number of clusters. To reduce the number of clusters generated by CC, we introduce ClusterMerge, where similar cluster models are merged. We also introduce enhanced ClusterCraft to reduce the generation of similar cluster models during the training. Moreover, IC-SYSID addresses cluster model instability by integrating a regularization term into the loss function and initializing cluster models with scaled Glorot initialization. It also utilizes a mini-batch deep learning approach to manage large SYSID datasets during local training. Through the experiments conducted on a real-world representing SYSID problem, where a fleet of vehicles collaboratively learns vehicle dynamics, we show that IC-SYSID achieves a high SYSID performance while preventing the learning of unstable clusters.

Figures

Figures reproduced from arXiv: 2505.16857 by the authors.

Figure 1
Figure 1. A snapshot of the realistic simulation environment [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
Figure 2
Figure 2. Comparison of C-SYSID with K = 2 and IC-SYSID with CC on Car Dataset (full-scale plot on the left, zoomed-in plot on the right) 6. Conclusion and Future Work This paper introduces IC-SYSID, a collaborative learning method for SYSID problems capable of learning stable ground truth cluster models without prior dataset knowledge. To prevent instability, IC-SYSID integrates a regularization term into the loss function a… view at source ↗
Figure 3
Figure 3. Comparison of C-SYSID with K = 4 and IC-SYSID with eCC on Car Dataset (full-scale plot on the left, zoomed-in plot on the right) Appendix The parameters of each cluster model used in Section 2.3 are as follows: A (1) =   0.5 0.3 0.1 0.0 0.2 0.0 0.1 0.0 0.3   B (1) =   1 0.5 0.1 1 0.75 1.5   A (2) =   −0.3 0.0 0.0 0.1 0.4 0.0 0.2 0.3 0.5   B (2) =   1.5 0.1 0.5 2.5 0.1 1.5   A (3) =   −0.1 0.1 0.1 0… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [24]

    Ke¸ ceci, M

    E. Ke¸ ceci, M. G¨ uzelkaya, T. Kumbasar, A novel federated learning framework for system identification, in: 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), 2024, pp. 1–6. 20

    work page 2024

  2. [1]

    Ljung, Perspectives on system identification, Annual Reviews in Control 34 (1) (2010) 1–12

    L. Ljung, Perspectives on system identification, Annual Reviews in Control 34 (1) (2010) 1–12

    work page 2010

  3. [2]

    K. J. Keesman, System identification: an introduction, Springer Science & Business Media, 2011

    work page 2011

  4. [3]

    T. B. Sch¨ on, A. Wills, B. Ninness, System identification of nonlinear state-space models, Automatica 47 (1) (2011) 39–49

    work page 2011

  5. [4]

    Bruder, C

    D. Bruder, C. D. Remy, R. Vasudevan, Nonlinear system identification of soft robot dynamics using koopman operator theory, in: 2019 International Conference on Robotics and Automation, 2019, pp. 6244–6250

    work page 2019

  6. [5]

    Chiuso, G

    A. Chiuso, G. Pillonetto, System identification: A machine learning perspective, Annual Review of Control, Robotics, and Autonomous Systems 2 (2019) 281–304

    work page 2019

  7. [6]

    T. Tuna, A. Beke, T. Kumbasar, Deep learning frameworks to learn prediction and simulation focused control system models, Applied Intelligence 52 (1) (2022) 662–679

    work page 2022

  8. [7]

    L. Xin, L. Ye, G. Chiu, S. Sundaram, Identifying the dynamics of a system by leveraging data from similar systems, in: American Control Conference, 2022, pp. 818–824

    work page 2022

Show all 24 references
  1. [8]

    T. T. Zhang, K. Kang, B. D. Lee, C. Tomlin, S. Levine, S. Tu, N. Matni, Multi-task imitation learning for linear dynamical systems, in: Learning for Dynamics and Control Conference, 2023, pp. 586–599

    2023

  2. [9]

    H. Wang, L. F. Toso, J. Anderson, Fedsysid: A federated approach to sample-efficient system identifi- cation, in: Learning for Dynamics and Control Conference, 2023, pp. 1308–1320

    2023

  3. [10]

    Papusha, E

    I. Papusha, E. Lavretsky, R. M. Murray, Collaborative system identification via parameter consensus, in: 2014 American Control Conference, 2014, pp. 13–19

    2014

  4. [11]

    McMahan, E

    B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Artificial intelligence and statistics, 2017, pp. 1273–1282

    2017

  5. [12]

    J. Le, X. Lei, N. Mu, H. Zhang, K. Zeng, X. Liao, Federated continuous learning with broad network architecture, IEEE Transactions on Cybernetics 51 (8) (2021) 3874–3888

    2021

  6. [13]

    Kaheni, M

    M. Kaheni, M. Lippi, A. Gasparri, M. Franceschelli, Selective trimmed average: A resilient federated learning algorithm with deterministic guarantees on the optimality approximation, IEEE Transactions on Cybernetics (2024)

    2024

  7. [14]

    Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, V. Chandra, Federated learning with non-iid data, arXiv:1806.00582 (2018)

    2018 arXiv

  8. [15]

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, V. Smith, Federated optimization in hetero- geneous networks, in: Proceedings of Machine Learning and Systems, Vol. 2, 2020, pp. 429–450

    2020

  9. [16]

    X. Li, Z. Qu, B. Tang, Z. Lu, Fedlga: Toward system-heterogeneity of federated learning via local gradient approximation, IEEE Transactions on Cybernetics (2023). 19

    2023

  10. [17]

    Yeganeh, A

    Y. Yeganeh, A. Farshad, N. Navab, S. Albarqouni, Inverse distance aggregation for federated learning with non-iid data, in: Domain Adaptation and Representation Transfer, and Distributed and Collabo- rative Learning, 2020, pp. 150–159

    2020

  11. [18]

    Zhang, L

    L. Zhang, L. Shen, L. Ding, D. Tao, L.-Y. Duan, Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10174–10183

    2022

  12. [19]

    Ghosh, J

    A. Ghosh, J. Chung, D. Yin, K. Ramchandran, An efficient framework for clustered federated learning, Advances in Neural Information Processing Systems 33 (2020) 19586–19597

    2020

  13. [20]

    Mansour, M

    Y. Mansour, M. Mohri, J. Ro, A. T. Suresh, Three approaches for personalization with applications to federated learning, arXiv:2002.10619 (2020)

    2020 arXiv

  14. [21]

    C. Li, G. Li, P. K. Varshney, Federated learning with soft clustering, IEEE Internet of Things Journal 9 (10) (2021) 7773–7782

    2021

  15. [22]

    Y. Ruan, C. Joe-Wong, Fedsoft: Soft clustered federated learning with proximal local updating, Pro- ceedings of the AAAI Conference on Artificial Intelligence 36 (7) (2022) 8124–8131

    2022

  16. [23]

    L. F. Toso, H. Wang, J. Anderson, Learning personalized models with clustered system identification, arXiv:2304.01395 (2023)

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.