Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Personalized Federated Learning under Model Dissimilarity Constraints

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read This paper introduces KARULA, a personalized federated learning method in which each client's model is constrained by data-driven Wasserstein-distance surrogates, and proves an $O(1/K)$ convergence rate to an $\epsilon$-stationary point…

desk verdict KARULA has a novel and sensible way to set pairwise model constraints from Wasserstein-based distributional similarity, but two load-bearing internal inconsistencies (constraint mismatch and a proof algebra slip) need fixing before the theory holds. read the letter →

arxiv 2505.07575 v2 pith:XGJNDAXE submitted 2025-05-12 cs.LG

classification cs.LG
keywords personalizedfederatedlearningstatisticalheterogeneity1-Wassersteindistancemodeldissimilarityconstraintslinearembeddingvariancereductioninexactprojectionnonconvexoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces KARULA, a personalized federated learning strategy in which the models of different clients are allowed to differ only up to bounds set by how dissimilar their data distributions are. Dissimilarity is measured by a linear-embedding surrogate for the 1-Wasserstein distance, so no raw client data are exchanged. The paper proves that, under standard smoothness and bounded-variance assumptions, the proposed inexact projected stochastic gradient algorithm reaches an $\epsilon$-stationary point at rate $O(1/K)$, with no assumption bounding client heterogeneity. On synthetic ridge regression and federated MNIST, KARULA reports lower estimation error and higher accuracy than local training, FedAvg, and IFCA. If these claims hold, the method offers a data-driven way to set personalization strength instead of choosing a fixed regularization by hand.

What carries the argument

The load-bearing object is the linear embedding $\Phi(D_i) = (N_0 \pi_i^\star D_i - D_0)/\sqrt{N_0}$, where $\pi_i^\star$ is the optimal transport plan between the reference empirical distribution and client $i$'s empirical distribution, and $D_0$ is the reference data matrix; its $\ell^1$ pairwise distance approximates $W_1(\hat{\mu}_i, \hat{\mu}_j)$. This embedding lets each client compute one transport plan instead of $n(n-1)/2$ plans, and transmits only the reference set and embeddings. The optimization machinery is an inexact projected stochastic gradient algorithm: a SAGA-style variance-reduced server update removes the bias of partial participation, and a $\delta$-inexact projection onto the constraint set $\mathcal{K}=\{\theta: \|\theta_i-\theta_j\|_2 \le t D_{ij}\}$ keeps the iterates feasible while contributing a controlled $\delta$ term to the final neighborhood.

What would settle it

Run the synthetic ridge regression experiment with ground-truth model parameters, replacing the embedding surrogate with the exact empirical 1-Wasserstein distance computed between client data sets; if KARULA's estimation error does not improve or becomes worse than with the surrogate, the Wasserstein bound is not the operative mechanism behind the reported gains. Alternatively, compare $\|\Phi(D_i)-\Phi(D_j)\|_1$ with exact $W_1$ on distributions with known distance while varying the reference distribution and sample size $N_0$; if the surrogate's ranking disagrees with true $W_1$, the constraints are not distributionally grounded.

Watch

Extended reading notes

Core claim

The central claim is that constraining personalized model differences by pairwise distributional distances is both principled and algorithmically tractable. Under quadratic functional growth and data-Lipschitz losses, the paper proves that the ideal models satisfy $\|\theta_i^\star-\theta_j^\star\|_2^2 \le (2L_X/\gamma) W_1(\mu_i,\mu_j)$, so distributional nearness justifies model nearness. In the federated setting it replaces the exact Wasserstein distance with $\|\Phi(D_i)-\Phi(D_j)\|_1$, where $\Phi$ is a linear embedding computed from a single optimal transport plan between each client's empirical data and a reference data set. The resulting constrained problem is solved by an inexact projected stochastic gradient method with variance reduction and partial participation; Theorem 4.3 bounds the minimum squared gradient mapping by $O(1/K)$ plus $\epsilon = 4\sigma^2(s+1)/s + 2\delta$.

Load-bearing premise

The entire scheme depends on treating the linear data embedding computed against a single reference data set as a faithful stand-in for the true 1-Wasserstein distance between client distributions; if that approximation is poor, the theoretical bound that motivates the constraints no longer governs the models the algorithm actually produces.

Editorial extensions

If this is right

  • Setting $t=0$ recovers a single global model and $t\to\infty$ gives fully local models, so KARULA continuously interpolates between the two extremes using constraints that are fully data-driven.
  • Because the convergence analysis does not bound client heterogeneity, the $O(1/K)$ rate holds for arbitrary distributional differences among clients, provided the loss is smooth and stochastic gradients have bounded variance.
  • The projection error $\delta$ appears additively in the stationarity neighborhood, giving a concrete trade-off: looser projections save per-round computation at the price of a larger final error.
  • The pairwise dissimilarity structure is preserved during training, so models can share information across overlapping subgroups without forcing a hard cluster assignment.
  • FedAvg and IFCA can be seen as special cases or degenerate limits of the constraint set, which helps explain the reported empirical advantage when client distributions overlap in complex ways.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: An immediate testable extension is to replace the fixed reference distribution with a learnable or client-adaptive reference; the theory's dependence on reference choice and reference sample size is unspecified, so the best reference may be distribution-selection-sensitive.
  • Editorial inference: The same constraint machinery could be applied to other dissimilarity surrogates, such as optimal-transport distances between feature representations rather than raw data, keeping the convergence result intact as long as the embedding defines a metric.
  • Editorial inference: The $\ell^1$ embedding inherits the slow $O(N^{-1/d})$ empirical-Wasserstein convergence, so KARULA's advantage over local models should be largest when per-client sample sizes are comparable to feature dimension and should shrink in very high dimensions.
  • Editorial inference: The variance-reduced inexact projection step is stated generically enough that it could be reused by other federated algorithms whose feasible sets are convex.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes KARULA, a personalized federated learning method that constrains pairwise differences between client models by data-driven dissimilarity parameters derived from a linear-embedding approximation of the 1-Wasserstein distance. The authors present an inexact projected stochastic gradient algorithm with variance reduction for partial client participation, prove an O(1/K) convergence bound to an epsilon-stationary point for smooth possibly nonconvex losses, and report experiments on synthetic ridge regression and federated MNIST showing improved accuracy over Local, FedAvg, and IFCA.

Significance. If the algorithm is made consistent with the stated problem and the Wasserstein-embedding approximation is justified, the method would be a useful peer-based PFL contribution: it avoids clustering assumptions, has a communication-light dissimilarity computation, and the convergence analysis explicitly accounts for stochastic gradients and inexact projections. The empirical results are plausible and support the method on the tested tasks.

major comments (4)
  1. [Sec. 3.1 Eq. (1) and Sec. 3.3 set K] The optimization problem is stated with constraints ||theta_i - theta_j||_2^2 <= t D_ij, while the feasible set actually used by Algorithm 1 and Theorem 4.3 is {theta : ||theta_i - theta_j||_2 <= t D_ij}. These feasible sets differ: for D_ij=2 and t=1, a pair with ||theta_i - theta_j||=1.5 satisfies the norm constraint but violates the squared one. Since Proposition 3.4 only bounds the squared distance, the theory justifies the squared formulation, whereas the algorithm and convergence theorem prove a result for a different problem. The formulation must be made consistent.
  2. [Sec. 3.2] The claim that ||Phi(D_i)-Phi(D_j)||_1 approximates W_1(mu_i, mu_j) is asserted with citations but without a statement of the approximation error, the conditions on the reference distribution mu_0 and sample size N_0, or a proof. Because these D_ij values directly determine the feasible set, a poor approximation would invalidate the transfer of the Wasserstein-based bound in Eq. (2) to the constraints actually used in the algorithm.
  3. [Sec. 4, Theorem 4.3 and Appendix A.3] The proof as written does not yield the stated bound. Substituting eta = 3/(8L) into the final displayed inequality 2/(eta(1-2*eta*L)) gives 64L/3, not 8L/3, and the proof uses an undefined factor alpha in the definition of the gradient mapping. The telescoping steps need to be rechecked and the constants corrected.
  4. [Sec. 3.3 and Limitations] The delta-inexact projection is required to return feasible points (theta in K), but the computation is said to be described in supplemental material that is not included, and the Limitations section concedes that feasibility is not always guaranteed. The theorem therefore may not apply to the implemented projection; the authors should either provide the projection algorithm and prove its feasibility guarantee, or adjust the theory to allow infeasible iterates.
minor comments (4)
  1. [Sec. 5.2] The sentence listing final test accuracies ends with '83.9 +/- 2.60%, respectively' without identifying which strategy this number belongs to (presumably IFCA).
  2. [Introduction] 'centerally constrained' should be 'centrally constrained'.
  3. [Fig. 3] The numeric dissimilarity values are shown without units or an explicit explanation of their scale; a brief caption note would help.
  4. [Sec. 5.1] Please report the number of random trials or seeds used to compute the mean and the +/- 2*SE intervals.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core derivation is a forward argument and the convergence guarantee does not encode fitted parameters.

full rationale

This paper's derivation chain is self-contained in the relevant circularity senses. Proposition 3.4 derives a squared-model-distance bound from quadratic functional growth and L_X-Lipschitz losses using Kantorovich-Rubinstein duality; the bound does not assume the constraint structure of Problem (1). The dissimilarity parameters D_ij are computed from a linear embedding of the empirical distributions, following external prior work (Kolouri et al. 2020, Liu et al. 2025), and are used as constraint coefficients; no fitted model-distance quantity is renamed as a prediction. The convergence proof (Theorem 4.3) is an adapted variance-reduced projected SGD analysis with inexact projections, and the stationarity neighborhood depends only on the stochastic-gradient variance sigma^2 and projection error delta; the free parameter t and the D_ij values do not enter the convergence guarantee, so the theorem is not forced by the empirical tuning. The empirical comparison uses cross-validation for hyperparameters in a standard way, so the reported accuracy is not a constructed consequence of the theory. The main substantive concerns—the mismatch between squared constraints in Problem (1) and norm constraints in K, and the unstated conditions for the Wasserstein-embedding approximation—are correctness and limitation issues, not circular reductions. No load-bearing self-citation or definitional equivalence was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces free parameters t, eta, the reference data set, the data metric, and the projection tolerance. The convergence theory relies on standard smoothness and bounded-variance assumptions plus the quadratic growth and Lipschitz assumptions behind Proposition 3.4. No new physical or mathematical entities are invented.

free parameters (5)
  • Regularization parameter t = Cross-validated on synthetic; set to 20 on MNIST
    Scales the dissimilarity thresholds and directly controls how much models may differ. Chosen by 5-fold cross-validation on the synthetic task and hand-set to t=20 on MNIST. Not derived from the theoretical bound.
  • Learning rate eta = 3/(8L) on synthetic with L estimated; 2e-4 on MNIST
    The algorithm requires the smoothness constant L for the step size; the paper does not specify how L is estimated in the synthetic experiments, and a different value is used on MNIST.
  • Reference data set D_0 and size N_0 = Not specified
    The Wasserstein embedding uses a reference data set; its construction, distribution, and sample size are not described, yet they define all dissimilarity parameters D_ij.
  • Data metric d_X = Euclidean norm on joint feature-label space
    The paper states this is used for simplicity; the choice of metric changes the Wasserstein approximation and thus all constraints.
  • Inexact projection tolerance delta = Not reported
    The convergence bound depends on delta, but the experiments do not state what delta was used or how the iterative projection was implemented.
assumptions (5)
  • domain assumption Assumption 3.1: quadratic functional growth with parameter gamma > 0
    Used in Proposition 3.4 to bound model distance by W1. Claimed to hold for losses with strongly convex envelopes; not verified for the neural network losses in the MNIST experiment.
  • domain assumption Assumption 3.2: loss is L_X-Lipschitz in data
    Also used in Proposition 3.4; for quadratic loss the parameter space must be bounded (Remark 3.3).
  • standard math Assumption 4.1: weighted losses are L-smooth
    Standard optimization assumption used in the convergence analysis; L must be known or estimated for the step size.
  • standard math Assumption 4.2: stochastic gradients have bounded variance sigma^2
    Standard assumption; the bound sigma^2 enters the final epsilon neighborhood, but no verification is provided in the experiments.
  • ad hoc to paper The embedding ||Phi(D_i)-Phi(D_j)||_1 defines a metric approximating W1
    Assumed from prior work (Kolouri et al. 2020, Liu et al. 2025). No theorem or conditions are given in this paper, and the reference distribution is unspecified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Personalized Federated Learning under Model Dissimilarity Constraints." pith.science (2026). https://pith.science/paper/XGJNDAXE

@misc{pith2026250507575,
  author       = {Pith},
  title        = {Pith review of: Personalized Federated Learning under Model Dissimilarity Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGJNDAXE}},
  note         = {Machine review of arXiv:2505.07575}
}
read the original abstract

One of the defining challenges in federated learning is that of statistical heterogeneity among clients. We address this problem with KARULA, a regularized strategy for personalized federated learning, which constrains the pairwise model dissimilarities between clients based on the difference in their distributions, as measured by a surrogate for the 1-Wasserstein distance adapted for the federated setting. This allows the strategy to adapt to highly complex interrelations between clients, that e.g., clustered approaches fail to capture. We propose an inexact projected stochastic gradient algorithm to solve the constrained problem that the strategy defines, and show theoretically that it converges with smooth, possibly non-convex losses to a neighborhood of a stationary point with rate O(1/K). We demonstrate the effectiveness of KARULA on synthetic and real federated data sets.

Figures

Figures reproduced from arXiv: 2505.07575 by the authors.

Figure 1
Figure 1. Heatmaps showing the true model dissimilarities, the dissimilarity parameters calculated by [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Training and test performance with ±2 × SE bands of FL strategies on classification task with 2NN model on Federated MNIST dataset with n = 30 clients. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Example handwritten digits from three clients and the dissimilarity parameters [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages

  1. [1]

    and Timmermann, A

    Ang, A. and Timmermann, A. (2012). Regime changes and financial markets. Annu. Rev. Financ. Econ. , 4(1):313--337

  2. [2]

    G., Aggarwal, V., Singh, A

    Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choudhary, S. (2019). Federated learning with personalization layers. arXiv preprint arXiv:1912.00818

  3. [3]

    Armacki, A., Bajovic, D., Jakovetic, D., and Kar, S. (2022). Personalized federated learning via convex clustering. In 2022 IEEE International Smart Cities Conference (ISC2) , pages 1--7. IEEE

  4. [4]

    J., Ritov, Y., and Tsybakov, A

    Bickel, P. J., Ritov, Y., and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics , 37(4):1705--1732

  5. [5]

    Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. (2019). LEAF : A benchmark for federated settings. In Workshop on Federated Learning for Data Privacy and Confidentiality

  6. [6]

    Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives . Advances in neural information processing systems , 27

  7. [7]

    M., and Mahdavi, M

    Deng, Y., Kamani, M. M., and Mahdavi, M. (2020). Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461

  8. [8]

    T., Tran, N., and Nguyen, J

    Dinh, C. T., Tran, N., and Nguyen, J. (2020). Personalized federated learning with moreau envelopes. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 21394--21405. Curran Associates, Inc

Show all 39 references
  1. [9]

    T., Vu, T

    Dinh, C. T., Vu, T. T., Tran, N. H., Dao, M. N., and Zhang, H. (2022). A new look and convergence rate of federated multitask learning with Laplacian regularization. IEEE Transactions on Neural Networks and Learning Systems

  2. [10]

    and Gursoy, G

    Elhussein, A. and Gursoy, G. (2024). A universal metric of dataset similarity for cross-silo federated learning. arXiv preprint arXiv:2404.18773

  3. [11]

    Fallah, A., Mokhtari, A., and Ozdaglar, A. (2020). Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Sys...

  4. [12]

    Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. (2020). An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 33:19586--19597

  5. [13]

    M., Ramage, D., Beaufays, F., Eichner, H., Rao, K., Mathews, R., and Augenstein, S

    Hard, A., Kiddon, C. M., Ramage, D., Beaufays, F., Eichner, H., Rao, K., Mathews, R., and Augenstein, S. (2018). Federated learning for mobile keyboard prediction

  6. [14]

    Hashemi, D., He, L., and Jaggi, M. (2024). CoBo : Collaborative learning via bilevel optimization. arXiv preprint arXiv:2409.05539

  7. [15]

    Jhunjhunwala, D., Sharma, P., Nagarkatti, A., and Joshi, G. (2022). Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence , pages 906--916. PMLR

  8. [16]

    Kantorovich, L. V. (1960). Mathematical methods of organizing and planning production. Management science , 6(4):366--422

  9. [17]

    P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A

    Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132--5143. PMLR

  10. [18]

    N., Viscoli, C

    Kernan, W. N., Viscoli, C. M., Makuch, R. W., Brass, L. M., and Horwitz, R. I. (1999). Stratified randomization for clinical trials. Journal of clinical epidemiology , 52(1):19--26

  11. [19]

    K., and Hoffmann, H

    Kolouri, S., Naderializadeh, N., Rohde, G. K., and Hoffmann, H. (2020). Wasserstein embedding for graph learning. arXiv preprint arXiv:2006.09430

  12. [20]

    and Leibler, R

    Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics , 22(1):79--86

  13. [21]

    Li, T., Hu, S., Beirami, A., and Smith, V. (2021). Ditto: Fair and robust federated learning through personalization. In International conference on machine learning , pages 6357--6368. PMLR

  14. [22]

    Lin, J. (1991). Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory , 37(1):145--151

  15. [23]

    Liu, X., Bai, Y., Lu, Y., Soltoggio, A., and Kolouri, S. (2025). Wasserstein task embedding for measuring task similarities. Neural Networks , 181:106796

  16. [24]

    A., Baraliakos, X., Baker, D., Hsu, B., and van der Heijde, D

    Machado, P., Landew \'e , R., Braun, J., Hermann, K.-G. A., Baraliakos, X., Baker, D., Hsu, B., and van der Heijde, D. (2011). A stratified model for health outcomes in ankylosing spondylitis. Annals of the rheumatic diseases , 70(10):1758--1764

  17. [25]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data . In Singh, A. and Zhu, J., editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics ,...

  18. [26]

    Melsen, W., Bootsma, M., Rovers, M., and Bonten, M. (2014). The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses. Clinical microbiology and infection , 20(2):123--129

  19. [27]

    Nesterov, Y. (2018). Lectures on Convex Optimization , volume 137 of Springer Optimization and Its Applications . Springer, Cham, 2 edition. eBook ISBN: 978-3-319-91578-4, Published: 19 November 2018

  20. [28]

    X., Duong, C., Phan, T., Tjiputra, E., and Tran, Q

    Nguyen, A., Do, T., Tran, M., Nguyen, B. X., Duong, C., Phan, T., Tjiputra, E., and Tran, Q. D. (2022). Deep federated learning for autonomous driving. In 2022 IEEE Intelligent Vehicles Symposium (IV) , pages 1824--1830. IEEE

  21. [29]

    Pan, S. J. (2020). Transfer learning. Learning , 21:1--2

  22. [30]

    A., Al-Fuqaha, A., and Qadir, J

    Qayyum, A., Ahmad, K., Ahsan, M. A., Al-Fuqaha, A., and Qadir, J. (2022). Collaborative federated learning for healthcare: Multi-modal covid-19 diagnosis at the edge. IEEE Open Journal of the Computer Society , 3:172--184

  23. [31]

    Rakotomamonjy, A., Nadjahi, K., and Ralaivola, L. (2023). Federated wasserstein distance. arXiv preprint arXiv:2310.01973

  24. [32]

    J., and Yu, B

    Raskutti, G., Wainwright, M. J., and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression over _q -balls. IEEE transactions on information theory , 57(10):6976--6994

  25. [33]

    J., Sra, S., Poczos, B., and Smola, A

    Reddi, S. J., Sra, S., Poczos, B., and Smola, A. J. (2016). Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Advances in neural information processing systems , 29

  26. [34]

    Shui, C., Abbasi, M., Robitaille, L.- \'E ., Wang, B., and Gagn \'e , C. (2019). A principled approach for learning task similarity in multitask learning. arXiv preprint arXiv:1903.09109

  27. [35]

    Villani, C. (2008). Optimal Transport . Springer Berlin, Heidelberg

  28. [36]

    Wang, J., Gao, R., and Xie, Y. (2021). Two-sample test using projected wasserstein distance. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 3320--3325. IEEE

  29. [37]

    Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv preprint arXiv:1806.00582

  30. [38]

    Zhong, X., Guo, S., Shan, H., Gao, L., Xue, D., and Zhao, N. (2018). Feature-based transfer learning based on distribution similarity. IEEE Access , 6:35551--35557

  31. [39]

    Zhou, P., Zou, Y., Yuan, X.-T., Feng, J., Xiong, C., and Hoi, S. (2021). Task similarity aware meta learning: Theory-inspired improvement on MAML . In Uncertainty in artificial intelligence , pages 23--33. PMLR

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.