REVIEW 4 major objections 4 minor 39 references
Personalized Federated Learning under Model Dissimilarity Constraints
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper introduces KARULA, a personalized federated learning method in which each client's model is constrained by data-driven Wasserstein-distance surrogates, and proves an $O(1/K)$ convergence rate to an $\epsilon$-stationary point…
desk verdict KARULA has a novel and sensible way to set pairwise model constraints from Wasserstein-based distributional similarity, but two load-bearing internal inconsistencies (constraint mismatch and a proof algebra slip) need fixing before the theory holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the linear embedding $\Phi(D_i) = (N_0 \pi_i^\star D_i - D_0)/\sqrt{N_0}$, where $\pi_i^\star$ is the optimal transport plan between the reference empirical distribution and client $i$'s empirical distribution, and $D_0$ is the reference data matrix; its $\ell^1$ pairwise distance approximates $W_1(\hat{\mu}_i, \hat{\mu}_j)$. This embedding lets each client compute one transport plan instead of $n(n-1)/2$ plans, and transmits only the reference set and embeddings. The optimization machinery is an inexact projected stochastic gradient algorithm: a SAGA-style variance-reduced server update removes the bias of partial participation, and a $\delta$-inexact projection onto the constraint set $\mathcal{K}=\{\theta: \|\theta_i-\theta_j\|_2 \le t D_{ij}\}$ keeps the iterates feasible while contributing a controlled $\delta$ term to the final neighborhood.
What would settle it
Run the synthetic ridge regression experiment with ground-truth model parameters, replacing the embedding surrogate with the exact empirical 1-Wasserstein distance computed between client data sets; if KARULA's estimation error does not improve or becomes worse than with the surrogate, the Wasserstein bound is not the operative mechanism behind the reported gains. Alternatively, compare $\|\Phi(D_i)-\Phi(D_j)\|_1$ with exact $W_1$ on distributions with known distance while varying the reference distribution and sample size $N_0$; if the surrogate's ranking disagrees with true $W_1$, the constraints are not distributionally grounded.
Extended reading notes
Core claim
The central claim is that constraining personalized model differences by pairwise distributional distances is both principled and algorithmically tractable. Under quadratic functional growth and data-Lipschitz losses, the paper proves that the ideal models satisfy $\|\theta_i^\star-\theta_j^\star\|_2^2 \le (2L_X/\gamma) W_1(\mu_i,\mu_j)$, so distributional nearness justifies model nearness. In the federated setting it replaces the exact Wasserstein distance with $\|\Phi(D_i)-\Phi(D_j)\|_1$, where $\Phi$ is a linear embedding computed from a single optimal transport plan between each client's empirical data and a reference data set. The resulting constrained problem is solved by an inexact projected stochastic gradient method with variance reduction and partial participation; Theorem 4.3 bounds the minimum squared gradient mapping by $O(1/K)$ plus $\epsilon = 4\sigma^2(s+1)/s + 2\delta$.
Load-bearing premise
The entire scheme depends on treating the linear data embedding computed against a single reference data set as a faithful stand-in for the true 1-Wasserstein distance between client distributions; if that approximation is poor, the theoretical bound that motivates the constraints no longer governs the models the algorithm actually produces.
Editorial extensions
If this is right
- Setting $t=0$ recovers a single global model and $t\to\infty$ gives fully local models, so KARULA continuously interpolates between the two extremes using constraints that are fully data-driven.
- Because the convergence analysis does not bound client heterogeneity, the $O(1/K)$ rate holds for arbitrary distributional differences among clients, provided the loss is smooth and stochastic gradients have bounded variance.
- The projection error $\delta$ appears additively in the stationarity neighborhood, giving a concrete trade-off: looser projections save per-round computation at the price of a larger final error.
- The pairwise dissimilarity structure is preserved during training, so models can share information across overlapping subgroups without forcing a hard cluster assignment.
- FedAvg and IFCA can be seen as special cases or degenerate limits of the constraint set, which helps explain the reported empirical advantage when client distributions overlap in complex ways.
Reading between the lines
- Editorial inference: An immediate testable extension is to replace the fixed reference distribution with a learnable or client-adaptive reference; the theory's dependence on reference choice and reference sample size is unspecified, so the best reference may be distribution-selection-sensitive.
- Editorial inference: The same constraint machinery could be applied to other dissimilarity surrogates, such as optimal-transport distances between feature representations rather than raw data, keeping the convergence result intact as long as the embedding defines a metric.
- Editorial inference: The $\ell^1$ embedding inherits the slow $O(N^{-1/d})$ empirical-Wasserstein convergence, so KARULA's advantage over local models should be largest when per-client sample sizes are comparable to feature dimension and should shrink in very high dimensions.
- Editorial inference: The variance-reduced inexact projection step is stated generically enough that it could be reused by other federated algorithms whose feasible sets are convex.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KARULA, a personalized federated learning method that constrains pairwise differences between client models by data-driven dissimilarity parameters derived from a linear-embedding approximation of the 1-Wasserstein distance. The authors present an inexact projected stochastic gradient algorithm with variance reduction for partial client participation, prove an O(1/K) convergence bound to an epsilon-stationary point for smooth possibly nonconvex losses, and report experiments on synthetic ridge regression and federated MNIST showing improved accuracy over Local, FedAvg, and IFCA.
Significance. If the algorithm is made consistent with the stated problem and the Wasserstein-embedding approximation is justified, the method would be a useful peer-based PFL contribution: it avoids clustering assumptions, has a communication-light dissimilarity computation, and the convergence analysis explicitly accounts for stochastic gradients and inexact projections. The empirical results are plausible and support the method on the tested tasks.
major comments (4)
- [Sec. 3.1 Eq. (1) and Sec. 3.3 set K] The optimization problem is stated with constraints ||theta_i - theta_j||_2^2 <= t D_ij, while the feasible set actually used by Algorithm 1 and Theorem 4.3 is {theta : ||theta_i - theta_j||_2 <= t D_ij}. These feasible sets differ: for D_ij=2 and t=1, a pair with ||theta_i - theta_j||=1.5 satisfies the norm constraint but violates the squared one. Since Proposition 3.4 only bounds the squared distance, the theory justifies the squared formulation, whereas the algorithm and convergence theorem prove a result for a different problem. The formulation must be made consistent.
- [Sec. 3.2] The claim that ||Phi(D_i)-Phi(D_j)||_1 approximates W_1(mu_i, mu_j) is asserted with citations but without a statement of the approximation error, the conditions on the reference distribution mu_0 and sample size N_0, or a proof. Because these D_ij values directly determine the feasible set, a poor approximation would invalidate the transfer of the Wasserstein-based bound in Eq. (2) to the constraints actually used in the algorithm.
- [Sec. 4, Theorem 4.3 and Appendix A.3] The proof as written does not yield the stated bound. Substituting eta = 3/(8L) into the final displayed inequality 2/(eta(1-2*eta*L)) gives 64L/3, not 8L/3, and the proof uses an undefined factor alpha in the definition of the gradient mapping. The telescoping steps need to be rechecked and the constants corrected.
- [Sec. 3.3 and Limitations] The delta-inexact projection is required to return feasible points (theta in K), but the computation is said to be described in supplemental material that is not included, and the Limitations section concedes that feasibility is not always guaranteed. The theorem therefore may not apply to the implemented projection; the authors should either provide the projection algorithm and prove its feasibility guarantee, or adjust the theory to allow infeasible iterates.
minor comments (4)
- [Sec. 5.2] The sentence listing final test accuracies ends with '83.9 +/- 2.60%, respectively' without identifying which strategy this number belongs to (presumably IFCA).
- [Introduction] 'centerally constrained' should be 'centrally constrained'.
- [Fig. 3] The numeric dissimilarity values are shown without units or an explicit explanation of their scale; a brief caption note would help.
- [Sec. 5.1] Please report the number of random trials or seeds used to compute the mean and the +/- 2*SE intervals.
Circularity Check
No circularity: the core derivation is a forward argument and the convergence guarantee does not encode fitted parameters.
full rationale
This paper's derivation chain is self-contained in the relevant circularity senses. Proposition 3.4 derives a squared-model-distance bound from quadratic functional growth and L_X-Lipschitz losses using Kantorovich-Rubinstein duality; the bound does not assume the constraint structure of Problem (1). The dissimilarity parameters D_ij are computed from a linear embedding of the empirical distributions, following external prior work (Kolouri et al. 2020, Liu et al. 2025), and are used as constraint coefficients; no fitted model-distance quantity is renamed as a prediction. The convergence proof (Theorem 4.3) is an adapted variance-reduced projected SGD analysis with inexact projections, and the stationarity neighborhood depends only on the stochastic-gradient variance sigma^2 and projection error delta; the free parameter t and the D_ij values do not enter the convergence guarantee, so the theorem is not forced by the empirical tuning. The empirical comparison uses cross-validation for hyperparameters in a standard way, so the reported accuracy is not a constructed consequence of the theory. The main substantive concerns—the mismatch between squared constraints in Problem (1) and norm constraints in K, and the unstated conditions for the Wasserstein-embedding approximation—are correctness and limitation issues, not circular reductions. No load-bearing self-citation or definitional equivalence was found.
Assumptions & free parameters
free parameters (5)
- Regularization parameter t =
Cross-validated on synthetic; set to 20 on MNIST
- Learning rate eta =
3/(8L) on synthetic with L estimated; 2e-4 on MNIST
- Reference data set D_0 and size N_0 =
Not specified
- Data metric d_X =
Euclidean norm on joint feature-label space
- Inexact projection tolerance delta =
Not reported
assumptions (5)
- domain assumption Assumption 3.1: quadratic functional growth with parameter gamma > 0
- domain assumption Assumption 3.2: loss is L_X-Lipschitz in data
- standard math Assumption 4.1: weighted losses are L-smooth
- standard math Assumption 4.2: stochastic gradients have bounded variance sigma^2
- ad hoc to paper The embedding ||Phi(D_i)-Phi(D_j)||_1 defines a metric approximating W1
Cite this review
Pith. "Pith review of Personalized Federated Learning under Model Dissimilarity Constraints." pith.science (2026). https://pith.science/paper/XGJNDAXE
@misc{pith2026250507575,
author = {Pith},
title = {Pith review of: Personalized Federated Learning under Model Dissimilarity Constraints},
year = {2026},
howpublished = {\url{https://pith.science/paper/XGJNDAXE}},
note = {Machine review of arXiv:2505.07575}
}
read the original abstract
One of the defining challenges in federated learning is that of statistical heterogeneity among clients. We address this problem with KARULA, a regularized strategy for personalized federated learning, which constrains the pairwise model dissimilarities between clients based on the difference in their distributions, as measured by a surrogate for the 1-Wasserstein distance adapted for the federated setting. This allows the strategy to adapt to highly complex interrelations between clients, that e.g., clustered approaches fail to capture. We propose an inexact projected stochastic gradient algorithm to solve the constrained problem that the strategy defines, and show theoretically that it converges with smooth, possibly non-convex losses to a neighborhood of a stationary point with rate O(1/K). We demonstrate the effectiveness of KARULA on synthetic and real federated data sets.
Figures
Reference graph
Works this paper leans on
-
[1]
Ang, A. and Timmermann, A. (2012). Regime changes and financial markets. Annu. Rev. Financ. Econ. , 4(1):313--337
work page 2012
-
[2]
Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choudhary, S. (2019). Federated learning with personalization layers. arXiv preprint arXiv:1912.00818
arXiv 2019
-
[3]
Armacki, A., Bajovic, D., Jakovetic, D., and Kar, S. (2022). Personalized federated learning via convex clustering. In 2022 IEEE International Smart Cities Conference (ISC2) , pages 1--7. IEEE
work page 2022
-
[4]
J., Ritov, Y., and Tsybakov, A
Bickel, P. J., Ritov, Y., and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics , 37(4):1705--1732
work page 2009
-
[5]
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. (2019). LEAF : A benchmark for federated settings. In Workshop on Federated Learning for Data Privacy and Confidentiality
work page 2019
-
[6]
Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives . Advances in neural information processing systems , 27
work page 2014
-
[7]
Deng, Y., Kamani, M. M., and Mahdavi, M. (2020). Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461
arXiv 2020
-
[8]
Dinh, C. T., Tran, N., and Nguyen, J. (2020). Personalized federated learning with moreau envelopes. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 21394--21405. Curran Associates, Inc
work page 2020
Show all 39 references
-
[9]
T., Vu, T
Dinh, C. T., Vu, T. T., Tran, N. H., Dao, M. N., and Zhang, H. (2022). A new look and convergence rate of federated multitask learning with Laplacian regularization. IEEE Transactions on Neural Networks and Learning Systems
2022
-
[10]
and Gursoy, G
Elhussein, A. and Gursoy, G. (2024). A universal metric of dataset similarity for cross-silo federated learning. arXiv preprint arXiv:2404.18773
2024
-
[11]
Fallah, A., Mokhtari, A., and Ozdaglar, A. (2020). Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Sys...
2020
-
[12]
Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. (2020). An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 33:19586--19597
2020
-
[13]
M., Ramage, D., Beaufays, F., Eichner, H., Rao, K., Mathews, R., and Augenstein, S
Hard, A., Kiddon, C. M., Ramage, D., Beaufays, F., Eichner, H., Rao, K., Mathews, R., and Augenstein, S. (2018). Federated learning for mobile keyboard prediction
2018
-
[14]
Hashemi, D., He, L., and Jaggi, M. (2024). CoBo : Collaborative learning via bilevel optimization. arXiv preprint arXiv:2409.05539
2024 arXiv
-
[15]
Jhunjhunwala, D., Sharma, P., Nagarkatti, A., and Joshi, G. (2022). Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence , pages 906--916. PMLR
2022
-
[16]
Kantorovich, L. V. (1960). Mathematical methods of organizing and planning production. Management science , 6(4):366--422
1960
-
[17]
P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132--5143. PMLR
2020
-
[18]
N., Viscoli, C
Kernan, W. N., Viscoli, C. M., Makuch, R. W., Brass, L. M., and Horwitz, R. I. (1999). Stratified randomization for clinical trials. Journal of clinical epidemiology , 52(1):19--26
1999
-
[19]
K., and Hoffmann, H
Kolouri, S., Naderializadeh, N., Rohde, G. K., and Hoffmann, H. (2020). Wasserstein embedding for graph learning. arXiv preprint arXiv:2006.09430
2020 arXiv
-
[20]
and Leibler, R
Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics , 22(1):79--86
1951
-
[21]
Li, T., Hu, S., Beirami, A., and Smith, V. (2021). Ditto: Fair and robust federated learning through personalization. In International conference on machine learning , pages 6357--6368. PMLR
2021
-
[22]
Lin, J. (1991). Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory , 37(1):145--151
1991
-
[23]
Liu, X., Bai, Y., Lu, Y., Soltoggio, A., and Kolouri, S. (2025). Wasserstein task embedding for measuring task similarities. Neural Networks , 181:106796
2025
-
[24]
A., Baraliakos, X., Baker, D., Hsu, B., and van der Heijde, D
Machado, P., Landew \'e , R., Braun, J., Hermann, K.-G. A., Baraliakos, X., Baker, D., Hsu, B., and van der Heijde, D. (2011). A stratified model for health outcomes in ankylosing spondylitis. Annals of the rheumatic diseases , 70(10):1758--1764
2011
-
[25]
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data . In Singh, A. and Zhu, J., editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics ,...
2017
-
[26]
Melsen, W., Bootsma, M., Rovers, M., and Bonten, M. (2014). The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses. Clinical microbiology and infection , 20(2):123--129
2014
-
[27]
Nesterov, Y. (2018). Lectures on Convex Optimization , volume 137 of Springer Optimization and Its Applications . Springer, Cham, 2 edition. eBook ISBN: 978-3-319-91578-4, Published: 19 November 2018
2018
-
[28]
X., Duong, C., Phan, T., Tjiputra, E., and Tran, Q
Nguyen, A., Do, T., Tran, M., Nguyen, B. X., Duong, C., Phan, T., Tjiputra, E., and Tran, Q. D. (2022). Deep federated learning for autonomous driving. In 2022 IEEE Intelligent Vehicles Symposium (IV) , pages 1824--1830. IEEE
2022
-
[29]
Pan, S. J. (2020). Transfer learning. Learning , 21:1--2
2020
-
[30]
A., Al-Fuqaha, A., and Qadir, J
Qayyum, A., Ahmad, K., Ahsan, M. A., Al-Fuqaha, A., and Qadir, J. (2022). Collaborative federated learning for healthcare: Multi-modal covid-19 diagnosis at the edge. IEEE Open Journal of the Computer Society , 3:172--184
2022
-
[31]
Rakotomamonjy, A., Nadjahi, K., and Ralaivola, L. (2023). Federated wasserstein distance. arXiv preprint arXiv:2310.01973
2023 arXiv
-
[32]
J., and Yu, B
Raskutti, G., Wainwright, M. J., and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression over _q -balls. IEEE transactions on information theory , 57(10):6976--6994
2011
-
[33]
J., Sra, S., Poczos, B., and Smola, A
Reddi, S. J., Sra, S., Poczos, B., and Smola, A. J. (2016). Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Advances in neural information processing systems , 29
2016
-
[34]
Shui, C., Abbasi, M., Robitaille, L.- \'E ., Wang, B., and Gagn \'e , C. (2019). A principled approach for learning task similarity in multitask learning. arXiv preprint arXiv:1903.09109
2019 arXiv
-
[35]
Villani, C. (2008). Optimal Transport . Springer Berlin, Heidelberg
2008
-
[36]
Wang, J., Gao, R., and Xie, Y. (2021). Two-sample test using projected wasserstein distance. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 3320--3325. IEEE
2021
-
[37]
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv preprint arXiv:1806.00582
2018 arXiv
-
[38]
Zhong, X., Guo, S., Shan, H., Gao, L., Xue, D., and Zhao, N. (2018). Feature-based transfer learning based on distribution similarity. IEEE Access , 6:35551--35557
2018
-
[39]
Zhou, P., Zou, Y., Yuan, X.-T., Feng, J., Xiong, C., and Hoi, S. (2021). Task similarity aware meta learning: Theory-inspired improvement on MAML . In Uncertainty in artificial intelligence , pages 23--33. PMLR
2021
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.