REVIEW 3 major objections 6 minor 37 references
PSI-PFL: Population Stability Index for Client Selection in non-IID Personalized Federated Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read PSI-PFL claims that selecting federated-learning clients by the Population Stability Index—a label-distribution divergence—improves global accuracy by up to 10% under non-IID data while making local performance fairer.
desk verdict A plausible client-selection heuristic whose headline gains are clouded by threshold selection on the test set and missing error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the client-level Population Stability Index for label skew, written as PSI_i^L = sum over classes c of (P(y=c) - P_i(y=c)) ln(P(y=c) / P_i(y=c)), which the paper notes is the same as the J-divergence. The server aggregates label frequencies from all clients, computes each client's PSI, forms the weighted average WPSI to summarize system-level non-IIDness, and selects clients whose PSI falls below a threshold tau. The threshold is chosen by training once for each candidate percentile of the client PSI distribution and keeping the best performer. This object does the work of turning raw label histograms into a per-client heterogeneity score that drives the selection decision and, the paper argues, explains the accuracy and fairness gains.
What would settle it
Run the same protocol but fix tau in advance, for example at the 25th percentile of the PSI distribution, or select tau on a disjoint validation split, then recompute the comparison against HACCS and FedCLS. If PSI-PFL's accuracy advantage shrinks to within the run-to-run noise across the five reported seeds, the headline 10% improvement is an artifact of peeking at test accuracy during threshold selection.
Extended reading notes
Core claim
The paper's central claim is that label skew between each client and the federation-wide label distribution can be measured client-by-client with the population stability index, and that restricting training to clients with PSI below a threshold tau yields a global model with up to 10% higher accuracy than existing selection and regularization baselines, along with fairer local performance. PSI for client i is defined as the summed divergence between the global and client label probability mass functions, with low PSI meaning homogeneous and high PSI meaning heterogeneous. Candidate tau values are taken from the 10th, 25th, 50th, 75th, and 90th percentiles of the client PSI distribution, and the tau with the best training outcome is kept. Across four datasets spanning tabular, image, and text modalities, with 10 to 100 clients, PSI-PFL consistently beats the baselines and lands within 3% average distance of centralized training under high heterogeneity, where the strongest baselines sit about 24% away.
Load-bearing premise
The load-bearing premise is that evaluating candidate tau thresholds on the same performance metric used for the final reported results is a fair procedure; if that is not valid, the 10% gain could reflect threshold tuning rather than a property of PSI-based selection.
Editorial extensions
If this is right
- Under label-skewed non-IID data, PSI-PFL raises global test accuracy by up to 10% relative to state-of-the-art selection and regularization baselines.
- Client fairness improves: at the most heterogeneous setting, PSI-PFL reports a client-parity value of 0.27 and an average distance of 3% from centralized training, while HACCS and FedCLS sit around 2.2 and 24%, respectively.
- The benefit holds across tabular (ACS Income, Dutch), image (CelebA), and text (Sent140) datasets.
- Low-to-medium-low thresholds, specifically the 10th and 25th percentiles of the PSI distribution, are the best operating points under medium-to-high non-IIDness.
- Larger federations, with 50 or 100 clients, amplify the accuracy advantage of PSI-based selection.
Reading between the lines
- The same PSI computation could be extended to feature or quantity skew by replacing label pmfs with feature distributions or sample-count ratios; the paper only tests label skew, so the 10% figure should not be read as covering other skew types.
- Because selection uses only label histograms, PSI-PFL adds little communication overhead; combining it with privacy techniques such as differential privacy or secure aggregation is a natural next step the paper leaves open.
- Selecting only low-PSI clients may systematically exclude clients with rare labels, so in extremely skewed federations the global model could drift away from minority subpopulations even while average accuracy rises.
- The threshold selection procedure, which trains once per candidate percentile and picks the best, could be turned into a cheap one-time calibration step at the start of training; the paper does not test whether the optimal tau remains stable across rounds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PSI-PFL, a client-selection strategy for federated learning under label distribution skew. The server computes a Population Stability Index (Eq. 1) between each client's label distribution and the global label distribution, selects clients with PSI below a threshold tau, trains a global model via FedAvg, and distributes it to all clients. The threshold tau is chosen from five percentiles of the client PSI distribution by maximizing global accuracy. Experiments on ACS Income, Dutch, CelebA, and Sent140 across alpha in {0.3, 0.7, 50} and client counts {10, 50, 100} compare against FedAvg, FedProx, adaptive optimizers, HACCS, FedCLS, and other baselines, reporting up to 10% relative accuracy gains and fairer local accuracy.
Significance. If the performance claims survive a properly validated threshold-selection protocol, this would be a useful contribution: a simple, communication-light divergence metric (only label frequencies are shared) yields strong gains in label-skewed FL, and the comparison with Hellinger, Jensen-Shannon, and EMD distances is of practical interest. The paper states that code will be released only after acceptance, so independent reproduction is currently impossible. The central claim, however, currently rests on test-set-based selection of tau and on point estimates without variance, so the significance cannot be fully assessed from the manuscript as written.
major comments (3)
- [Section 3.2 (step 3) and Section 4.2] The optimal tau is selected by training on the evaluation data and picking the threshold with the highest performance metric, and all reported results are "based on the optimal tau threshold." With only five candidate thresholds, the maximum of five correlated accuracy estimates is positively biased relative to any fixed threshold, and no baseline is given an analogous model-selection step. This directly affects the headline "up to 10%" claim and the fairness metrics (CP, AD) reported at the selected tau. Please use a held-out validation split for tau selection and then evaluate on a separate test split, or give baselines the same tuning opportunity; alternatively, report results averaged over the five thresholds as a robustness check.
- [Section 4.1 and all results figures] The paper states that five random seeds are used, but no standard deviations, confidence intervals, or significance tests are reported anywhere in the main text or appendix; every figure shows only point estimates. Consequently, the assertions "consistently outperforms" and "up to 10% relative increase" cannot be distinguished from seed noise. Please report mean and standard deviation across seeds and run paired statistical tests (e.g., Wilcoxon signed-rank) against each baseline, especially for the configurations that produce the maximum gain.
- [Observation 2 (Section 4.2) and Appendix C (paragraph after Fig. 13)] There is a direct contradiction in the threshold analysis. Observation 2 states that "Low and Medium-low PSI-PFL thresholds facilitate obtaining higher global accuracy for all client and alpha configurations," while Appendix C states that "using high and medium-high tau values systematically yields better test accuracy values" and claims this is the same behavior as in the ACSIncome results. This inconsistency needs to be resolved, either by correcting a typo or by clarifying that the two data settings exhibit different threshold preferences; as written, it undermines the claimed consistency of the method.
minor comments (6)
- [Section 3.1] Eq. (1) is the symmetric J-divergence, not the standard binned Population Stability Index used in credit scoring; although the authors acknowledge the equivalence to J-divergence, the terminology is likely to confuse readers, and a one-sentence clarification of why the name PSI is retained would help.
- [Table 1] The number of examples for Sent140 is given as "1,6000,000," which appears to be a typo (likely 1,600,000 or 16,000,000); please correct.
- [Appendix C and Section 4.3] The abbreviation "PSI-FL" appears in the paragraph after Fig. 14 and "PS-PFL" appears in Section 4.3; both should be "PSI-PFL" for consistency.
- [Section 4.1] The OS name "beastbianv2" looks like a typo for a Debian-based distribution; please verify the exact name.
- [Section 4.1 and Appendix B] The experimental setup lists model architectures but omits key training hyperparameters (learning rate, batch size, local epochs, communication rounds, server optimizer settings). Including these is necessary for reproduction, even if the code is later released.
- [Abstract and Section 5] The paper claims "code will be released after acceptance" but provides no repository or pseudocode; since the evaluation protocol is one of the main weaknesses, a public anonymized implementation would substantially strengthen the manuscript.
Circularity Check
Optimal tau is selected on the same global-accuracy metric used for the headline, making the up-to-10% gain a best-of-five in-sample maximum; the underlying PSI-PFL derivation is otherwise self-contained.
-
fitted input called prediction
[Section 3.2 (step 3) and Section 4.2 (first paragraph)]
"For each candidate τ, train the FL model and evaluate its performance. Select the optimal τ threshold based on the highest value of our chosen performance metric (e.g., global accuracy). ... The results in the following sections are based on the optimal τ threshold."
The global-accuracy metric used in step 3 to select τ is the same metric later reported as PSI-PFL's global test accuracy and summarized as 'up to 10%' relative improvement. Because only five candidate percentiles of the client PSI distribution are tried and the best one is chosen post hoc, the reported result is an in-sample maximum rather than the performance of a fixed a priori threshold. The comparison against baselines, which are not given an analogous selection step, inherits this positive bias. No held-out split or multiple-comparison correction is described, so the headline gain is partly a tuning artifact rather than an independently predicted property of PSI-based client selection.
full rationale
PSI-PFL's client-selection derivation is self-contained: Eq. 1 defines PSI from client and global label pmfs, and the selection rule (clients with PSI below τ) does not reference the accuracy metric used for evaluation. The method is benchmarked against external baselines and does not rely on a load-bearing self-citation chain or an imported uniqueness theorem. The main concern is on the evaluation side: τ is chosen by training and evaluating on the global-accuracy metric, and all later results are based on the optimal τ, so the reported 'up to 10%' gain is a best-of-five in-sample maximum. This is a statistical circularity in the evaluation protocol, not a definitional equivalence in the method's derivation. Separately, the Section 3.3 study showing that weighted PSI predicts the Dirichlet α is a tautological sanity check: α generates the client label pmfs from which PSI is computed, so a strong correlation is expected by construction; the paper does not use this as evidence for the selection accuracy. Overall, the central derivation is independent, but the headline performance claim is partially compromised by the test-set-based τ selection.
Assumptions & free parameters
free parameters (1)
- PSI threshold tau =
Selected per experiment from candidate percentiles (10th, 25th, 50th, 75th, 90th); no fixed value reported
assumptions (3)
- domain assumption The aggregated label distribution P over all clients is a valid reference for measuring each client's heterogeneity.
- domain assumption Selecting clients whose label distribution is closest to the global distribution improves the global model.
- domain assumption Label frequencies shared by clients are sufficient statistics for the non-IIDness that harms FL performance.
Cite this review
Pith. "Pith review of PSI-PFL: Population Stability Index for Client Selection in non-IID Personalized Federated Learning." pith.science (2026). https://pith.science/paper/I2VKVZOE
@misc{pith2026250600440,
author = {Pith},
title = {Pith review of: PSI-PFL: Population Stability Index for Client Selection in non-IID Personalized Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/I2VKVZOE}},
note = {Machine review of arXiv:2506.00440}
}
read the original abstract
Federated Learning (FL) enables decentralized machine learning (ML) model training while preserving data privacy by keeping data localized across clients. However, non-independent and identically distributed (non-IID) data across clients poses a significant challenge, leading to skewed model updates and performance degradation. Addressing this, we propose PSI-PFL, a novel client selection framework for Personalized Federated Learning (PFL) that leverages the Population Stability Index (PSI) to quantify and mitigate data heterogeneity (so-called non-IIDness). Our approach selects more homogeneous clients based on PSI, reducing the impact of label skew, one of the most detrimental factors in FL performance. Experimental results over multiple data modalities (tabular, image, text) demonstrate that PSI-PFL significantly improves global model accuracy, outperforming state-of-the-art baselines by up to 10\% under non-IID scenarios while ensuring fairer local performance. PSI-PFL enhances FL performance and offers practical benefits in applications where data privacy and heterogeneity are critical.
Figures
Figures from the paper (18 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusmão, et al. Flower: A friendly federated learning research framework.arXiv preprint arXiv:2007.14390, 2020
arXiv 2007
-
[3]
J. Böhler and F. Kerschbaum. Secure multi-party computation of differentially private heavy hitters. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2361–2377, 2021
work page 2021
-
[4]
A. Chen, Y. Fu, Z. Sha, and G. Lu. An emd-based adaptive client selection algo- rithm for federated learning in heterogeneous data scenarios.Frontiers in Plant Science, 13:908814, 2022. PSI-PFL: PSI for Client Selection in non-IID PFL 15
work page 2022
-
[5]
Learning with privacy at scale
Differential Privacy Team. Learning with privacy at scale. https: //machinelearning.apple.com/research/learning-with-privacy-at-scale ,
-
[6]
F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34:6478– 6490, 2021
work page 2021
-
[7]
J. du Pisanie, J. S. Allison, and J. Visagie. A proposed simulation technique for population stability testing in credit risk scorecards.Mathematics, 11(2):492, 2023
work page 2023
-
[8]
Ú. Erlingsson, V. Pihur, and A. Korolova. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, pages 1054–1067, 2014
work page 2014
Show all 37 references
-
[9]
D. M. J. G., D. Solans, M. Heikkila, A. Vitaletti, N. Kourtellis, A. Anagnostopou- los, and I. Chatzigiannakis. Non-iid data in federated learning: A survey with taxonomy, metrics, methods, frameworks and future directions, 2024
2024
-
[10]
A. Go, R. Bhayani, and L. Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009
2009
-
[11]
Goussakov
R. Goussakov. Hellinger distance-based similarity measures for recommender sys- tems, 2020
2020
-
[12]
Hsieh, A
K. Hsieh, A. Phanishayee, O. Mutlu, and P. Gibbons. The non-iid data quagmire of decentralized machine learning. InInternational Conference on Machine Learning, pages 4387–4398. PMLR, 2020
2020
-
[13]
T.-M. H. Hsu, H. Qi, and M. Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019
1909 arXiv
-
[14]
Jee Cho, J
Y. Jee Cho, J. Wang, and G. Joshi. Towards understanding biased client selection in federated learning. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 ofProceeding...
2022
-
[15]
G. D. M. Jimenez, A. Anagnostopoulos, I. Chatzigiannakis, and A. Vitaletti. Fedartml: A tool to facilitate the generation of non-iid datasets in a controlled way to support federated learning research.IEEE Access, 2024
2024
-
[16]
Li and H
C. Li and H. Wu. FedCLS: A federated learning client selection algorithm based on cluster label information. In2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), pages 1–5. IEEE, 2022
2022
-
[17]
Q. Li, Y. Diao, Q. Chen, and B. He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978, IEEE, 2022. IEEE, IEEE
2022
-
[18]
T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020
2020
-
[19]
J. Lin. Divergence measures based on the shannon entropy.IEEE Transactions on Information theory, 37(1):145–151, 1991
1991
-
[20]
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015
2015
-
[21]
McMahan, E
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017
2017
-
[22]
J. Pei, W. Liu, J. Li, L. Wang, and C. Liu. A review of federated learning methods in heterogeneous scenarios.IEEE Transactions on Consumer Electronics, 2024. 16 Authors Suppressed Due to Excessive Length
2024
-
[23]
Reddi, Z
S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečn` y, S. Ku- mar, and H. B. McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020
2003 arXiv
-
[24]
Rubner, C
Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover’s distance as a metric for image retrieval.International journal of computer vision, 40:99–121, 2000
2000
-
[25]
Y. Shi, H. Yu, and C. Leung. Towards fairness-aware federated learning.IEEE Transactions on Neural Networks and Learning Systems, 2023
2023
-
[26]
N. Siddiqi. Credit risk scorecards: developing and implementing intelligent credit scoring, volume 3. John Wiley & Sons, 2012
2012
-
[27]
A. Z. Tan, H. Yu, L. Cui, and Q. Yang. Towards personalized federated learning. IEEE transactions on neural networks and learning systems, 34(12):9587–9603, 2022
2022
-
[28]
Q. Tan, S. Wu, and Y. Tao. Privacy-enhanced federated learning for non-iid data. Mathematics, 11(19):4123, 2023
2023
-
[29]
Van der Laan
P. Van der Laan. The 2001 census in the netherlands: Integration of registers and surveys. In CONFERENCE AT THE CATHIE MARSH CENTRE., pages 1–24, 2001
2001
-
[30]
Distribution- regularized federated learning on non-iid data
Y.Wang,Y.Tong,Z.Zhou,R.Zhang,S.J.Pan,L.Fan,andQ.Yang. Distribution- regularized federated learning on non-iid data. In2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2113–2125. IEEE, 2023
2023
-
[31]
Wolfrath, N
J. Wolfrath, N. Sreekumar, D. Kumar, Y. Wang, and A. Chandra. HACCS: Heterogeneity-aware clustered client selection for accelerated federated learning. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 985–995. IEEE, 2022
2022
-
[32]
Y. Xu, Y. Li, H. Luo, X. Fan, and X. Liu. Fblg: A local graph based approach for handling dual skewed non-iid data in federated learning. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, K. Larson, Ed. International Joint C...
2024
-
[33]
Yurdakul.Statistical properties of population stability index
B. Yurdakul.Statistical properties of population stability index. Western Michigan University, 2018
2018
-
[34]
D. Zeng, S. Liang, X. Hu, H. Wang, and Z. Xu. Fedlab: A flexible federated learning framework. Journal of Machine Learning Research, 24(100):1–7, 2023
2023
-
[35]
H. Zhao. Non-iid quantum federated learning with one-shot communication com- plexity. Quantum Machine Intelligence, 5(1):3, 2023. PSI-PFL: PSI for Client Selection in non-IID PFL 17 7 Appendix This section presents the appendix related to the data distribution analysis, the ar...
2023
-
[100]
PSI-FL demonstrates the highest average local performance among the tech- niques compared, highlighting its effectiveness in federated learning scenarios
than with fewer clients in the presence of high non-IDness levels. PSI-FL demonstrates the highest average local performance among the tech- niques compared, highlighting its effectiveness in federated learning scenarios. Additionally, PSI-FL outperforms the best-performing ba...
-
[2017]
Accessed: 2024-10-16
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.