{"id":"8aba8679-b33f-45f9-9ac3-7b72c80e8349","arxiv_id":"2506.00440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PSI-PFL uses the Population Stability Index to select federated-learning clients with label distributions closest to the global average, reporting accuracy and fairness gains under label skew.","lead":"Researchers propose a federated learning client-selection method that measures how different each client's label distribution is from the global average and trains only the closest, most homogeneous clients. Across four datasets, they report accuracy gains of up to 10% over strong baselines and more even local performance in non-IID settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Optimal τ is selected on the same test set used for final comparison, so the reported up-to-10% gain may be a best-of-five selection artifact rather than a property of PSI-based client selection.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing issue: the optimal τ is chosen by evaluating on the same data used to report final accuracy. This is not a minor methodological detail; it directly affects the validity of the central claim (up to 10% improvement). The five candidate thresholds give the method multiple chances to fit the test set, while baselines are evaluated at their default settings, so the comparison is unfair. The paper's systematic threshold-search procedure is a strength, but it must be separated from the final evaluation. A held-out protocol or an average-over-thresholds report would settle whether the gain is real. No other concern is as directly tied to the headline result. The paper provides no released code and no error bars, but those are secondary; the threshold-selection bias is the single issue that could invalidate the claimed improvement. The conditional verdict is appropriate because the concern is addressable with additional experiments.","tokens_in":11941,"tokens_out":4959,"duration_ms":49568,"concrete_test":"Use a nested evaluation protocol: for each dataset, α, and client count, split the client population (or the test examples) into a τ-selection validation set and a held-out evaluation set. Select τ using only the validation set, then report PSI-PFL accuracy on the held-out set, and compare with baselines evaluated on the same held-out set. As a secondary check, report the accuracy averaged over all five candidate τ values rather than the best; if the held-out or averaged gain is much smaller than 10% or non-significant, the central claim is a tuning artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 (step 3) instructs: 'For each candidate τ, train the FL model and evaluate its performance. Select the optimal τ threshold based on the highest value of our chosen performance metric.' Section 4.2 then states that all reported results are 'based on the optimal τ threshold.' With only five candidate thresholds (10th, 25th, 50th, 75th, 90th percentiles of the client PSI distribution), selecting the best on the evaluation set introduces a positive selection bias: the maximum of five correlated estimates is expected to exceed the performance of any fixed threshold, and no baseline is given an analogous model-selection step. Because the same test data is used both to choose τ and to compare against baselines, the headline 'up to 10% relative increase' can be inflated even if PSI-based selection has no real advantage. The fairness metrics (CP, AD) are also reported at the selected τ and inherit the same bias. No held-out split, cross-validation, or multiple-comparison correction is described, and no error bars are reported despite five random seeds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PSI-PFL, a client-selection strategy for federated learning under label distribution skew. The server computes a Population Stability Index (Eq. 1) between each client's label distribution and the global label distribution, selects clients with PSI below a threshold tau, trains a global model via FedAvg, and distributes it to all clients. The threshold tau is chosen from five percentiles of the client PSI distribution by maximizing global accuracy. Experiments on ACS Income, Dutch, CelebA, and Sent140 across alpha in {0.3, 0.7, 50} and client counts {10, 50, 100} compare against FedAvg, FedProx, adaptive optimizers, HACCS, FedCLS, and other baselines, reporting up to 10% relative accuracy gains and fairer local accuracy.","tokens_in":12142,"tokens_out":5539,"duration_ms":51810,"significance":"If the performance claims survive a properly validated threshold-selection protocol, this would be a useful contribution: a simple, communication-light divergence metric (only label frequencies are shared) yields strong gains in label-skewed FL, and the comparison with Hellinger, Jensen-Shannon, and EMD distances is of practical interest. The paper states that code will be released only after acceptance, so independent reproduction is currently impossible. The central claim, however, currently rests on test-set-based selection of tau and on point estimates without variance, so the significance cannot be fully assessed from the manuscript as written.","major_comments":[{"comment":"The optimal tau is selected by training on the evaluation data and picking the threshold with the highest performance metric, and all reported results are \"based on the optimal tau threshold.\" With only five candidate thresholds, the maximum of five correlated accuracy estimates is positively biased relative to any fixed threshold, and no baseline is given an analogous model-selection step. This directly affects the headline \"up to 10%\" claim and the fairness metrics (CP, AD) reported at the selected tau. Please use a held-out validation split for tau selection and then evaluate on a separate test split, or give baselines the same tuning opportunity; alternatively, report results averaged over the five thresholds as a robustness check.","section":"Section 3.2 (step 3) and Section 4.2"},{"comment":"The paper states that five random seeds are used, but no standard deviations, confidence intervals, or significance tests are reported anywhere in the main text or appendix; every figure shows only point estimates. Consequently, the assertions \"consistently outperforms\" and \"up to 10% relative increase\" cannot be distinguished from seed noise. Please report mean and standard deviation across seeds and run paired statistical tests (e.g., Wilcoxon signed-rank) against each baseline, especially for the configurations that produce the maximum gain.","section":"Section 4.1 and all results figures"},{"comment":"There is a direct contradiction in the threshold analysis. Observation 2 states that \"Low and Medium-low PSI-PFL thresholds facilitate obtaining higher global accuracy for all client and alpha configurations,\" while Appendix C states that \"using high and medium-high tau values systematically yields better test accuracy values\" and claims this is the same behavior as in the ACSIncome results. This inconsistency needs to be resolved, either by correcting a typo or by clarifying that the two data settings exhibit different threshold preferences; as written, it undermines the claimed consistency of the method.","section":"Observation 2 (Section 4.2) and Appendix C (paragraph after Fig. 13)"}],"minor_comments":[{"comment":"Eq. (1) is the symmetric J-divergence, not the standard binned Population Stability Index used in credit scoring; although the authors acknowledge the equivalence to J-divergence, the terminology is likely to confuse readers, and a one-sentence clarification of why the name PSI is retained would help.","section":"Section 3.1"},{"comment":"The number of examples for Sent140 is given as \"1,6000,000,\" which appears to be a typo (likely 1,600,000 or 16,000,000); please correct.","section":"Table 1"},{"comment":"The abbreviation \"PSI-FL\" appears in the paragraph after Fig. 14 and \"PS-PFL\" appears in Section 4.3; both should be \"PSI-PFL\" for consistency.","section":"Appendix C and Section 4.3"},{"comment":"The OS name \"beastbianv2\" looks like a typo for a Debian-based distribution; please verify the exact name.","section":"Section 4.1"},{"comment":"The experimental setup lists model architectures but omits key training hyperparameters (learning rate, batch size, local epochs, communication rounds, server optimizer settings). Including these is necessary for reproduction, even if the code is later released.","section":"Section 4.1 and Appendix B"},{"comment":"The paper claims \"code will be released after acceptance\" but provides no repository or pseudocode; since the evaluation protocol is one of the main weaknesses, a public anonymized implementation would substantially strengthen the manuscript.","section":"Abstract and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a practical problem and the core idea is simple and plausible, but the evaluation protocol is the key weakness. The test-set-based tau selection and the absence of error bars are load-bearing issues that can likely be fixed with additional experiments, so I recommend major revision rather than rejection. The contradictory threshold observations in Section 4.2 versus Appendix C should also be carefully fixed, as they may indicate a deeper inconsistency in the reported behavior. I would also ask the editor to consider whether the journal permits code to be withheld until acceptance, since reproducibility is already limited by missing hyperparameters."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on PSI-PFL. The paper does one simple thing: use PSI (which they correctly note is the same as J divergence) to select clients with low label skew relative to the global distribution, and it reports consistent accuracy and fairness gains over a decent set of baselines on four datasets spanning tabular, image, and text. The equations are correct, the writing is clear, and they are honest about the metric's provenance. That's the good part.\n\nThe soft spot is the evaluation protocol. In Section 3.2, step 3, they pick tau by training the model and evaluating on the chosen performance metric; then Section 4.2 reports all results \"based on the optimal tau threshold.\" With only five candidate taus, selecting the best on the same test set used for comparison introduces positive selection bias, and no baseline is given an analogous model-selection step. That makes the headline \"up to 10%\" gain potentially inflated, and this issue is load-bearing for the main claim, not a footnote. The fix is straightforward: hold out a validation set for tau selection, report results on a separate test set, or at least correct for multiple comparisons and show error bars across the five seeds. The absence of standard deviations or significance tests is a real weakness, especially since they ran five seeds.\n\nThere is also a smaller circularity in the \"PSI predicts alpha\" study: alpha generates the label pmfs, and PSI is computed from those same pmfs, so the feature importance result is expected. It shows PSI tracks the generative parameter, but it is not evidence that PSI is uniquely good. The paper also promises code \"after acceptance,\" which is fine for a workshop, but for a journal-level claim it should be released.\n\nI would send this to a serious referee. The idea is plausible, the experiments are broad, and the main flaw is fixable with a clean evaluation protocol. If the tau-selection issue is resolved and the gains survive a held-out split, this becomes a useful operational contribution to the client-selection literature. As it stands, I'd read the 10% claim skeptically until then.","headline":"A plausible client-selection heuristic whose headline gains are clouded by threshold selection on the test set and missing error bars.","tokens_in":12656,"tokens_out":1728,"would_cite":false,"duration_ms":16802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PSI-PFL claims that selecting federated-learning clients by the Population Stability Index—a label-distribution divergence—improves global accuracy by up to 10% under non-IID data while making local performance fairer.","keywords":["Federated Learning","Personalized Federated Learning","Population Stability Index","Client Selection","Non-IID Data","Label Skew","Data Heterogeneity","Client Fairness"],"falsifier":"Run the same protocol but fix tau in advance, for example at the 25th percentile of the PSI distribution, or select tau on a disjoint validation split, then recompute the comparison against HACCS and FedCLS. If PSI-PFL's accuracy advantage shrinks to within the run-to-run noise across the five reported seeds, the headline 10% improvement is an artifact of peeking at test accuracy during threshold selection.","tokens_in":11758,"feed_emoji":"🎯","tokens_out":7338,"duration_ms":67161,"temperature":0.7,"pith_summary":"The paper proposes using the Population Stability Index (PSI), a divergence measure borrowed from credit scoring, to quantify label skew in federated learning and to select which clients train the global model. It claims that PSI-PFL outperforms state-of-the-art baselines by up to 10% in global accuracy under non-IID data while improving fairness across clients. The paper also argues that PSI is a better diagnostic of non-IIDness than Hellinger distance, Jensen-Shannon distance, and Earth mover's distance. If the claim holds, a cheap per-client statistic computed from label frequencies alone could substantially improve federated learning under data heterogeneity.","feed_headline":"Label-skew metric picks better FL clients: up to 10% accuracy gain","feed_subtitle":"PSI-PFL filters clients by label-skew divergence, beating strong baselines and landing within 3% of centralized accuracy.","key_machinery":"The machinery is the client-level Population Stability Index for label skew, written as PSI_i^L = sum over classes c of (P(y=c) - P_i(y=c)) ln(P(y=c) / P_i(y=c)), which the paper notes is the same as the J-divergence. The server aggregates label frequencies from all clients, computes each client's PSI, forms the weighted average WPSI to summarize system-level non-IIDness, and selects clients whose PSI falls below a threshold tau. The threshold is chosen by training once for each candidate percentile of the client PSI distribution and keeping the best performer. This object does the work of turning raw label histograms into a per-client heterogeneity score that drives the selection decision and, the paper argues, explains the accuracy and fairness gains.","core_discovery":"The paper's central claim is that label skew between each client and the federation-wide label distribution can be measured client-by-client with the population stability index, and that restricting training to clients with PSI below a threshold tau yields a global model with up to 10% higher accuracy than existing selection and regularization baselines, along with fairer local performance. PSI for client i is defined as the summed divergence between the global and client label probability mass functions, with low PSI meaning homogeneous and high PSI meaning heterogeneous. Candidate tau values are taken from the 10th, 25th, 50th, 75th, and 90th percentiles of the client PSI distribution, and the tau with the best training outcome is kept. Across four datasets spanning tabular, image, and text modalities, with 10 to 100 clients, PSI-PFL consistently beats the baselines and lands within 3% average distance of centralized training under high heterogeneity, where the strongest baselines sit about 24% away.","pith_inferences":["The same PSI computation could be extended to feature or quantity skew by replacing label pmfs with feature distributions or sample-count ratios; the paper only tests label skew, so the 10% figure should not be read as covering other skew types.","Because selection uses only label histograms, PSI-PFL adds little communication overhead; combining it with privacy techniques such as differential privacy or secure aggregation is a natural next step the paper leaves open.","Selecting only low-PSI clients may systematically exclude clients with rare labels, so in extremely skewed federations the global model could drift away from minority subpopulations even while average accuracy rises.","The threshold selection procedure, which trains once per candidate percentile and picks the best, could be turned into a cheap one-time calibration step at the start of training; the paper does not test whether the optimal tau remains stable across rounds."],"forward_implications":["Under label-skewed non-IID data, PSI-PFL raises global test accuracy by up to 10% relative to state-of-the-art selection and regularization baselines.","Client fairness improves: at the most heterogeneous setting, PSI-PFL reports a client-parity value of 0.27 and an average distance of 3% from centralized training, while HACCS and FedCLS sit around 2.2 and 24%, respectively.","The benefit holds across tabular (ACS Income, Dutch), image (CelebA), and text (Sent140) datasets.","Low-to-medium-low thresholds, specifically the 10th and 25th percentiles of the PSI distribution, are the best operating points under medium-to-high non-IIDness.","Larger federations, with 50 or 100 clients, amplify the accuracy advantage of PSI-based selection."],"supporting_citations":[{"why":"Defines the population stability index used to score each client's label skew.","marker":"[26]"},{"why":"Documents non-IID data as a central federated learning challenge and identifies label skew as particularly harmful, motivating client selection.","marker":"[9]"},{"why":"Provides the standard non-IID partitioning protocol used to generate heterogeneous client datasets in the experiments.","marker":"[15]"},{"why":"Classifies personalized federated learning into regularization-based and selection-based strategies, the taxonomy in which PSI-PFL is positioned.","marker":"[27]"},{"why":"Power-of-Choice, a selection baseline whose convergence and accuracy behavior PSI-PFL is compared against.","marker":"[14]"},{"why":"HACCS, a clustered client-selection baseline that PSI-PFL must beat for its accuracy claim.","marker":"[31]"},{"why":"FedCLS, a label-based selection baseline that PSI-PFL outperforms in the reported results.","marker":"[16]"}],"fun_headline_variants":["PSI client selection lifts non-IID FL accuracy by up to 10%","Choose FL clients by PSI to cut label-skew damage: 10% gain","Population Stability Index filters clients, beats FL baselines by 10%","Label-skew metric for client selection lands within 3% of centralized","PSI-PFL: pick homogeneous clients, beat strong baselines by 10%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that evaluating candidate tau thresholds on the same performance metric used for the final reported results is a fair procedure; if that is not valid, the 10% gain could reflect threshold tuning rather than a property of PSI-based selection.","fun_headline_variants_meta":{"raw":{"variants":["PSI client selection lifts non-IID FL accuracy by up to 10%","Choose FL clients by PSI to cut label-skew damage: 10% gain","Population Stability Index filters clients, beats FL baselines by 10%","Label-skew metric for client selection lands within 3% of centralized","PSI-PFL: pick homogeneous clients, beat strong baselines by 10%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1357,"prompt_tokens":938,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":554,"tokens_out":419,"duration_ms":4132,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:04:48.310215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol but fix tau in advance, for example at the 25th percentile of the PSI distribution, or select tau on a disjoint validation split, then recompute the comparison against HACCS and FedCLS. If PSI-PFL's accuracy advantage shrinks to within the run-to-run noise across the five reported seeds, the headline 10% improvement is an artifact of peeking at test accuracy during threshold selection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the population stability index used to score each client's label skew."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents non-IID data as a central federated learning challenge and identifies label skew as particularly harmful, motivating client selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard non-IID partitioning protocol used to generate heterogeneous client datasets in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classifies personalized federated learning into regularization-based and selection-based strategies, the taxonomy in which PSI-PFL is positioned."},{"cited_title":"Jee Cho, J","cited_arxiv_id":null,"evidence_quote":"Power-of-Choice, a selection baseline whose convergence and accuracy behavior PSI-PFL is compared against."},{"cited_title":"Wolfrath, N","cited_arxiv_id":null,"evidence_quote":"HACCS, a clustered client-selection baseline that PSI-PFL must beat for its accuracy claim."},{"cited_title":"Li and H","cited_arxiv_id":null,"evidence_quote":"FedCLS, a label-based selection baseline that PSI-PFL outperforms in the reported results."}],"review_version":1}