{"id":"e39c6a0e-67be-4724-9937-c4f4d145d1d7","arxiv_id":"2412.15010","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FEDMPR, combining magnitude pruning, dropout, and noise injection in local training, reports accuracy gains over standard federated baselines on several image benchmarks, though not consistently in all settings.","lead":"This paper proposes FEDMPR, a federated learning method that prunes and regularizes client models to cope with data differences between clients. It also introduces a new gender classification benchmark, CelebA-Gender, drawn from CelebA with controlled within-class attribute shifts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's mask lifecycle is underspecified and likely does not match the evaluated method; without code, the reported gains cannot be attributed to FEDMPR as described.","rationale":"I focused on the mask lifecycle because the abstract's claim of consistent outperformance presupposes a well-defined, reproducible method. Algorithm 1's mask handling is the least specified component: m_c is initialized, applied multiplicatively each round, and never updated, while pruning is described only as zeroing weights. The two coherent readings (identity mask or stale mask) either make the pseudocode misleading or should degrade performance, so neither supports the reported results without code. The reader's weakest_assumption identifies the same issue. I also note Table II contains direct counterexamples to 'consistently outperforms'—for example, CIFAR10 with α=0.1, 10 clients, FEDMPR (72.82) trails FedDyn (74.39) and SCAFFOLD (74.35), and CelebA-Gender with α=0.5, 100 clients, FEDMPR (52.76) trails FedProx (62.8), FedDC (53.4), SCAFFOLD (59.2), and FedDyn (58.2). These reinforce the reject verdict. Since the reader already rejected, my stress-test does not change the verdict.","tokens_in":12382,"tokens_out":8348,"duration_ms":73371,"concrete_test":"Run FEDMPR on CIFAR10 high-CS with two implementations: (A) m_c = all ones, never updated; (B) m_c updated after each PruneWeights call to equal the support of the pruned local model. If neither reproduces Table I(b)'s 75.22% within a few points, or if (A) and (B) differ materially, the paper's mask handling is not the mechanism behind the reported results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Section III-A) initializes a per-client mask m_c once and never updates it, yet PruneWeights (Algorithm 2) returns only pruned weight vectors, not masks. If m_c is non-trivial, re-applying it at the start of every round (w_t^c <- m_c * w_{t-1}) resets a fixed subset of weights to zero before local training, which should interfere with the global model's ability to learn in those positions; if m_c is the all-ones vector, the 'Apply mask' step is a no-op and the pseudocode does not describe the evaluated method. The paper gives no initial value for m_c, no update rule, and no code release, so the improvements reported in Tables I and II (e.g., 75.22% on CIFAR10 high-CS) cannot be reproduced from the description. This makes the central claim of consistent outperformance unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FEDMPR, a federated learning method that combines magnitude pruning, dropout, and noise injection in each client's local training, aiming to improve the global model's robustness under covariate shift without personalization. The authors also introduce a new benchmark dataset, CelebA-Gender, designed to isolate within-class attribute-based distribution shifts. Experiments are reported for low- and high-covariate-shift settings with two clients, as well as for Dirichlet-partitioned data with 10 and 100 clients, comparing against FedAvg, FedProx, FedDC, SCAFFOLD, FedDyn, and FedSelect on CIFAR-10, MNIST, Fashion-MNIST, SVHN, RAF-DB, and CelebA-Gender. The central claim is that FEDMPR consistently outperforms standard FL baselines in heterogeneous settings.","tokens_in":12640,"tokens_out":6017,"duration_ms":44852,"significance":"If the claimed improvements were robust, FEDMPR would be an attractive, simple add-on to local training: it requires no change to aggregation and no personalization, and the CelebA-Gender dataset could provide a new testbed for within-class covariate shift. However, the paper's own tables contain multiple counterexamples to the 'consistent outperformance' claim, the algorithmic description is incomplete, and the headline hyperparameters are selected on the test dataset. The evidence as presented does not support the central conclusion, and the new benchmark is insufficiently specified to be independently usable.","major_comments":[{"comment":"The claim in the abstract and Section VI that FEDMPR 'consistently outperforms standard FL baselines' is contradicted by the reported results. On CelebA-Gender with α=0.5 and 100 clients, FEDMPR achieves 52.76% whereas FedProx achieves 62.80% and FedDyn 58.20%. On CIFAR-10 with α=0.1 and 10 clients, FEDMPR (72.82%) is below SCAFFOLD (74.35%) and FedDyn (74.39%). These are not marginal edge cases; they appear in a table that also favors FEDMPR in other rows. The abstract and conclusion must be revised to identify the regimes where the method helps and to address these reversals.","section":"Section IV, Table II"},{"comment":"The mask lifecycle is underspecified. The pseudocode initializes per-client masks m_c without defining their values or update rule, and PruneWeights (Algorithm 2) returns only pruned weight vectors, not masks. The step 'Apply mask: w_t^c ← m_c ⊙ w_{t-1}' implies a fixed binary mask, but if m_c is non-trivial, re-applying it in every round permanently zeroes a subset of weights before local training, with no reconciliation across clients; if m_c is the all-ones vector, the step is a no-op and the algorithm does not match the described pruning. Without a precise mask definition or released code, the reported accuracy gains cannot be reproduced or attributed to the described method.","section":"Section III-A, Algorithm 1"},{"comment":"The hyperparameter configuration (p=0.4, d=0.2, n=0.4) is selected as the best on CIFAR-10 under 'best accuracy under many runs' and then applied to all datasets. The CIFAR-10 results are therefore fitted rather than predictive, and the claimed consistent gains on other datasets are not independent validations. The paper should report sensitivity across datasets or describe a proper validation protocol.","section":"Section V, Table III"},{"comment":"The new CelebA-Gender benchmark is a stated contribution, but its construction is insufficiently specified: the paper does not state how the five attributes are combined to form the mutually exclusive and mutually inclusive splits, how many images fall into each subset, or which feature spaces were used for the reported FID and CMMD scores. The provided link is anonymous and cannot be verified. This limits the utility of the dataset as a benchmark.","section":"Section III-C"}],"minor_comments":[{"comment":"Reference [2] identifies FedProx as 'Federated minimax optimization' by Sharma et al., which is not the FedProx paper; the correct citation is T. Li et al., 'Federated optimization in heterogeneous networks' (2020).","section":"Section II-B, References"},{"comment":"The paragraph says 'Figures 6 illustrate the impact of the number of samples per class', but Figure 5 is the accuracy-versus-samples plot and Figure 6 shows t-SNE embeddings; the figure cross-references are mixed.","section":"Section IV, text after Table I"},{"comment":"The phrase 'especially when there are relatively the contribution of few clients' is ungrammatical and should be rewritten.","section":"Section I"},{"comment":"Table II reports top-1 accuracy but does not state the number of runs or standard deviations, unlike Table I; this makes it difficult to assess the reliability of the differences.","section":"Table II caption"},{"comment":"'Initializem i masks' should read 'Initialize m_i masks' (missing space and underscore).","section":"Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is not supported by its own data, and the algorithm description is incomplete. The reference to FedProx is wrong, which suggests a need for careful proofreading if resubmitted. The anonymous dataset link does not allow verification. I would not encourage resubmission without major changes, but the authors could potentially revise the claims and provide code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a mixed bag. It proposes FEDMPR, a combination of magnitude pruning, dropout, and noise injection on the client side during federated training, plus a new benchmark, CelebA-Gender, that creates controlled within-class covariate shift by varying attribute overlap. The benchmark is the most solid piece: it addresses a real gap, and the FID/CMMD checks suggest the two conditions differ as intended. The empirical coverage is broad (six datasets, two-client and Dirichlet partitionings), and in the two-client high-covariate-shift setting the method often wins by large margins (CIFAR-10 75.2 vs 51.3 for FedAvg). That part is worth a look.\n\nThe soft spots are real, though. The abstract's \"consistently outperforms\" is false on the paper's own numbers: Table II has several 100-client, alpha=0.5 cells where FEDMPR trails FedProx, FedDyn, or FedSelect (e.g., 52.8 vs 62.8 on CelebA-Gender). The headline claim needs to be scoped down to something like \"helps most in high covariate shift with few clients.\"\n\nMore serious: the method description doesn't line up with a reproducible algorithm. Algorithm 1 initializes a per-client mask m_c once and then applies it as a Hadamard product with the broadcast global model every round, but there is no initial value, no update rule, and Algorithm 2 returns only pruned weights, not masks. If m_c is all-ones, the apply step is a no-op and the method is just round-wise magnitude pruning; if it isn't, the pseudocode describes something that would likely interfere with learning. Either way, the evaluated method isn't fully specified, and there is no code for the method. On top of that, the hyperparameters (p=0.4, d=0.2, n=0.4) were selected on CIFAR-10 test accuracy, so the headline CIFAR-10 result is fitted; other datasets give some independent check, but the selection procedure should be stated clearly.\n\nThe ablations add a bit of confusion: prune+dropout alone looks worse than dropout alone (71.4 vs 78.2 high-CS), and adding noise recovers it. That's not a fatal flaw, but it means the interactions matter and the explanation is thin.\n\nBottom line: this deserves a serious referee because the benchmark and the high-CS results are potentially useful, but the paper needs major revision: fix the algorithm description, report the mask initialization and update, release code or give exact details, and temper the consistency claim.","headline":"Useful new benchmark and strong gains in high-covariate-shift settings, but the method is under-specified and the 'consistent outperformance' claim doesn't survive its own Table II.","tokens_in":13104,"tokens_out":3539,"would_cite":true,"duration_ms":29143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that adding magnitude pruning, dropout, and noise injection to local training makes federated models markedly more robust when client data distributions drift apart.","keywords":["federated learning","covariate shift","data heterogeneity","magnitude pruning","dropout","noise injection","CelebA-Gender","regularization"],"falsifier":"Run the high-covariate-shift two-client MNIST experiment (client A digits 0-4, client B 5-9) with FEDMPR, then test the aggregated global model on the full test set; if accuracy does not clearly exceed the 50% chance level, the proposed aggregation of masked weights fails. A second decisive check: recompute each client's mask every round instead of freezing it, and observe whether the performance gap over FedAvg collapses.","tokens_in":12204,"feed_emoji":"🧠","tokens_out":3966,"duration_ms":27721,"temperature":0.7,"pith_summary":"This paper tries to establish that a simple client-side training recipe—magnitude-based weight pruning combined with dropout and Gaussian noise injection—makes federated learning models markedly more accurate when clients hold very different data distributions. The authors test this FEDMPR procedure across six benchmarks, including a new gender-classification dataset they construct to isolate covariate shift. Across low- and high-shift settings, with as few as two clients or as many as one hundred, FEDMPR consistently beats standard federated baselines without personalizing the global model. If the claim holds, it means standard federated averaging can be made robust to covariate shift by changing only local training, not the aggregation rule.","feed_headline":"Magnitude pruning makes federated learning robust to covariate shift","feed_subtitle":"A client-side mix of pruning, dropout, and noise injection beats seven FL baselines on six heterogeneous benchmarks.","key_machinery":"The central mechanism is FEDMPR's per-client training loop: each round, a client applies its fixed per-client mask $m_c$ (initialized at the start and never updated), trains the masked network locally under dropout and Gaussian noise injected inside each ResNet-18 basic block, and prunes weights again if the model's sparsity falls below a threshold $\\beta$. The three components work together: pruning zeros small-magnitude weights to reduce the influence of client-specific parameters, dropout enforces functional redundancy across subnetworks, and noise injection regularizes responses to weight perturbations during aggregation.","core_discovery":"In the paper's own terms, FEDMPR consistently outperforms standard FL baselines, yielding more robust and generalizable models in heterogeneous settings. The central empirical discovery is that sparsifying each client's local model with iterative magnitude pruning, while training with dropout and noise injection, reduces the damage caused by aggregating divergent local optima. The improvements are largest under high covariate shift, where FEDMPR raises two-client CIFAR-10 accuracy from 51.28% (FedAvg) to 75.22%, and from 91.18% to 96.93% on the new CelebA-Gender benchmark. The paper attributes this to removing redundant low-magnitude weights that amplify aggregation misalignment, while dropout creates redundant pathways and noise injection tolerates aggregation perturbations.","pith_inferences":["If pruning's benefit comes from dampening the influence of client-specific weights, then server-side masking or sparse aggregation might yield similar gains with less computation—a testable extension the paper does not run.","The stale per-client masks, never updated after initialization, may actually help stabilization by keeping the same subnetwork identity over rounds; updating masks every round could reveal whether the effect is due to sparsity or to mask persistence.","CelebA-Gender's attribute construction suggests a general recipe for building heterogeneity benchmarks where shift is controlled by attribute co-occurrence rather than label skew.","The noise injection and dropout combination resembles a form of implicit ensemble; measuring test-time robustness to weight perturbations could connect FEDMPR to flat-minima explanations of generalization."],"forward_implications":["FEDMPR can be dropped into standard federated averaging without personalization, and still outperforms FedSelect, FedProx, FedDyn, SCAFFOLD, and FedDC in most tested settings.","The gains hold under extreme heterogeneity: two clients with disjoint classes, 90% stragglers, and Dirichlet partitions with $\\alpha=0.1$ across 100 clients.","CelebA-Gender supplies a controlled benchmark for within-class covariate shift based on multi-attribute overlap, complementing class-imbalance benchmarks.","Ablations attribute most of the benefit to dropout (ratio 0.2), with pruning mattering most in high-shift regimes and noise injection providing smaller consistent gains.","The method also improves generalization and feature separability in early training, as shown by t-SNE embeddings at round 5."],"supporting_citations":[{"why":"Defines FedAvg, the standard aggregation baseline that FEDMPR extends and compares against.","marker":"[1]"},{"why":"Proximal regularization baseline (FedProx) that motivates constraining local updates and is a key comparison.","marker":"[2]"},{"why":"FedSelect, a pruning-based personalized FL approach used as the closest sparsity-aligned baseline for comparison.","marker":"[8]"},{"why":"SCAFFOLD, a control-variate regularization baseline for non-IID and covariate-shift settings.","marker":"[17]"},{"why":"The lottery ticket hypothesis, the source of iterative magnitude pruning that FEDMPR reformulates round-wise.","marker":"[20]"},{"why":"CelebA face attribute dataset from which the new CelebA-Gender benchmark is derived.","marker":"[4]"},{"why":"FID metric used to quantify similarity between client data distributions in the proposed benchmarks.","marker":"[46]"},{"why":"CMMD metric used alongside FID to measure inter-client distribution similarity.","marker":"[26]"}],"fun_headline_variants":["Pruning plus noise injection stabilizes federated learning","Sparse local models beat covariate shift in federated learning","Hybrid pruning technique makes FL robust to data drift","Dropout, noise, and pruning boost federated learning under shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The algorithm keeps each client's pruning mask fixed from initialization and never records which weights were pruned, so the paper assumes that server-averaging these differently pruned weight vectors still yields a coherent global model—an assumption that is load-bearing and unexplained.","fun_headline_variants_meta":{"raw":{"variants":["Pruning plus noise injection stabilizes federated learning","Sparse local models beat covariate shift in federated learning","Hybrid pruning technique makes FL robust to data drift","Dropout, noise, and pruning boost federated learning under shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":2150,"prompt_tokens":933,"completion_tokens":1217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1149}},"tokens_in":549,"tokens_out":1217,"duration_ms":9037,"temperature":1.0,"reasoning_tokens":1149,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:42:13.224836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the high-covariate-shift two-client MNIST experiment (client A digits 0-4, client B 5-9) with FEDMPR, then test the aggregated global model on the full test set; if accuracy does not clearly exceed the 50% chance level, the proposed aggregation of masked weights fails. A second decisive check: recompute each client's mask every round instead of freezing it, and observe whether the performance gap over FedAvg collapses.","supporting_citations":[{"cited_title":"McMahan, E","cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, the standard aggregation baseline that FEDMPR extends and compares against."},{"cited_title":"Sharma, R","cited_arxiv_id":null,"evidence_quote":"Proximal regularization baseline (FedProx) that motivates constraining local updates and is a key comparison."},{"cited_title":"Tamirisa, C","cited_arxiv_id":null,"evidence_quote":"FedSelect, a pruning-based personalized FL approach used as the closest sparsity-aligned baseline for comparison."},{"cited_title":"Frankle and M","cited_arxiv_id":null,"evidence_quote":"The lottery ticket hypothesis, the source of iterative magnitude pruning that FEDMPR reformulates round-wise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CelebA face attribute dataset from which the new CelebA-Gender benchmark is derived."},{"cited_title":"Heusel, H","cited_arxiv_id":null,"evidence_quote":"FID metric used to quantify similarity between client data distributions in the proposed benchmarks."},{"cited_title":"Jayasumana, S","cited_arxiv_id":null,"evidence_quote":"CMMD metric used alongside FID to measure inter-client distribution similarity."}],"review_version":1}