{"id":"d0b72c83-65cb-4f6f-98b9-d02350d4c0e4","arxiv_id":"2506.16458","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SecureFed combines PCA-based anomaly detection, validation loss, and gradient magnitude into a trust score that controls zone-weighted aggregation, improving accuracy over vanilla FedAvg in a small MNIST label-flipping experiment.","lead":"SecureFed is a two-phase defense for federated learning that uses PCA-based anomaly scores and trust-weighted aggregation to down-weight suspicious client updates. The paper reports small accuracy gains over vanilla federated averaging on MNIST with label-flipping attackers, but the evidence is thin.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4 rewards high-norm updates as trustworthy, but the tested label-flipping attack can produce large-norm malicious updates; the claimed 92.50% accuracy is not tied to a sound trust mechanism.","rationale":"The reader's weakest_assumption correctly identifies the gradient-magnitude/trustworthiness assumption as fragile and notes the non-IID limitation. My concern is sharper: Eq. 4 does not merely assume an unreliable correlation; it gives gradient magnitude a positive coefficient. For the paper's own attack model, malicious label-flipping updates can have large norms, so the third term can actively push poisoned clients into Zone 1. This is load-bearing because if malicious clients do not have lower Gc, the mechanism credited for the headline result cannot be the source of the claimed robustness; the 92.50% accuracy would be an unexplained observation from a single, unreported run. The proposed check is cheap and would settle the question. The paper also lacks code, hyperparameters, standard deviations, and competing baseline defenses, which independently justify the reader's REJECT verdict, but the Eq. 4 flaw is the most specific internal threat to the central claim.","tokens_in":7662,"tokens_out":7005,"duration_ms":77612,"concrete_test":"Re-run the reported 20-client, 3-round MNIST experiment with 48% single-class label flipping, recording per-client Gc and the zone assigned in each round. Compare mean Gc for malicious vs benign clients and compute the fraction of malicious clients placed in Zone 1; then re-run with gamma=0 in Eq. 4 and check whether Table I's 92.50% accuracy changes. If malicious clients have equal or larger Gc, or appear in Zone 1, the positive gradient-magnitude term is not performing the claimed filtering; if the gamma=0 result is unchanged, the paper's stated mechanism is not the source of the reported gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the trust score in Eq. 4 of Section IV-B2: Tc = alpha*(1 - Ac/tau*) + beta*(1 - Lc/max L) + gamma*Gc/max G, with Gc = ||grad W_c^r||. The paper states repeatedly that low-gradient updates are suspicious and high-gradient updates are reliable, and Phase 2 assigns clients with larger Gc to Zone 1. Under the only attack tested, single-class label flipping on MNIST, a malicious client trains on mislabeled data for several local steps; its update must move the decision boundary for an entire digit class, so its L2 norm is not systematically low. It can easily be as large as or larger than benign updates. Because the third term enters with a positive sign, a poisoned update with large norm is rewarded in the trust score, directly counteracting the anomaly and validation-loss terms. The ablation reports only a 75% detection rate at 48% malicious, and no per-zone statistics are given, so it is unknown whether Zone 1 actually contains mostly benign clients. Section VI concedes the gradient-divergence assumption fails under non-IID data, but the problem is more basic: even in the IID setting, gradient magnitude is not a reliable proxy for trustworthiness under the very attack used. The reported 92.50% accuracy is therefore not tied to the mechanism as described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SecureFed, a two-phase framework for detecting and mitigating malicious clients in federated learning. In Phase 1, the server applies PCA to client weight updates, computes an anomaly score, and estimates a threshold using a so-called synthetic dataset. In Phase 2, each client's update is temporarily applied to the global model, evaluated on that dataset to obtain a validation loss, and combined with the anomaly score and the update's gradient magnitude into a trust score. Clients are then assigned to one of three learning zones with different aggregation weights. The authors report that with 48% malicious clients performing single-class label flipping on IID MNIST, SecureFed achieves 92.50% test accuracy versus 84.42% for FedAvg, with a detection rate of 75% in the ablation study. The paper claims improvements in precision, recall, and F1 without compromising benign accuracy.","tokens_in":8003,"tokens_out":2955,"duration_ms":32841,"significance":"If the claimed robustness results were reproducible and generalizable, a two-phase anomaly-detection-plus-trust-zone aggregation framework could be a useful addition to the FL defense literature. The paper does not ship code, proofs, or a reproducible experimental protocol, so its contribution is currently limited to a conceptual framework plus preliminary empirical observations. The idea of combining dimensionality reduction with server-side validation is not new, and the paper does not compare against the robust aggregation baselines it cites (Krum, median/trimmed mean, FLTrust, MAB-RFL). The framework does, however, propose an explicit trust score and zone-based weighting scheme, which could be a testable design if the mechanism were consistent with the threat model.","major_comments":[{"comment":"The trust score assigns a positive weight to the gradient magnitude term: T_c = alpha*(1 - A_c/tau*) + beta*(1 - L_c/max L) + gamma*G_c/max G. Under the only attack tested (single-class label flipping on MNIST, Section V-A), a malicious client training on mislabeled data must move the decision boundary for an entire digit class, so its update can have a large L2 norm. The positive gamma*G_c/max G term therefore rewards exactly the kind of poisoned update this framework is designed to detect, and the claimed 92.50% accuracy is not tied to the trust mechanism as described. The authors need to justify this term theoretically and show empirically (e.g., per-zone statistics on malicious vs benign norms) that it does not counteract the anomaly and validation-loss terms.","section":"Section IV-B2 (trust score formula)"},{"comment":"The experimental evaluation uses three global rounds, one dataset (MNIST), one IID partitioning scheme, and one attack type (single-class label flipping), with no repeated seeds, no error bars, and no statistical significance tests. These results cannot support the claim in Section VI that SecureFed 'consistently outperforms' vanilla FL, especially since the gap at 48% malicious (92.50% vs 84.42%) could be an artifact of the particular initialization or round count. The authors should run more rounds, multiple datasets, multiple random seeds, and report means and variances, and ideally compare with the robust aggregation methods cited in Section II-C.","section":"Section V-A (experimental setup) and Table I"},{"comment":"The anomaly score function f_anomaly and the threshold estimation procedure tau* = Validate(D_s, {A_c}) are not defined. The text only mentions PCA and K-Means in Section V-A, but does not specify how the anomaly score is computed from the reduced-dimensional representation, how the threshold is selected, or how the 'synthetic' dataset D_s is used in validation. Without these definitions, the detection rate in Table II cannot be reproduced, so the main empirical claim is not independently verifiable as written.","section":"Algorithm 1, lines 9-10 and Section IV-B1"},{"comment":"The reported detection rate of 75% appears to be computed in-sample on the same runs used to train and evaluate SecureFed, but the methodology for computing this rate is not provided (e.g., whether it is the fraction of malicious clients assigned to Zone 3, or a different criterion). Moreover, the zone-weighted aggregation still includes Zone 2 and Zone 3 clients with weights alpha_2 and alpha_3, so the global accuracy could be high even when detection is imperfect. The paper should report the confusion matrix of zone assignments for benign and malicious clients, and clarify how the detection rate is defined, before claiming that the framework 'successfully reduces the effects of poisoned updates'.","section":"Table II (ablation study)"},{"comment":"The paper concedes that the gradient-divergence assumption fails under non-IID data, but the problem is more basic: even in the IID setting, the trust score's dependence on gradient magnitude is not a sound indicator of trustworthiness under the tested label-flipping attack. Additionally, the 'synthetic dataset' D_s is actually a second handwritten-digits dataset (reference [23]), not a synthetic one, and the server's access to and labeling of this dataset is a non-trivial assumption that should be stated clearly. The central mechanism needs to be revised or its behavior under the actual attack distribution demonstrated before the claims in Table I can be considered supported.","section":"Section VI (Discussion) and threat model"}],"minor_comments":[{"comment":"There are numerous typographical and grammatical errors, e.g., 'training cycle continues Based on' in the abstract, 'approhces' in Section IV-A, and inconsistent capitalization of 'SecureFed' in the abstract. The paper would benefit from careful proofreading.","section":"Abstract and Section I (Introduction)"},{"comment":"The paper cites Krum, median/trimmed mean, FoolsGold, FLAME, and FLTrust as related defenses but never evaluates any of them in the experiments. At minimum, the discussion should explain why these baselines were not compared, or the experiments should include them.","section":"Section II-C (Comparison with existing works)"},{"comment":"The hyperparameters alpha, beta, gamma, tau_high, tau_low, alpha_1, alpha_2, alpha_3, and the number of PCA components (described as 'retaining top-5 components' in Section V-A) are never reported in the paper. Without these values, the experiments cannot be reproduced.","section":"Section IV-B2 and Figure 2"},{"comment":"The table reports precision, recall, and F1 for the aggregated model, but does not state what the positive class is in this classification context. It would be clearer to define these metrics with respect to the classification task (e.g., digit classification) rather than leaving the reader to infer the setting.","section":"Section V-B and Table I"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication. The main technical problem—the trust score's positive dependence on gradient magnitude rewarding poisoned updates—undermines the central claim, and the experimental evidence is far too thin to compensate. I would encourage the authors to rebuild the mechanism, run a proper evaluation with multiple seeds and baselines, and report all hyperparameters before resubmitting. The use of a real Kaggle handwriting dataset labeled as 'synthetic' also needs clarification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: SecureFed is an incremental recombination of known FL defense pieces — PCA-based outlier detection, synthetic validation loss, and trust-weighted aggregation. The specific three-term trust score and zone routing are new, but the paper doesn't demonstrate they work as described.\n\nWhat's good: The framework is genuinely server-side, simple to describe, and the ablation in Table II gives some evidence each component matters. Dropping synthetic validation hurts detection rate the most (75% to 38%), which is a concrete finding. The writing is clear enough about the pipeline. The authors also acknowledge the non-IID limitation up front in Section VI.\n\nWhere it falls short: The experimental support is not close to the claim. Three global rounds, one dataset (MNIST, IID), one attack (label flipping), no error bars, no comparison to Krum, Trimmed Mean, FLTrust, FLAME, or FedDMC — several of which are cited. The anomaly score A_c and threshold tau* are never defined; the reader has to guess what 'Validate(D_s, {A_c})' means. Hyperparameters alpha, beta, gamma, tau_high, tau_low, alpha1-3 are unreported, so the 75% detection rate could easily be tuned in-sample. The validation dataset is 'a standard dataset with similar feature vectors to the training dataset' — essentially a second handwriting dataset, so the synthetic validation is not independent of the test distribution.\n\nThe deeper problem is the trust score itself. Eq. 4 adds +gamma * G_c / max(G), so larger gradient norms increase trust. But the only attack tested is single-class label flipping over several local steps; a malicious update that must flip an entire digit class will have a large norm, not a small one. The paper's own narrative — 'low-gradient patterns usually associated with adversarial manipulation' — doesn't hold for this attack, and the stress-test note is right that the mechanism rewards the very updates it should down-weight. The ablation shows only a 75% detection rate at 48% malicious, with no per-zone statistics, so we don't know whether Zone 1 is actually clean.\n\nIn short: the idea is not absurd, but the evidence is far too thin for the conclusions, and the mechanism has a plausible internal weakness that the experiment doesn't address. This is a workshop-level draft, not a refereed-paper result. I would not cite it, and I would not send it to peer review in its current form. If the authors add baselines, error bars, more rounds/datasets/attacks, and report the hyperparameters, it could become a modest contribution worth a second look.\n\nRecommendation: desk reject, with an encouraging note to resubmit after real experiments.","headline":"A plausible server-side defense that is under-supported by a three-round MNIST experiment and built on a trust score whose gradient-magnitude term works against the tested attack.","tokens_in":8464,"tokens_out":2962,"would_cite":false,"duration_ms":29011,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SecureFed claims that its two-phase anomaly detection and trust-weighted aggregation keep a federated model at 92.5% accuracy when 48% of clients launch poisoning attacks, versus 84.4% for vanilla averaging.","keywords":["federated learning","poisoning attacks","anomaly detection","principal component analysis","learning zones","trust score","label flipping","secure aggregation"],"falsifier":"Run the Section V experiment with a non-IID MNIST partition (for example, each client sees only two digit classes) and 48% label-flipping clients; if honest clients routinely land in Zone 3, or malicious-client detection falls to near chance, the central claim is falsified.","tokens_in":7518,"feed_emoji":"🛡️","tokens_out":14594,"duration_ms":127089,"temperature":0.7,"pith_summary":"SecureFed proposes a two-phase server-side defense for federated learning against clients that deliberately poison their updates. In the first phase, the server reduces each client's weight update with PCA and scores it for outlier behavior, then builds a temporary model from those updates and evaluates it on a reference handwritten-digits dataset to get validation losses. In the second phase, those signals are combined with the update's gradient magnitude into a trust score, and clients are routed into three learning zones that determine how much weight their update carries in aggregation. The paper's central evidence is that with 48% label-flipping clients on an independent-and-identically-distributed (IID) MNIST partition, SecureFed reaches 92.50% accuracy and a 0.92 F1 score, versus 84.42% accuracy and 0.84 F1 for vanilla federated averaging, while the fully benign configuration stays at 95.49%. If this holds, a privacy-preserving learning system can withstand a client population that is nearly half adversarial with only a small accuracy cost.","feed_headline":"92.5% accuracy with 48% malicious clients: two-phase defense","feed_subtitle":"Two-phase anomaly scoring and trust-weighted averaging keeps 92.5% accuracy when 48% of clients attack.","key_machinery":"The load-bearing mechanism is the trust score and the learning zones it creates. The score $T_c = \\alpha(1 - A_c/\\tau^*) + \\beta(1 - L_c/\\max L) + \\gamma G_c/\\max G$ fuses three per-client signals: the PCA-based anomaly score $A_c$, the validation loss $L_c$ of a temporary model built from that client's update, and the update's gradient magnitude $G_c$. Clients with scores above a high threshold join Zone 1 and receive the highest aggregation weight; those between thresholds sit in Zone 2 with reduced weight; those below join Zone 3 and are progressively removed. The global update is formed by zone-weighted aggregation $W^{r+1} = \\frac{\\sum_{c\\in C}\\alpha_{z(c)}n_c W_c^r}{\\sum_{c\\in C}\\alpha_{z(c)}n_c}$. The design work of this mechanism is to soften client exclusion: instead of dropping an update outright, it scales its influence, which is what the paper claims preserves benign performance while suppressing poisoning.","core_discovery":"On its own terms, SecureFed is an architecture claim rather than a new statistical identity: the paper asserts that poisoning attacks in federated learning can be both detected and damped by coupling dimensionality reduction with trust-weighted aggregation. Phase 1 applies PCA to client updates, uses K-Means-based anomaly clustering on the reduced representations to form an anomaly score $A_c$, and calibrates a detection threshold $\\tau^*$ on the reference dataset. Phase 2 constructs a temporary model from each update, evaluates its validation loss $L_c$ on the reference set, computes the gradient magnitude $G_c = \\|\\nabla W_c^r\\|$, and forms the trust score $T_c = \\alpha(1 - A_c/\\tau^*) + \\beta(1 - L_c/\\max L) + \\gamma G_c/\\max G$. Thresholds on $T_c$ place each client in Zone 1, 2, or 3, and aggregation weights updates by a zone factor. The paper reports that this configuration keeps benign accuracy at 95.49%, lifts accuracy under 48% malicious clients from 84.42% (vanilla) to 92.50%, raises F1 from 0.84 to 0.92, and that its ablation study identifies synthetic validation as the largest contributor, since detection rate drops from 75% to 38.25% when it is removed.","pith_inferences":["A natural but untested corollary is that the trust score will misfire on non-IID data: an honest client with sparse or homogeneous local data may have both a small gradient and an outlying PCA pattern, exactly the signature the framework reads as malicious.","Because the reference set is a different handwritten-digits corpus rather than data drawn from the training distribution, the detection threshold and validation losses depend on distribution shift; using a clean held-out subset of the training data would likely change the reported detection-rate numbers.","Backdoor attacks are discussed as motivation but never tested; since backdoors are designed to leave primary-task accuracy intact, SecureFed's PCA and validation-loss signals should be expected to be weaker against them than against label flipping."],"forward_implications":["With 48% malicious clients, SecureFed's accuracy (92.50%) stays within 3 points of its benign baseline (95.49%), whereas vanilla FedAvg drops more than 11 points to 84.42%.","The ablation study predicts that removing synthetic validation is the most damaging change: detection rate falls from 75% to 38.25%, so the reference-dataset check carries much of the defense.","Replacing the graded trust score with a binary filter loses 3.23 accuracy points (89.27% versus 92.50%), which supports the paper's claim that soft zone weighting is preferable to hard exclusion.","The framework does not require changing client-side training or communication, so it could be dropped into existing federated learning pipelines as a server-side module."],"supporting_citations":[{"why":"Supplies the PCA dimensionality-reduction method used to convert client weight updates into the low-dimensional anomaly patterns scored in Phase 1.","marker":"[5]"},{"why":"Supplies the reference handwritten-digits dataset used for synthetic validation: temporary models are scored against it to produce validation losses and the detection threshold.","marker":"[23]"},{"why":"Defines the single-class label-flipping poisoning attack that the experiments use to create malicious clients.","marker":"[24]"}],"fun_headline_variants":["SecureFed: 92.5% accuracy despite 48% adversarial clients","Two-phase defense lifts poisoned FL accuracy from 84% to 92.5%","Anomaly scores and trust zones keep federated learning resilient","SecureFed's synthetic validation is key to attack detection","Trust-weighted aggregation mitigates poisoning in federated learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All three trust signals assume that an outlier PCA pattern or a low gradient magnitude marks an attacker, and that assumption has only been tested on IID MNIST with label-flipping attacks; the authors themselves note it becomes unreliable when client data is highly non-IID.","fun_headline_variants_meta":{"raw":{"variants":["SecureFed: 92.5% accuracy despite 48% adversarial clients","Two-phase defense lifts poisoned FL accuracy from 84% to 92.5%","Anomaly scores and trust zones keep federated learning resilient","SecureFed's synthetic validation is key to attack detection","Trust-weighted aggregation mitigates poisoning in federated learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2798,"prompt_tokens":1015,"completion_tokens":1783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1693}},"tokens_in":631,"tokens_out":1783,"duration_ms":14125,"temperature":1.0,"reasoning_tokens":1693,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:25:52.982565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section V experiment with a non-IID MNIST partition (for example, each client sees only two digit classes) and 48% label-flipping clients; if honest clients routinely land in Zone 3, or malicious-client detection falls to near chance, the central claim is falsified.","supporting_citations":[{"cited_title":"Jolliffe, Principal Component Analysis , 2nd ed","cited_arxiv_id":null,"evidence_quote":"Supplies the PCA dimensionality-reduction method used to convert client weight updates into the low-dimensional anomaly patterns scored in Phase 1."},{"cited_title":"Handwritten digits dataset (not in mnist),","cited_arxiv_id":null,"evidence_quote":"Supplies the reference handwritten-digits dataset used for synthetic validation: temporary models are scored against it to produce validation losses and the detection threshold."}],"review_version":1}