{"id":"563fdec9-4e6b-4186-b173-85af52a81eca","arxiv_id":"2501.12123","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FL-CLEANER filters Byzantine and backdoor model updates in federated learning under non-IID data by scoring clients with a conditional variational autoencoder on activation-map reconstruction errors and clustering the scores.","lead":"Federated learning trains a shared AI model across devices without collecting their data, but malicious devices can corrupt it. This paper proposes FL-CLEANER, a server-side filter that detects poisoned model updates even when client data is unevenly distributed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under the paper's white-box threat model (§3.1.2), an attacker knows the defense, but §4.3 assumes an unproven reconstruction-error gap and only non-adaptive attacks are tested. An adaptive attacker exploiting Algorithm 1's δ-chain is the load-bearing missing case.","rationale":"Reader's weakest_assumption points to exactly this gap, and I agree. The paper deserves credit for a clear mechanism, a small-trigger-set CVAE training design with warmup and KLD annealing, and tested defenses over two non-IID splits and two backdoors; but those tests are fixed-parameter and non-adaptive. The threat model explicitly includes full knowledge of the defense, so the absence of an adaptive evaluation is not an optional robustness check—it is the difference between the claimed first defense and a defense against the four tested recipes. The backdoor results are limited to the Dirichlet split and no error bars/seeds are reported, which adds uncertainty but the adaptive-gap issue is the load-bearing one. No code or data are released, so the 100% recall numbers cannot be independently checked; I do not treat that as evidence of misconduct, only as a reason the conditional verdict should remain. Because the reader already set CONDITIONAL and my concern is the same, I recommend UNCHANGED rather than reject: the paper could plausibly pass with an adaptive attack evaluation and a formal or rigorous empirical bound on the gap.","tokens_in":18144,"tokens_out":6808,"duration_ms":72334,"concrete_test":"Reproduce the MNIST Dirichlet and custom non-IID settings (100 clients, 30% attackers, λ=0.3, 250-sample trigger set) and add a white-box adaptive attacker that, at each round, receives the server's GeoMed, CVAE, and δ, then uses projected gradient descent on the malicious weights to minimize the trigger-set reconstruction-error score while preserving a damaging effect (e.g., sign-flip or scaling). If any adapted malicious update is accepted by Algorithm 1 and either degrades global accuracy or raises ASR, the gap assumption of §4.3 is falsified; if all adapted updates are rejected, the central claim gains the missing evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.2 grants the adversary full knowledge of the aggregation method and the deployed defense, but Section 4.3's detection mechanism rests on an empirical assertion—\"in the reconstruction error space, there is an identifiable relative error gap between the two\"—with no formal bound on the gap or on the variance of benign errors. Algorithm 1 accepts clients while sorted_re[i+1] ≤ sorted_re[i] + δ, with δ = λ(max−min). The evaluation in Section 5.4 covers only Sign-Flipping, Additive Noise, Same Value, and Scaling attacks with fixed parameters (ξ=1, σ=0.1, c=0.01, a=10); these are static, non-adaptive recipes. An adversary who knows GeoMed, the CVAE, and the cluster rule can optimize a malicious update to keep its normalized activation-map reconstruction error inside the benign chain while preserving the attack's harmful effect. This is not a stronger adversary than assumed; it is the capability the paper's own threat model claims to defend against. Without a theoretical bound on the benign/malicious error gap, or an experiment with an adaptive attacker, the central claim of 100% Byzantine recall and \"first defense\" in non-IID FL is not supported at the claimed strength. The same gap concern applies to backdoor defenses, because a backdoor update kept below the δ-chain would not be filtered.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FL-CLEANER is a server-side defense for non-IID federated learning. In each round, the server extracts activation maps from each client's updated model on a small trigger set, GeoMed-normalizes them, and uses a conditionally trained variational autoencoder to assign each client a reconstruction error. A one-parameter 'trust propagation' algorithm then clusters clients by walking through the sorted errors and stopping at the first gap larger than delta = lambda(max - min); all clients beyond the gap are rejected before FedAvg aggregation. The paper evaluates this method on MNIST and FashionMNIST under two non-IID distributions (Dirichlet label skew and a custom two-class quantity skew), against four Byzantine attacks (sign-flipping, additive noise, same-value, scaling) and two backdoor attacks (DBA and Neurotoxin), comparing with FedCAM, FedCVAE, FLEDGE, FedGuard, FLAME, and FLDetector. The reported results are 100% Byzantine recall in all tested cells, 0% FPR in most cells, less than 1% benign misclassification in attack-free settings, and near-zero backdoor ASR with preserved accuracy.","tokens_in":1848,"tokens_out":2484,"duration_ms":102985,"significance":"The intended contribution is timely and practically relevant: non-IID data are the rule in real FL deployments, and prior CVAE-based defenses such as FedCAM are shown here to fail in that regime. The design choices--using GeoMed-normalized activation maps rather than raw weights, KLD annealing and warmup to stabilize CVAE training on a small server-side trigger set, and a threshold-free clustering rule with a single interpretable parameter--are sensible and are tested against several relevant baselines. The paper also states clear security requirements (R1-R3) and reports recall, FPR, and accuracy separately, which is good experimental hygiene. However, the contribution is empirical rather than formal: no code or proofs are provided, and the central claims rest on single unseeded runs against non-adaptive attackers. The absence of an adaptive adversary under the paper's own white-box threat model, together with the uncontrolled accuracy baseline and test-set overlap, currently prevents the 'first defense' claim from being accepted at full strength. These are addressable with additional experiments and clearer reporting rather than a change of approach.","major_comments":[{"comment":"The white-box threat model in §3.1.2 grants the adversary full knowledge of the aggregation method and the deployed defense, but no adaptive attacker is evaluated. The detection mechanism rests on the §4.3 assertion that 'in the reconstruction error space, there is an identifiable relative error gap' between benign and malicious NAMs, and Algorithm 1 accepts clients while sorted_re[i+1] ≤ sorted_re[i] + δ, with δ = λ(max - min). The attacks tested in §5.4 are static recipes with fixed parameters (ξ = 1, σ = 0.1, c = 0.01, a = 10). An adversary with the assumed knowledge can, in principle, craft updates whose normalized activation-map reconstruction errors stay inside the benign chain while still poisoning the global model. To support the 100%-recall and 'first defense' claims, please either provide a formal bound on the benign/malicious error gap under the stated assumptions, or evaluate an adaptive attack that optimizes a malicious update subject to a reconstruction-error constraint.","section":"§3.1.2, §4.3, §5.4"},{"comment":"The accuracy comparison for requirement R2 is confounded. The text says that FL-CLEANER experiments use only 70% of the training data because 30% of clients are attackers, whereas the no-defense/no-attack benchmark uses 100% of the data. The claim that FL-CLEANER 'matches benchmark accuracy' (e.g., 95.4% vs. 95.6% on MNIST) is therefore not a clean measure of whether the defense disrupts training. Please add a no-attack control with the identical 70/30 client split and the same sampling, or explicitly quantify the accuracy loss attributable to the defense independent of the data split.","section":"§5.4"},{"comment":"All reported values are single point estimates with no seeds, standard deviations, or confidence intervals. The paper's headline claims are exact--100% recall, 0% FPR, and less than 1% benign misclassification--and the pipeline contains multiple stochastic components (client sampling, local SGD, CVAE training, trigger-set selection). A single run cannot establish these rates. Report results over at least five seeds, including the worst-case and average recall and FPR per attack and per distribution.","section":"§5.4, Figures 6-7, Tables 1-2"},{"comment":"The trigger set is drawn from the test set, and accuracy and ASR are then measured on that same test set. This creates a form of test-set leakage: the CVAE is trained on the evaluation distribution, which can inflate both the reconstruction-error separation and the reported accuracy preservation. Please use an auxiliary public set disjoint from the evaluation set, or explicitly hold out the trigger-set samples from all reported test metrics and verify that the results are unchanged.","section":"§5.1.3, Tables 1-2"},{"comment":"The backdoor evaluation covers only the Dirichlet non-IID distribution; the custom two-class distribution is used only for Byzantine attacks. Since the abstract and conclusion claim robustness in non-IID environments generally, either add backdoor experiments under the second distribution or restrict the backdoor claim to the tested distribution. This is a scope issue, but it is load-bearing for the breadth of the central claim.","section":"§5.3, Tables 1-2"}],"minor_comments":[{"comment":"Several typos remain, including 'as whown in in Fig. 3' (§4.2), 'Additionnally' (§4.2), 'mdoel' (§4.3), and 'udpates' (§2.1.2); please proofread the manuscript.","section":"§4.2, §4.3, §2.1.2"},{"comment":"The symbol α is used both for the Dirichlet concentration parameter and for the scale factor of the custom inverse-law distribution; rename one of them to avoid ambiguity.","section":"§5.1.2"},{"comment":"In step 2(b)(ii), the CVAE condition is written as 'Yt, the label'; this should be the trigger-set labels YT, not the round index.","section":"§4.1"},{"comment":"Please specify which layers are used for activation maps and how the maps are vectorized for the CVAE; the current description 'we use all the layers of the model' is under-specified for replication.","section":"§5.1.3"},{"comment":"The histogram in Figure 4 should state which attack, layer, trigger-set size, and normalization are used to produce the benign and attacker error distributions; otherwise it is only illustrative.","section":"Figure 4"},{"comment":"The sentence about FLEDGE--'It focuses on backdoor attacks but provides results against Byzantine attacks but only a fraction of their data is non-IID'--is grammatically broken and should be rewritten.","section":"§1.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an empirical extension of the authors' own FedCAM, and the 'first defense' claim should be checked against a broader comparison of recent non-IID poisoning defenses. No authorship or citation anomalies are apparent to me; the main concerns are the missing adaptive-attack evaluation, the uncontrolled 70% versus 100% data baseline, the lack of seed variance, and the test-set overlap, all of which are fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real step forward for defense in non-IID FL: the CV AE training recipe and the trust-propagation clusterer are concrete improvements over FedCAM, and the Byzantine results on the four static attacks (sign-flip, noise, same-value, scaling) are clean—100% recall and near-zero FPR across the tested cells, with a trigger set of 250 images instead of FedCVAE's 10k. Second, the paper's headline claim—'first defense' for both Byzantine and backdoor under non-IID—is broader than the evidence, and the missing adaptive-attacker experiment is the load-bearing gap.\n\nWhat is actually new: the warmup epochs and KLD annealing to avoid posterior collapse on activation-map NAMs, layer selection over all layers, and the chain-based clustering that replaces the mean-error threshold. That clustering is simple and interpretable—one parameter λ—and it handles the no-attack case better than thresholding. The comparison against FedCAM, FedCVAE, FedGuard, FLEDGE is useful and honestly parameterized for those baselines. The authors also disclose the 70%-versus-100% training-data caveat for the no-defense accuracy baseline; most papers would hide that.\n\nNow the soft spots, in order. The stress-test note is right: §3.1.2 grants the adversary full knowledge of the defense, but §4.3 rests on an empirical reconstruction-error gap with no bound, and Algorithm 1 stops at the first δ-gap. An adaptive attacker who knows GeoMed, the CVAE, and λ can try to keep its normalized activation-map error inside the benign chain. That is not a stronger attacker than the threat model already assumes—it is the stated capability. No such attack is tested, and the backdoor evaluation covers only the Dirichlet split, not the harsher custom split. Second, all Byzantine results are point estimates with no error bars or seeded repeats, and no code or data are shipped, so the 100% recall claims cannot be independently checked. Third, λ is fixed at 0.3 with no ablation, and trigger-set size is deferred to future work. These are standard fixable issues, not fatal ones.\n\nWho is this for: people building FL poisoning defenses, and anyone comparing non-IID anomaly-detection methods. It deserves a serious referee. My recommendation: send it out, and require the authors to add an adaptive adversary, ablations on λ and trigger set, multiple seeds, and to release code. The core mechanism is promising; the paper would be stronger and more honest if the claims were trimmed to match the tested attacks.","headline":"A solid non-IID FL defense with strong results on static Byzantine attacks, but the 'first defense' claim outruns the evidence, and the adaptive-attacker gap is real.","tokens_in":18967,"tokens_out":2783,"would_cite":true,"duration_ms":27799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FL-CLEANER filters both Byzantine and backdoor attackers in non-IID federated learning, achieving 100% recall on tested Byzantine attacks with under 1% benign misclassification.","keywords":["non-IID federated learning","Byzantine attacks","backdoor attacks","conditional variational autoencoder","activation maps","trust propagation clustering","poisoning defense"],"falsifier":"Craft an adaptive Byzantine update whose activation maps fall inside the benign reconstruction-error range (for example by optimizing the update to minimize CVAE reconstruction error on the same trigger set), or measure reconstruction-error gaps across a broad non-IID population and find rounds where honest clients' error differences exceed delta; if the trusted cluster then captures an attacker or drops a benign client, the central claim is refuted.","tokens_in":17942,"feed_emoji":"🛡️","tokens_out":3952,"duration_ms":41736,"temperature":0.7,"pith_summary":"Federated learning is vulnerable to malicious clients who send poisoned updates, and existing defenses often assume that client data are identically distributed, which fails in realistic non-IID settings. This paper proposes FL-CLEANER, a server-side defense that detects both Byzantine and backdoor attackers by analyzing how well each client's updated model reconstructs the activation maps of a small server-held trigger set. The central claim is that this method is the first to achieve, in non-IID environments, 100% detection of all tested Byzantine attacks in every round while misclassifying fewer than 1% of benign clients, and to suppress state-of-the-art backdoor attacks to near-zero success without hurting main-task accuracy. If true, this would give federated learning deployments a practical filtering defense that works under the heterogeneous data conditions common in real applications.","feed_headline":"Defense catches 100% of Byzantine updates in non-IID federated learning","feed_subtitle":"Activation-map reconstruction errors plus trust-propagation clustering filter malicious updates with under 1% benign misclassification.","key_machinery":"The central object is the reconstruction error of a client's activation maps under a class-conditioned CVAE that the server trains before federated training begins. Activation maps are normalized by subtracting the geometric median and applying a sigmoid, yielding normalized activation maps (NAMs); the CVAE learns the distribution of benign NAMs from a small trigger set (250 images for Byzantine attacks, 500 for backdoor attacks). A trust-propagation clustering algorithm takes the sorted client reconstruction errors, starts with the lowest-error client, and sequentially accepts clients whose error is within delta = lambda(max - min) of the last accepted client, stopping at the first gap larger than delta; this single parameter lambda is the only tuning knob, and the gap marks the frontier between benign and malicious clients.","core_discovery":"The paper claims that a conditional variational autoencoder (CVAE), trained on GeoMed-normalized activation maps of a server-side trigger set with KLD annealing and warmup epochs, produces reconstruction errors that separate benign clients from malicious ones in non-IID federated learning. A trust-propagation clustering algorithm then sorts clients by error and accepts a chain of clients until the first gap larger than a data-adaptive threshold, forming one benign cluster and flagging the rest as attackers. The paper reports that this scheme detects 100% of Byzantine attackers (sign-flipping, additive noise, same-value, and scaling attacks) in all tested scenarios on MNIST and FashionMNIST under two non-IID distributions, with zero false positives when attackers are present and under 1% benign misclassification when they are absent. It further reports that the defense reduces the success rate of distributed backdoor attacks and Neurotoxin to near zero while preserving test accuracy close to the no-attack baseline.","pith_inferences":["The trust-propagation step is essentially a gap-statistic heuristic: its correctness depends on a persistent, ordered gap in reconstruction-error space, and an adaptive adversary that crafts updates whose activation maps reconstruct like benign ones could plausibly shrink or eliminate that gap; the paper does not evaluate such an adversary.","The method's reliance on a server-held trigger set means that if the trigger set's class distribution drifts from the client data distribution over time, benign reconstruction errors could widen and inflate the false-positive rate; this is an implicit risk the paper does not test.","A natural extension would be to test FL-CLEANER on label-flipping and model-replacement attacks where the attacker can optimize updates to be close to benign in activation space, or to apply the scoring mechanism to regression and non-image tasks by defining a suitable activation-map representation.","One testable improvement would be to replace the fixed delta = lambda(max-min) rule with a statistical gap test that adapts to the variance of benign errors, which could make the defense robust to broader non-IID distributions than the two considered."],"forward_implications":["If the central claim holds, a federated server can filter all tested Byzantine attackers in every round under non-IID data, satisfying the requirement that no malicious update reaches aggregation.","The defense preserves benign client inclusion, avoiding the roughly 50% false-positive rate that mean-threshold CVAE defenses incur in IID and non-IID settings alike.","Backdoor attacks such as distributed backdoors and Neurotoxin can be neutralized to near-zero attacker success rate while keeping main-task accuracy close to the no-defense baseline.","The server needs only a small trigger set (250–500 images), making the defense feasible in settings where the server lacks a large auxiliary dataset.","The single dynamic parameter delta is interpretable, allowing deployment without per-round threshold tuning."],"supporting_citations":[{"why":"The prior activation-map CVAE defense that FL-CLEANER extends; the paper shows its failure in non-IID settings and builds its training strategy on it.","marker":"[16]"},{"why":"The CVAE anomaly-detection baseline for Byzantine attacks; FL-CLEANER compares against it and improves on its trigger-set and thresholding limitations.","marker":"[12]"},{"why":"The FedAvg aggregation algorithm that FL-CLEANER wraps; the global model is updated from the filtered clients' updates using FedAvg.","marker":"[8]"},{"why":"FedGuard, a client-decoder-based defense baseline; FL-CLEANER compares its accuracy, recall, and false-positive rates against it.","marker":"[13]"},{"why":"FLEDGE, a clustering and kernel-density baseline that handles backdoor and partial non-IID scenarios; used as a comparison defense.","marker":"[14]"},{"why":"FLAME, a backdoor defense that filters attackers; compared against FL-CLEANER on backdoor attacker success rate and accuracy.","marker":"[28]"},{"why":"FLDetector, a model-poisoning defense that detects malicious clients; compared as a backdoor defense baseline.","marker":"[29]"},{"why":"The distributed backdoor attack (DBA) whose success rate FL-CLEANER suppresses to near zero in experiments.","marker":"[64]"},{"why":"The Neurotoxin backdoor attack, which persists through aggregation; FL-CLEANER reduces its attack success rate to near zero.","marker":"[65]"},{"why":"The geometric median, used to normalize activation maps before feeding them into the CVAE, providing a robust centroid that ignores outlier updates.","marker":"[10]"}],"fun_headline_variants":["FL-CLEANER: 100% Byzantine detection in non-IID FL","Activation map errors catch Byzantine and backdoor attacks","Non-IID federated learning: new defense blocks all Byzantine attacks","CVAE-based defense filters malicious updates in non-IID FL","Trust propagation clustering defeats Byzantine and backdoor in FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defense assumes that in the reconstruction-error space there is always a clear, ordered gap between benign and malicious clients' scores, so that the first jump larger than the dynamic step size cleanly separates the two groups; if attackers can close that gap, or if benign clients' errors vary more than the chain step, the filter will admit an attacker or reject a benign client.","fun_headline_variants_meta":{"raw":{"variants":["FL-CLEANER: 100% Byzantine detection in non-IID FL","Activation map errors catch Byzantine and backdoor attacks","Non-IID federated learning: new defense blocks all Byzantine attacks","CVAE-based defense filters malicious updates in non-IID FL","Trust propagation clustering defeats Byzantine and backdoor in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3815,"prompt_tokens":997,"completion_tokens":2818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":2730}},"tokens_in":613,"tokens_out":2818,"duration_ms":20704,"temperature":1.0,"reasoning_tokens":2730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:29:28.994662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Craft an adaptive Byzantine update whose activation maps fall inside the benign reconstruction-error range (for example by optimizing the update to minimize CVAE reconstruction error on the same trigger set), or measure reconstruction-error gaps across a broad non-IID population and find rounds where honest clients' error differences exceed delta; if the trusted cluster then captures an attacker or drops a benign client, the central claim is refuted.","supporting_citations":[{"cited_title":"FedCAM - Identifying Malicious Models in Federated Learning Environments Conditionally to Their Activation Maps","cited_arxiv_id":null,"evidence_quote":"The prior activation-map CVAE defense that FL-CLEANER extends; the paper shows its failure in non-IID settings and builds its training strategy on it."},{"cited_title":"Detecting malicious model updates from federated learning on conditional variational autoencoder","cited_arxiv_id":null,"evidence_quote":"The CVAE anomaly-detection baseline for Byzantine attacks; FL-CLEANER compares against it and improves on its trigger-set and thresholding limitations."},{"cited_title":"Fedguard: Selective parameter aggregation for poisoning attack mitigation in federated learning","cited_arxiv_id":null,"evidence_quote":"FedGuard, a client-decoder-based defense baseline; FL-CLEANER compares its accuracy, recall, and false-positive rates against it."},{"cited_title":"FLEDGE: Ledger-based Federated Learning Resilient to Inference and Backdoor Attacks","cited_arxiv_id":"2310.02113","evidence_quote":"FLEDGE, a clustering and kernel-density baseline that handles backdoor and partial non-IID scenarios; used as a comparison defense."},{"cited_title":"FLAME: Taming backdoors in federated learning","cited_arxiv_id":null,"evidence_quote":"FLAME, a backdoor defense that filters attackers; compared against FL-CLEANER on backdoor attacker success rate and accuracy."},{"cited_title":"Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients, 2022","cited_arxiv_id":null,"evidence_quote":"FLDetector, a model-poisoning defense that detects malicious clients; compared as a backdoor defense baseline."},{"cited_title":"Distributed backdoor attacks on federated graph learning and certified defenses, 2024","cited_arxiv_id":null,"evidence_quote":"The distributed backdoor attack (DBA) whose success rate FL-CLEANER suppresses to near zero in experiments."},{"cited_title":"Mahoney, Joseph E","cited_arxiv_id":null,"evidence_quote":"The Neurotoxin backdoor attack, which persists through aggregation; FL-CLEANER reduces its attack success rate to near zero."},{"cited_title":"The hidden vulnerability of distributed learning in byzantium","cited_arxiv_id":null,"evidence_quote":"The geometric median, used to normalize activation maps before feeding them into the CVAE, providing a robust centroid that ignores outlier updates."}],"review_version":1}