{"id":"6a9ee473-49f8-422f-84c2-1975ccd73b40","arxiv_id":"2505.09110","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"SafeFL generates a synthetic dataset from the global model trajectory and uses loss-based median or clustering rules to detect malicious federated learning clients.","lead":"This paper introduces SafeFL, a defense that builds a synthetic dataset from the history of global models in federated learning and then flags clients whose local models have unusually high loss on that synthetic data. A generalist reader might care because it promises to detect poisoning attacks without requiring the server to hold a clean copy of clients' data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SafeFL-ML's Table 1 metrics are arithmetically impossible under the stated algorithm: Eqs. (3)-(4) force at least 50 of 100 clients to be flagged every round, bounding DACC <= 0.80 and FPR >= 0.29 at 30% malicious, yet the table reports DACC up to 1.00 with FPR 0.00.","rationale":"The paper's idea, using a trajectory-derived synthetic dataset to score client models, is interesting, and the evaluation is broad (five datasets, eleven attacks, ablations, fairness checks). Those are real assets. But the arithmetic check above is decisive: it uses only the paper's own equations and table, so it does not depend on code availability or on out-of-distribution assumptions. The reader's weakest_assumption concerned contamination of the first-epsilon-rounds trajectory when malicious clients approach 50%; that is acknowledged in Section 6 and is a threat-model boundary rather than a formal contradiction. The internal inconsistency of SafeFL-ML's reported metrics is more load-bearing: if Table 1 cannot be reproduced from Eqs. (3)-(4), the empirical support for the central claim fails regardless of the trajectory assumption. The reader flagged this same point in their rationale, but chose the trajectory assumption as the headline weakness; I would elevate the Table 1 inconsistency to primary. A genuine fix would be to release code or to specify the actual detection threshold and re-report the metrics; if the numbers then change materially, the DACC near 1.00 claim would need revision. As written, REJECT is the right verdict.","tokens_in":50449,"tokens_out":13917,"duration_ms":133149,"concrete_test":"Apply the median-split rule of Eqs. (3)-(4) to per-client losses in the paper's own setup (n=100, 30% malicious, CIFAR-10, round 750, loss plots in Figures 3-7): flag every client with loss at or above the median and compute DACC/FPR/FNR. The feasible region for SafeFL-ML is DACC <= 0.80 and FPR >= 0.286 under attack, and DACC <= 0.50 for the no-attack rows. If the recomputed metrics do not match Table 1 (e.g., CIFAR-10 Trim: DACC 0.90, FPR 0.03, FNR 0.13), then the tables were not produced by the described algorithm; the authors should then release runnable code or state the actual threshold rule used to generate Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing concern is that the headline detection numbers cannot be produced by the algorithm as specified, so the central empirical claim is unsupported by the paper's own formalism. SafeFL-ML (Section 4.4, Eqs. (3)-(4)) classifies as malicious every client whose loss on the synthetic dataset is at or above the median of all n losses, forcing at least 50 of the n=100 clients to be flagged in every round, irrespective of the loss distribution. At the default 30% malicious ratio, even perfect separation (all 30 malicious losses above the median) yields 20 benign false positives, bounding DACC at (50+30)/100 = 0.80 and FPR at 20/70 about 0.29; with no attack, each round flags half of all benign clients, giving DACC = 0.50. Table 1 reports SafeFL-ML no-attack DACC of 0.92-0.99 and attack-row values up to 1.00 with FPR as low as 0.00 (e.g., CIFAR-10 Trim: DACC 0.90, FPR 0.03, FNR 0.13). The reported FPR/FNR pair also self-contradicts: FPR 0.03 and FNR 0.13 imply only about 28 flagged clients, not at least 50. The same discrepancy appears in Tables 13 and 15 and in the SafeFL-ML curves of Figures 1-2. Either a different, unspecified threshold rule generated the tables, or the metrics were computed differently than stated; in either case the near-perfect DACC claim is not reproducible from the described method. Compounding this, Appendix A.2's 'adaptive attack' is implemented following [60], an attack aimed at median/trimmed-mean aggregation rather than at SafeFL's loss-based screening, so the claimed worst-case adversary is not actually tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SafeFL, a server-side detection method for malicious clients in federated learning. The server records the first epsilon global models (aggregated from the largest cluster of local models), synthesizes a dataset D_syn from this trajectory via the SynGen algorithm, and then in each subsequent round evaluates every client's local model on D_syn. SafeFL-ML aggregates only clients whose loss is at or below the median loss; SafeFL-CL instead keeps clients in the largest loss cluster. Experiments compare SafeFL with seven detection-based and three prevention-based baselines on five datasets under six attack types, plus ablations and additional attacks in the appendix, and report near-perfect detection accuracy for SafeFL.","tokens_in":50911,"tokens_out":8137,"duration_ms":79745,"significance":"The core idea of replacing a clean server-side validation set with a synthetic dataset distilled from the global-model trajectory is original and potentially useful, since one of the main criticisms of existing defenses such as FLTrust is the unrealistic assumption that the server holds a representative clean dataset. The paper also provides a broad evaluation grid covering five datasets, several attacks, and numerous ablations. However, the central empirical claim is invalidated by an internal arithmetic inconsistency in SafeFL-ML's decision rule, and the paper's 'adaptive attack' evaluation does not exercise the full-knowledge threat model stated in Section 3. The positive contributions are the synthetic-dataset formulation and the breadth of the experimental design, but the manuscript in its current form does not support its headline detection results.","major_comments":[{"comment":"The reported SafeFL-ML metrics are arithmetically inconsistent with the stated decision rule. Eq. (4) sets r_i^t = 0 for every client whose loss is strictly greater than the median l_Med^t, and the text and Algorithm 2 confirm that such clients are treated as malicious. For n = 100 clients, at most 50 clients can have loss above the median, so at most 50 clients can be flagged in any round. In the default setting with 30% malicious and 70% benign clients, even if the 30 malicious clients are exactly the 30 highest losses, 20 benign clients must also be flagged; this gives DACC = (30 + 50) / 100 = 0.80, FPR = 20 / 70 ≈ 0.286, and FNR = 0 as the best possible outcome. With no attack, the rule flags half the benign clients, giving DACC = 0.50. Table 1 reports SafeFL-ML no-attack DACC values of 0.92–0.99 and attack-row DACC values up to 1.00 with FPR as low as 0.00; for example, the CIFAR-10 Trim attack row reports DACC 0.90, FPR 0.03, and FNR 0.13. These numbers are impossible under Eq. (4): the reported FPR/FNR pair implies only about 28 flagged clients (26.1 true positives plus 2.1 false positives), whereas the median rule forces about 50 flagged clients. The same discrepancy appears in Tables 13 and 15 and in the SafeFL-ML curves of Figures 1–2. Either a different, unspecified threshold rule produced the tables, or the metrics were computed differently from the stated procedure; as written, the method cannot produce the headline detection results.","section":"§4.4, Eq. (4) and Table 1"},{"comment":"The Adaptive attack, which the contributions describe as a 'strong adaptive attack,' is not adaptive to SafeFL. Section 3 states that the full-knowledge attacker knows the aggregation rule and designs an attack to deceive the FL process, and Section 5.1.2 says the attacker 'designs an adaptive attack to disrupt and deceive the FL process' with SafeFL as the target. However, Appendix A.2 says only that the attack is implemented 'following the methodology outlined in [60]'; reference [60] (Shejwalkar and Houmansadr) constructs attacks against robust aggregation rules such as median and trimmed mean, not against a defense that evaluates local models on a synthetic dataset. The Adaptive-attack rows in Tables 1–3, 10, and 12 therefore do not test the full-knowledge threat model stated in Section 3, and the claim that SafeFL resists a worst-case adversary that knows SafeFL's detection mechanism is unsupported by the reported experiments.","section":"§5.1.2 and Appendix A.2"},{"comment":"The global model trajectory used for synthetic data generation is collected by aggregating only the largest cluster of local models for the first epsilon rounds, but the paper provides no correctness argument and no dedicated experiment establishing that this cluster is benign. Because the threat model in Section 3 lets malicious clients transmit arbitrary local models, an adversary that knows the clustering mechanism could attempt to place malicious updates inside the largest cluster during the trajectory-collection phase; the default 30% malicious setting and the tested attacks do not rule this out. Since D_syn is generated from this trajectory, poisoning the trajectory would poison the synthetic dataset and invalidate the subsequent loss-based separation. The paper should either prove a separation condition under which the largest cluster is guaranteed to be benign, or empirically evaluate trajectory poisoning by an adversary that is actually adaptive to SafeFL.","section":"§4.2, Eq. (1), and Algorithm 2, lines 14–17"}],"minor_comments":[{"comment":"The bullet numbering in Appendix A.2 uses the label 'e)' twice, first for the Scaling+DBA attack and then for the Adaptive attack; the numbering should be corrected.","section":"Appendix A.2"},{"comment":"The rows for Scaling+DBA and Trim+DBA in Table 10(b) are identical for every defense and every client count, which appears to be a copy-paste error and should be verified.","section":"Table 10(b)"},{"comment":"Figures 10(c)–(f) in the appendix appear to be empty, with only captions and no plotted data or axes; these subfigures should be filled in or removed.","section":"Figures 10(c)–(f)"},{"comment":"The first sentence of Section 4.2 says 'the server possesses its own distinct dataset' before explaining that the server will generate a synthetic dataset; this wording is confusing and should be revised to avoid implying the server has a clean validation set.","section":"§4.2"}],"recommendation":"reject","confidential_remarks":"The breadth of the evaluation is a strength, but the SafeFL-ML metric inconsistency is a load-bearing error: the central empirical claim cannot be reproduced from the paper's own formal description. I would not return this for minor revision. If the authors can identify the actual decision rule that generated the reported tables and rerun the experiments with code, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading this one. First, the central evaluation is internally inconsistent: SafeFL-ML's reported DACC and FPR in Table 1 cannot be produced by the algorithm the paper defines. Equations (3)-(4) flag every client whose loss is at or above the median, so with 100 clients and 30% malicious, at least 50 clients are flagged each round, capping DACC at 0.80 and FPR at about 0.29 even with perfect separation. The paper's own text admits this ('identifies half of the clients as suspicious in each round'), yet Table 1 reports no-attack DACC up to 0.99 and attack-row FPR as low as 0.00. Those numbers had to come from a different threshold rule or a different metric than the one stated. Second, the 'adaptive attack' is not adaptive to SafeFL: Appendix A.2 simply says it follows [60], an attack aimed at median/trimmed-mean aggregation, so the worst-case evaluation does not actually target the loss-based detector.\n\nThat's the bad news. The good news: the core idea is genuinely new. Using trajectory-based dataset condensation to synthesize a server-side reference dataset, and then screening client models on that dataset, addresses a real limitation of detection defenses that need a clean validation set matching the client distribution. That angle is worth exploring, and the paper's breadth—five datasets, eleven attacks, ten baselines, plus ablations on client ratio, non-IID degree, trajectory length, and storage—is more than most papers in this area manage. The writing is clear and the related work is well surveyed.\n\nThe soft spots beyond the arithmetic: no code, no error bars, and the assumption that the first epsilon rounds' largest cluster is benign is fragile when malicious clients approach 40%, which is exactly the upper end they test. The clustering variant SafeFL-CL is more plausible than SafeFL-ML, but it inherits the same unverified synthetic-data pipeline.\n\nBottom line: I would not accept this as-is, and I would trust none of the headline numbers until the threshold rule is clarified and code is released. But the idea is novel enough that I'd send it to a serious referee rather than desk reject: a reviewer can force the authors to reconcile their equations with their tables, and the trajectory-condensation approach deserves a fair test. If the numbers hold up after correction, it would be a useful contribution.","headline":"The detection results in Table 1 are arithmetically impossible under SafeFL-ML's own median-loss rule, so the paper's empirical claims cannot be trusted as written—but the trajectory-condensation idea is new enough to warrant refereeing.","tokens_in":51412,"tokens_out":4087,"would_cite":false,"duration_ms":38618,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SafeFL detects malicious federated-learning clients by scoring their local models on a synthetic dataset generated from the global model trajectory.","keywords":["federated learning","poisoning attacks","malicious client detection","synthetic dataset generation","model trajectory","backdoor attacks","Byzantine robustness","server-side defense"],"falsifier":"Run SafeFL with 50 percent or more of clients malicious while keeping the same clustering and loss rules, so the largest-cluster assumption in Eq. (1) is violated; detection accuracy should drop sharply if the mechanism is what the paper claims. A sharper test is to craft malicious local models that carry a backdoor but are explicitly optimized to have loss close to the benign median on $D_{\\mathrm{syn}}$, then check whether both SafeFL-ML and SafeFL-CL still flag them.","tokens_in":50262,"feed_emoji":"🛡️","tokens_out":4722,"duration_ms":46013,"temperature":0.7,"pith_summary":"The paper proposes SafeFL, a detection-based defense in federated learning that lets the server identify malicious clients without holding a clean sample of the clients' data. The server collects the first few global models, condenses them into a synthetic dataset, and then scores every submitted local model on that dataset: malicious models are claimed to incur noticeably different, usually higher, loss than benign ones. Two variants are given, one thresholding at the median loss and one clustering the losses and keeping the largest cluster. On five datasets including CIFAR-10, FEMNIST, STL-10, and Tiny-ImageNet, and against eleven attacks including an adaptive full-knowledge attack, SafeFL-CL is reported to reach detection accuracy near 1.00 while preserving benign-client accuracy. If correct, this removes a key unrealistic assumption of prior server-based detectors: the server no longer needs a validation set drawn from the clients' distribution.","feed_headline":"Synthetic trajectory data exposes malicious FL clients","feed_subtitle":"The server builds a fake dataset from model history and flags clients whose loss stands out, keeping accuracy high.","key_machinery":"The central object is a synthetic dataset produced by trajectory matching. The server treats pairs $(w_\\alpha, w_{\\alpha+\\Delta})$ from the collected global-model trajectory as input-output targets and trains synthetic data so that starting from $w_\\alpha$ and training for $\\Delta$ steps on the synthetic set reproduces $w_{\\alpha+\\Delta}$. The resulting synthetic set $D_{\\mathrm{syn}}$ is then the evaluation instrument: the loss of each local model on $D_{\\mathrm{syn}}$ is what separates malicious from benign behavior. The machinery also includes the largest-cluster aggregation used during trajectory collection, plus the two detection rules, median-loss weighting and loss clustering.","core_discovery":"SafeFL's central claim is that the trajectory of global models alone, with no access to client data, can be turned into a usable test set for client screening. During the first epsilon rounds the server clusters received local models, aggregates only the largest cluster to form each global model, and then uses those global models as targets in a dataset-condensation optimization that produces synthetic images and labels. Once the synthetic set exists, each client's local model is evaluated on it; benign models are those whose loss falls below the median (SafeFL-ML) or in the largest loss cluster (SafeFL-CL). The paper reports that malicious local models show loss clearly separated from benign ones across all tested attacks, and that SafeFL-CL achieves near-perfect detection while keeping final global-model accuracy comparable to no-attack training.","pith_inferences":["If the loss-separation result generalizes, the same synthetic-trajectory trick could screen for other deviant client behavior, such as buggy or stale local models, using only the model updates themselves.","A direct stress test of the mechanism would be an attack that explicitly optimizes malicious local models to keep their loss inside the benign cluster on $D_{\\mathrm{syn}}$ while still implanting a backdoor; the paper's adaptive attack is full-knowledge, but this specific objective is not reported.","Because the synthetic set is built from the largest cluster in the first $\\epsilon$ rounds, the method has a natural threshold near 50 percent malicious clients; adding a small trusted anchor set during trajectory collection could push that threshold higher and is a testable extension.","The paper's privacy discussion suggests a concrete follow-up: measuring how much the synthetic dataset leaks about client data, and whether differentially private global models still separate malicious from benign losses."],"forward_implications":["Detection no longer requires the server to hold a clean validation dataset drawn from the clients' distribution, so it applies in highly heterogeneous or data-free settings.","Because clients are re-assessed every round and only excluded from that round's aggregation, benign clients that occasionally exhibit high loss are not permanently removed from the system.","SafeFL-CL's loss-clustering rule yields near-perfect detection accuracy and low false-positive rates across the tested attacks, keeping final global-model accuracy close to the no-attack FedAvg baseline.","The detection filter remains effective when different aggregation rules such as Median, Trimmed Mean, or Krum are applied after filtering, making it compatible with prevention-based defenses.","Longer global-model trajectories and larger synthetic datasets improve detection, with diminishing returns after roughly 25 trajectory rounds and 100 synthetic samples."],"supporting_citations":[{"why":"Supplies the trajectory-matching objective, training synthetic data so the model moves from $w_\\alpha$ to $w_{\\alpha+\\Delta}$, which SafeFL adapts to generate $D_{\\mathrm{syn}}$.","marker":"[16]"},{"why":"Shows how global model dynamics can be used to synthesize data in federated learning, and motivates using the same network $f$ for generation.","marker":"[57]"},{"why":"Defines the FLTrust baseline that assumes the server holds a clean validation set, the unrealistic assumption SafeFL is designed to remove.","marker":"[14]"},{"why":"Provides the FLDetector prediction-based detection baseline that SafeFL is compared against on detection accuracy.","marker":"[73]"},{"why":"Provides the FLAME clustering-based defense baseline, a main comparison target under backdoor and poisoning attacks.","marker":"[54]"},{"why":"Defines the Median and Trimmed-mean aggregation rules used as prevention baselines and as aggregation options inside SafeFL.","marker":"[70]"},{"why":"Provides the Trim attack and the general local-model-poisoning framework used to craft malicious client updates in the evaluation.","marker":"[24]"},{"why":"Defines the adaptive attack methodology used as the worst-case full-knowledge threat model for testing SafeFL.","marker":"[60]"}],"fun_headline_variants":["SafeFL: synthetic history catches malicious FL clients","Trajectory-built synthetic set detects bad FL clients","No raw data needed: SafeFL finds FL attackers from model history","SafeFL outperforms earlier detection with synthetic screening"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"During the first $\\epsilon$ rounds, clustering local models into the largest cluster isolates the benign clients, so the global-model trajectory used to make the synthetic dataset is uncontaminated; if malicious clients form the largest cluster or approach half the participants, the synthetic dataset is poisoned and detection fails.","fun_headline_variants_meta":{"raw":{"variants":["SafeFL: synthetic history catches malicious FL clients","Trajectory-built synthetic set detects bad FL clients","No raw data needed: SafeFL finds FL attackers from model history","SafeFL outperforms earlier detection with synthetic screening"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000463,"raw_usage":{"total_tokens":2273,"prompt_tokens":863,"completion_tokens":1410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":1346}},"tokens_in":479,"tokens_out":1410,"duration_ms":14077,"temperature":1.0,"reasoning_tokens":1346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:40:05.427775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SafeFL with 50 percent or more of clients malicious while keeping the same clustering and loss rules, so the largest-cluster assumption in Eq. (1) is violated; detection accuracy should drop sharply if the mechanism is what the paper claims. A sharper test is to craft malicious local models that carry a backdoor but are explicitly optimized to have loss close to the benign median on $D_{\\mathrm{syn}}$, then check whether both SafeFL-ML and SafeFL-CL still flag them.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the trajectory-matching objective, training synthetic data so the model moves from $w_\\alpha$ to $w_{\\alpha+\\Delta}$, which SafeFL adapts to generate $D_{\\mathrm{syn}}$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how global model dynamics can be used to synthesize data in federated learning, and motivates using the same network $f$ for generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the FLTrust baseline that assumes the server holds a clean validation set, the unrealistic assumption SafeFL is designed to remove."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FLDetector prediction-based detection baseline that SafeFL is compared against on detection accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Median and Trimmed-mean aggregation rules used as prevention baselines and as aggregation options inside SafeFL."}],"review_version":1}