{"id":"b3d16721-3d2a-4dc8-977e-6eab6ff6c35e","arxiv_id":"2508.10042","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FIDELIS uses a client-agreed isolation forest judge model, trained on gradient statistics from a public dataset, to detect and filter poisoned federated learning updates.","lead":"This paper introduces FIDELIS, a blockchain-based system where clients jointly agree on an isolation forest judge model that flags poisoned updates before federated aggregation. If it works, it would let federated learning drop the trusted central server while still blocking label-flipping attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge-model consensus scores only clean-data pass/fail, so permissive judges pass and the protocol cannot guarantee a detector is selected.","rationale":"The reader's weakest assumption is that gradient statistics from the public dataset represent benign private-data updates; that is a valid external-validity concern. However, the more load-bearing problem is internal: the judge-selection mechanism described in §IV-C scores candidates only on whether they avoid false positives on a clean test set. A constant-1 judge, or any isolation forest with no splits, passes this test and therefore can be selected. Since malicious clients are assumed non-majority but can still submit candidate judges, the protocol does not guarantee that the winning judge has any anomaly-detection ability. This directly threatens the central claim of robust poisoning defense. The lack of any blockchain implementation means the experimental evaluation never exercises this selection step, so the gap is not observed. This concern is distinct from, but related to, the reader's data-representativeness assumption; both point to the judge model's validity, but the selection flaw would persist even if the public data were perfectly representative. The verdict remains CONDITIONAL: the framework should be revised to include a detection-capability criterion (e.g., a labeled validation set of simulated attacks) and re-evaluated with the actual consensus process.","tokens_in":9239,"tokens_out":9279,"duration_ms":110664,"concrete_test":"Implement the §IV-C consensus in a 20-client simulation (e.g., 4 malicious). Benign clients submit judges trained per Algorithm 1; attackers submit a constant-1 judge. Compute the pass/fail score on the clean public test set. If the constant-1 judge receives the same score as honest judges (or wins a tie), the selection protocol fails to exclude degenerate detectors. Then run the full FIDELIS pipeline with that constant-1 judge and report global accuracy; if accuracy remains high, the defense's reported effectiveness is not attributable to the judge model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-C specifies that each client tests each proposed judge on the clean public test set and passes it with 1 if no test sample is flagged anomalous, else 0. The winning judge is the one with the highest total of pass votes. This score is a false-positive-only criterion: any judge that never marks clean data anomalous receives the same full score as a well-trained detector, and a degenerate judge (e.g., one that always outputs 1) also passes. The protocol therefore does not 'ensure that the most accurate model is chosen' as claimed; it does not even compare true-positive rates on poisoned updates. A malicious minority can propose such a permissive judge and, with no tie-break specified, it may be selected. Since the selected judge is then used to gate all model updates, the central guarantee that poisoned updates are excluded is unsubstantiated. The experiments do not instantiate the consensus process (no blockchain or homomorphic voting), so this selection behavior is never tested. This is a logical gap independent of the public-data representativeness question: even with perfectly representative clean data, the scoring function fails to discriminate detectors from non-detectors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FIDELIS, a blockchain-based federated learning framework that removes the trusted global server and uses an isolation-forest 'judge model' to detect poisoned model updates before aggregation. Each client trains a local judge on gradient statistics derived from a public dataset plus a local model on private data; clients then vote via homomorphic encryption to select a single judge, and a majority vote decides which model updates are accepted. Experiments on Oxford-IIIT Pets (public) and Cat-vs-Dog (client data) report judge true-positive/F1 rates, global model accuracy under label-flipping attacks, and linear runtime scaling for judge creation.","tokens_in":9534,"tokens_out":3336,"duration_ms":43098,"significance":"If the claims hold, FIDELIS would be a useful step toward decentralized poisoning defense for federated learning, combining a concrete anomaly-detection method with a blockchain consensus layer. The paper is explicit about the adversary model and provides an algorithmic description of judge training, which helps reproducibility. The scalability experiment is a strength. However, the central robustness claim is not yet supported: the judge-selection consensus is logically flawed, the attack experiments lack baselines and comparisons to existing defenses, and the representativeness of the public dataset is untested. These issues are fixable but are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The judge-selection consensus scores only false-positive behavior. Each judge is tested on the clean public test set P_test and receives a pass (1) if no sample is flagged anomalous. A degenerate judge that always outputs benign receives the same full score as a well-trained detector, and permissive judges are indistinguishable from accurate ones. The claim that \"this process ensures that the most accurate model is chosen\" is therefore unsupported; true-positive rates on poisoned updates are never compared, and no tie-breaking rule is specified. A malicious minority could propose such permissive judges, and the selected judge would then fail to exclude poisoned updates. Since the experiments do not instantiate the consensus process, this selection behavior is never tested.","section":"§IV-C"},{"comment":"The global-accuracy experiment has no baseline without defense and no comparison to existing robust aggregation methods (e.g., Krum, Trimmed Mean, Zeno, Raza et al.). The reported >98% accuracy with up to 35% malicious clients may simply reflect the weakness of the attack: each malicious client flips labels on only 35% of its own dataset, so the effective fraction of poisoned data is at most 15.75% even with 45% malicious clients. Without a FedAvg-no-defense baseline, the reader cannot attribute the result to FIDELIS. The paper should report the accuracy of vanilla FedAvg under the same attack and compare against standard byzantine-robust baselines.","section":"§V-C, Fig. 5"},{"comment":"The judge model is trained exclusively on gradient statistics from the Oxford-IIIT Pets public dataset, while client models are trained on Cat-vs-Dog private data. Algorithm 1 instructs clients to \"simulate this process of collecting data points for other clients\" from the public dataset, implicitly assuming that the public data's gradient distribution matches the benign clients' private-data distribution. This assumption is load-bearing: if the public data is not representative, the judge will either reject benign updates or accept poisoned ones. The paper provides no experiments with different public datasets, different data distributions, or non-IID client data, and does not test how the judge's accuracy degrades under distribution shift.","section":"§V-B, Algorithm 1"},{"comment":"The consensus mechanism for accepting model updates is described but never instantiated in the experiments. There is no blockchain, homomorphic encryption, or voting simulation in the evaluation; the experimental pipeline appears to be a centralized judge-based filter followed by FedAvg. Consequently, the paper's scalability results (Fig. 6) measure judge creation, not the consensus/verification process, and the claimed decentralization is not empirically validated. At minimum, the authors should simulate the voting and tallying steps and report the communication/computation overhead, or clearly state which components were not implemented.","section":"§IV-D, §V"}],"minor_comments":[{"comment":"Notation is inconsistent: Table II defines private data as D_i, but the text says clients train on \"their own private data Pi.\" Also, Algorithm 1's \"1-D list Ti\" is unclear since each simulation produces a 45-dimensional feature vector.","section":"§IV-B"},{"comment":"Typo: \"Isolation Forrest\" should be \"Isolation Forest.\"","section":"Table I"},{"comment":"\"majority of runtime is compromised of CNN training\" should be \"composed of.\"","section":"§V-D"},{"comment":"The y-axis is labeled \"Accuracy (%)\" but the curves report TP and F1; this should be clarified to avoid conflating true-positive rate with classification accuracy.","section":"Fig. 4"},{"comment":"The number of simulations n and the isolation forest hyperparameters (contamination, n_estimators, etc.) are not specified. These are needed for reproducibility, especially since the five-stat summary totals 45 features per simulation.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope but needs substantially stronger empirical support. The judge-selection flaw in §IV-C is the most serious concern because it affects the protocol's guarantee even under the authors' assumptions. The authors should either add a true-positive component to judge scoring and test it, or weaken the claimed guarantee. The missing baselines and lack of consensus-protocol implementation are also critical for a systems paper. With these additions, the contribution could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Open with punchline: the combination is new, but the judge-selection tally is a false-positive-only test, so the protocol can pick a detector that never flags anything. The empirical support is also missing a baseline.\n\nWhat's good: the idea of having every client train an isolation forest on gradient statistics and then using blockchain votes to pick one judge is genuinely not in the cited literature. The adversarial model is explicit, the writing is clear, and the runtime scalability result is a useful data point. The related-work table is honest about what prior systems do.\n\nWhere it's soft: the scoring in Section IV-C is the load-bearing weak point. A judge gets a pass vote if it doesn't flag any clean test sample; there's no check for whether it catches poisoned updates. A degenerate judge that always outputs 1 gets full marks. So the claim that the most accurate model is chosen doesn't follow. This is not a minor implementation detail; it's a gap in the core mechanism.\n\nThe experiments don't close that gap. No baseline without defense, no comparison to Krum, trimmed mean, Zeno, or Raza et al., and no instantiation of the blockchain or homomorphic voting. Without those, Figure 5's 98% accuracy is hard to interpret. The public-dataset representativeness assumption is also untested; the judge is trained on Oxford-IIIT Pets and applied to client updates from a cat/dog dataset, which is a reasonable start but not a demonstration of generality.\n\nThese are all fixable. A baseline and a true-positive-aware selection score would make the central claim testable. As written, I'd treat it as a promising framework sketch, not a demonstrated defense. That said, the flaw is instructive and the paper deserves a proper referee rather than a desk reject.","headline":"Novel combination of blockchain-based judge selection and isolation forest detection, but the judge-selection score is a false-positive-only test that can select a degenerate detector, and the experiments have no baselines.","tokens_in":9980,"tokens_out":2878,"would_cite":false,"duration_ms":31215,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FIDELIS claims that a blockchain-consensus isolation-forest judge model, trained on gradient statistics from a public dataset, detects poisoned updates and holds global accuracy above 98% with up to 35% malicious clients.","keywords":["federated learning","blockchain","data poisoning","label-flipping","isolation forest","judge model","consensus","anomaly detection"],"falsifier":"Run FIDELIS with a public dataset unrelated to the private task (for example, a public pet-image dataset while clients train on CIFAR-10) and measure the judge model's true-positive rate; if it drops well below the reported 95–100%, the representativeness assumption fails. Alternatively, launch a backdoor (non-label-flipping) poisoning attack under the same protocol: the paper only evaluates label-flipping, so a successful backdoor would show the robustness claim is attack-specific.","tokens_in":9163,"feed_emoji":"🛡️","tokens_out":5675,"duration_ms":61984,"temperature":0.7,"pith_summary":"FIDELIS proposes a decentralized defense against label-flipping poisoning in federated learning. Instead of trusting a central server to detect malicious model updates, each client trains an isolation-forest judge model on gradient statistics extracted from a public dataset, and the blockchain lets clients vote—via encrypted homomorphic tallies—on which judge model to use. All model updates are then scored by the winning judge, and only updates that pass are aggregated with FedAvg. The paper claims this keeps global model accuracy above 98% even when 35% of clients are malicious, and that judge-model creation scales linearly with the number of clients. If correct, it removes the single point of trust that current server-based detection methods rely on.","feed_headline":"Consensus judge model keeps federated learning above 98% accuracy","feed_subtitle":"Each client trains an isolation-forest detector and blockchain votes pick one, cutting the trusted server.","key_machinery":"The judge model: an isolation forest trained on 45-dimensional gradient-statistic summaries (nine batch statistics times a five-stat summary) extracted by training the initial model on a public dataset. Each client builds one; the blockchain's consensus protocol—ElGamal-encrypted votes homomorphically tallied—selects a single judge model used to flag anomalous model updates before FedAvg aggregation.","core_discovery":"The central claim is that the movement of gradients during benign training, summarized by nine per-batch statistics (mean, standard deviation, minimum, maximum, range, skew, kurtosis, L1, L2) further reduced to five-stat summaries, gives an isolation forest enough signal to distinguish poisoned from benign model updates. Each client produces such a judge model, and consensus is reached by testing every candidate judge on a clean public test set and homomorphically tallying encrypted votes, so no single client controls the detector. The winning judge model then labels each submitted update; any update marked anomalous is excluded from federated averaging. Experiments with ResNet-18 on a dog-v","pith_inferences":["The public-dataset representativeness assumption is untested; if clients' private data is non-IID relative to the public set, judge accuracy likely degrades, so a natural extension is to evaluate under distribution shift.","An adaptive attacker who knows the public dataset could craft poisoned updates whose gradient statistics mimic benign ones, potentially bypassing the isolation forest; the paper does not address this.","The consensus protocol assumes malicious clients lack a majority; if that assumption weakens, all judge selection and acceptance can be gamed, a boundary the paper explicitly sets but does not stress-test.","Using multiple public datasets or a validation committee could improve representativeness and resilience over the single-dataset design."],"forward_implications":["If correct, federated learning can run without a trusted aggregator while still filtering poisoned updates, enabling FL in fully untrusted IoT settings.","The judge-model consensus protocol gives a standardized, reproducible detection method, addressing the lack of standardization noted in the paper.","The linear scalability of judge creation means the defense can be applied as client counts grow, at least up to 200 clients as tested.","The high global accuracy at 35% malicious clients suggests the defense tolerates substantial poisoning without model degradation.","The judge layer is defined independently of the aggregation rule, so it can be paired with robust aggregators like FedLAW or RFA, which the paper explicitly considers."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation formula used to combine accepted model updates.","marker":"[2]"},{"why":"Defines data poisoning attacks in federated learning, framing the threat model the paper defends against.","marker":"[1]"},{"why":"Proposes a trusted third-party auditor/reference model approach that FIDELIS replaces with a decentralized judge.","marker":"[6]"},{"why":"Introduces Zeno's reference batch scoring, a baseline for comparing updates against clean data.","marker":"[7]"},{"why":"Presents a blockchain FL architecture with validation consensus, the closest work FIDELIS extends.","marker":"[19]"},{"why":"Krum is a robust aggregation baseline that motivates the need for outlier filtering in FL.","marker":"[16]"}],"fun_headline_variants":["Blockchain votes pick poison-detecting judge model","Federated learning stays 98% accurate with blockchain judge","Decentralized judge model blocks poisoning attacks in federated learning","Isolation-forest judge, voted on by blockchain, filters poisoned updates","No trusted server: clients vote on judge to keep federated learning safe"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that gradient statistics from training on the public dataset represent the distribution of benign clients' gradient updates on their private data, across rounds and attack intensities; if that match fails, the judge either misses poisoned updates or rejects benign ones.","fun_headline_variants_meta":{"raw":{"variants":["Blockchain votes pick poison-detecting judge model","Federated learning stays 98% accurate with blockchain judge","Decentralized judge model blocks poisoning attacks in federated learning","Isolation-forest judge, voted on by blockchain, filters poisoned updates","No trusted server: clients vote on judge to keep federated learning safe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2624,"prompt_tokens":656,"completion_tokens":1968,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":1881}},"tokens_in":400,"tokens_out":1968,"duration_ms":15097,"temperature":1.0,"reasoning_tokens":1881,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:31:26.675453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FIDELIS with a public dataset unrelated to the private task (for example, a public pet-image dataset while clients train on CIFAR-10) and measure the judge model's true-positive rate; if it drops well below the reported 95–100%, the representativeness assumption fails. Alternatively, launch a backdoor (non-label-flipping) poisoning attack under the same protocol: the paper only evaluates label-flipping, so a successful backdoor would show the robustness claim is attack-specific.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the FedAvg aggregation formula used to combine accepted model updates."},{"cited_title":"Data poisoning attacks against federated learning systems,","cited_arxiv_id":null,"evidence_quote":"Defines data poisoning attacks in federated learning, framing the threat model the paper defends against."},{"cited_title":"Using Anomaly Detection to Detect Poisoning Attacks in Federated Learning Applications","cited_arxiv_id":"2207.08486","evidence_quote":"Proposes a trusted third-party auditor/reference model approach that FIDELIS replaces with a decentralized judge."},{"cited_title":"Zeno: Distributed stochastic gradient descent with suspicion-based fault- tolerance,","cited_arxiv_id":null,"evidence_quote":"Introduces Zeno's reference batch scoring, a baseline for comparing updates against clean data."},{"cited_title":"Trustworthy federated learning via blockchain,","cited_arxiv_id":null,"evidence_quote":"Presents a blockchain FL architecture with validation consensus, the closest work FIDELIS extends."},{"cited_title":"Machine learning with adversaries: Byzantine tolerant gradient descent,","cited_arxiv_id":null,"evidence_quote":"Krum is a robust aggregation baseline that motivates the need for outlier filtering in FL."}],"review_version":1}