{"id":"b8c4b4e6-9363-452f-a721-55bb5d610796","arxiv_id":"2506.19892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"RepuNet is a decentralized reputation mechanism that weights or excludes model updates based on locally measured model similarity, parameter changes, latency, and message volume, and the emulated results show improved resistance to poisoning, delay, and flooding attacks in most tested scenarios.","lead":"This preprint describes RepuNet, a reputation system that scores neighboring nodes in decentralized federated learning by tracking model similarity, parameter changes, arrival latency, and message volume, then uses those scores to weight or exclude model updates. It reports that the system detects and limits model poisoning, delay, and flooding attacks in emulated federations of up to 25 nodes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F1 numbers are not traceable to any reported table, and the main poisoning comparison mixes rounds (RepuNet at r8 vs no-defense at r11), so the central effectiveness claim is not yet evidenced.","rationale":"I read the paper in good faith. The RepuNet design is plausible: the four metrics are locally observable, the reputation update is formalized in equations, and integration into the Nebula platform provides a concrete deployment path. The authors also honestly note limitations such as intermittent attacks allowing partial reintegration and the need for stricter memory-based penalties. Those are real strengths. However, the paper's central empirical assertion is that RepuNet detects and mitigates attacks, with specific F1 numbers. That assertion is what the experiments must establish, and the experimental report has a load-bearing inconsistency: the main quantitative table, Table VI, compares with-RepuNet at round 8 against without-RepuNet at round 11, so the Delta-F1 cannot be interpreted as a clean effect of the defense. The abstract's 95%/76% F1 values do not appear anywhere in the body, and if they are detection-F1 values for malicious nodes, that metric is never defined or reported, making the headline unreproducible from the manuscript alone. This is more immediate than the reader's identified weakest assumption about attackers mimicking honest metrics or colluding in the feedback channel; those are important secondary risks, but they cannot even be assessed until the basic empirical comparison is valid. A concrete check is straightforward: recompute per-round F1 curves and compare at identical rounds, and pin down which quantity the abstract's numbers refer to. This would settle whether the central claim survives. I therefore agree with the CONDITIONAL verdict but with the condition that the quantitative evidence be re-presented or made available.","tokens_in":15258,"tokens_out":4063,"duration_ms":48762,"concrete_test":"Re-run or re-extract the per-round F1 curves for the Table V scenarios from the Nebula logs. Compute with-RepuNet and without-RepuNet F1 at the same rounds (r8 and r11) and report the standard deviation over at least 3 seeds; then identify explicitly which quantity the abstract's 95% (MNIST) and 76% (CIFAR-10) figures refer to and place that metric in the results. If the same-round comparison does not show a positive Delta-F1 or the 95%/76% numbers cannot be reproduced, the central claim should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on the quantitative evaluation, but the evaluation as written cannot support it. In Table VI, the 'F1 w/rep' column is measured at round 8 and the 'F1 w/o rep' column at round 11; the reported Delta-F1 therefore confounds defense effectiveness with training progress, since the two systems are compared at different points in the learning curve. The abstract states 'F1 scores above 95% for MNIST scenarios and approximately 76% for CIFAR-10 cases,' yet no table or figure reports any 0.95 or 0.76 value: Table VI's best with-defense F1 is 0.6879, and the CIFAR-10 rows (5.1-5.3) are 0.5360, 0.5122, and 0.5166. If the abstract's numbers refer to a different quantity, such as malicious-node detection F1, that quantity is never defined or reported, so the headline cannot be verified. Additionally, no error bars or repeated-seed statistics are given, and only the model-poisoning attack has a quantitative global-model comparison; delay and flooding sections report reputation trajectories and aggregated-model counts but no F1 or accuracy comparison against no-defense. Because the claim 'improves global model F1 compared with no defense' is checked only through this inconsistent and incomplete comparison, the evidence is not yet adequate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RepuNet is a decentralized reputation system for Decentralized Federated Learning (DFL) that scores each neighboring node from four locally observable metrics: model similarity, fraction of parameters changed, model arrival latency, and incoming message flow. The reputation score determines whether a received model is excluded from aggregation and how it is weighted, with optional distributed feedback from peers. The system is integrated into the Nebula platform and evaluated on MNIST and CIFAR-10 under non-IID data partitions, fully connected and random topologies, and three attack families (model poisoning, delay, flooding), with claimed F1 improvements over an undefended baseline.","tokens_in":15626,"tokens_out":3804,"duration_ms":38830,"significance":"If the central claims were properly evidenced, RepuNet would be a practically attractive defense for DFL: it avoids blockchain overhead, is locally computable, and addresses communication-side attacks that are often neglected. The paper is also commendable for implementing the system in a real platform (Nebula), testing multiple topologies, attack intensities, and activation timings, and for reporting reputation trajectories and aggregation counts. However, the quantitative evidence as presented is not yet adequate: the headline F1 numbers in the abstract are not traceable to any table, the main poisoning comparison mixes evaluation rounds, and the delay/flooding evaluations do not compare final model quality against a no-defense baseline. These issues are correctable with additional experiments and careful re-reporting, so the contribution remains plausible but unproven in its current form.","major_comments":[{"comment":"The abstract states that RepuNet achieves 'F1 scores above 95% for MNIST scenarios and approximately 76% for CIFAR-10 cases', but no such values appear in Table VI or elsewhere in the body: the best MNIST F1 with RepuNet reported is 0.6879 (scenario 7.2), and the CIFAR-10 rows (5.1–5.3) report 0.5360, 0.5122, and 0.5166. If the abstract refers to a different quantity (e.g., malicious-node detection F1), that quantity is never defined or reported, so the headline claim cannot be verified.","section":"Abstract vs. Section V-B, Table VI"},{"comment":"The comparison in Table VI is confounded by different evaluation rounds: the 'F1 w/rep (r8)' column is measured at round 8 while the 'F1 w/o rep (r11)' column is measured at round 11. Since the undefended model continues training for three additional rounds, the reported ΔF1 conflates RepuNet's defensive effect with extra training progress. The authors need to report both systems at the same round (or otherwise show that the round difference does not materially affect the comparison).","section":"Table VI, Section V-B"},{"comment":"For the delay and flooding attacks, the evaluation reports only reputation trajectories (Figures 5, 6, 9), aggregated-model counts (Figures 8, 10), and CPU usage (Table XI); there is no global-model F1 or accuracy comparison against an undefended baseline. The paper's central claim that RepuNet 'mitigates' these attacks in terms of final model quality is therefore not quantitatively evidenced for two of the three attack types. The authors should either provide such baseline comparisons or explicitly limit their claims to reputation-based exclusion effects.","section":"Sections V-C and V-D"},{"comment":"The system relies on several hand-chosen parameters—the exclusion threshold of 0.6 (Section IV-E), the 150% latency tolerance multiplier (Section IV-C.3), the 1.05 and 1.10 multipliers in Equation (4), the bootstrapping penalty δ=0.05 in Equation (13), and the feedback weight η in Equation (25)—but no sensitivity analysis is performed. Without evidence that the results are stable with respect to these parameters, the claim that RepuNet is adaptive rather than configuration-sensitive is not established. A parameter-sweep study for at least the exclusion threshold and latency multiplier would address this.","section":"Sections IV-C, IV-D, IV-E and Table III"}],"minor_comments":[{"comment":"The statement that 'only nodes with an active reputation module are allowed to issue evaluations' guarantees that all received scores originate from 'valid and trustworthy participants' is unsupported: an attacker can also run an active reputation module and send dishonest feedback. This should be clarified as a design assumption, and the feedback mechanism should be tested under collusion if it is claimed as a robustness feature.","section":"Section IV-F"},{"comment":"The definition of the latency tolerance threshold as '150% of the average latency' is stated only in prose; if this is a fixed multiplier, it should be included in the formal equations or in Table III to allow reproducibility.","section":"Section IV-C.3"},{"comment":"The term 'increase mean' in the amplification formula is not defined; the text says it 'adjusts automatically based on the system phase' but never specifies how. Please provide a formal definition.","section":"Equation (17)"},{"comment":"The notation μ_t and σ_t in Equations (3)–(4) is ambiguous: it is not clear whether these are thresholds per round, and how they are initialized when no history is available.","section":"Section IV-C.2"},{"comment":"Several figures and tables are referenced out of order (e.g., Figure 7 is cited before Figures 5 and 6, and Table X is mentioned in Section V-D but appears before Figure 9); the numbering should be re-sequenced to match the order of first mention.","section":"Figures and tables"},{"comment":"There are minor language issues, including 'delayer attack' (should likely be 'delay attack'), inconsistent hyphenation of 'F1-score' vs. 'F1 score', and the use of 'X' in Table I to denote negative/absent features, which is not explained in the caption.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between the abstract's headline F1 numbers and the values in Table VI is substantial enough that I would ask the authors to reconcile them explicitly; it may indicate that the abstract reports a different metric or a different set of experiments than those in the body. The comparison at different rounds (r8 vs r11) should also be fixed before the paper is considered further. The core idea is reasonable and the platform integration is a strength, but the evaluation currently overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe abstract sells this paper with F1 numbers that simply aren't in the body. No table or figure reports 0.95 for MNIST or 0.76 for CIFAR-10; Table VI's best with-defense F1 is 0.6879. And the main poisoning comparison is apples-to-oranges: RepuNet is measured at round 8, no-defense at round 11, so the ΔF1 confounds defense with training progress. That is a load-bearing flaw, because the only non-circular evidence for the system is the global model improvement.\n\nBut the design itself is worth a look. RepuNet combines four locally observable metrics — model similarity, parameter-change fraction, arrival latency, and message volume — into a decentralized reputation score with dynamic weighting and progressive reintegration, and it avoids blockchain overhead. That is a genuine extension over the systems in Table I, which mostly use one or two metrics and often rely on a central server or blockchain. The integration into Nebula and the sweep across topologies, attacker proportions, and activation timings is also solid. The reputation trajectories show a consistent pattern: malicious nodes drop, honest nodes stay high. That pattern is partly by construction, since reputation is computed from the same metrics that define attack behavior, but it does show the mechanism behaves coherently.\n\nThe soft spots beyond the F1 mismatch: thresholds like the 0.6 exclusion value and the 150% latency tolerance are hand-chosen rather than derived; there are no repeated-seed runs or error bars anywhere; the delay and flooding sections report reputation and aggregated-model counts but never compare global model performance against a no-defense baseline; and the feedback channel is assumed trustworthy because only nodes with an active reputation module can send evaluations, which does not stop a malicious node from running the module and colluding. The intermittent-attack results (7.1, 7.2) are honest in showing negligible effect, which is to the authors' credit.\n\nThe paper deserves a serious referee — the idea is relevant and the evaluation is fixable in principle — but the current evidence does not support the abstract's claims. I would ask for a revised version with matched rounds, error bars, a defined detection-F1 if that is what the abstract means, and a global-model comparison for delay and flooding before trusting any of the quantitative conclusions.","headline":"The abstract's F1 numbers don't appear in any table and the main poisoning comparison mixes rounds, so the central effectiveness claim is not yet supported; the design is novel enough to warrant a serious revised submission.","tokens_in":16100,"tokens_out":2374,"would_cite":false,"duration_ms":23964,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that local reputation scoring from four observable metrics can detect and mitigate model poisoning, delay, and flooding in decentralized federated learning, with reported F1 scores above 95% for MNIST and around 76% for…","keywords":["decentralized federated learning","reputation system","model poisoning","delay attack","flooding attack","anomaly detection","reputation-based aggregation","non-IID data"],"falsifier":"A concrete experiment that would settle the central claim is an adaptive poisoning attack in which malicious nodes craft updates whose model similarity and parameter-change fraction remain within the historical mean plus one standard deviation of honest updates while still degrading the global model. If RepuNet's detection F1 then drops to near the no-defense baseline, the claim that these four metrics can detect poisoning is refuted; if it stays above 90% on MNIST, the claim survives.","tokens_in":15105,"feed_emoji":"🛡️","tokens_out":8098,"duration_ms":81544,"temperature":0.7,"pith_summary":"Decentralized federated learning lets each node choose whom to aggregate models from, and malicious nodes can exploit that freedom. RepuNet is a reputation mechanism each node runs locally, scoring every neighbor on model similarity, parameter-change fraction, model arrival latency, and message volume. Those scores decide how much weight a neighbor's model gets, or whether it is excluded entirely, and they are updated each round so a node that behaves well can recover. In emulated federations of up to 25 nodes, the paper reports that RepuNet identified malicious behavior with F1 above 95% on MNIST and about 76% on CIFAR-10, and in many poisoning scenarios improved the global F1 by more than 0.25 over running no defense. The point is that threat detection and mitigation in DFL can be done with lightweight local observations, without a central server or blockchain.","feed_headline":"Four local metrics stop malicious nodes in decentralized learning","feed_subtitle":"Per-node scores from four observed metrics filter poisoning, delay, and flooding attacks.","key_machinery":"RepuNet's central object is a per-neighbor reputation score computed inside each node every round. The score combines four normalized metrics: model similarity (a weighted average of cosine, Euclidean, Manhattan, and Pearson comparisons between the local and received models), fraction of parameters changed (a sigmoid penalty on deviations from the node's own history), model arrival latency (a sigmoid penalty when current latency exceeds the historical mean), and incoming message flow (an exponential penalty when message volume rises above a dynamic margin). Each metric is mapped to [0,1], dynamic weights are set in proportion to how far that metric deviates from historical behavior, and the final reputation is a weighted combination of the reputation history and the current score. Optional neighbor feedback, averaged and blended with the local score, can refine the estimate. Reputations below a threshold (typically 0.6) cause the sender's model to be discarded; otherwise the model is aggregated with weight proportional to reputation, and a node can be re-admitted if its behavior improves in later rounds.","core_discovery":"The paper's central claim is that three classes of attacks leave locally measurable traces, and that a reputation score built from those traces can protect the federation without a central authority. During model poisoning, the received model diverges from honest updates, so similarity falls and the fraction of changed parameters rises; during delay attacks, arrival latency exceeds historical patterns; during flooding, message counts jump above a dynamic margin. RepuNet turns each trace into a normalized score, weights the metrics by how anomalous they are, and excludes or down-weights models from low-reputation neighbors. Its own experiments report that malicious nodes' reputations drop sharply within a round or two after an attack starts, while honest nodes stay near the top of the scale, and that in most poisoning scenarios the global F1 with RepuNet is substantially higher than without it. Using MNIST and CIFAR-10 under non-IID partitions, the paper reports detection F1 scores above 95% for MNIST and about 76% for CIFAR-10.","pith_inferences":["This is an extension, not a paper claim: an adaptive adversary that keeps model similarity and parameter-change fraction inside the honest historical band while still injecting poisoned labels would test whether the four metrics are truly sufficient; RepuNet's fixed reputation threshold would likely need revision if such an attacker evades it.","Another testable consequence: if the same four metrics were used for peer selection in addition to aggregation weighting, the reputation could also steer which neighbors are worth connecting to in sparse or dynamic topologies, something RepuNet does not currently do.","The CIFAR-10 detection gap (76% versus over 95% on MNIST) suggests that data heterogeneity, not attack strength, may set the practical limit on metric-based detection; testing with harder non-IID splits or distribution-aware similarity baselines would sharpen that boundary."],"forward_implications":["Reputation-based aggregation can replace binary filtering in DFL: low-reputation models are down-weighted or excluded, while accepted models retain a vote proportional to reputation.","Because reputation is recomputed every round and exclusion is not disconnection, a node that returns to honest behavior can be progressively re-admitted, so transient failures need not be permanent.","The system runs without blockchain or a central coordinator, keeping per-node overhead low enough for federations of tens of nodes.","Across poisoning scenarios, the reported gap in F1 with versus without RepuNet reaches over 0.25 in many cases, and the largest gains appear when attacks start early or the malicious fraction is high."],"supporting_citations":[{"why":"Establishes the federated learning setting and the practice of aggregating locally trained models.","marker":"[1]"},{"why":"Defines decentralized federated learning and the absence of a central coordinator that makes peer selection a vulnerability.","marker":"[2]"},{"why":"Provides a baseline for detecting falsified updates and motivates the model-quality metrics.","marker":"[3]"},{"why":"Defines delay attacks in hierarchical federated learning and supplies the latency-based defense RepuNet compares against.","marker":"[4]"},{"why":"Establishes flooding/DDoS mitigation as a decentralized learning problem and motivates the message-volume metric.","marker":"[5]"},{"why":"Supplies the emulation platform on which all RepuNet experiments were run.","marker":"[10]"},{"why":"Represents the model-filtering defense that RepuNet contrasts with its reputation-weighted aggregation.","marker":"[17]"}],"fun_headline_variants":["Four local signals flag bad nodes in decentralized federated learning","RepuNet: Reputation scores cut malicious nodes in decentralized learning","No central server: local metrics detect poisoning, delay, and flooding attacks","Local trust scores neutralize adversaries in decentralized federated learning","Decentralized defense: RepuNet scores peers to stop model poisoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That malicious behavior reliably changes the four measured metrics enough to cross the fixed thresholds, and that the optional reputation-feedback channel cannot be flooded by colluding nodes because only nodes with an active reputation module issue evaluations.","fun_headline_variants_meta":{"raw":{"variants":["Four local signals flag bad nodes in decentralized federated learning","RepuNet: Reputation scores cut malicious nodes in decentralized learning","No central server: local metrics detect poisoning, delay, and flooding attacks","Local trust scores neutralize adversaries in decentralized federated learning","Decentralized defense: RepuNet scores peers to stop model poisoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3185,"prompt_tokens":991,"completion_tokens":2194,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2106}},"tokens_in":607,"tokens_out":2194,"duration_ms":15256,"temperature":1.0,"reasoning_tokens":2106,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:41.310782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete experiment that would settle the central claim is an adaptive poisoning attack in which malicious nodes craft updates whose model similarity and parameter-change fraction remain within the historical mean plus one standard deviation of honest updates while still degrading the global model. If RepuNet's detection F1 then drops to near the no-defense baseline, the claim that these four metrics can detect poisoning is refuted; if it stays above 90% on MNIST, the claim survives.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Establishes the federated learning setting and the practice of aggregating locally trained models."},{"cited_title":"Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,","cited_arxiv_id":null,"evidence_quote":"Defines decentralized federated learning and the absence of a central coordinator that makes peer selection a vulnerability."},{"cited_title":"Detection of false data injection attacks in smart grid: A secure federated deep learning approach,","cited_arxiv_id":null,"evidence_quote":"Provides a baseline for detecting falsified updates and motivates the model-quality metrics."},{"cited_title":"Delay- aware hierarchical federated learning,","cited_arxiv_id":null,"evidence_quote":"Defines delay attacks in hierarchical federated learning and supplies the latency-based defense RepuNet compares against."},{"cited_title":"FLEAM: A federated learning empowered architecture to mitigate DDoS in industrial IoT,","cited_arxiv_id":null,"evidence_quote":"Establishes flooding/DDoS mitigation as a decentralized learning problem and motivates the message-volume metric."},{"cited_title":"Nebula: A platform for decentralized federated learn- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the emulation platform on which all RepuNet experiments were run."},{"cited_title":"Poisoning attacks on federated learning-based IoT intrusion detection system,","cited_arxiv_id":null,"evidence_quote":"Represents the model-filtering defense that RepuNet contrasts with its reputation-weighted aggregation."}],"review_version":1}