{"id":"85558309-7326-4125-99b0-ad4555931d7e","arxiv_id":"2608.08574","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"R2CFL couples worker reputation to robust update filtering in crowdsourced federated learning so that stealthy attackers lose influence, with reputation dynamics claimed to mirror the defense's true and false positive rates.","lead":"This paper proposes R2CFL, a reputation system for crowdsourced federated learning that ties each worker's trust score to whether their model updates pass a robust filter, and adds a reputation-weighted nearest-neighbor defense (R2-NNM). It may interest anyone building open federated learning systems that must resist stealthy attackers who slowly earn trust while poisoning the shared model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reputation-fidelity claim is validated by simulating the assumption: Table 2 is derived from Eq. (14)'s Bernoulli TPR/FPR, so it cannot support the headline until end-to-end reputation trajectories are measured.","rationale":"The reader's weakest_assumption identifies the load-bearing point: Section 4.3 validates reputation fidelity by assuming the very Bernoulli acceptance process whose fidelity is claimed. This is an evidential gap, not a demonstrated contradiction, because the paper does provide concrete and separate evidence that R2-NNM matches NNM in TPR/FPR and convergence (Table 1, Figure 2), and it promises code that could settle the issue. The conclusion's own caveat about not adapting to round-varying adversarial pressure reinforces that the adaptive-attacker guarantee advertised in the abstract is not established by the reported experiments. For these reasons, the reader's CONDITIONAL verdict is appropriate: acceptance should require end-to-end reputation trajectories, a reputation-aware adaptive attacker, and full hyperparameter disclosure. I find no basis to escalate to REJECT, since the detection experiments are concrete and the central issue is a missing validation step rather than an internal inconsistency in the main defense results.","tokens_in":17272,"tokens_out":8576,"duration_ms":97495,"concrete_test":"Run the released R2-NNM code end-to-end on the A3FL FashionMNIST non-IID configuration (n=20, f=6, attack window rounds 60-80), recording per round the acceptance indicator Reliable_i^t and local reputation L_rep,i^t for every benign and malicious client. Then (a) estimate per-round TPR/FPR and test serial independence of each client's acceptance sequence with a runs test; (b) compare the empirical final-round separation Delta = R_B_bar - R_M_bar against Table 2's values, using the same seeds and disclosing alpha, tau, psi, xi, lambda. If the end-to-end Delta is materially smaller than the Bernoulli-model value, or if acceptance is significantly autocorrelated, the claim that reputation faithfully reflects defense TPR/FPR and prevents gradual trust accumulation is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is that R2-NNM 'faithfully captures' the underlying defense's TPR/FPR and 'prevents stealthy attackers from gradually accumulating trust.' The only evidence offered for this, Table 2, is produced by Section 4.3's simulation in which Reliable_i^t is sampled as an independent Bernoulli variable with probability 1-FPR for benign clients and 1-TPR for malicious clients (Eq. 14), then fed into the EMA reputation update (Eq. 10). Under those inputs, high benign and near-zero malicious reputation is a mathematical consequence, so Table 2 cannot independently establish the faithfulness claim. For the claim to hold, the actual R2-NNM acceptance decisions must be distributed as independent Bernoulli draws with those exact, constant TPR/FPR values. That is not justified in the paper: acceptance is a deterministic function of update geometry, and therefore temporally correlated, since the same clients, local data, and attack recipes recur across rounds; Table 4 itself shows FPR varying from 0.00 to 0.13 with attack onset, so TPR/FPR are not constant over time; and an adversary who observes reputation feedback can adapt, a case the conclusion explicitly disclaims ('does not adapt to round-varying adversarial pressure'). No end-to-end reputation trajectories from actual R2-NNM runs are reported anywhere. The simulation is also underspecified: Eq. (9) defines C_i as a mix of the soft score \\hat C_i and Reliable_i, but Section 4.3 only states that Reliable_i is sampled, leaving alpha, tau, psi, xi, and lambda unspecified; if alpha=0 was used in Table 2, the reported separation concerns a degenerate reputation rule that omits the geometric component of R2-NNM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes R2CFL, a reputation-driven framework for crowdsourced federated learning that couples reputation evolution with a robust aggregation defense called R2-NNM. In R2-NNM, each client update is smoothed by mixing with its k nearest neighbors using reputation-dependent weights, followed by Multi-Krum selection. The reputation model is data-free: per-round acceptance by the defense (plus a soft geometric score) is combined into a local reputation via EMA with a warm-up, and a task-level asymmetric EMA update is defined. The paper claims that this mechanism prevents stealthy attackers from accumulating trust and that, when integrated with detect-and-filter defenses, the reputation scores faithfully reflect the underlying defense's TPR/FPR. Experiments compare R2-NNM with several Byzantine-robust and backdoor defenses on FashionMNIST, GTSRB, and CIFAR-10 under IID/non-IID settings, showing that R2-NNM closely matches NNM's detection performance. Section 4.3 then presents a simulation study that feeds defense-specific TPR/FPR into the reputation update (Eq. (14)) and reports final reputation separation in Table 2, which the paper interprets as confirming the reputation-fidelity claim.","tokens_in":17676,"tokens_out":3125,"duration_ms":32347,"significance":"If the central claim were fully supported, the paper would make a useful contribution: it links reputation to the statistical behavior of a robust aggregation rule, avoids external validation data, and provides a concrete feedback loop that could hinder stealthy attackers in open FL settings. The paper has tangible strengths: a public code repository, experiments across three datasets and multiple attacks, ablations of key parameters, and an honest concluding limitation about non-stationary attack pressure. However, the headline claim that reputation 'faithfully captures' the defense's TPR/FPR is currently validated only by a simulation that assumes the very statistics it purports to recover, so the main novelty is not yet established as an empirical fact about the proposed mechanism.","major_comments":[{"comment":"The reputation-fidelity claim is not independently validated because Table 2 is generated by simulating the acceptance indicator with Eq. (14): for benign clients the acceptance probability is 1-FPR and for malicious clients 1-TPR, and these samples are then fed into the EMA update of Eq. (10). With these inputs, high benign reputation and near-zero malicious reputation are a mathematical consequence of the EMA update, not a property of R2-NNM's actual acceptance behavior. The paper provides no end-to-end reputation trajectories from real R2-NNM runs where Reliable_i^t comes from the actual Multi-Krum selection of Eq. (5). The Bernoulli model of Eq. (14) is also not justified by the mechanism: acceptance is a deterministic function of update geometry, and the same clients, local data, and attack recipes recur across rounds, so acceptance events are likely correlated rather than independent. Table 4 itself shows FPR varying from 0.00 to 0.13 with attack onset, contradicting the constant-FPR assumption used in the simulation. To support the abstract's claim, the authors should report actual reputation dynamics from end-to-end runs and compare them with the TPR/FPR-based simulation.","section":"§4.3, Eq. (14), Table 2"},{"comment":"The reputation model is underspecified at a load-bearing point. Eq. (10) defines a per-round local reputation L_t_rep,i, while Eq. (11) defines a task-level global reputation R_i updated using a quantity 'L_rep,i' that is never formally tied to the per-round sequence; no aggregation of the per-round values into the task-level signal is given. Section 4.3 says it 'simulates the robust local evaluation (Eq. (10)) over multiple FL rounds' and Table 2 reports final 'L_t_rep,i scores' at round 100 (with the text in §4.3 saying 'here T=50'), but the relationship between these reported scores and the task-level update of Eq. (11) is unclear. This makes it difficult to interpret the simulated results as predictions for the actual deployed reputation system, and it prevents the reader from reproducing Table 2 from the stated equations.","section":"§3.5, Eqs. (10)–(11), §4.3"},{"comment":"The claimed security benefit of the reputation-aware mixing is not demonstrated by the experimental comparisons. R2-NNM's TPR/FPR are essentially identical to those of plain NNM in Table 1, and in Table 4 the ASR for R2-NNM remains around 0.096–0.098 under A3FL, i.e., the backdoor still has a measurable success rate; no statistical test or repeated-seed comparison is provided to show that the slight ASR differences are meaningful. The conclusion's own limitation statement concedes that the design 'does not adapt to round-varying adversarial pressure.' The paper's central defensive advantage—preventing stealthy attackers from gradually accumulating trust—is therefore supported only by the simulated reputation separation of Table 2, not by attack-success or reputation measurements in the actual system.","section":"§4.2, Table 4, §5"}],"minor_comments":[{"comment":"The table caption says 'Final L_t_rep,i scores at round 100' while the text in §4.3 states 'T denotes the final round (here T=50)'; these numbers should be reconciled.","section":"§4.3, Table 2 caption"},{"comment":"The normalization in the reputation-weighted term divides by sum over L_rep,l, but it is not stated whether L_rep denotes the local per-round score or the global task-level score; this should be clarified.","section":"Eq. (2)"},{"comment":"In the AutoDFL baseline description, 'the top-m updates are selected' but the variable is then confusingly referred to as 'k' in the same paragraph; the notation should be aligned with the rest of the paper.","section":"§4.1, baselines"},{"comment":"The legend in Figure 2 lists eight methods; the curves are difficult to distinguish in the gray-scale print, and the paper would benefit from labeled final-point markers or a table of final ACC/ASR values for each subplot.","section":"§4.2, Fig. 2"},{"comment":"For t > T_warm the factor gamma_t = kappa * t / T_warm continues to grow beyond kappa, yet the text says reputation is 'increasingly influenced' only after warm-up; a cap or a different schedule should be stated explicitly.","section":"§3.4, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The paper's detection experiments are solid and the R2-NNM defense is a reasonable engineering contribution, but the central novelty—the 'faithful' reputation model—rests on a simulation that assumes the conclusion. I would ask for end-to-end reputation measurements before considering acceptance. The manuscript also has several self-citations to the same group's previous work; that is not unusual but the related-work discussion would benefit from a more distanced treatment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know before reading: R2-NNM is a real but incremental extension of NNM, and the paper's central advertised result—that reputation faithfully mirrors the defense's TPR/FPR—is not actually demonstrated end-to-end. Table 2 is produced by sampling each defense's measured TPR/FPR and feeding that signal into the EMA update. The separation in reputation is an arithmetic consequence of the inputs, not an independent validation.\n\nThe genuine novelty is the coupling of nearest-neighbor mixing with a data-free reputation signal based on filter acceptance, plus the progressive activation schedule and asymmetric task-level update. That combination is new relative to NNM, AutoDFL, and SSMTD. The detection experiments are credible: three datasets, three attacks, IID and non-IID, and R2-NNM tracks NNM and M-Krum without degrading convergence. The ablation on k, n, f, and attack schedule is useful, and the code is available. The authors are honestly exploring a design space.\n\nThe soft spot is load-bearing. Section 4.3 samples Reliable_i^t as a Bernoulli variable with probability 1-FPR for benign clients and 1-TPR for malicious clients, then runs it through Eq. (10). Under those inputs, benign reputations high and malicious reputations near zero is what the equations produce. It does not test whether actual R2-NNM acceptance decisions behave that way. Acceptance is a deterministic function of update geometry, likely temporally correlated, and Table 4 itself shows FPR changing with attack onset, so constant TPR/FPR is not justified. No end-to-end reputation trajectories from actual R2-NNM runs are reported anywhere. This is fixable: run R2-NNM under the same attacks, record the real acceptance signal per client per round, and compare the resulting reputation curves to the simulated ones.\n\nAlso worth asking for: a truly reputation-aware adaptive adversary, since the abstract says stealthy attackers cannot accumulate trust but the conclusion explicitly disclaims round-varying adversarial pressure. Minor but real: several reputation hyperparameters (at least tau, and psi/xi if the task-level update is used) are not specified in the main experimental setup, and the detection tables report no error bars.\n\nWho gets value: people working on Byzantine-robust aggregation, reputation mechanisms for crowdsourced or blockchain federated learning, and NNM variants. It is a defense contribution within an established subfield, not a new paradigm. It deserves a serious referee, but the referee should require the end-to-end reputation validation before accepting.","headline":"A solid, incremental NNM variant whose headline reputation-fidelity claim is validated by a simulation that assumes the answer; end-to-end reputation trajectories would settle it.","tokens_in":18235,"tokens_out":3106,"would_cite":true,"duration_ms":35598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In crowdsourced federated learning, deriving worker reputation from the acceptance decisions of a robust filtering defense prevents stealthy attackers from accumulating trust, and makes reputation scores mirror the defense's TPR/FPR.","keywords":["federated learning","crowdsourcing","reputation systems","Byzantine attacks","backdoor attacks","robust aggregation","nearest neighbor mixing","stealthy adversaries"],"falsifier":"Run R2CFL end-to-end with a stealthy attacker who monitors its own reputation and alternates between clean and poisoned updates, then check whether malicious reputation stays below roughly 0.3 while benign reputation stays above 0.7 after 100 rounds; alternatively, measure the autocorrelation of the acceptance indicator and show that substantial positive autocorrelation is present, which would violate the independent-Bernoulli assumption behind the reported separation.","tokens_in":17003,"feed_emoji":"🛡️","tokens_out":5468,"duration_ms":58268,"temperature":0.7,"pith_summary":"The paper proposes R2CFL, a reputation-driven framework for crowdsourced federated learning in which worker trust is updated from the same robust statistical filter that decides which model updates get aggregated. The central claim is that tying reputation to the acceptance and rejection decisions of a detect-and-filter defense stops stealthy attackers from gradually building credibility, because each failed poisoning attempt pushes their reputation down. If correct, reputation becomes an empirical estimate of the probability that a worker's update would be accepted by a robust aggregation rule, giving task publishers a principled signal for worker selection. Experiments on three image datasets under three adaptive attacks show that the proposed R2-NNM defense matches state-of-the-art defenses in detection while keeping false positives low, and that the reputation scores it produces closely track the defense's true and false positive rates.","feed_headline":"Reputation that mirrors robust filtering stops stealthy FL attackers","feed_subtitle":"Tying trust scores to a defense's TPR/FPR stops gradual poisoning takeover in crowdsourced FL.","key_machinery":"The central object is R2-NNM, the Robust Reputation-Aware Nearest Neighbor Mixing defense. Each local update is first smoothed by a weighted average of its k nearest neighbors, where the weights blend uniform mixing with reputation-derived weights modulated by a warm-up factor; the smoothed updates are then passed through Multi-Krum, which retains the m updates with the smallest pairwise distances. The reputation signal is computed without any external validation data: each client receives a binary acceptance flag from the filter and a soft sigmoid-transformed median/MAD deviation score, and the two are combined and updated with an exponential moving average. This makes reputation an empirical estimate of the probability of being consistently selected by the robust aggregation rule, which is the mechanism that couples filtering outcomes to future trust.","core_discovery":"The paper's discovery is that reputation in CrowdFL can be derived from the statistical behavior of the robust aggregation filter itself, rather than from external performance signals such as validation accuracy. The authors build a closed loop: a reputation-aware nearest neighbor mixing step smooths each update using neighbors weighted by current reputation, then a Multi-Krum selection chooses the trusted subset; clients accepted by the filter gain reputation and rejected clients lose it. Under adaptive stealthy attacks (OMP, Neurotoxin, and A3FL), the proposed R2-NNM matches the detection trade-off of plain NNM and achieves low false positives, while the final reputation gap between benign and malicious workers (around 0.72 to 0.74) mirrors the defense's TPR and FPR. The authors claim this prevents stealthy attackers from accumulating influence and that, when attached to any detect-and-filter defense, the reputation model faithfully captures that defense's robustness.","pith_inferences":["The independence assumption used to simulate reputation dynamics (each round's acceptance is an independent Bernoulli event with fixed probabilities) is likely the main fragility: in real runs, acceptance events are correlated over time, and a stealthy attacker who observes its own reputation could alternate between benign and poisoned updates to keep its score above the selection threshold.","A natural testable extension is to run the full selection loop end-to-end, using the computed reputation scores to choose which workers participate in future tasks, and measure whether long-term accuracy under intermittent attacks improves compared with a system that filters but ignores reputation.","The same 'reputation equals defense statistics' principle could be applied to defenses with round-varying randomized decisions, such as those that inject noise, where TPR and FPR fluctuate; the paper's current model assumes these rates are constant across rounds.","The authors note that the design does not adapt to round-varying adversarial pressure; learning the warmth and selection hyperparameters online, for instance with a reinforcement-learning controller, is a plausible way to maintain the robustness-fairness balance under non-stationary attacks."],"forward_implications":["If a worker consistently fails the robust filter, reputation decays toward zero, so the worker loses selection priority and influence in future tasks even when an individual poisoned update is not flagged in a given round.","Reputation becomes a defense-calibrated quantity: a benign worker's score approaches 1 minus the false positive rate, while a malicious worker's score approaches 1 minus the true positive rate, giving task publishers a stated likelihood of acceptance rather than an opaque heuristic.","Because the loop is closed, an attacker that occasionally succeeds pays a long-term cost, making gradual trust-accumulation strategies less sustainable than in reputation systems decoupled from filtering.","The reputation update rule can be attached to other detect-and-filter defenses, and the final reputation gap then reflects that defense's TPR/FPR, as the paper demonstrates for M-Krum, FLAME, and NNM.","Scaling the federation from 20 to 100 workers keeps the false positive rate roughly stable, while the reputation mechanism tracks the underlying detector without artificially improving or degrading its trade-off."],"supporting_citations":[{"why":"Supplies the nearest-neighbor mixing recipe that R2-NNM extends and uses as its primary baseline.","marker":"[1]"},{"why":"Defines Multi-Krum, the Byzantine-robust selection rule used as the second stage of the R2-NNM pipeline.","marker":"[5]"},{"why":"Provides a reputation-aware decentralized FL framework whose workflow R2CFL follows and whose reputation model is a key comparison baseline.","marker":"[9]"},{"why":"Introduces FLAME, a clustering-based backdoor defense used as a baseline and a target for reputation characterization.","marker":"[22]"},{"why":"Introduces DeepSight, a clustering-based backdoor defense evaluated as a baseline, which the paper finds detects no malicious clients.","marker":"[24]"},{"why":"Defines the optimized model poisoning attack used to stress-test the defense and reputation model.","marker":"[25]"},{"why":"Defines the A3FL backdoor attack, the worst-case scenario used for the reputation separation analysis.","marker":"[30]"},{"why":"Defines the Neurotoxin durable backdoor attack used in the attack evaluation.","marker":"[31]"}],"fun_headline_variants":["Trust scores mirroring filter's TPR/FPR stop stealthy FL takeover","Reputation loop from robust filter blocks gradual poisoning attacks","R2-NNM ties reputation to defense stats, defeats adaptive attackers","Stealthy attackers lose trust when reputation mirrors filter's TPR/FPR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that reputation faithfully mirrors a defense's TPR/FPR rests on simulating each client's acceptance as an independent coin flip with fixed probabilities across rounds, so if acceptance events are correlated over time or an attacker adapts to reputation feedback, the clean separation between benign and malicious scores may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Trust scores mirroring filter's TPR/FPR stop stealthy FL takeover","Reputation loop from robust filter blocks gradual poisoning attacks","R2-NNM ties reputation to defense stats, defeats adaptive attackers","Stealthy attackers lose trust when reputation mirrors filter's TPR/FPR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":2016,"prompt_tokens":918,"completion_tokens":1098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":1022}},"tokens_in":534,"tokens_out":1098,"duration_ms":7948,"temperature":1.0,"reasoning_tokens":1022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:31:13.037607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run R2CFL end-to-end with a stealthy attacker who monitors its own reputation and alternates between clean and poisoned updates, then check whether malicious reputation stays below roughly 0.3 while benign reputation stays above 0.7 after 100 rounds; alternatively, measure the autocorrelation of the acceptance indicator and show that substantial positive autocorrelation is present, which would violate the independent-Bernoulli assumption behind the reported separation.","supporting_citations":[{"cited_title":"In: International Conference on Artificial Intelligence and Statistics","cited_arxiv_id":null,"evidence_quote":"Supplies the nearest-neighbor mixing recipe that R2-NNM extends and uses as its primary baseline."},{"cited_title":"In: IEEE Network Operations and Management Symposium (NOMS)","cited_arxiv_id":null,"evidence_quote":"Provides a reputation-aware decentralized FL framework whose workflow R2CFL follows and whose reputation model is a key comparison baseline."},{"cited_title":"In: 31st USENIX Security Symposium (USENIX Security)","cited_arxiv_id":null,"evidence_quote":"Introduces FLAME, a clustering-based backdoor defense used as a baseline and a target for reputation characterization."},{"cited_title":"In: Ndss (2021)","cited_arxiv_id":null,"evidence_quote":"Defines the optimized model poisoning attack used to stress-test the defense and reputation model."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Defines the A3FL backdoor attack, the worst-case scenario used for the reputation separation analysis."},{"cited_title":"In: Proceedings of the 39th International Confer- ence on Machine Learning (ICML)","cited_arxiv_id":null,"evidence_quote":"Defines the Neurotoxin durable backdoor attack used in the attack evaluation."}],"review_version":1}