{"id":"bbff311e-42ab-4966-991a-ff5548f4a914","arxiv_id":"2505.22108","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Compliance-weighted noise allocation in federated healthcare learning claims no accuracy loss versus uniform noise, but its differential privacy guarantee applies only to the aggregator dataset, not client data.","lead":"This paper proposes a federated learning framework in which each hospital receives a noise level based on a compliance score, allowing less compliant clinics to join without forcing extra noise on highly compliant ones. The study reports that including lower-compliance sites can improve accuracy on two small medical image datasets, but the formal privacy guarantee covers only the server's small aggregator dataset, not the patient data held by clients.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central no-utility-cost claim is not evidenced: Experiment 1 vs. Experiment 4 confounds compliance weighting with a 4x increase in training data, and the claimed +0.1 pp match to uniform DP is not traceable in Table 3.","rationale":"The reader's weakest assumption centers on privacy. I partially agree, but I think the more load-bearing vulnerability is the utility evidence. The abstract explicitly limits formal guarantees to the aggregator dataset and states that client-level DP requires secure aggregation, so the privacy scope is disclosed; a rejection on privacy grounds depends on reading the title and introduction's 'privacy-preserving' framing as overclaiming beyond the stated caveats. The central claim as quoted includes 'no utility cost,' and this is where the experiments are structurally weak. The comparison in the paper is under-powered: no standard deviations are reported despite five seeds; the compliance scores are assigned by hand; the noise schedule is an uncalibrated linear formula; and the main 'inclusion helps' comparison conflates compliance with data volume and quality. The natural controlled experiment, with the same clients and data and only the noise allocation varied, would settle whether compliance weighting itself has any effect. If it does not, the method reduces to a trust-scoring wrapper around server-side DP. This is a concrete and testable weakness, so I view it as more load-bearing than the already-acknowledged privacy limitation. I therefore keep the reader's rejection verdict while shifting the emphasis from the privacy caveat to the unsupported utility claim.","tokens_in":9395,"tokens_out":7086,"duration_ms":84240,"concrete_test":"Hold the client set and data fixed: use the same 16 clients and the same per-client data in both arms, and compare compliance-weighted per-client noise to uniform DP noise with the same mean noise multiplier (for example, set uniform noise equal to the average of the compliance-derived multipliers). Report per-strategy accuracy, standard deviation over at least 10 seeds, and realized per-client and global epsilon. If the difference is within sampling noise for every strategy, the no-utility-cost claim survives; if accuracy changes by more than the standard error or direction differs across strategies, the central claim fails. As a second check, run the same 16 clients without any compliance weighting to measure how much of the Experiment 1 gain comes from added data volume alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'no utility cost' claim is not supported by the reported experiments. Experiment 1 introduces 12 lower-compliance clients whose data were artificially degraded (random cropping, Gaussian noise, contrast reduction) and compares them to Experiment 4 with only the 4 compliant clients. This changes at least three variables at once: number of clients, total data volume, and compliance-weighted noise. The +4.5 to +6.8 pp gains can be explained by a 4x larger training set, not by the allocation mechanism. The claim that compliance-weighted allocation matched uniform server-side DP at equal mean noise (+0.1 pp) is not traceable in Table 3: per-strategy differences between Experiment 1 and Experiment 6 range from -7.89 to +18.54 pp on BreastMNIST and from -9.46 to +6.54 pp on PneumoniaMNIST, with no mean-noise levels, standard deviations, or per-configuration epsilons reported. The noise multiplier formula Nm = (1.0 - Sc) + Min Noise Multiplier is asserted without calibration or sensitivity analysis, so the 'no utility cost' conclusion cannot be separated from the particular arbitrary noise schedule. The aggregator-only DP limitation, by contrast, is disclosed in the abstract and Section 5, so it is a scope caveat rather than an unstated hidden assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a compliance-aware federated learning framework that maps each institution's compliance score Sc to a per-step Gaussian noise multiplier Nm = (1.0 - Sc) + Min Noise Multiplier for server-side DP-SGD, and introduces a web-based compliance scoring tool aligned with several healthcare and security standards. The method is evaluated with five FL strategies on PneumoniaMNIST and BreastMNIST over six experimental configurations and five seeds; Table 3 reports accuracy, precision, recall, and F1 for each configuration. The authors claim that including 12 lower-compliance clients improves accuracy over a compliant-only baseline and that compliance-weighted noise allocation matches uniform server-side DP at equal mean noise, yielding 'no utility cost.' The formal DP guarantee is explicitly scoped to a small aggregator dataset under a semi-honest aggregator, with client-level DP deferred to future work.","tokens_in":9606,"tokens_out":11740,"duration_ms":124656,"significance":"If the central claim were established, the framework would be a useful practical contribution: it offers a concrete compliance scoring template, a formal DP accounting, and a mechanism for resource-constrained clinics to participate without dedicated DP hardware. The use of public benchmarks, five FL strategies, and five seeds provides a reasonable evaluation skeleton, and the limitations section is candid about the aggregator-only guarantee. However, the significance is currently undermined by the experimental design: the headline comparison is confounded with data volume, the equal-mean-noise comparison is not reproducible from Table 3, and the natural controlled comparison within Table 3 (Experiment 1 vs Experiment 3) actually suggests that compliance-weighted noise degrades accuracy for most strategies. The privacy guarantee, with cumulative epsilon on the order of 500 to 1400, is also much weaker than the 'privacy-preserving' framing suggests. The framework and scoring tool may still be of interest, but the empirical validation as submitted does not support the paper's main claims.","major_comments":[{"comment":"The central 'no utility cost' claim rests on comparing Experiment 1 (4 compliant + 12 non-compliant clients) with Experiment 4 (4 compliant clients only). These conditions differ in client count and training-set size, so the reported gains of +4.5 to +6.8 pp on BreastMNIST could be due to the roughly four-fold larger dataset rather than to the compliance-weighted mechanism. If, as Table 1 and the description of a fixed 16-client split imply, Experiment 1 and Experiment 3 use the same 16 client subsets and differ only in the assigned compliance scores, then Table 3 shows that compliance-weighted noise degrades accuracy substantially for most strategies (e.g., FedMedian: 70.12 vs 83.67 on PneumoniaMNIST and 50.01 vs 73.24 on BreastMNIST; FedAvg: 82.43 vs 85.64 and 66.98 vs 73.68). The paper never reports this controlled comparison, and the headline claim is therefore not supported by the current experiments.","section":"Abstract, §3 (Table 3)"},{"comment":"The claim that compliance-weighted allocation 'matched uniform server-side DP at equal mean noise (+0.1 pp)' is not traceable from the reported data. No mean-noise levels, per-client noise multipliers, or per-configuration epsilon values are given, and the per-strategy accuracy differences between Experiment 1 and Experiment 6 in Table 3 are large and mixed (for example, on BreastMNIST they range from about -7.9 to +18.5 pp). To support this claim, the authors need to specify how equal mean noise was enforced and report seed-level means and standard deviations; Table 3 currently reports single numbers despite the stated five-seed protocol.","section":"§3 (Table 3), Abstract"},{"comment":"The formal DP guarantee covers only the small aggregator dataset under a semi-honest aggregator, because clients send un-noised updates to the server and noise is added after receipt. This is disclosed in the abstract and limitations, but the introduction and discussion repeatedly describe the framework as privacy-preserving without that qualification. The reported cumulative epsilon values of about 1434 (BreastMNIST) and 513 (PneumoniaMNIST) at delta=10^-5 are very large and should be discussed as providing only weak privacy for the aggregator dataset; the practically relevant client-level privacy scenario requires secure aggregation, which is not implemented, and the limitations section also notes that first-round client updates lack DP.","section":"§2 (Algorithm 1), §5"},{"comment":"The mapping Nm = (1.0 - Sc) + Min Noise Multiplier is asserted without calibration or sensitivity analysis, and the compliance scores come from the authors' own scoring tool with arbitrary weights (Equation 1, Table 2). Because Nm is a direct algebraic function of Sc, the observation that lower-compliance clients receive more noise is true by construction; what needs to be demonstrated is that this particular schedule preserves utility, and the paper provides no sensitivity analysis over the compliance weights or the noise schedule. This makes the 'no utility cost' conclusion dependent on unexamined modeling choices.","section":"§2 (Noise Multiplier Calculation, Compliance Scoring Mechanism)"}],"minor_comments":[{"comment":"The sentence 'gaining 1%–15% accuracy across strategies' overstates Table 3, where Experiment 1 versus Experiment 4 shows a decrease for FedMedian on both datasets (70.12 vs 79.44 on PneumoniaMNIST; 50.01 vs 62.84 on BreastMNIST).","section":"§4 Discussion"},{"comment":"The sentence 'Notably ,The experiment' contains a punctuation and spacing error that should be corrected.","section":"§3 Results"},{"comment":"The text 'see Algo 18' should refer to Algorithm 1 in the pseudocode.","section":"§2 Experimental Setup"},{"comment":"The parameter 'eta' in the DPTrain call is used for a noise multiplier, but the notation is not defined in the pseudocode or the surrounding text.","section":"§2 Pseudocode"}],"recommendation":"reject","confidential_remarks":"To the editor: I recommend rejection. The central utility claim is not merely unsupported; under the natural reading of Table 1, the paper's own Table 3 suggests that compliance-weighted noise degrades accuracy compared with treating the same clients as compliant (Experiment 1 vs Experiment 3). This is a load-bearing problem that would require new experiments and a substantial reframing to address. I would not object to a future submission that positions the contribution as a compliance-scoring tool with a properly controlled utility analysis against uniform DP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper is a reasonable engineering sketch, but the central empirical claim does not hold. The compliance-scoring checklist and the per-client noise-multiplier idea could be useful if someone builds on them. The experiments as run do not show that compliance-weighted noise has no utility cost.\n\nWhat's actually new is modest. Per-client adaptive DP noise is already in the literature, and the authors cite Li et al. 2024 and Wang et al. 2020. The contribution is a domain-specific compliance scoring tool, a weighted checklist across HIPAA/GDPR/NIST/etc., mapped linearly to a noise multiplier. That is a legitimate design pattern for healthcare FL, but it is not a new privacy mechanism. I would credit the authors for being transparent: the abstract and Section 5 clearly state that the formal DP guarantee covers only the aggregator dataset, not client data, and that client-level DP would require secure aggregation. So the privacy concern is disclosed, though it does mean the 'privacy-preserving' framing should be read narrowly.\n\nThe soft spot is the utility claim, and it is load-bearing. Experiment 1 adds 12 lower-compliance clients with degraded data and compares against Experiment 4 with only 4 compliant clients. That changes number of clients, total data volume, and compliance-weighted noise at once. The +4.5 to +6.8 pp gains can be explained by a 4x larger training set. The claim that compliance-weighted noise matched uniform server-side DP at equal mean noise (+0.1 pp) is not traceable in Table 3: per-strategy differences between Experiments 1 and 6 range from -7.89 to +18.54 pp on BreastMNIST, and no mean-noise levels or standard deviations are reported. The noise formula Nm = (1 - Sc) + min is asserted without derivation or calibration, and compliance scores are manually assigned. Table 3 reports single numbers despite five seeds, and the abstract itself notes the pooled gain is not significant. None of this looks fraudulent, and the authors flag parts of it, but the central conclusion is not supported by the evidence as presented.\n\nWho gets value? Someone building a compliance-aware FL demo might want to look at the scoring tool and the experiment design, but a reader should not take the utility results at face value. The paper needs a reanalysis where client count and data volume are matched or ablated, and where noise levels and error bars are reported.\n\nRecommendation: I would not accept this for peer review in current form. The central comparison is confounded, and a serious referee would spend most of the time asking for obvious control experiments. I would keep the skeleton in mind, but the paper as written overclaims.","headline":"Useful engineering skeleton for compliance-aware server-side DP in healthcare FL, but the headline 'no utility cost' claim is not supported by the reported experiments.","tokens_in":10267,"tokens_out":3648,"would_cite":false,"duration_ms":38674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that compliance-weighted server-side differential privacy lets lower-compliance institutions join federated learning without degrading accuracy, by scaling the noise each client's update receives to that client's…","keywords":["federated learning","differential privacy","compliance scoring","adaptive noise allocation","server-side DP","medical image classification","healthcare AI","inclusivity"],"falsifier":"Capture a client's raw update on the server before the compliance-weighted noise is added and run a gradient-inversion attack on it: if recognizable patient images are recovered, the framework's client-data privacy claim fails for a curious aggregator, and the first round is especially exposed because the paper states it has no DP. Separately, repeating the 12-plus-4 versus 4-only comparison with many more seeds would show whether the reported accuracy gains are real or within random variation.","tokens_in":1679,"feed_emoji":"🩺","tokens_out":2525,"duration_ms":107668,"temperature":0.7,"pith_summary":"The paper is trying to establish that healthcare federated learning does not have to choose between privacy and inclusion. Its mechanism assigns each institution a compliance score and maps that score to a per-client Gaussian noise scale, with noise applied on the server after clients upload their model updates. On two medical image benchmarks with 16 simulated clinics, the authors report that adding twelve lower-compliance clinics to four high-compliance ones improved BreastMNIST accuracy by 4.5 to 6.8 points for four of five aggregation strategies, and that compliance-weighted noise matched uniform server-side differential privacy at equal mean noise within 0.1 points. In other words, the paper argues that per-site noise control can be auditable and comes with no measured utility penalty. The formal $(\\epsilon,\\delta)$ bound, however, applies only to the aggregator's small dataset; the paper states that client-level privacy would require secure aggregation, which is left to future work.","feed_headline":"Per-clinic noise lets low-compliance clinics join federated learning","feed_subtitle":"Adding 12 lower-compliance clinics matched or beat a compliant-only baseline, so inclusivity need not cost accuracy.","key_machinery":"The load-bearing object is the compliance-to-noise mapping: $S_c=\\frac{\\sum_i w_i s_i}{\\sum_i w_i}$ and $N_m=(1-S_c)+\\text{min}$, where $S_c$ is a weighted average of 12 auditable compliance factors and min is a tiny baseline noise (set to $10^{-10}$ in the experiments). This mapping turns an institutional assessment into a deterministic noise scale, so per-site privacy burden is auditable and adjustable. The companion mechanism is server-side DP-SGD: the server trains one epoch on a small aggregator dataset using the noise-adjusted client updates, which lets the global model adapt to perturbed updates before aggregation. Together they let an aggregator add client-specific noise without requiring clinics to run DP-compliant training hardware themselves.","core_discovery":"On its own terms, the paper's central discovery is that a compliance score can be treated as a privacy budget: lower compliance means more noise, higher compliance means almost none, and this allocation is more inclusive than uniform noise. Concretely, a weighted compliance score $S_c$ computed from factors such as encryption, anonymization, and interoperability maps to a noise multiplier $N_m=(1-S_c)+\\text{min}$, and the server runs DP-SGD on a small aggregator dataset with those per-client noise levels before global aggregation. Across six experimental configurations on PneumoniaMNIST and BreastMNIST, the configuration with 12 lower-compliance plus 4 compliant clients beat the compliant-only configuration by +4.5, +6.8, +5.2, and +1.6 points for FedAvg, FedMedian, FedProx, and FedYogi, and by -4.1 for FedAdam; the pooled gain was +2.8 points with five seeds, reported as not significant. When matched for mean noise, compliance-weighted allocation matched uniform server-side DP within 0.1 points, and the first-round noise cost 1.3 points on BreastMNIST and 2.5 points on PneumoniaMNIST under FedAvg.","pith_inferences":["Beyond the paper: the compliance-to-noise map can be read as a data-quality filter; replacing subjective compliance scores with objective data-quality metrics would turn the mechanism into a general robustness strategy against low-quality or malicious clients.","Beyond the paper: matching uniform noise at equal mean noise does not imply equal effective privacy, because a curious server observes raw client updates before noise is applied; under that threat model the realized protection is weaker than the reported aggregator-dataset epsilon.","Beyond the paper: since each client receives a different noise scale, the framework invites per-client privacy accounting, tracking each institution's cumulative epsilon rather than a single global number."],"forward_implications":["A clinic with compliance score 0.3 automatically receives about 0.7 noise multiplier while a score-1.0 clinic receives near-zero noise, making the privacy burden visible and proportional to the trust score.","Because all DP noise is added on the server, resource-constrained clinics can participate without purchasing specialized DP-compliant hardware or running local DP-SGD.","Adding 12 lower-compliance clients to 4 compliant clients raised BreastMNIST accuracy by 4.5 to 6.8 points for FedAvg, FedMedian, FedProx, and FedYogi and lowered FedAdam by 4.1 points, with the pooled +2.8-point gain not significant at five seeds.","At equal mean noise, compliance-weighted allocation matched uniform server-side DP within 0.1 points of accuracy, so the paper claims no utility penalty from the adaptive allocation.","The formal $(\\epsilon,\\delta)$ privacy guarantee covers the aggregator dataset under a semi-honest aggregator; protecting client-level data requires secure aggregation, which the paper leaves to future work."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation algorithm that all experimental configurations build on.","marker":"[22]"},{"why":"Provides the PneumoniaMNIST and BreastMNIST benchmarks used for every reported accuracy number.","marker":"[33]"},{"why":"Defines the $(\\epsilon,\\delta)$ differential-privacy guarantee the framework claims for the aggregator dataset.","marker":"[9]"},{"why":"Supplies the DP-SGD implementation used on the server to add noise to client updates.","marker":"[34]"},{"why":"Is the prior server-side DP approach the paper adapts into compliance-weighted noise allocation.","marker":"[31]"},{"why":"Documents the disparate accuracy cost of uniform DP noise, which motivates per-client allocation.","marker":"[3]"},{"why":"Supplies the federated learning orchestration used to run clients, rounds, and aggregation.","marker":"[4]"}],"fun_headline_variants":["Compliance-weighted noise lets low-compliance clinics join FL without accuracy loss","Privacy budget per clinic: lower compliance, more noise, same accuracy","No utility cost for inclusive federated learning with compliance-aware noise","Matching uniform DP: compliance-based noise boosts inclusion, preserves accuracy","Let low-compliance clinics in: noise scaled by compliance keeps FL accuracy"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The privacy claim rests on the server being semi-honest and on each clinic's compliance score being accurate and honestly reported, because clients send un-noised updates to the server and the promised noise is added only after those updates arrive.","fun_headline_variants_meta":{"raw":{"variants":["Compliance-weighted noise lets low-compliance clinics join FL without accuracy loss","Privacy budget per clinic: lower compliance, more noise, same accuracy","No utility cost for inclusive federated learning with compliance-aware noise","Matching uniform DP: compliance-based noise boosts inclusion, preserves accuracy","Let low-compliance clinics in: noise scaled by compliance keeps FL accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1410,"prompt_tokens":1195,"completion_tokens":215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":811,"completion_tokens_details":{"reasoning_tokens":125}},"tokens_in":811,"tokens_out":215,"duration_ms":2911,"temperature":1.0,"reasoning_tokens":125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:15:10.378969+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Capture a client's raw update on the server before the compliance-weighted noise is added and run a gradient-inversion attack on it: if recognizable patient images are recovered, the framework's client-data privacy claim fails for a curious aggregator, and the first round is especially exposed because the paper states it has no DP. Separately, repeating the 12-plus-4 versus 4-only comparison with many more seeds would show whether the reported accuracy gains are real or within random variation.","supporting_citations":[{"cited_title":"In: Artificial intelligence and statistics","cited_arxiv_id":null,"evidence_quote":"Supplies the FedAvg aggregation algorithm that all experimental configurations build on."},{"cited_title":"Scientific Data 10(1), 41 (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the PneumoniaMNIST and BreastMNIST benchmarks used for every reported accuracy number."},{"cited_title":"Foundations and Trends® in Theoretical Computer Science 9(3–4), 211–407 (2014)","cited_arxiv_id":null,"evidence_quote":"Defines the $(\\epsilon,\\delta)$ differential-privacy guarantee the framework claims for the aggregator dataset."},{"cited_title":"In: Procee dings of the 14th ACM Conference on Recommender Systems","cited_arxiv_id":null,"evidence_quote":"Is the prior server-side DP approach the paper adapts into compliance-weighted noise allocation."},{"cited_title":"Differential Privacy Has Disparate Impact on Model Accuracy","cited_arxiv_id":"1905.12101","evidence_quote":"Documents the disparate accuracy cost of uniform DP noise, which motivates per-client allocation."}],"review_version":1}