{"id":"34b8e095-794e-4014-a84b-0d0d68c6e986","arxiv_id":"2412.06157","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A comprehensive survey of membership inference attacks and defenses in federated learning, proposing update-based and trend-based attack taxonomies and four defense categories.","lead":"This paper is a survey that organizes existing research on membership inference attacks and defenses in federated learning into two attack families and four defense categories. It is a useful reference for researchers entering the area, though its completeness claim is weakened by internal inconsistencies and a non-transparent selection process.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The update/trend taxonomy is not shown to be disjoint or exhaustive: Zhu et al. [59] is classified under both categories, so the survey's unique organizational map needs a decision rule or a restatement.","rationale":"The reader's weakest assumption is the same one I would stress: the update-based versus trend-based dichotomy is not established as exhaustive and disjoint. The duplicate listing of Zhu et al. [59] is a concrete place to inspect. If the original paper's method is a single attack using both gradients and loss trajectories, then the two category names describe evidence sources rather than disjoint classes; if it is two attacks, the table conflates attack instances with papers. Either way, the current text does not give readers a reproducible way to classify future attacks or to verify that all published FL MIAs are covered. The absence of a search or inclusion protocol reinforces this, because 'comprehensive' cannot be checked without one. These issues are fixable, and the survey is still useful as a curated map of the area, so a reject is not warranted. The reader's CONDITIONAL verdict already captures the appropriate bar, so I would leave the verdict unchanged.","tokens_in":33487,"tokens_out":6087,"duration_ms":62006,"concrete_test":"Re-annotate every attack entry in Tables 3-5 from the cited source papers: for each entry, record the input type actually used (single snapshot of parameters or gradients, multiple snapshots, or a sequence of scalar indicators over rounds). Then check whether any entry appears in more than one leaf; Zhu et al. [59] currently does. If the source paper contains multiple distinct algorithms, the survey should state that rows classify algorithms, not papers, and should separate the algorithms. If one algorithm fits both leaves, the taxonomy is not disjoint and needs a priority rule or a third category. Additionally, scan the FL MIA papers cited in the survey (e.g., [47], [82], [97]) and one external search for 'membership inference federated learning' 2017-2024; if any published attack cannot be assigned to either leaf, the exhaustiveness half of the claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim includes a 'unique taxonomy' that partitions FL MIAs into update-based and trend-based attacks (Section 3.5). For that taxonomy to organize the field, the two categories should be well-defined, mutually exclusive, and able to cover all surveyed attacks. They are not shown to be. Zhu et al. [59] is listed under 'Gradient difference' in Table 3 (update-based) and under 'Loss trajectory' in Table 3 (trend-based), and Section 4 describes the same work as both extending gradient differences across rounds (Section 4.1.1) and using multi-round model updates and loss trajectories (Section 4.2.1). This may be because the original paper proposes two attacks, but the survey does not say so and gives no rule for multi-method papers. Moreover, the definitions themselves overlap: 'leverage one or more historical versions of the target model' versus 'analyze the trajectory of specific indicators' both describe sequences of historical models, and a gradient difference over consecutive rounds is also a trajectory. No argument establishes exhaustiveness, and the survey lacks a literature-selection protocol that would support the 'most studies' claim. The taxonomy's value as an organizational map is therefore load-bearing and currently under-specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript surveys membership inference attacks (MIAs) and defenses in federated learning (FL). It proposes a two-category taxonomy for attacks—update-based and trend-based—and a four-category taxonomy for defenses (partial sharing, secure aggregation, noise perturbation, anomaly detection). It compares these with centralized learning settings, presents summary tables of attacks and defenses, and discusses future research directions. The paper claims to be the first survey specifically focused on MIAs and defenses in the FL domain.","tokens_in":33744,"tokens_out":2925,"duration_ms":28915,"significance":"If the taxonomy is accepted, the survey provides a useful organizational map of a rapidly growing area. The paper collects a broad set of recent works, including 2023–2024 results, and structures them into attack/defense tables that are convenient for readers entering the field. The comparisons with centralized learning and the discussion of open problems are valuable. However, the central contribution—the unique attack taxonomy—is not defined with enough precision to be reliable, and there are several inconsistencies in the reported metadata. These issues must be addressed before the paper can serve as a dependable reference.","major_comments":[{"comment":"The proposed update-based versus trend-based taxonomy is not shown to be disjoint or exhaustive. The same work, Zhu et al. [59], is classified under both 'Gradient difference' (update-based) in Table 3 and 'Loss trajectory' (trend-based) in Table 3, and Section 4 describes this work both as extending gradient differences across rounds (Section 4.1.1) and as using multi-round model updates and loss trajectories (Section 4.2.1). The paper provides no decision rule for papers that propose multiple attack methods, and no argument establishes that every surveyed attack falls into exactly one category. Because the paper's central claim is that this taxonomy is unique and organizes the field, the taxonomy must be either redefined so the categories are disjoint, or the paper must explicitly state how multi-method papers are handled and justify exhaustiveness.","section":"Section 3.5, Table 3, Sections 4.1.1 and 4.2.1"},{"comment":"The claim that this is 'the first survey of MIAs and defenses in the FL domain' requires qualification. The paper itself cites Reference [39] (a CSUR survey on MIAs in machine learning that includes FL works) and Reference [40] (a survey on defenses against MIAs). While these works are not dedicated exclusively to FL, the novelty claim as stated is too strong. The authors should either provide a precise definition of what counts as an FL-specific survey or soften the claim to 'the first dedicated FL-focused survey' with a comparison to the coverage in [39] and [40].","section":"Abstract and Section 1 (Contributions)"},{"comment":"The text repeatedly refers to 'Hue et al. [29]' when describing the source inference attack, but the reference list and the rest of the paper use 'Hu et al. [29]' (Hongsheng Hu et al.). This name error, combined with the inconsistent year in Table 4 for Truex et al. [97], suggests that the survey's citation metadata was not carefully cross-checked. For a survey whose value depends on accurate attribution, these errors need to be corrected across the manuscript.","section":"Section 4.2.1"}],"minor_comments":[{"comment":"The phrase 'deep leaning' should be 'deep learning'.","section":"Abstract"},{"comment":"Truex et al. [97] is listed with year 2018, but the reference list gives 2021 (IEEE TSC, with an online date of 2019). Please use the published version year consistently.","section":"Table 4"},{"comment":"The technique column for Zari et al. [35] reads 'Predication sequence'; this should be 'Prediction sequence'.","section":"Table 5"},{"comment":"The legend for the 'Unique' column uses the symbols '  ', 'G #', and '#' with spacing that is likely a formatting artifact; please ensure the legend is printed correctly and the symbols are consistent with the table body.","section":"Table 6"},{"comment":"In the paragraph 'Reasons for Information Leakage', the phrase 'in the black setting' should be 'in the black-box setting'.","section":"Section 6.1"},{"comment":"The definition of 'update-based attacks' as leveraging 'one or more historical versions of the target model' overlaps with the definition of 'trend-based attacks' as analyzing 'the trajectory of specific indicators'; consider clarifying the distinction by emphasizing the type of evidence (parameter/gradient values vs. derived scalar indicators) rather than the use of historical snapshots.","section":"Section 3.5 and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of ACM Computing Surveys and could be a valuable reference after a thorough revision. The main concern is the under-specified taxonomy; the authors need to either make the categories disjoint with clear decision rules or explicitly frame the taxonomy as a pragmatic organization rather than a strict partition. The citation inconsistencies, while not fatal, should be fixed during revision. I would not recommend acceptance in the present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a careful read for anyone coming into FL privacy. The survey is the first I know that focuses specifically on MIAs and defenses in FL, and it brings together recent work (including 2024 papers) that earlier surveys missed. The CL-vs-FL comparison in Tables 1, 6, and 9 is a nice organizational contribution, and the per-attack summaries in Tables 4 and 5 will help newcomers find entry points. The four-category defense structure (partial sharing, secure aggregation, noise perturbation, anomaly detection) is sensible and covers the main countermeasures. Those are real strengths.\n\nThe soft spots are real but fixable. The stress-test note lands: the two attack categories overlap. 'Leverage one or more historical versions' and 'analyze the trajectory' both describe observing a sequence of models. Zhu et al. [59] appears under both gradient-difference and loss-trajectory with no explanation of whether that paper proposes two attacks. The authors need a decision rule for multi-method papers or should describe the categories as overlapping perspectives rather than a clean partition. The 'unique taxonomy' claim is also a bit strong—similar categories appear in prior surveys—but the FL-specific framing is new enough.\n\nThere are smaller consistency issues: Table 4 lists Truex et al. [97] as 2018 while the reference list says 2021, reference [28] has an empty author field, and the paper never describes a systematic literature selection protocol, which makes the 'most studies' claim hard to verify. These are minor cleanliness problems, not load-bearing flaws.\n\nThe central argument—that FL MIA research is distinct from CL and deserves its own survey—holds. The taxonomy problem is an organizational weakness, not a fatal one. I'd recommend peer review with the expectation of moderate revisions: tighten the taxonomy definitions, acknowledge multi-attack papers, add a selection method, and fix the citation nits. The paper will be a useful resource for grad students and researchers entering FL privacy; specialists will use it as a reference but won't find much new.","headline":"Useful FL-specific MIA survey, but the update/trend taxonomy needs a cleaner decision rule before the 'unique' claim holds.","tokens_in":34256,"tokens_out":2578,"would_cite":true,"duration_ms":24663,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey of membership inference in federated learning claims the field splits into update-based and trend-based attacks, with four defense families.","keywords":["membership inference attacks","federated learning","privacy attacks","survey taxonomy","model updates","differential privacy","secure aggregation","source-level membership inference"],"falsifier":"Find a published federated-learning membership inference attack that cannot be assigned to either the update-based or the trend-based branch; the taxonomy is also falsified if no rule is given for resolving the paper's own dual listing of Zhu et al. [59], which currently appears in both branches.","tokens_in":33312,"feed_emoji":"🕵️","tokens_out":6264,"duration_ms":60427,"temperature":0.7,"pith_summary":"Federated learning lets phones and hospitals train a shared model without uploading raw data, but the model updates themselves can betray whether a specific record was part of someone's training set. This survey claims to be the first comprehensive review of those membership inference attacks and their defenses specifically in federated learning. It organizes the attack literature into two families: update-based attacks, which read membership from exchanged gradients or model snapshots, and trend-based attacks, which watch how a prediction, loss, or bias parameter evolves across training rounds. On the defense side it groups countermeasures into partial sharing, secure aggregation, noise perturbation, and anomaly detection. If the map is right, researchers gain a single structured reference for what has been tried, how attacks differ from centralized-learning attacks, and where the open problems are.","feed_headline":"Survey sorts federated-learning privacy attacks into two families","feed_subtitle":"Update-based and trend-based attacks are cataloged, with four defenses from noise to encryption.","key_machinery":"The machinery that carries the survey is the two-branch attack taxonomy. Update-based attacks treat one or more exchanged model updates as evidence; they include original-gradient attacks, gradient-difference attacks, shadow-training attacks on a local or global model, and attacks that modify the model's structure (for example, embedding ReLU-gated neurons that only member samples activate). Trend-based attacks instead follow an indicator across rounds—the prediction score of the ground-truth label, the training loss, adversarial robustness, or final-layer bias—and compare its trajectory between members and non-members. On the defense side, the organizing machinery is the four-category split into partial sharing, secure aggregation, noise perturbation, and anomaly detection, with the survey using these categories to tabulate each method's threat model, advantages, and limitations.","core_discovery":"On its own terms, this survey establishes a systematic map of membership inference attacks and defenses in federated learning, which it argues no earlier survey has provided. Its organizing proposal is that every FL membership inference attack exploits either the exchanged update itself—original gradients, gradient differences across rounds, shadow-trained models, or deliberately modified model structure—or the trajectory of an indicator such as prediction confidence, loss, adversarial robustness, or bias across training rounds. It pairs this with four defense families: withholding part of the update, cryptographically hiding updates, adding noise, and detecting malicious updates. The survey also argues that FL membership inference differs from centralized learning because attackers are insiders during training, see historical model versions, and can attack actively, and because a second, stronger target exists: source-level membership, which identifies which client's data a record belongs to.","pith_inferences":["Editorial inference: the taxonomy is presented as dividing the field, but the paper's own Table 3 lists Zhu et al. [59] under both update-based and trend-based attacks, so a stated rule for primary assignment, or a third 'hybrid' branch, would be needed to make the partition truly disjoint.","Editorial inference: a practical extension the paper gestures at but does not build is a shared evaluation harness where the same datasets, networks, and attack protocols compare all four defense families; the survey's comparison tables show why current numbers cannot be cross-compared.","Editorial inference: source-level membership—identifying which client's data holds a record—looks like the FL-specific privacy harm most likely to grow, since it leaks both the record and the institution, and tracking how defenses trade off against it is a natural next study."],"forward_implications":["A reader can use the two-branch attack taxonomy to place any new FL membership inference attack and to see which threat models it addresses, including record-level versus source-level goals and passive versus active strategies.","Each defense family protects a different slice of the threat space, so the survey implies that practical privacy in FL will usually require combining partial sharing, noise, or aggregation rather than relying on one mechanism.","Because FL attacks run during training and use historical model versions, defenses validated in centralized learning should not be assumed to transfer to the federated setting.","The field lacks a unified evaluation benchmark, and current attack metrics such as accuracy can mislead; future evaluations should include false-positive rates and test realistic non-IID, partial-participation settings.","Emerging frameworks such as peer-to-peer, blockchain-based, and vertical FL remain understudied, making them the most likely places where new membership inference attacks will appear."],"supporting_citations":[{"why":"Introduces FedAvg, the federated training protocol against which most surveyed attacks and defenses are defined.","marker":"[12]"},{"why":"Shows gradient differences in collaborative learning leak membership and evaluates partial sharing as a defense, anchoring both taxonomy branches.","marker":"[17]"},{"why":"Defines the shadow-training membership inference attack from centralized learning that several FL update-based attacks adapt.","marker":"[22]"},{"why":"Pioneers passive and active white-box membership inference on FL gradients and intermediate outputs, the origin of the update-based category.","marker":"[26]"},{"why":"Proposes source-level membership inference by comparing loss across local models, anchoring the trend-based loss-trajectory branch.","marker":"[29]"},{"why":"Provides gradient-difference and cosine attacks showing that members and non-members have distinguishable gradient distributions in overparameterized models.","marker":"[33]"},{"why":"The most relevant prior survey, covering membership inference in centralized learning, and the main contrast used to show the FL-specific gap.","marker":"[39]"},{"why":"A recent evaluation extending membership inference across communication rounds and non-target clients, cited in both attack branches of the taxonomy.","marker":"[59]"},{"why":"Introduces SecAgg, the secure aggregation protocol that the survey analyzes as a defense and notes can still leak to clients with global-model access.","marker":"[127]"}],"fun_headline_variants":["Federated learning survey: two attack families, four defenses","FL membership attacks: survey splits by update vs trend","Survey catalogs FL membership inference attacks and defenses","Two families of FL membership attacks, four defenses","FL privacy survey: attack taxonomy and defense groups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's value rests on the claim that every published membership inference attack in federated learning fits exactly one of the two categories, update-based or trend-based, with no attack left out and no attack needing dual listing.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning survey: two attack families, four defenses","FL membership attacks: survey splits by update vs trend","Survey catalogs FL membership inference attacks and defenses","Two families of FL membership attacks, four defenses","FL privacy survey: attack taxonomy and defense groups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3347,"prompt_tokens":895,"completion_tokens":2452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2377}},"tokens_in":511,"tokens_out":2452,"duration_ms":17557,"temperature":1.0,"reasoning_tokens":2377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:56:48.222294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a published federated-learning membership inference attack that cannot be assigned to either the update-based or the trend-based branch; the taxonomy is also falsified if no rule is given for resolving the paper's own dual listing of Zhu et al. [59], which currently appears in both branches.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces SecAgg, the secure aggregation protocol that the survey analyzes as a defense and notes can still leak to clients with global-model access."}],"review_version":1}