{"id":"c5f2671e-e2e9-4814-bd87-cc6445b7c569","arxiv_id":"2412.05183","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Privacy drift, the change in membership inference risk during incremental training, correlates strongly with training accuracy across centralized and federated settings.","lead":"This paper introduces 'privacy drift', the idea that a machine learning model's exposure to membership inference attacks changes as it is trained on new data over time. Experiments in centralized and federated learning show that this exposure tracks the model's training accuracy closely, no matter how many clients are used.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MIA AUC values that ground the headline correlation and client-count independence are produced by an unspecified attack pipeline, so the central empirical claim may be measuring the attack rather than privacy drift.","rationale":"The paper's contribution is explicitly claimed as 'Introduction and Formalization of Privacy Drift,' but Section IV contains no formal definition; the only quantified quantity is MIA AUC from Section V. The central strongest claim—independence from client count and strong accuracy-AUC correlation—is therefore only as strong as the MIA measurement. The reader flagged the missing attack details in Sections III-B and V; I agree and sharpen it: because the defender exposes confidence scores (Section III-A), a naive confidence-based attack would yield an AUC that tracks training accuracy almost by construction, making 'privacy drift' a restatement of confidence or overfitting drift. This does not require assuming bad faith; a standard Shokri-style or loss-based attack may be fine, but the paper has to say so. Independent support: the code link and eight permutations are real evidence, but they cannot be used to validate the correlation until the attack is identified. A concrete code inspection plus an independent MIA re-run would settle whether the effect is robust. Given the reader already conditioned acceptance on this, my recommendation is no change to the conditional verdict.","tokens_in":7264,"tokens_out":4203,"duration_ms":47220,"concrete_test":"Run the released code (github.com/soumyaxyz/Privacy-Preserving-Federated-Learning) with the exact MIA construction, then re-compute MIA AUC using a second, independent attack (e.g., a loss-threshold attack or LiRA with shadow models) on the same incremental checkpoints, ideally with 95% confidence intervals over permutations and seeds. If the training-accuracy/AUC Pearson correlations in Figures 4 and 6 remain high (greater than about 0.8) under the independent attack and the client-count independence still holds, the concern is resolved; if the correlation changes materially or reverses for some client counts, the claimed phenomenon is at least partly an artifact of the original attack setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Section V-C—that MIA AUC strongly tracks training accuracy and that this 'privacy drift' is independent of FL client count—rests entirely on the MIA AUC values, yet the paper never specifies how those values are produced. Section III-B only states that the adversary has query access and 'creates an attack model'; it does not describe the attack architecture, shadow training procedure, feature set, member/non-member balance, or whether a fresh attack is trained at each incremental phase. Because the attacker is given API access to confidence values (Section III-A), a confidence-threshold or likelihood-based attack would make MIA AUC increase almost mechanically as training accuracy and confidence rise, so the reported correlation could be an artifact of the attack definition rather than evidence about the target model's private-information leakage. The 'formalization' of privacy drift promised in the introduction is also absent: no definition or equation for privacy drift appears in Section IV, so the only operational proxy is this underspecified AUC. Without the attack pipeline, the paper's headline correlation and client-count independence cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the concept of \"privacy drift\" for federated learning (FL), defined informally as the evolving vulnerability of an incrementally trained model to membership inference attacks (MIA). Using CIFAR-100 converted to a 20-superclass task, the authors partition data into four non-IID splits, train EfficientNet models sequentially over permutations of these splits, and measure training accuracy, test accuracy, and MIA AUC under uniform and additive test paradigms. They also vary the number of FL clients (centralized, 2, 5, 10) and report that MIA AUC tracks training accuracy and that this behavior is independent of client count. The paper claims to formalize privacy drift and to provide empirical evidence of its dynamics.","tokens_in":7471,"tokens_out":3749,"duration_ms":40274,"significance":"If the central empirical claim holds—that MIA vulnerability in incremental FL tracks training accuracy and is insensitive to the number of clients—the paper would provide a useful framing for privacy monitoring in dynamically updated models. The experimental design is controlled and the code is publicly available, which are strengths. However, the absence of a formal definition of privacy drift, the lack of any specification of the MIA attack pipeline, and the absence of statistical inference for the client-count independence claims currently prevent the paper from delivering on its stated contributions. The non-circular portion of the work—the correlation between training accuracy and MIA AUC—is plausibly interesting, but it is not yet convincingly established given the underspecified attack methodology.","major_comments":[{"comment":"The paper promises in the contribution list (Section I) to \"introduce and formalize\" privacy drift, but Section IV, titled \"Privacy Drift in FL: Dynamics and Trade-offs,\" contains no formal definition, equation, or operational metric. The Introduction defines privacy drift as a \"gradual increase\" in vulnerability, whereas Section IV describes \"fluctuations\" and a \"complex and non-monotonic\" relationship. This inconsistency leaves the central construct undefined. Please provide a precise mathematical definition, such as a drift score or rate based on MIA AUC or another specified measure, and state exactly which hypothesis (e.g., monotonic increase, correlation with training accuracy, or client-count independence) the experiments are designed to test.","section":"Section I and Section IV"},{"comment":"The MIA AUC values that underlie all of the paper's quantitative claims are produced by an attack pipeline that is never described. Section III-B only states that the adversary has query access and \"creates an attack model,\" but does not specify the attack architecture (e.g., shadow model training as in Shokri et al. [4]), the features used (confidence vector, predicted label, etc.), the number of shadow models, the member/non-member balance, or whether a fresh attack is trained at each incremental phase. Without this information, the reported AUC values could be artifacts of a misconfigured or biased attack setup, and the claimed correlation with training accuracy might reflect the attack's own behavior rather than a property of the target model. Please include a full description of the attack pipeline, ideally with pseudocode, and validate the attack by reporting its performance on a control model trained on data disjoint from the target members.","section":"Section III-B and Section V"},{"comment":"The headline claim that \"privacy drift is independent of the number of clients in FL\" is not supported by any statistical analysis. Figure 6 shows variation in the correlation across the individual experiments, but the text reports no error bars, confidence intervals, hypothesis tests, or effect sizes. The statement that there is a \"strong correlation\" is qualitative; no Pearson r values or p-values are given for Figures 4 or 6. To support the client-count independence claim, please report the per-permutation and per-client-number correlation coefficients with measures of uncertainty, and formally test whether the correlation differs across client counts (e.g., via bootstrap resampling or Fisher z-transformation).","section":"Section V-C"},{"comment":"Figures 5 and 7 show results averaged over eight permutations, but no variance or standard deviation is displayed. With only eight permutations of four partitions, the stability and representativeness of the averaged curves are unclear. The choice of eight permutations is also not justified; the text says this is \"a reasonable number of variations.\" Please add error bars or shaded regions to the averaged plots, report the per-permutation results (at least in an appendix), and justify the permutation sampling procedure.","section":"Section V-A and Figures 5–7"}],"minor_comments":[{"comment":"The abstract says experiments are on CIFAR-100, but the paper actually reduces to the 20 superclasses and calls this CIFAR-20. Please clarify this in the abstract and first mention of the dataset.","section":"Abstract and Section V-A"},{"comment":"\"fed-average\" should be written as \"FedAvg\" with the appropriate citation to McMahan et al. [20], and the model aggregation method should be described in one sentence for readers unfamiliar with the abbreviation.","section":"Section V-C"},{"comment":"The term \"honest but curious\" is typically used in secure multiparty computation to denote a semi-honest adversary; here it is used more loosely to mean an attacker who follows the query interface but attempts inference. Please define the term explicitly in the context of this threat model.","section":"Section III-B"},{"comment":"The caption states that Figure 4 presents the Pearson correlation, but no numerical values or axes units are described in the text. Please report the actual correlation coefficients, or at minimum state the ranges observed, in the caption or the main text.","section":"Figure 4"},{"comment":"The paragraph on \"ML Model Attack Accuracy and the Privacy Drift\" is largely definitional and repeats the abstract. It could be condensed, and the listed factors (data drift, model evolution, etc.) should be mapped to specific experiments in Section V.","section":"Section II-C"},{"comment":"Reference [18] is cited as prior work by the same authors; in Section IV this is referred to as \"Ahamed et al.\" but since it is the authors' own work, consider phrasing it as \"we previously showed\" to avoid ambiguous self-citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a preliminary version of a study that extends the authors' prior work (arXiv:2407.19119). The novelty is limited to the incremental-training aspect and the client-count variation, but the lack of attack-pipeline details and statistical rigor would need to be addressed before publication. The GitHub repository announced in the footnote should be checked by the editors to verify that it contains the exact scripts used to generate Figures 3–7. If the authors can supply the missing formalization and attack details, a revised version could be a legitimate empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper tracks MIA AUC over incremental training phases in federated learning and finds it tracks training accuracy across permutations and client counts. The 'privacy drift' label is new; the underlying observation is not — it's the familiar overfitting-to-membership-inference link from Shokri et al. and Yeom et al. (the latter isn't cited). The experiments are systematic: eight permutations of four non-IID splits, two test paradigms, centralized plus 2/5/10 clients, and the code is public. That's real reproducible work, and the claim that the correlation holds across those settings is supported by the plots.\n\nThe main problems are at the measurement layer. Section III-B says the adversary 'creates an attack model' but never specifies its architecture, shadow training, features, or member/non-member balance. Since the defender's API returns confidence values, a confidence-threshold attack would make MIA AUC rise mechanically with training accuracy. Until that pipeline is described, the central correlation could be an artifact of the attack definition. Also, the promised 'formalization' of privacy drift never appears — no definition or equation, just prose. And there are no error bars, significance tests, or baseline comparisons against, say, a randomly guessing attacker or standard central training. The client-count independence claim is stated from visual inspection of three client counts.\n\nNone of this sinks the empirical pattern; it just means the paper is a preliminary empirical note, not the framework paper it claims to be. For practitioners deploying FL and wanting a cheap privacy-risk monitor, training accuracy as a proxy is useful, but they should read this with the attack-pipeline caveat.\n\nI'd send it to peer review — a serious reviewer can ask for the missing details, and the public code makes verification feasible. But I wouldn't cite it as establishing 'privacy drift' as a formal concept.","headline":"A useful empirical observation wrapped in a new term; the correlation is real, but the paper oversells 'formalization' and leaves the MIA pipeline a black box.","tokens_in":8021,"tokens_out":1872,"would_cite":false,"duration_ms":19779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Privacy leaks track training accuracy as models learn incrementally","keywords":["privacy drift","membership inference attack","federated learning","incremental learning","concept drift","accuracy-privacy trade-off","non-IID data"],"falsifier":"Re-run the incremental training protocol with a fully specified membership inference attack (e.g., shadow-model training with balanced member/non-member sets) and test whether the correlation between training accuracy and MIA AUC persists; if the correlation disappears or AUC stays near chance under a properly specified attack, the privacy drift phenomenon as defined is an artifact of the attack configuration rather than a property of the evolving model.","tokens_in":7093,"feed_emoji":"📈","tokens_out":7196,"duration_ms":60281,"temperature":0.7,"pith_summary":"This paper introduces \"privacy drift,\" a concept analogous to concept drift: as a model is trained incrementally on new data, its vulnerability to membership inference attacks changes over time. Using a CIFAR-20 dataset split into four non-IID partitions, the authors show that the area under the curve (AUC) of membership inference attacks tracks training accuracy across training phases. This correlation holds in both centralized training and federated learning with two, five, and ten clients, leading the authors to conclude that privacy drift is independent of the number of clients. The finding matters because it suggests that accuracy improvements in incrementally updated models can quietly increase privacy risk, so static privacy evaluations may miss a moving target.","feed_headline":"Privacy leaks track training accuracy as models learn incrementally","feed_subtitle":"In federated and centralized settings alike, membership inference risk moves with accuracy, independent of client count.","key_machinery":"The central object is \"privacy drift,\" defined as the gradual variation in a model's vulnerability to membership inference over its training lifecycle, measured by the AUC of a membership inference attack. The experimental machinery consists of a CIFAR-20 dataset (the 20 superclasses of CIFAR-100) split into four non-IID partitions (A, B, C, D), on which an EfficientNet model is trained sequentially in different permutations, with test sets evaluated under both a uniform paradigm (a fixed mixed test set) and an additive paradigm (test sets that grow with each training phase). Privacy drift is operationalized by computing the Pearson correlation between training accuracy and MIA AUC across training phases, in centralized mode and in federated mode using federated averaging, with client counts of two, five, and ten.","core_discovery":"The paper's central claim is that privacy drift is a real, measurable phenomenon: the leakage of private information from a machine learning model, quantified by membership inference attack AUC, varies as the model undergoes incremental training on non-IID data. Across eight permutations of four data partitions and under two testing paradigms (uniform and additive), the authors observe a strong Pearson correlation between training accuracy and MIA AUC, even when test accuracy diverges from training accuracy. Repeating the same experiments in a federated setting with one, two, five, and ten clients, they find the correlation persists, and they conclude that the presence of privacy drift does not depend on the number of participating clients. The paper frames this as a dynamic accuracy-privacy trade-off: improvements in model performance tend to be accompanied by greater susceptibility to membership inference.","pith_inferences":["If the correlation reflects the model's confidence calibration, then interventions that sharpen confidence (e.g., temperature scaling) could alter privacy drift without changing accuracy; this is a testable extension the paper does not explore.","Privacy drift could double as an early-warning diagnostic: a sudden spike in training accuracy during incremental updates may predict an imminent increase in membership inference risk.","The client-count independence implies that privacy-preserving defenses in FL should target the training dynamics themselves (e.g., per-phase differential privacy budgets) rather than assuming more clients dilute leakage.","A formal definition of privacy drift as a rate of change of leakage per unit of accuracy gain would make the concept quantifiable for comparing learning algorithms."],"forward_implications":["Training accuracy can serve as a live proxy signal for privacy risk in incrementally updated models, since MIA AUC tracks it across phases and settings.","Privacy evaluations of continually learning systems must be repeated over time; a single static assessment at deployment will miss drift in leakage risk.","Federated learning does not remove privacy drift: the correlation holds whether training is centralized or distributed among up to ten clients.","The accuracy-privacy trade-off is phase-dependent, so techniques that boost accuracy mid-training may increase membership inference risk in the same period."],"supporting_citations":[{"why":"Defines the concept drift phenomenon that privacy drift is modeled on.","marker":"[2]"},{"why":"Supplies the membership inference attack methodology used to measure privacy leakage.","marker":"[4]"},{"why":"Prior work establishing the accuracy-privacy trade-off in federated learning and its independence from client count, extended here to the temporal dimension.","marker":"[18]"},{"why":"Provides the federated averaging aggregation algorithm used in the federated experiments.","marker":"[20]"},{"why":"Describes an approach for membership inference attacks and mitigation in federated learning, used as a basis for attack measurement.","marker":"[3]"}],"fun_headline_variants":["Privacy drift: leakage tracks training accuracy","Incremental learning inflates membership inference risk","Accuracy up, privacy down: drift in federated learning","Privacy leak correlates with accuracy in incremental models","Client count irrelevant: privacy drift follows accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the membership inference attack AUC values it reports are a faithful measure of private information leakage, but it does not describe how the attack model is trained or how member and non-member test sets are balanced, so the observed correlation with training accuracy could be an artifact of the attack setup.","fun_headline_variants_meta":{"raw":{"variants":["Privacy drift: leakage tracks training accuracy","Incremental learning inflates membership inference risk","Accuracy up, privacy down: drift in federated learning","Privacy leak correlates with accuracy in incremental models","Client count irrelevant: privacy drift follows accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1281,"prompt_tokens":923,"completion_tokens":358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":539,"tokens_out":358,"duration_ms":4124,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:50:11.962040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the incremental training protocol with a fully specified membership inference attack (e.g., shadow-model training with balanced member/non-member sets) and test whether the correlation between training accuracy and MIA AUC persists; if the correlation disappears or AUC stays near chance under a properly specified attack, the privacy drift phenomenon as defined is an artifact of the attack configuration rather than a property of the evolving model.","supporting_citations":[{"cited_title":"Incremental learning from noisy data,","cited_arxiv_id":null,"evidence_quote":"Defines the concept drift phenomenon that privacy drift is modeled on."},{"cited_title":"Mia-bad: An approach for enhancing membership inference attack and its mit- igation with federated learning,","cited_arxiv_id":null,"evidence_quote":"Describes an approach for membership inference attacks and mitigation in federated learning, used as a basis for attack measurement."}],"review_version":1}