{"id":"ae5fc5cb-70f1-4eaa-92e7-9021440ec80f","arxiv_id":"2502.08008","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning BERT with the fixed-mini-batch FSRDP differential privacy accountant in a 4-client federated setup yields roughly 1-2% lower accuracy than RDP while using constant memory.","lead":"Researchers propose FLIP, a framework that adds a human privacy expert to tune privacy and learning parameters in differentially private federated learning, and test it by fine-tuning BERT on GLUE tasks. They adopt a fixed-mini-batch privacy accountant (FSRDP) to keep memory use constant, reporting about 1-2% accuracy loss versus the standard RDP accountant.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RDP baseline appears to be run with fixed-size minibatches even though RDP's privacy analysis assumes Poisson subsampling; if so, the reported ε=6/10 for RDP is invalid and the 1.33%/1.9% accuracy gaps are not measured at equal privacy.","rationale":"The reader's weakest assumption focused on whether FSRDP's noise scales are correctly applied under add-remove adjacency in multi-round federated training. I identify a closely related but more specific accounting mismatch: the RDP baseline appears to be run with fixed-size minibatches even though the RDP accountant's privacy analysis assumes Poisson subsampling. This is load-bearing because the paper's headline numbers are equal-privacy accuracy gaps between FSRDP and RDP. If the RDP runs used fixed-size batches without using a fixed-size-capable accountant, the RDP column's true ε is larger than reported, making FSRDP look worse than it is at matched privacy (or making RDP look better than it should). This does not contradict the reader's conditional verdict; rather, it provides a sharper technical condition that should be verified before the quantitative claims are accepted. The concern is concrete and testable by inspecting the released code, so it is appropriate to keep the verdict conditional rather than rejecting outright. I agree with the reader partially: the general area is privacy-accounting validity, but the specific failure mode I identify concerns the RDP baseline rather than FSRDP itself.","tokens_in":12008,"tokens_out":10698,"duration_ms":97712,"concrete_test":"Inspect the public repository (github.com/KasraAhmadi/FL-Privacy-LLM) training loop to determine how minibatches are constructed for the RDP baseline. If the RDP DataLoader uses a fixed batch_size=550 sampler, rerun the RDP experiments under proper Poisson subsampling (each example included independently with probability 550/N) and recompute the Table I accuracy and the headline accuracy gaps. Alternatively, recompute the true privacy level of the current fixed-batch RDP runs using FSRDP or the fixed-size RDP accountant of Wang et al. [25]; if the true ε exceeds the nominal 6/10, the equal-privacy comparison is invalid and the reported 1.33%/1.9% gaps need correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim compares FSRDP and RDP at the same nominal privacy levels, so both accountants must be evaluated under the sampling scheme each one assumes. Section 2.3 states that RDP's privacy analysis relies on Poisson subsampling, which produces random-size minibatches, while FSRDP is specifically designed for fixed-size minibatches. However, Section 4.1 reports a single fixed 'batch size of 550' and says noise is 'scaled by the standard deviation divided by the batch size (550)'; no Poisson sampler is described for the RDP runs. With a standard fixed-size DataLoader (the default in Flower and Transformers), each RDP minibatch has exactly 550 examples, but the RDP accountant's noise values in Table III were computed under the assumption that each example is included independently with probability 550/N. Because fixed-size subsampling without replacement provides weaker privacy amplification than Poisson subsampling, the true ε for the RDP column would be larger than the nominal 6 or 10. The reported average accuracy reductions of 1.33% (ε=10) and 1.9% (ε=6) would then not be equal-privacy comparisons: RDP would be operating at a weaker privacy level while FSRDP is correctly calibrated. This directly affects the central claim that adopting FSRDP costs only a small accuracy reduction relative to the state-of-the-art RDP accountant. The manuscript must either confirm that Poisson sampling was used for RDP, or recompute the RDP baseline with a valid fixed-size accountant before the accuracy gap can be interpreted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLIP, an interactive framework for privacy-preserving federated learning that introduces a human privacy practitioner to select DP and FL parameters and adopts the FSRDP accountant to support fixed-size minibatches with constant memory usage. The authors fine-tune BERT-base (109M parameters) on three GLUE tasks (QQP, QNLI, SST2) with four clients and four data-partitioning policies, comparing non-private training, RDP, and FSRDP at epsilon=10 and epsilon=6. They report maximum accuracies over five federated rounds and noise-scale tables, and claim average accuracy reductions of 1.33% (epsilon=10) and 1.9% (epsilon=6) for FSRDP relative to RDP, with stable memory usage.","tokens_in":12330,"tokens_out":6930,"duration_ms":59907,"significance":"If the results are valid, the paper provides a useful empirical study of a practical deployment concern: variable-size minibatches in DP-SGD can cause memory instability on resource-constrained federated clients. The open-source implementation, the coverage of multiple datasets and partition policies, and the explicit comparison between two accountants are commendable. However, the central quantitative claim—that FSRDP achieves nearly the same accuracy as RDP at the same privacy level—depends on a valid RDP baseline, which the manuscript does not establish because the RDP runs appear to use fixed-size minibatches rather than the Poisson subsampling assumed by the RDP accountant. The human-practitioner component, a headline contribution, is not empirically evaluated. The paper's value is therefore conditional on a substantial revision that addresses the baseline validity and measurement rigor.","major_comments":[{"comment":"The RDP baseline does not appear to satisfy the privacy guarantee attributed to it. Section 2.3 states that RDP's privacy amplification relies on subsampling, which in standard implementations (e.g., Opacus) is Poisson subsampling and produces random-size minibatches. Section 4.1, however, reports a fixed batch size of 550 and describes noise scaled by the standard deviation divided by 550, with no mention of a Poisson sampler. If the RDP runs used fixed-size minibatches without replacement, then the noise values in Table III for RDP are too low for the stated epsilon under the add-remove adjacency relation, and the actual privacy cost is larger than the nominal epsilon=6 or 10. Consequently, the reported average accuracy reductions of 1.33% and 1.9% are not measured at equal privacy. Please either run the RDP baseline with true Poisson subsampling, or use a fixed-size-valid accountant (e.g., Wang et al. or FSRDP) for the baseline, and recompute the comparison.","section":"Section 2.3, Section 4.1, Table III"},{"comment":"The accuracy results are reported as single numbers without error bars or multiple seeds. The differences between RDP and FSRDP are often one to two percentage points, which may be within run-to-run variability for fine-tuning BERT on these tasks. Moreover, the abstract's average accuracy reductions (1.33% for epsilon=10 and 1.9% for epsilon=6) do not match the values computed from Table I: a simple average of the 12 RDP-vs-FSRDP differences gives about 1.4% and 2.0% respectively. Please report means and standard deviations over at least three seeds, and correct the abstract numbers to match the reported tables.","section":"Section 4.2, Table I, Abstract"},{"comment":"The paper claims in the abstract and elsewhere to have 'achieved stable memory usage,' but no memory measurements are reported in the experiments. Figure 1, which illustrates the memory-stability property, appears to be taken from the FSRDP paper [11] without explicit attribution in the caption, and the current experiments do not measure memory consumption during federated training. Please either add empirical memory measurements in the FL setting or temper the claim to state that FSRDP offers a constant-memory property by design, as established in [11], rather than as a result demonstrated here.","section":"Abstract, Section 4.1, Figure 1"},{"comment":"The framework's key novelty—the integration of a human privacy practitioner—is not empirically validated. Section 3.3 describes the practitioner's role in choosing epsilon, batch size, and accountant, but Section 4 contains no experimental comparison between FLIP with and without practitioner input. The experiments only vary partition policies and accountants, which are pre-defined configurations rather than interactive human decisions. Please add an ablation that quantifies the effect of the practitioner's involvement, or explicitly reframe this component as a design proposal that is not evaluated in the current study.","section":"Section 3.3, Section 4"}],"minor_comments":[{"comment":"There is a typo in the opening sentence: 'we reivew We examine' should be corrected to a single clean sentence such as 'We examine three key areas of the literature.'","section":"Section 2, first paragraph"},{"comment":"The phrase 'fine-tuneing' should be 'fine-tuning'.","section":"Section 1.1"},{"comment":"The noise injection description is ambiguous: 'scaled by the standard deviation divided by the batch size (550)' should clarify the exact DP-SGD formula used, including whether the clipping norm C appears in the numerator and whether noise is added to the summed or averaged gradient.","section":"Section 4.1"},{"comment":"In Section 2.3, the citation 'RDP [12], [21]' incorrectly groups Abadi et al. [21] with the RDP definition; RDP should be attributed solely to Mironov [12], while [21] is the DP-SGD paper.","section":"References"},{"comment":"BERT-base has 109 million parameters and is often not considered a 'large language model' in the current literature; consider using 'pre-trained transformer' or providing a justification for the terminology.","section":"Title and Abstract"},{"comment":"The paper says it uses 'the GLUE dataset' but only evaluates three of its tasks (QQP, QNLI, SST2); please state explicitly 'three GLUE tasks' to avoid overgeneralization.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The central comparison relies on a self-cited accountant [11] whose authors overlap with the current paper's authors. While citing one's own prior work is normal, the paper would be stronger if it used an independent implementation or at least explicitly acknowledged the self-reference and provided cross-checks. The RDP baseline validity issue is the most serious technical problem and must be resolved before the paper can be considered for publication. The abstract numbers also need to be reconciled with Table I. The paper's scope as an empirical systems study is reasonable, but the revision should focus on making the privacy accounting airtight and the utility comparisons statistically grounded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nThe real substance here is the experiments: the authors take the FSRDP accountant (fixed-size minibatch RDP from their own NeurIPS paper) and run it through a federated fine-tuning pipeline for BERT on three GLUE tasks with four data-partition policies. The noise tables and accuracy tables are new, the setup is described well enough to reproduce, and the code is public. That is a legitimate extension, and for someone deploying DP-SGD on memory-constrained clients, the constant-memory point is relevant.\n\nThe problem is that the RDP baseline is probably not run under the sampling scheme that gives it the claimed privacy. RDP's analysis in this paper assumes Poisson subsampling, which gives variable-size minibatches. The experiments use a fixed batch size of 550 with no Poisson sampler described. If the RDP noise was precomputed under Poisson but applied to fixed-size batches, then the actual privacy level of the RDP runs is weaker than the nominal epsilon=6 or 10. So the reported 1.33% and 1.9% average accuracy reductions are not measured at equal privacy. The paper comes close to this point in the Section 4.2.2 discussion of adjacency relations, but it never addresses the sampling mismatch.\n\nOther soft spots are smaller. The averages in the abstract don't match Table 1 (I get roughly 1.4% and 2.0%). There are no error bars or multiple seeds. The 'stable memory usage' claim is asserted from the FSRDP paper rather than measured here. And the human privacy practitioner is a conceptual framework, not an experimentally tested component.\n\nThe self-citation of [11] is worth flagging, but the accuracy numbers are genuine measurements, so I don't see it as disqualifying.\n\nBottom line: this deserves a serious referee, but it needs major revision. The authors need to either run RDP with actual Poisson subsampling, or use a fixed-size accountant for the baseline, before the accuracy-gap claims can be interpreted. They also need error bars and a direct memory measurement. The work is useful enough to justify the cleanup.\n\nI'd bring it to reading group to talk about the sampling issue, but I wouldn't cite it in its current form.","headline":"A useful first application of fixed-minibatch DP accounting to federated fine-tuning, but the RDP baseline is likely miscalibrated and the headline accuracy gaps don't hold at equal privacy.","tokens_in":12903,"tokens_out":3939,"would_cite":false,"duration_ms":32084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed-minibatch privacy accountant can replace the standard variable-batch RDP method in federated fine-tuning of BERT, losing only 1.33% average accuracy at epsilon=10 while keeping memory use constant.","keywords":["differential privacy","federated learning","fixed-size minibatch","Rényi differential privacy","privacy-utility trade-off","BERT fine-tuning","GLUE benchmark","privacy practitioner"],"falsifier":"An independent computation of FSRDP noise for the exact configurations in Table III should reproduce the reported standard deviations; if it yields lower noise for the same $(\\epsilon, \\delta)$, or if a membership-inference audit of the fine-tuned models indicates an effective epsilon larger than the target (10 or 6), then the claimed privacy-utility trade-off would collapse. Alternatively, a controlled comparison of FLIP with and without practitioner input would test the framework's second claim about the value of human expertise.","tokens_in":11812,"feed_emoji":"🔒","tokens_out":8628,"duration_ms":67090,"temperature":0.7,"pith_summary":"This paper introduces FLIP, a framework that puts a human privacy practitioner in charge of choosing differential-privacy and federated-learning parameters before fine-tuning a large language model. Its main technical claim is that a fixed-minibatch privacy accountant, FSRDP, can replace the standard variable-batch RDP accountant during federated fine-tuning of BERT, holding memory usage constant across training rounds while losing only 1.33 percentage points of accuracy on average at $\\epsilon = 10$ and 1.9 points at $\\epsilon = 6$ relative to RDP. The experiments span three GLUE tasks and four data-partitioning policies with four clients, and show that data distribution and accountant choice interact with privacy level. If the claim holds, resource-limited devices can join private federated fine-tuning without out-of-memory failures and with only a small utility penalty.","feed_headline":"Fixed-batch DP costs only 1.33% accuracy in federated BERT fine-tuning","feed_subtitle":"Constant-memory privacy for phone-scale clients comes within 1.9 points of non-private accuracy.","key_machinery":"The machinery that carries the argument is the FSRDP privacy accountant, a method for computing Rényi differential privacy guarantees for DP-SGD when minibatches are drawn with a fixed size rather than by Poisson subsampling. The paper uses FSRDP to pre-compute the Gaussian noise standard deviation needed for a target privacy cost $(\\epsilon, \\delta)$, injects that noise into client-side clipped gradients via DP-SGD after each federated training round, and measures the resulting accuracy. The key property FSRDP supplies is constant memory usage across training iterations, which the paper contrasts with the variable-size minibatches of RDP that can cause out-of-memory failures on low-end devices. The comparison is made concrete by Table III, which lists the required noise for each accountant, dataset, and partitioning policy at $\\epsilon = 6$ and $\\epsilon = 10$.","core_discovery":"The paper's central claim is that FLIP is the first privacy-preserving federated learning framework to adopt a fixed-minibatch differential privacy accountant, FSRDP, in place of the Poisson-subsampling-based RDP accountant, and that this substitution costs an average of 1.33% accuracy at $\\epsilon = 10$ and 1.9% at $\\epsilon = 6$ compared with the RDP baseline. A supporting claim is that a privacy practitioner who selects privacy cost, batch size, and data-partition strategy based on application needs can reduce the accuracy gap to the non-private model from as much as 5% down to about 2%. The paper also argues that when privacy guarantees must extend to the replace-one adjacency relation, RDP would require roughly double the noise in Table III while FSRDP's listed noise already covers both add-remove and replace-one adjacency, so the utility advantage of RDP disappears in that setting. These claims are supported by fine-tuning BERT on QQP, QNLI, and SST2 under IID, Linear, Square, and Exponential client data partitions.","pith_inferences":["The paper never directly tests whether a human privacy practitioner outperforms a fixed heuristic; a natural experiment would compare FLIP to an automated rule that selects the same parameters from data-distribution statistics.","Because the memory-stability claim is demonstrated in a simulated single-GPU setting, a test on actual mobile hardware is needed to confirm that constant memory usage prevents real out-of-memory failures.","The reported accuracy decline in later training rounds under FSRDP suggests that an adaptive noise schedule could recover some utility loss; the paper mentions this possibility but does not evaluate it.","Combining Table III with the discussion in reference [11] implies that FSRDP becomes strictly preferable to RDP when both add-remove and replace-one adjacency must be satisfied, since FSRDP's noise does not change while RDP's would double."],"forward_implications":["Federated fine-tuning can be run with a fixed minibatch size and constant memory footprint, allowing phones and other low-end clients to participate without out-of-memory failures.","The average accuracy reduction of 1.33% at $\\epsilon = 10$ and 1.9% at $\\epsilon = 6$ relative to RDP is small enough that fixed-minibatch accounting is a practical drop-in replacement for many applications.","Data partitioning matters: with FSRDP at $\\epsilon = 6$ and large datasets, IID partitions achieve the best accuracy, while with RDP, exponential partitions are best; practitioners need to know their data distribution to set parameters.","If a deployment requires protection under both add-remove and replace-one adjacency, FSRDP's pre-computed noise works as-is, whereas RDP would need roughly twice the noise, negating its accuracy advantage.","The privacy practitioner's parameter choices can reduce the gap to non-private accuracy from about 5% to about 2%, so interactive parameter tuning has measurable value in private federated fine-tuning."],"supporting_citations":[{"why":"Supplies the FSRDP accountant and the particular noise values used to achieve the target epsilon under fixed-size minibatches.","marker":"[11]"},{"why":"Supplies the RDP accountant that serves as the state-of-the-art baseline for accuracy and noise comparison.","marker":"[12]"},{"why":"Supplies the DP-SGD algorithm through which both accountants' noise is applied to clipped gradients.","marker":"[21]"},{"why":"Supplies the FedAvg aggregation rule used to combine client updates in the federated simulation.","marker":"[19]"},{"why":"Supplies the BERT model that is fine-tuned in all experiments.","marker":"[18]"},{"why":"Supplies the GLUE benchmark and the three datasets (QQP, QNLI, SST2) used for evaluation.","marker":"[15]"},{"why":"Provides an earlier fixed-size subsampling analysis for (epsilon, delta)-DP that the paper cites as composing poorly over multiple steps, motivating FSRDP.","marker":"[24]"},{"why":"Provides an earlier fixed-minibatch RDP accountant that the paper cites as less tight, motivating FSRDP.","marker":"[25]"}],"fun_headline_variants":["Constant-memory DP for federated LLMs costs 1.33% accuracy","Human-guided DP cuts federated BERT privacy cost to 1.33%","First fixed-memory DP framework for federated LLMs","Trade-off tool trims DP accuracy hit to 1.33% in federated BERT","FSRDP: Fixed-noise DP for phone-scale federated LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the FSRDP accountant's pre-computed noise scales truly deliver the claimed $(\\epsilon, \\delta)$-differential privacy for the multi-round federated DP-SGD procedure under the add-remove adjacency relation, since the paper takes those noise values from reference [11] and does not independently verify them.","fun_headline_variants_meta":{"raw":{"variants":["Constant-memory DP for federated LLMs costs 1.33% accuracy","Human-guided DP cuts federated BERT privacy cost to 1.33%","First fixed-memory DP framework for federated LLMs","Trade-off tool trims DP accuracy hit to 1.33% in federated BERT","FSRDP: Fixed-noise DP for phone-scale federated LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1417,"prompt_tokens":1089,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":705,"tokens_out":328,"duration_ms":4318,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:08:53.210413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent computation of FSRDP noise for the exact configurations in Table III should reproduce the reported standard deviations; if it yields lower noise for the same $(\\epsilon, \\delta)$, or if a membership-inference audit of the fine-tuned models indicates an effective epsilon larger than the target (10 or 6), then the claimed privacy-utility trade-off would collapse. Alternatively, a controlled comparison of FLIP with and without practitioner input would test the framework's second claim about the value of human expertise.","supporting_citations":[{"cited_title":"Differentially pri- vate stochastic gradient descent with fixed-size minibatches: Tighter RDP guarantees with or without replacement,","cited_arxiv_id":null,"evidence_quote":"Supplies the FSRDP accountant and the particular noise values used to achieve the target epsilon under fixed-size minibatches."},{"cited_title":"R ´enyi differential privacy,","cited_arxiv_id":null,"evidence_quote":"Supplies the RDP accountant that serves as the state-of-the-art baseline for accuracy and noise comparison."},{"cited_title":"Communication-efficient learning of deep networks from decentral- ized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the FedAvg aggregation rule used to combine client updates in the federated simulation."},{"cited_title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the GLUE benchmark and the three datasets (QQP, QNLI, SST2) used for evaluation."},{"cited_title":"Privacy amplification by sub- sampling: Tight analyses via couplings and divergences,","cited_arxiv_id":null,"evidence_quote":"Provides an earlier fixed-size subsampling analysis for (epsilon, delta)-DP that the paper cites as composing poorly over multiple steps, motivating FSRDP."},{"cited_title":"Subsampled r´enyi differential privacy and analytical moments accountant,","cited_arxiv_id":null,"evidence_quote":"Provides an earlier fixed-minibatch RDP accountant that the paper cites as less tight, motivating FSRDP."}],"review_version":1}