Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Optimizing Canaries for Privacy Auditing with Metagradient Descent

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Metagradient-optimized canaries improve black-box one-run privacy audit lower bounds for DP-SGD image classifiers by more than 2x in some regimes, and the canaries transfer from a small non-private ResNet-9 to larger DP-trained Wide…

desk verdict Useful audit-optimization idea, but the Steinke column in Table 2 rests on an invalid use of the theorem; the MMC column is the sound part and still shows the intended effect. read the letter →

arxiv 2507.15836 v1 pith:Z4AKNNPH submitted 2025-07-21 cs.LG cs.CR

Matteo Boglioni , Terrance Liu , Andrew Ilyas , Zhiwei Steven Wu This is my paper
classification cs.LGcs.CR
keywords privacyauditingdifferentialDP-SGDcanaryoptimizationmetagradientdescentmembershipinferenceblack-boxone-run
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the special 'canary' examples an auditor inserts into a training set can themselves be designed, rather than randomly sampled, to make differential-privacy audits more sensitive. It proposes using metagradient descent, differentiating through the training process with respect to the canary pixels, to maximize the loss gap between canaries that are included in training and canaries that are left out. On CIFAR-10 with DP-SGD-trained Wide ResNets, the optimized canaries raise the empirical epsilon lower bound by more than 2x relative to random or mislabeled canaries under both one-run audit procedures tested. Because optimization runs on a small non-private ResNet-9, the canaries are cheap to produce and transfer to larger private models and to private finetuning. If the transferability holds broadly, auditors can precompute strong canaries for a model family and tighten black-box privacy lower bounds at negligible additional cost.

What carries the argument

The load-bearing object is the loss-gap surrogate objective of Equation (3), $\phi(w)=\sum_i (\mathbb{1}\{z_i\in C_{\mathrm{in}}\}-\mathbb{1}\{z_i\in C_{\mathrm{out}}\})L(w,z_i)$, which rewards canaries that the trained model fits when inserted and fails to fit when withheld. The optimization machinery is metagradient descent: canary pixels are treated as continuous metaparameters $z$, the training algorithm maps them to final weights $w=A(z)$, and the REPLAY method [EIC+25] computes the metagradient $\nabla_z \phi(A(z))$ efficiently enough for a small ResNet-9. Each meta-iteration randomly splits the canaries into included and withheld sets, trains a non-private model, computes the metagradient, and updates the canary pixels, so the resulting canaries are robust across random seeds and data orderings. The surrogate exists because the audit procedures themselves are non-differentiable; the loss gap is a smooth stand-in for audit success.

What would settle it

Run the same two audit protocols with metagradient canaries optimized on a different dataset, or with a different DP-SGD noise multiplier, and check whether the empirical epsilon still beats random and mislabeled canaries; if the advantage disappears or reverses while the theoretical bound is unchanged, the transferability claim would fail. A cheaper check is to increase the number of audit runs from five to roughly fifty and see whether the median advantage over baselines survives.

Watch

Extended reading notes

Core claim

The central claim is that the canary set, not just the auditing procedure, is a tunable lever for black-box privacy auditing, and that gradient-based optimization can turn it into a stronger audit. The paper defines a surrogate objective $\phi(w)=\sum_i (\mathbb{1}\{z_i\in C_{\mathrm{in}}\}-\mathbb{1}\{z_i\in C_{\mathrm{out}}\})L(w,z_i)$, the difference in training loss between canaries placed in and out of the training set, and optimizes it by computing metagradients of the final model loss with respect to canary pixels using the REPLAY method. This yields canaries that the model memorizes when present and fails to generalize to when absent, which is exactly the signal that membership inference exploits. The authors show that in a black-box, last-iterate, one-run setting, these canaries outperform random and mislabeled baselines under both the [SNJ23] and [MMC24] audit procedures, and that canaries optimized on a small non-private model remain effective for auditing larger DP-trained Wide ResNets, both from scratch and in private finetuning.

Load-bearing premise

The load-bearing premise is that canaries optimized to maximize the in/out loss gap during non-private ResNet-9 training on CIFAR-10 will also maximize the audit score of a larger DP-SGD-trained Wide ResNet, a transfer across architecture, privacy regime, and data sampling that the paper supports empirically but only on one dataset and five audit runs per cell.

Editorial extensions

If this is right

  • Black-box auditors can treat canary selection as part of the audit design and obtain tighter empirical epsilon lower bounds without any additional access to the training process.
  • Because canaries optimized on a small non-private model transfer to larger DP-trained models, the optimization cost is paid once per canary set rather than once per audit run.
  • The improvement appears under both one-run audit protocols examined, so the gain is not tied to a single guessing game.
  • The method also strengthens audits of private finetuning of pretrained networks, where inserted canaries must survive adaptation of an already-trained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the loss-gap surrogate frames canary optimization as targeted memorization, so stronger optimizers or larger canary budgets should push the empirical lower bound closer to the theoretical epsilon ceiling; this is a testable prediction the paper does not make.
  • Beyond the paper: the transferability result hints that canaries could be precomputed once for an architecture family and reused across datasets or tasks, turning canary optimization into an infrastructure cost for auditors; only transfer from ResNet-9 to Wide ResNet on CIFAR-10 is shown.
  • Beyond the paper: the same metagradient recipe may extend to multi-run audits, text models with discrete token canaries, or federated learning if a differentiable relaxation of the insertion mechanism can be designed; the paper leaves these settings unexamined.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to optimize the canary set used in one-run, black-box privacy auditing of DP-SGD image classifiers. The method embeds canaries as continuous metaparameters, trains a small non-private ResNet-9 with a surrogate objective equal to the loss gap between included and excluded canaries (Eq. 3), and computes metagradients through training using the REPLAY method of Engstrom et al. The optimized canaries are then transferred to audit larger Wide ResNet models trained with DP-SGD, using the one-run auditing procedures of Steinke et al. (Algorithm 2) and Mahloujifar et al. (Algorithm 3). The paper reports that the metagradient canaries improve empirical epsilon lower bounds over random and mislabeled canaries by up to roughly a factor of two in certain DP training and DP finetuning regimes (Table 2), and that the canaries are effective even when transferred across architectures and from non-private to private training.

Significance. If the reported effect is real and the audit numbers are valid lower bounds, the paper makes a useful practical contribution: it gives an efficient way to strengthen black-box one-run audits of DP-SGD without changing the auditing procedure or requiring access to intermediate model states. The use of external audit procedures to compute epsilon, the transfer experiments from ResNet-9 to Wide ResNet, and the inclusion of both average and median summaries are strengths. However, the significance is substantially tempered by a load-bearing validity issue in the Steinke et al. audit implementation and by the thin empirical support (five runs per cell, no error bars or per-run values). The central idea is not circular, because the reported epsilons come from the external auditing procedures rather than from fitting parameters to those target values, but the surrogate objective is closely aligned with the audit score, so the direction of improvement is unsurprising; the magnitude and reliability need stronger evidence.

major comments (4)
  1. [Section 4, Algorithm 2, Theorem 1] The statement in Section 4 that Algorithm 2 is run by splitting C randomly in half, instead of sampling each canary independently with probability 1/2, changes the distribution of the membership vector S. Theorem 1 assumes S is uniform over all of {−1,1}^m, whereas the balanced split makes S uniform over exactly-half subsets. The theorem's binomial null distribution is then not the correct null distribution for the executed audit: for m=2 and ε=0, a guess vector with one positive and one negative is correct on both entries with probability 1/2 under the balanced prior but with probability 1/4 under the theorem's binomial model. Consequently, the empirical epsilons in the Steinke columns of Table 2 are not valid calibrated lower bounds as stated, and the claims of improvement 'regardless of auditing procedure' and 'over 2x' are currently supported only by the MMC24 columns. Please either revert to the Bernoulli sampling assumed by the theorem, derive and use the correct hypergeometric null distribution for the balanced design, or explicitly restrict the claims to the MMC24 procedure.
  2. [Table 2] Table 2 reports only the average and median over five runs, with no standard errors, confidence intervals, or per-run values. The large gaps between averages and medians in several baseline cells (e.g., Steinke random under DP training: 0.204 vs. 0.001; MMC24 random under DP training: 0.150 vs. 0.000) indicate that the averages are driven by a small number of high-epsilon runs. Since audit epsilons are nonnegative and skewed, the claim that metagradient canaries 'consistently surpass' baselines by up to 2x is not established by these summary statistics alone. Please report the full distribution of the five runs, or at least bootstrap confidence intervals and a paired comparison across seeds, so the reader can assess the stability of the reported improvements.
  3. [Algorithm 5 and Section 4] The meta-optimization procedure is underspecified. Algorithm 5 takes as input a number of meta-iterations N and an initialization C0, and Section 4 only says the canaries are optimized 'following Engstrom et al.' The values of N, the choice of C0 (random noise, random training images, etc.), the inner training schedule, the REPLAY memory and checkpointing settings, and the canary update rule are not reported. Without these details the method is not reproducible, and the sensitivity of the transfer results to these choices is unknown. Please add a complete hyperparameter table and an ablation over at least N and C0.
  4. [Section 5, Table 2] The central transferability assumption — that canaries maximizing Eq. (3) on non-private ResNet-9 training improve audits of DP-SGD-trained Wide ResNets — is tested only on CIFAR-10, with one DP-SGD noise multiplier per regime (σ=3.0 for DP training and σ=1.75 for DP finetuning) and five audit runs per cell. This is a correctness-risk concern rather than a circularity: Eq. (3) is aligned with the cross-entropy audit score, so the direction of improvement is expected, but the claimed magnitude and generality rest on a single dataset and a single privacy regime. A concrete strengthening would be to vary the noise multiplier and dataset, or to compare against canaries optimized directly on the audited architecture. At minimum, the abstract and conclusion should soften the generality claim to reflect the narrow empirical support.
minor comments (5)
  1. [Algorithm 5] Line 6 of Algorithm 5 refers to ϕ(θ_t), but only w_t and ϕ(w_t) are defined in the surrounding text; the notation should be made consistent.
  2. [Algorithm 5] Line 7 of Algorithm 5 updates Ci+1 from Ci, but the loop variable is Ct; please use a consistent index (e.g., Ct+1 ← update(Ct, ∇Ct)).
  3. [Section 3] The text says the optimization proceeds in 'T > 1 metasteps', but Algorithm 5 uses the parameter N for the number of iterations; please unify the notation.
  4. [Section 2.1.2 and Algorithm 2] The prose in Section 2.1.2 says canaries are 'randomly sampled from C with probability 1/2', while Algorithm 2 and Section 4 describe a balanced split; this inconsistency should be resolved, especially given the validity issue in the major comments.
  5. [Table 1] Table 1 gives learning rate, batch size, clipping norm, and noise multiplier, but not the number of training steps/epochs or the sampling probability q; these are needed for reproduction of the audited DP-SGD runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audit bounds are measured by external procedures, and the optimized canaries are not fitted to the reported epsilons.

full rationale

The paper's central derivation is self-contained. Canaries are optimized on a non-private ResNet-9 using the surrogate objective in Equation (3), which is the loss gap between inserted and non-inserted canaries. The reported empirical epsilons are then computed by the external one-run auditing procedures of Steinke et al. [SNJ23] and Mahloujifar et al. [MMC24] on separately trained DP-SGD Wide ResNet models. The optimization objective and the audit score share the same cross-entropy-based loss gap, so the direction of improvement is unsurprising, but the reported lower bounds are not fitted outputs of the optimization: they are measured on held-out target models with independent audit machinery. The self-citations present (Engstrom et al. [EIC+25] for the REPLAY metagradient primitive, and Liu et al. [LBF+25] as related work) are not load-bearing equivalence arguments; they provide computational tools and context, not the paper's empirical conclusions. The Section 4 balanced-split modification to Algorithm 2 deviates from Theorem 1's uniform-S assumption, which is a validity and correctness risk for the SNJ23 column, but it is not a circularity and therefore does not affect the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper contributes an empirical method. The audit lower-bound theorems and the REPLAY metagradient engine are imported from prior work. The key self-imposed premise is that optimizing the loss-gap surrogate on non-private small models transfers to larger DP-SGD models, an assumption validated only in a narrow experimental setting. No free parameters are fit to the reported epsilon values; instead, key optimization choices (canary initialization, meta-iteration count, REPLAY hyperparameters) are left unspecified, which limits reproducibility.

free parameters (2)
  • Canary initialization C0 = Not reported
    Algorithm 5 requires an initial canary set; the paper does not state whether it uses random noise or natural images. This choice affects the final canaries and the reported results.
  • Number of meta-iterations N = Not reported
    Algorithm 5 iterates N times to optimize canaries; the value is not given and directly controls optimization quality.
assumptions (4)
  • standard math The one-run auditing procedures of Steinke et al. [SNJ23] and Mahluojifar et al. [MMC24] produce valid lower bounds on the privacy parameter given the guess vector and Theorem 1 / Algorithm 4.
    The empirical epsilon values are computed via these procedures; the paper relies on their validity without re-deriving them.
  • domain assumption REPLAY [EIC+25] computes metagradients of the training loss with respect to canary pixels accurately and at scale.
    Section 3 assumes efficient metagradient computation; the correctness of REPLAY is taken from prior work and not independently verified here.
  • ad hoc to paper Optimizing the surrogate objective (Eq. 3) on non-private SGD with a small ResNet-9 transfers to improved audit scores in DP-SGD with a larger Wide ResNet.
    This transferability is the core empirical premise of the method. It is validated only on CIFAR-10 with one DP configuration and is not theoretically justified.
  • domain assumption Negative cross-entropy loss is an appropriate score for membership inference under the audited procedures.
    Section 4 sets s(x) = -CE loss, as in prior work; the surrogate objective uses the same loss, linking optimization to evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Canaries for Privacy Auditing with Metagradient Descent." pith.science (2026). https://pith.science/paper/Z4AKNNPH

@misc{pith2026250715836,
  author       = {Pith},
  title        = {Pith review of: Optimizing Canaries for Privacy Auditing with Metagradient Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4AKNNPH}},
  note         = {Machine review of arXiv:2507.15836}
}
read the original abstract

In this work we study black-box privacy auditing, where the goal is to lower bound the privacy parameter of a differentially private learning algorithm using only the algorithm's outputs (i.e., final trained model). For DP-SGD (the most successful method for training differentially private deep learning models), the canonical approach auditing uses membership inference-an auditor comes with a small set of special "canary" examples, inserts a random subset of them into the training set, and then tries to discern which of their canaries were included in the training set (typically via a membership inference attack). The auditor's success rate then provides a lower bound on the privacy parameters of the learning algorithm. Our main contribution is a method for optimizing the auditor's canary set to improve privacy auditing, leveraging recent work on metagradient optimization. Our empirical evaluation demonstrates that by using such optimized canaries, we can improve empirical lower bounds for differentially private image classification models by over 2x in certain instances. Furthermore, we demonstrate that our method is transferable and efficient: canaries optimized for non-private SGD with a small model architecture remain effective when auditing larger models trained with DP-SGD.

Figures

Figures reproduced from arXiv: 2507.15836 by the authors.

Figure 1
Figure 1. An illustration of the metagradient. We embed the canaries into a continuous metaparameter [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. We evaluate the effectiveness of our metagradient canaries for the purpose of auditing [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A model-agnostic audit detects synthetic data disclosures via feature-match and membership-inference tests that separate true from phantom leaks and give empirical differential-privacy lower bounds.

Reference graph

Works this paper leans on

10 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning with differential privacy

    [ACG+16] Martin Abadi et al. “Deep learning with differential privacy”. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 2016, pp. 308–318. [BBC+25] Borja Balle et al. JAX-Privacy: Algorithms for Privacy-Preserving Machine Learning in JAX . Ver- sion 0.4.0

    work page 2016

  2. [3]

    Stabilizing differentiable architecture search via perturbation- based regularization

    URL: https://openreview.net/forum?id=xzKFnsJIXL. [CH20] Xiangning Chen and Cho-Jui Hsieh. “Stabilizing differentiable architecture search via perturbation- based regularization”. In: International conference on machine learning . PMLR. 2020, pp. 1554–

    work page 2020

  3. [7]

    Tight auditing of differentially private machine learning

    [NHS+23] Milad Nasr et al. “Tight auditing of differentially private machine learning”. In: 32nd USENIX Security Symposium (USENIX Security 23). 2023, pp. 1631–1648. 12 [NSB+25] Milad Nasr et al. “The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD”. In: The Thirteenth International Conference on ...

    work page 2023

  4. [8]

    Adversary instantiation: Lower bounds for differentially private machine learning

    URL: https://openreview.net/forum?id=DwqoBkj2Mw. [NST+21] Milad Nasr et al. “Adversary instantiation: Lower bounds for differentially private machine learning”. In: 2021 IEEE Symposium on security and privacy (SP). IEEE. 2021, pp. 866–882. [PTC+25] Ashwinee Panda et al. “Privacy Auditing of Large Language Models”. In: The Thirteenth Inter- national Confer...

    work page 2021

  5. [9]

    Meta-learning with implicit gradients

    URL: https://openreview.net/forum? id=60Vd7QOXlM. [RFK+19] Aravind Rajeswaran et al. “Meta-learning with implicit gradients”. In: Advances in neural in- formation processing systems 32 (2019). [SNJ23] Thomas Steinke, Milad Nasr, and Matthew Jagielski. “Privacy auditing with one (1) training run”. In: Proceedings of the 37th International Conference on Neu...

    work page 2019

  6. [1135]

    Forward and reverse gradient-based hyperparameter optimization

    [FDF+17] Luca Franceschi et al. “Forward and reverse gradient-based hyperparameter optimization”. In: International conference on machine learning. PMLR. 2017, pp. 1165–1173. [JUO20] Matthew Jagielski, Jonathan Ullman, and Alina Oprea. “Auditing differentially private ma- chine learning: How private is private sgd?” In: Advances in Neural Information Proc...

    arXiv 2020

  7. [1565]

    Unlocking high-accuracy differentially private image classification through scale

    [DBH+22] Soham De et al. “Unlocking high-accuracy differentially private image classification through scale”. In: arXiv preprint arXiv:2204.13650 (2022). [DCA+18] Luke N Darlow et al. “Cinic-10 is not imagenet or cifar-10”. In: arXiv preprint arXiv:1810.03505 (2018). [DDS+09] Jia Deng et al. “Imagenet: A large-scale hierarchical image database”. In: 2009 ...

    arXiv 2022

  8. [2022]

    Wide residual networks

    arXiv: 2202.12219 [cs.LG]. URL: https://arxiv.org/abs/2202.12219. [ZK16] Sergey Zagoruyko and Nikos Komodakis. “Wide residual networks”. In: arXiv preprint arXiv:1605.07146 (2016). 13

    arXiv 2016

Show all 10 references
  1. [2023]

    It’s Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss

    [Mut24] Meenatchi Sundaram Muthu Selva Annamalai. “It’s Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss”. In: Proceedings of the 2024 Workshop on Ar- tificial Intelligence and Security . AISec ’24. Salt Lake City, UT, USA: Association for Comput...

    2024

  2. [2025]

    Gradient-based optimization of hyperparameters

    URL: http://github.com/google-deepmind/jax_privacy. [Ben00] Yoshua Bengio. “Gradient-based optimization of hyperparameters”. In: Neural computation 12.8 (2000), pp. 1889–1900. [BGD+18] Benjamin Bichsel et al. “DP-Finder: Finding Differential Privacy Violations by Sampling and ...

    2000

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.