REVIEW 3 major objections 6 minor 45 references
BESA: Boosting Encoder Stealing Attack with Perturbation Recovery
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read BESA claims to boost existing encoder stealing attacks against perturbation-based defenses, raising surrogate encoder accuracy by up to 24.63%.
desk verdict A plausible two-stage attack idea that is undermined by inconsistent baseline numbers and a mischaracterized baseline; worth a careful revision, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two modules carry the argument. Perturbation detection uses K binary meta-classifiers, each trained to distinguish clean feature vectors from feature vectors altered by a specific defense, using outputs from 128 shadow encoders with varied architectures and contrastive learning algorithms. Perturbation recovery uses a generator composed of several fully-connected blocks with LeakyReLU and BatchNorm, inspired by MagNet, trained with a cosine-similarity loss to output the clean feature vector when given the defended one. The two modules are inserted between the API query and the surrogate encoder optimizer, and the meta-classifier's prediction selects which generator to apply.
What would settle it
Protect a target encoder with a perturbation defense that is unlike the three simulated ones, for example a defense that applies a random per-query rotation or input-dependent adversarial perturbation to the returned feature vector, and compare the downstream accuracy of the surrogate trained with BESA against the same surrogate trained without BESA. If the accuracy gain disappears or the meta-classifier's detection accuracy falls near chance, the transferability assumption underlying BESA fails.
Extended reading notes
Core claim
The central discovery is that perturbation-based defenses are not an obstacle in themselves: the defensive transformation leaves characteristic, detectable traces in the feature vectors, and those traces can be inverted. The paper shows that binary meta-classifiers, trained on pairs of clean and defended outputs from many shadow encoders, identify the active defense with over 98% accuracy in most settings, and that a fully-connected generative model with BatchNorm and LeakyReLU layers can map perturbed features back to clean ones. Wrapping these two modules around StolenEncoder, Cont-Steal, or SSLGuard raises the surrogate encoder's downstream accuracy by up to 24.63% against Top-K, rounding, and noise poisoning, and the improvement persists in a positive direction when one of the three defenses is held out during training.
Load-bearing premise
The entire attack rests on the assumption that the defense used by the real target encoder produces feature-vector traces similar enough to the traces of the defenses simulated on shadow encoders, so that the trained meta-classifiers and generators transfer to the real target.
Editorial extensions
If this is right
- Wrapping BESA around StolenEncoder, Cont-Steal, or SSLGuard improves surrogate encoder accuracy across MNIST, Fashion-MNIST, CIFAR-10, SVHN, and ImageNette, with the largest gain of 24.63 percentage points on MNIST.
- BESA remains effective when two defenses are combined and applied per query, with the meta-classifier deciding on each returned feature vector.
- Holding out one of the three defenses during training still leaves BESA improving the surrogate accuracy in the reported unknown-defense settings.
- The meta-classifiers and generators are trained once offline and can be reused against different target encoders, so the online attack adds only lightweight inference.
- On the more complex datasets the gain shrinks but remains positive, up to 16.26 percentage points on ImageNette and SVHN.
Reading between the lines
- Beyond the paper, the same detect-and-recover pattern could be tried on other protected representations, such as language-model embeddings or differential-privacy-noised outputs, since the paper only evaluates image encoders and three perturbation families.
- The reported unknown-defense gains may partly come from shared statistical traces among the three simulated defenses, such as zeros, decimal truncation, and Gaussian noise; a genuinely novel defense family is the untested boundary.
- If service providers randomize defense parameters per query or use input-dependent adversarial perturbations, the meta-classifier's detection confidence could drop, suggesting an adaptive version that fine-tunes the generator online.
- The detection results imply that perturbation defenses leak information through feature statistics; defenses that decorrelate the perturbation from the input distribution would be the natural countermeasure to test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BESA, a booster for encoder stealing attacks that combines a perturbation detection module (meta-classifiers trained on shadow encoders) and a perturbation recovery module (a MagNet-inspired generator) to recover clean feature vectors from outputs modified by perturbation-based defenses. BESA is integrated into three existing encoder stealing attacks (SSLGuard, StolenEncoder, Cont-Steal) and evaluated on five datasets against Top-K, rounding, noise poisoning, and hybrid defenses, plus an 'unknown defense' setting where one defense is held out. The paper reports accuracy improvements of up to 24.63% over the underlying attacks and claims that the method generalizes to unseen defenses.
Significance. If the reported results hold, BESA would provide a practical means of boosting encoder stealing against a common family of defenses, and the modular design (separate detection and recovery) is a plausible and useful contribution. The evaluation is broad in terms of datasets and attack baselines, and the paper includes hybrid defenses and a held-out defense scenario, which is a step beyond previous encoder stealing work. However, the experimental evidence contains internal inconsistencies and the generalization claim is not adequately supported: the held-out defenses are drawn from the same three families used in training, and baseline numbers differ between the main table and the unknown-defense table. No comparison is made with existing defense-penetrating extraction attacks. The central idea is interesting, but the paper currently does not substantiate its stronger claims.
major comments (3)
- [Section V-B1, Tables I and II] The no-BESA baseline accuracies for the same attack and defense differ between the two tables. For MNIST with SSLGuard, Top-K gives 79.65% in Table I but 57.43% in the Un-Top-K column of Table II; RD gives 72.67% vs 64.94%; NP gives 75.38% vs 69.44%. Since the attack protocol and target encoder should be identical, these discrepancies indicate that the two tables use different setups or that at least one set of numbers is incorrect. The reported BESA gains in the unknown-defense rows are therefore computed against weaker baselines and do not isolate the effect of BESA.
- [Section V-B1, Section III-B] The 'unknown defense' experiments do not test a genuinely novel defense because the held-out defenses (Un-Top-K, Un-RD, Un-NP) are the same three perturbation families used for shadow training, and the defense parameters (K, rounding precision, noise variance) are not varied. The threat model in Section III-B assumes the attacker's reconstructed set includes the deployed defense, yet the paper goes further and claims effectiveness against strategies not in that set. The current experiments cannot support that claim; a concrete test would hold out a defense type not belonging to any of the three families (e.g., randomized smoothing or DP-based perturbation) or vary parameters outside the training range.
- [Section II-A, Section V-B1] The paper motivates BESA by the lack of prior work on stealing defended encoders and cites D-DAE [21] as a defense-penetrating extraction attack for logits. However, no comparison is made against D-DAE or any other recovery-based attack, even when adapted to feature vectors. Without such a baseline, the claimed improvements could stem from the specific generator design or from the detection module, and the contribution relative to existing defense-penetrating methods is not established.
minor comments (6)
- [Section V-A] The sentence says 'four commonly used datasets' but lists five: MNIST, Fashion-MNIST, CIFAR-10, SVHN, and ImageNette.
- [Section IV-B, Eq. (5)] The L1-norm loss is written without an absolute value or norm, so the formula does not match the description in the text.
- [Section I and Section V-B2] The introduction claims detection accuracy 'over 99%' while the experimental section reports 'over 98%'; these numbers should be reconciled.
- [Table II] The header 'MINST' should be 'MNIST'.
- [Section V-C1] The text references 'SVH' but should read 'SVHN'.
- [Section V-A] The paper does not state the defense parameters used in the experiments (e.g., the value of K for Top-K, the rounding precision, and the noise variance sigma-squared), which hinders reproduction.
Circularity Check
No significant circularity: the detection and recovery modules are trained on shadow encoders and evaluated on held-out encoders and defense configurations, so the reported gains are empirical transfer results rather than consequences of the method's own definitions.
full rationale
The paper's derivation chain is an empirical attack pipeline rather than a circular construction. Algorithm 1 and Section IV train the meta-classifiers and the generator G^k on shadow encoders {E_m} with known defenses {f^k}; the generator minimizes Eq. (2) between recovered and clean feature vectors. The headline claim is the downstream surrogate encoder accuracy reported in Tables I and II, which is not the training objective of G^k and is measured on target encoders not used to train the modules. The 'unknown defense' experiments hold out one defense family from the training subset (e.g., Un-NP is trained on Top-K and RD), so the improvements under those conditions are transfer results, not identities forced by the training data. There is no load-bearing self-citation chain: the cited MagNet work [22] is used only for architectural inspiration, and the other cited works are external baselines. The main concerns are correctness risks rather than circularity: Table II baselines differ from Table I for the same attack/defense/dataset (e.g., SSLGuard on MNIST Top-K is 79.65% in Table I vs. 57.43% in the Un-Top-k column of Table II), and the held-out defenses are all drawn from the same three perturbation families used in shadow training. These issues affect whether the generalization claim is well controlled, but they do not make the reported result true by definition or by construction.
Assumptions & free parameters
free parameters (4)
- Meta-classifier confidence threshold =
0.5
- Number of shadow encoders M =
128
- Generator loss function =
cosine similarity
- Surrogate encoder architecture =
ResNet-50
assumptions (4)
- domain assumption The defense strategy used by the service provider is either in the reconstructed set f De or similar enough for transfer.
- domain assumption Meta-classifiers trained on shadow encoders generalize to the target encoder without knowledge of its architecture or training data.
- domain assumption The perturbation defenses are deterministic or learnable mappings that can be inverted by a generator trained on shadow encoder pairs.
- ad hoc to paper The attacker can simulate the exact defense mechanisms and parameters used by the target.
Cite this review
Pith. "Pith review of BESA: Boosting Encoder Stealing Attack with Perturbation Recovery." pith.science (2026). https://pith.science/paper/ZBJEND37
@misc{pith2026250604556,
author = {Pith},
title = {Pith review of: BESA: Boosting Encoder Stealing Attack with Perturbation Recovery},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZBJEND37}},
note = {Machine review of arXiv:2506.04556}
}
read the original abstract
To boost the encoder stealing attack under the perturbation-based defense that hinders the attack performance, we propose a boosting encoder stealing attack with perturbation recovery named BESA. It aims to overcome perturbation-based defenses. The core of BESA consists of two modules: perturbation detection and perturbation recovery, which can be combined with canonical encoder stealing attacks. The perturbation detection module utilizes the feature vectors obtained from the target encoder to infer the defense mechanism employed by the service provider. Once the defense mechanism is detected, the perturbation recovery module leverages the well-designed generative model to restore a clean feature vector from the perturbed one. Through extensive evaluations based on various datasets, we demonstrate that BESA significantly enhances the surrogate encoder accuracy of existing encoder stealing attacks by up to 24.63\% when facing state-of-the-art defenses and combinations of multiple defenses.
Figures
Reference graph
Works this paper leans on
-
[21]
D-DAE: defense- penetrating model extraction attacks,
Y . Chen, R. Guan, X. Gong, J. Dong, and M. Xue, “D-DAE: defense- penetrating model extraction attacks,” inSP. IEEE, 2023, pp. 382–399
work page 2023
-
[1]
A. Baevski, A. Babu, W. Hsu, and M. Auli, “Efficient self-supervised learning with contextualized target representations for vision, speech and language,” inICML, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 1416–1429
work page 2023
-
[2]
Reaas: Enabling adversarially robust downstream classifiers via robust encoder as a service,
W. Qu, J. Jia, and N. Z. Gong, “Reaas: Enabling adversarially robust downstream classifiers via robust encoder as a service,” inNDSS. The Internet Society, 2023
work page 2023
-
[3]
Can’t steal? cont- steal! contrastive stealing attacks against image encoders,
Z. Sha, X. He, N. Yu, M. Backes, and Y . Zhang, “Can’t steal? cont- steal! contrastive stealing attacks against image encoders,” inCVPR. Computer Vision Foundation /IEEE, 2023, pp. 16 373–16 383
work page 2023
-
[4]
AWEncoder: Adversarial Watermarking Pre-trained Encoders in Contrastive Learning
T. Zhang, H. Wu, X. Lu, and G. Sun, “Awencoder: Adversarial wa- termarking pre-trained encoders in contrastive learning,”CoRR, vol. abs/2208.03948, 2022
work page Pith review arXiv 2022
-
[5]
10 Security and Privacy Problems in Large Foundation Models
J. Jia, H. Liu, and N. Z. Gong, “10 security and privacy problems in self-supervised learning,”CoRR, vol. abs/2110.15444, 2021
work page Pith review arXiv 2021
-
[6]
On the difficulty of defending self-supervised learning against model extraction,
A. Dziedzic, N. Dhawan, M. A. Kaleem, J. Guan, and N. Papernot, “On the difficulty of defending self-supervised learning against model extraction,” inICML, ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 5757–5776
work page 2022
-
[8]
L. Chen, Y . Zhang, Y . Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” inCVPR. IEEE, 2022, pp. 18 689–18 698
work page 2022
Show all 45 references
-
[9]
Encodermi: Membership inference against pre-trained encoders in contrastive learning,
H. Liu, J. Jia, W. Qu, and N. Z. Gong, “Encodermi: Membership inference against pre-trained encoders in contrastive learning,” inCCS. ACM, 2021, pp. 2081–2095
2021
-
[10]
Semi-leak: Membership inference attacks against semi-supervised learning,
X. He, H. Liu, N. Z. Gong, and Y . Zhang, “Semi-leak: Membership inference attacks against semi-supervised learning,” inECCV (31), ser. Lecture Notes in Computer Science, vol. 13691. Springer, 2022, pp. 365–381
2022
-
[11]
Badencoder: Backdoor attacks to pre- trained encoders in self-supervised learning,
J. Jia, Y . Liu, and N. Z. Gong, “Badencoder: Backdoor attacks to pre- trained encoders in self-supervised learning,” inSP. IEEE, 2022, pp. 2043–2059
2022
-
[12]
Backdoor attacks on self-supervised learning,
A. Saha, A. Tejankar, S. A. Koohpayegani, and H. Pirsiavash, “Backdoor attacks on self-supervised learning,” inCVPR. IEEE, 2022, pp. 13 327– 13 336
2022
-
[13]
An embarrassingly simple backdoor attack on self-supervised learning,
C. Li, R. Pang, Z. Xi, T. Du, S. Ji, Y . Yao, and T. Wang, “An embarrassingly simple backdoor attack on self-supervised learning,” in CVPR. Computer Vision Foundation /IEEE, 2023, pp. 4367–4378
2023
-
[14]
PRADA: protecting against DNN model stealing attacks,
M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “PRADA: protecting against DNN model stealing attacks,” inEuroS&P. IEEE, 2019, pp. 512–527
2019
-
[15]
Modelguard: Information-theoretic defense against model extraction attacks,
M. Tang, A. Dai, L. DiValentin, A. Ding, A. Hass, N. Z. Gong, and Y . Chen, “Modelguard: Information-theoretic defense against model extraction attacks,” inUSENIX Security Symposium. USENIX As- sociation, 2024
2024
-
[16]
Plmmark: A secure and robust black-box watermarking framework for pre-trained language models,
P. Li, P. Cheng, F. Li, W. Du, H. Zhao, and G. Liu, “Plmmark: A secure and robust black-box watermarking framework for pre-trained language models,” inAAAI. AAAI Press, 2023, pp. 14 991–14 999
2023
-
[17]
Ssl-auth: An authentication framework by fragile watermarking for pre-trained encoders in self- supervised learning,
X. Li, C. Yin, L. Fang, R. Wang, and C. Lin, “Ssl-auth: An authentication framework by fragile watermarking for pre-trained encoders in self- supervised learning,”CoRR, vol. abs/2308.04673, 2023
2023 arXiv
-
[18]
Water- marking vision-language pre-trained models for multi-modal embedding as a service,
Y . Tang, J. Yu, K. Gai, X. Qu, Y . Hu, G. Xiong, and Q. Wu, “Water- marking vision-language pre-trained models for multi-modal embedding as a service,”CoRR, vol. abs/2311.05863, 2023
2023 arXiv
-
[19]
Threat modeling ai/ml systems and dependencies,
A. Marshall, J. Parikh, E. Kiciman, and R. Kumar, “Threat modeling ai/ml systems and dependencies,”Security documentation, 2019
2019
-
[20]
Stolenencoder: Stealing pre- trained encoders in self-supervised learning,
Y . Liu, J. Jia, H. Liu, and N. Z. Gong, “Stolenencoder: Stealing pre- trained encoders in self-supervised learning,” inCCS. ACM, 2022, pp. 2115–2128
2022
-
[22]
Magnet: A two-pronged defense against adversarial examples,
D. Meng and H. Chen, “Magnet: A two-pronged defense against adversarial examples,” inCCS. ACM, 2017, pp. 135–147
2017
-
[23]
Model extraction attacks and defenses on cloud-based machine learning models,
X. Gong, Q. Wang, Y . Chen, W. Yang, and X. Jiang, “Model extraction attacks and defenses on cloud-based machine learning models,”IEEE Commun. Mag., vol. 58, no. 12, pp. 83–89, 2020
2020
-
[24]
Inversenet: Augmenting model extraction attacks with training data inversion,
X. Gong, Y . Chen, W. Yang, G. Mei, and Q. Wang, “Inversenet: Augmenting model extraction attacks with training data inversion,” in IJCAI. ijcai.org, 2021, pp. 2439–2447
2021
-
[25]
Stealing machine learning models via prediction apis,
F. Tram `er, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis,” inUSENIX Security Symposium. USENIX Association, 2016, pp. 601–618
2016
-
[26]
Black-box attacks on sequential recommenders via data-free model extraction,
Z. Yue, Z. He, H. Zeng, and J. J. McAuley, “Black-box attacks on sequential recommenders via data-free model extraction,” inRecSys. ACM, 2021, pp. 44–54
2021
-
[27]
Divtheft: An ensemble model stealing attack by divide-and-conquer,
Z. Ma, X. Liu, Y . Liu, X. Liu, Z. Qin, and K. Ren, “Divtheft: An ensemble model stealing attack by divide-and-conquer,”IEEE Trans. Dependable Secur . Comput., vol. 20, no. 6, pp. 4810–4822, 2023. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11
2023
-
[28]
Data- free model extraction attacks in the context of object detection,
H. Shah, A. G, P. Kulkarni, Y . Govindarajulu, and M. Parmar, “Data- free model extraction attacks in the context of object detection,”CoRR, vol. abs/2308.05127, 2023
2023 arXiv
-
[29]
MAZE: data-free model stealing attack using zeroth-order gradient estimation,
S. Kariyappa, A. Prakash, and M. K. Qureshi, “MAZE: data-free model stealing attack using zeroth-order gradient estimation,” inCVPR. Computer Vision Foundation /IEEE, 2021, pp. 13 814–13 823
2021
-
[30]
Data-free model extraction,
J. Truong, P. Maini, R. J. Walls, and N. Papernot, “Data-free model extraction,” inCVPR. Computer Vision Foundation / IEEE, 2021, pp. 4771–4780
2021
-
[31]
Entangled watermarks as a defense against model extraction,
H. Jia, C. A. Choquette-Choo, V . Chandrasekaran, and N. Papernot, “Entangled watermarks as a defense against model extraction,” in USENIX Security Symposium. USENIX Association, 2021, pp. 1937– 1954
2021
-
[32]
Good artists copy, great artists steal: Model extraction attacks against image translation generative adversarial networks,
S. Szyller, V . Duddu, T. Gr ¨ondahl, and N. Asokan, “Good artists copy, great artists steal: Model extraction attacks against image translation generative adversarial networks,”CoRR, vol. abs/2104.12623, 2021
2021 arXiv
-
[33]
Fe-dast: Fast and effective data-free substitute training for black-box adversarial attacks,
M. Yu and S. Sun, “Fe-dast: Fast and effective data-free substitute training for black-box adversarial attacks,”Comput. Secur ., vol. 113, p. 102555, 2022
2022
-
[34]
QUDA: query-limited data-free model extraction,
Z. Lin, K. Xu, C. Fang, H. Zheng, A. A. Jaheezuddin, and J. Shi, “QUDA: query-limited data-free model extraction,” inAsiaCCS. ACM, 2023, pp. 913–924
2023
-
[35]
Knockoff nets: Stealing func- tionality of black-box models,
T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing func- tionality of black-box models,” inCVPR. Computer Vision Foundation / IEEE, 2019, pp. 4954–4963
2019
-
[36]
Practical black-box attacks against machine learning,
N. Papernot, P. D. McDaniel, I. J. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in AsiaCCS. ACM, 2017, pp. 506–519
2017
-
[37]
DST: dynamic substitute training for data-free black-box attack,
W. Wang, X. Qian, Y . Fu, and X. Xue, “DST: dynamic substitute training for data-free black-box attack,” inCVPR. IEEE, 2022, pp. 14 341– 14 350
2022
-
[38]
Bucks for buckets (B4B): active defenses against stealing encoders,
J. Dubinski, S. Pawlak, F. Boenisch, T. Trzcinski, and A. Dziedzic, “Bucks for buckets (B4B): active defenses against stealing encoders,” CoRR, vol. abs/2310.08571, 2023
2023 arXiv
-
[39]
Sslguard: A watermarking scheme for self-supervised learning pre-trained encoders,
T. Cong, X. He, and Y . Zhang, “Sslguard: A watermarking scheme for self-supervised learning pre-trained encoders,” inCCS. ACM, 2022, pp. 579–593
2022
-
[40]
Are you copying my model? protecting the copyright of large language models for eaas via backdoor watermark,
W. Peng, J. Yi, F. Wu, S. Wu, B. Zhu, L. Lyu, B. Jiao, T. Xu, G. Sun, and X. Xie, “Are you copying my model? protecting the copyright of large language models for eaas via backdoor watermark,” inACL (1). Association for Computational Linguistics, 2023, pp. 7653–7668
2023
-
[41]
Learning multiple layers of features from tiny images
A. Krizhevsky, G. Hintonet al., “Learning multiple layers of features from tiny images.” Toronto, ON, Canada, 2009
2009
-
[42]
Reading digits in natural images with unsupervised feature learning,
N. Yuval, “Reading digits in natural images with unsupervised feature learning,” inNeurIPS, 2011
2011
-
[43]
Auto-encoding variational bayes,
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in ICLR, 2014
2014
-
[44]
Generative adversarial nets,
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y . Bengio, “Generative adversarial nets,” inNeurIPS, 2014, pp. 2672–2680
2014
-
[45]
Rectifier nonlinearities improve neural network acoustic models,
A. L. Maas, A. Y . Hannun, A. Y . Nget al., “Rectifier nonlinearities improve neural network acoustic models,” inICML, ser. Proceedings of Machine Learning Research, vol. 30. PMLR, 2013, p. 3
2013
-
[46]
A survey of machine unlearning,
T. T. Nguyen, T. T. Huynh, P. L. Nguyen, A. W. Liew, H. Yin, and Q. V . H. Nguyen, “A survey of machine unlearning,”CoRR, vol. abs/2209.02299, 2022. Xuhao Renreceived his B.S. degree in the School of Information Engineering of Sichuan Agricultural University, Sichuan, China, i...
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.