Pith. sign in

REVIEW 3 major objections 6 minor 62 references

High Accuracy and High Fidelity Extraction of Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A query-only attack recovers a two-layer ReLU network's exact weights from ordinary logit access, giving the first practical functionally-equivalent extraction.

desk verdict First logit-only functionally equivalent extraction attack for two-layer ReLU nets; strong taxonomy and learning attack, but Section 6.5 has a real sign-recovery bug in the printed algorithm. read the letter →

arxiv 1909.01838 v2 pith:HKGSGP5I submitted 2019-09-03 cs.LG cs.CRstat.ML

Matthew Jagielski , Nicholas Carlini , David Berthelot , Alex Kurakin , Nicolas Papernot This is my paper · ORCID
classification cs.LGcs.CRstat.ML
keywords modelextractionfunctionallyequivalentReLUnetworkspiecewiselinearanalysisquery-basedattackstealingfidelityvsaccuracyadversarialexampletransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a deployed neural network can be stolen from its own answers, and separates model extraction into two goals: accuracy (matching task performance) and fidelity (matching the victim on every input), which the authors argue are in conflict. For accuracy, they show that training on oracle-supplied labels beats training on ground-truth labels, with a 16× query-efficiency gain from semi-supervised learning, demonstrated against a 193-million-parameter production image classifier. For fidelity, they argue that learning-based attacks are fundamentally capped near 93 percent agreement because training non-determinism prevents perfectly retraining the same function, even with identical seeds and data. Their central claim is that a two-layer ReLU network can instead be copied exactly, weights recovered up to the inherent scale-equivalence class, using only ordinary logit queries, by locating each ReLU's activation boundary, measuring weight ratios across that boundary, and solving the last layer algebraically. If correct, the attack turns black-box logit access into functionally identical white-box access, making a whole class of remote classifiers copyable end to end.

What carries the argument

The load-bearing object is the piecewise-linear geometry of a two-layer ReLU network: within each linear region every hidden unit's pre-activation has a fixed sign, and the boundaries between adjacent regions are the critical points where exactly one unit's input $A^{(0)}_i x + B^{(0)}_i$ is zero. The argument is carried by Algorithm 1, a 2-linearity test that, given a line segment containing exactly one kink, fits the two adjacent linear segments and computes their intersection in $O(1)$ queries, then verifies the candidate by comparing the predicted function value there against one fresh query, which is also what rejects segments containing more than one kink. Around each located kink the second derivative, the jump in the gradient across the boundary, equals $\pm A^{(0)}_{ji} A^{(1)}_i$, so taking ratios of these jumps across input directions recovers each row of $A^{(0)}$ up to sign and scale.

What would settle it

Run the attack against an oracle that returns logits rounded to single precision or as probabilities with any truncation: Algorithm 1's equality check between the predicted value at the candidate kink and the fresh queried value will fail, so no critical point is accepted and weight recovery cannot proceed. A second concrete test is to train a two-layer network whose first-layer weight matrix contains two identical rows, violating linear independence, and observe that the per-neuron isolation step can no longer attribute a kink to a single ReLU.

Watch

Extended reading notes

Core claim

The paper's central claim is that functionally-equivalent extraction of a neural network is achievable in practice: given an oracle that returns exact 64-bit logits for arbitrary inputs, an adversary can recover all weights of a two-layer ReLU network up to the inherent equivalence class (positive scaling and permutation symmetries), with no training and no side channel. The recovery works because ReLU networks are piecewise linear, so the logit function's kinks are exactly the hyperplanes where individual hidden units sit at zero; by sweeping one-dimensional lines through input space and locating each kink with a constant-query 2-linearity test, the attacker measures the gradient jump across each boundary to obtain ratios of the incoming weights, resolves signs through directional second derivatives and a null-space trick, and then solves the final layer by least squares. In experiments on trained MNIST and CIFAR-10 models the extracted network matches the victim's prediction on 100 percent of test inputs for networks up to 50,000 parameters and on 99.98 percent at 100,000 parameters, with first-layer weights recovered to roughly 23 bits of precision and adversarial examples transferring to the victim 100 percent of the time.

Load-bearing premise

The attack collapses if the victim does not return exact 64-bit logits for arbitrary queries, if the model is not exactly a two-layer ReLU network, or if the rows of the first weight matrix are linearly dependent; the paper itself notes that information-limiting defenses (labels only, top-k scores, or perturbed outputs) break the functionally-equivalent attack.

Editorial extensions

If this is right

  • Any two-layer ReLU classifier served through a logits-returning API can be copied outright: an attacker obtains a model that agrees with the victim on every input up to floating-point precision, using a number of queries proportional to the parameter count, $O(dh)$.
  • Downstream white-box attacks that need the victim's internal weights, such as adversarial-example crafting, membership inference on confidence scores, and inspection of overlearned internal representations, become available to a black-box adversary after extraction.
  • Learning-based extraction is bounded: even with the exact training set, hyperparameters, and random seeds, training non-determinism caps fidelity near 93 percent in the paper's Fashion-MNIST experiments, so exact copies must come from direct weight recovery rather than retraining.
  • The hybrid strategy shows the direct attack's bias-propagation errors are repairable: fixing the extracted first layer and fine-tuning a single bias parameter with gradient descent raises worst-case fidelity from 80 percent to 99.75 percent and preserves 100 percent adversarial-example transferability.
  • Because the attack is query-optimal up to a constant factor, defenses must limit the information returned per query; the paper shows that returning only top-k scores, labels, or perturbed outputs breaks the functionally-equivalent attack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a quantitative bits-per-query model of this attack is testable, degrading logit precision from 64-bit to 32-bit to 16-bit should raise the fraction of weight rows recovered with large error, and measuring that curve would give defenders a precise trade-off instead of the all-or-nothing verdict the paper reports.
  • Editorial inference: the layer-attribution difficulty the paper flags suggests a staircase extension, once the first layer is recovered, its activations on the already-collected queries are known, so the second layer's kinks become visible as kinks in the output with known inputs to that layer; whether this peels further layers or drowns in accumulated finite-difference error is the paper's main o
  • Editorial inference: the accuracy results indicate that oracle labels leak information beyond ground-truth labels, so re-running the ImageNet recipe with different distillation temperatures would separate dark-knowledge transfer from mere label replication, testing how much of the stolen accuracy gain survives when the victim's outputs are flattened.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies model extraction with two objectives: accuracy and fidelity. It proposes a learning-based extraction attack that uses the victim model as a label oracle and combines semi-supervised techniques (rotation loss and MixMatch) to improve query efficiency, validating it on the WSL ImageNet model, SVHN, and CIFAR-10. It then argues from theory and experiments that learning-based approaches cannot achieve perfect fidelity because of training nondeterminism. To reach the high-fidelity limit, the paper presents a functionally-equivalent extraction attack for two-layer ReLU networks that recovers the first-layer weights directly from logit queries, followed by algebraic recovery of the second layer. The empirical section reports near-perfect test-set fidelity on MNIST and CIFAR-10, 100% adversarial-example transferability, and a hybrid strategy that repairs numerical errors with a small amount of learning.

Significance. If the direct extraction attack is correct, it is a substantial contribution: it removes the gradient-oracle requirement of Milli et al. and the side-channel requirement of Batina et al., and it achieves query complexity O(dh), which is optimal up to constants for a model with dh first-layer parameters. The hardness statements in Theorems 1 and 2 are useful formal bounds on worst-case extraction and equivalence testing. The large-scale learning-based experiments on the WSL model also provide credible evidence that accuracy extraction can be improved with semi-supervised methods. However, the presented algorithm is not a complete specification as written: the global sign-recovery step in Section 6.5 contains a load-bearing error, and the empirical evaluation measures test-set and adversarial fidelity rather than functional equivalence on all inputs. These issues should be resolved before the headline claim is accepted as stated.

major comments (3)
  1. [Section 6.5] The global sign-recovery procedure is incorrect as printed. The text says to choose z with A(0)z=0 and then asserts that at z, z+v_i, and z-v_i 'all hidden units are 0 except for ReLU_i.' A hidden unit's pre-activation is A(0)_k x + B(0)_k, so A(0)z=0 only gives pre-activations equal to B(0)_k; when the biases are nonzero, the ReLUs at z are generally not all at zero. The correct condition is to solve the affine system A(0)z+B(0)=0, which is solvable because h<d, and then choose v_i such that A(0)_k v_i = delta_{ki}. As written, an implementer following Section 6.5 will fail sign recovery on any network with nonzero biases, and the networks trained in the paper all have biases. Since this step is necessary for the claimed exact extraction, the algorithm is not a complete specification. The text should also specify how the unknown global sign of each recovered row is handled when constructing v_i, and should base the sign decision on a comparison of OL(z+v_i) and OL(z-v_i) rather than on the unstated assumption that all other units are exactly at zero.
  2. [Section 6.4.1] The absolute-value recovery equations mix vectors and scalars in a way that is not implementable as written. The quantity d^2 OL / d e_j^2 is a K-dimensional logit vector, because A(1)_i is a row in R^K and the formula contains the vector A(1)_i. The text then divides 'quantities' |A(0)_1i A(1)_i| and |A(0)_2i A(1)_i| to obtain a scalar ratio |A(0)_1i / A(0)_2i|. The paper should specify how the K-dimensional logit differences are reduced to scalars, for example by using a fixed nonzero output component or the norm of the logit difference. Without this specification, the ratio operation is undefined and an implementer cannot reconstruct the row weights from the second-derivative measurements.
  3. [Section 6.7] The empirical evaluation does not directly verify functional equivalence, which is agreement on all inputs. Section 6.7 reports test-set fidelity and adversarial-example transferability, and for the 100,000-parameter model the test fidelity is 99.98%; the weight-recovery precision is reported as an average number of bits. To support the claim of a 'practically functionally-equivalent extraction attack,' the paper should report agreement on out-of-distribution or uniformly random inputs (and, ideally, the maximum logit discrepancy) for each model size, or explicitly qualify the result as near-functional equivalence subject to numerical error. Releasing the implementation would make this check reproducible, especially given the subtlety of Algorithm 1 and the sign-recovery step.
minor comments (6)
  1. [Section 6.5] The notation 'viA(0)=ei' appears to have the multiplication order reversed; it should be A(0)v_i=e_i (or the transpose should be defined consistently).
  2. [Algorithm 1] Lines 4 and 5 use variables a and b before they are defined; these should be t1 and t2. The displayed formulas for x and y-hat are also garbled by line breaks and should be rewritten with all variables defined.
  3. [Section 7.2] The objective written as 'minimizing E_x || f_theta(x) - W1 ReLU(Ahat(0)x + Bhat(0) + W0) + W2 ||' uses W0..2, W1, and W2 without defining their shapes or initialization; this should be clarified.
  4. [Table 8] The caption says the table reports the transferability rate of adversarial examples, while the text says the hybrid strategy increases fidelity and reports a fidelity value of 99.75%; the table should either include the fidelity numbers or the caption should be corrected.
  5. [Section 6.1] The assumption that the input dimension exceeds the hidden dimension by at least 100 is stated informally; the paper should state explicitly that all reported experiments satisfy this assumption and discuss what happens as h approaches d.
  6. [Section 6.7] The statement that gradient estimation fails roughly 1 in 10,000 times is not accompanied by a confidence interval or per-model variance; because the paper bases a design decision on this rate, reporting the spread across the five runs would be helpful.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity found; the extraction claims are measured against independent oracles, self-citations are non-load-bearing, and the Section 6.5 sign-recovery issue is a correctness gap, not a circular one.

full rationale

Applying the circularity tests, I find no load-bearing circular step. The direct functionally-equivalent attack (Section 6) recovers A^(0), B^(0), A^(1), B^(1) from finite-difference queries to the oracle; its success is measured as agreement with independently trained victim models (Section 6.7), and no parameter of the attack is fitted to that fidelity number. The learning-based attacks (Section 4) use the victim as a label source, which is the threat model rather than a circular reduction, and the reported accuracy/fidelity improvements are empirical comparisons against baselines trained on original labels. The cited prior work [19, 25] is external; the self-citations ([38] MixMatch, [44] prototypicality) are methods or analysis tools and are not used to justify the central extraction claim. The paper's own limitation in Section 8 ('Our functionally-equivalent attack is broken by all of these measures.') is a stated boundary, not circular dependence. The one substantive concern is a correctness gap, not circularity: Section 6.5 chooses z with A^(0)z = 0 and asserts all hidden units are 0, but the hidden pre-activations are A^(0)z + B^(0); with nonzero biases the three queries do not implement the described sign test. That is an omitted condition or possible bug in the algorithm as printed, not a reduction of the claimed result to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The extraction attack rests on assumptions about the oracle: exact double-precision logits, two-layer ReLU structure, and linear independence of the input weight rows. These are stated in Section 6.1. The learning attack assumes access to task-relevant unlabeled data. The distillation temperature T=1.5 is the only hand-tuned parameter reported.

free parameters (2)
  • distillation temperature T = 1.5
    Tuned by random search in Section 4.1 for the fully-supervised learning attack.
  • finite-difference step epsilon
    Used in Algorithm 1 and Section 6.4 for gradient estimation; no default value is specified, but it must be small relative to the weights.
assumptions (4)
  • domain assumption The oracle returns exact 64-bit logits for any query
    Section 6.1: 'We assume the adversary is able to observe the raw logit outputs as 64-bit floating point values.' This is required for the finite-difference-based critical point search.
  • domain assumption The victim model is a two-layer fully-connected ReLU network with h < d
    Section 6.1: 'we only aim to extract neural networks with one hidden layer using the ReLU activation function.' The null-space step requires h < d.
  • domain assumption The rows of A^(0) are linearly independent
    Section 6.1: 'Our attack implicitly assumes that the rows of A(0) are linearly independent.' This is used to recover signs and solve for biases.
  • domain assumption The adversary has access to task-relevant unlabeled data
    Section 3.4.1: 'we assume that the adversary is able to collect public unlabeled data to mount their attack.' Used in the learning-based attack.

how reviews work

0 comments
Cite this review

Pith. "Pith review of High Accuracy and High Fidelity Extraction of Neural Networks." pith.science (2026). https://pith.science/paper/HKGSGP5I

@misc{pith2026190901838,
  author       = {Pith},
  title        = {Pith review of: High Accuracy and High Fidelity Extraction of Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKGSGP5I}},
  note         = {Machine review of arXiv:1909.01838}
}
read the original abstract

In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access. We taxonomize model extraction attacks around two objectives: *accuracy*, i.e., performing well on the underlying learning task, and *fidelity*, i.e., matching the predictions of the remote victim classifier on any input. To extract a high-accuracy model, we develop a learning-based attack exploiting the victim to supervise the training of an extracted model. Through analytical and empirical arguments, we then explain the inherent limitations that prevent any learning-based strategy from extracting a truly high-fidelity model---i.e., extracting a functionally-equivalent model whose predictions are identical to those of the victim model on all possible inputs. Addressing these limitations, we expand on prior work to develop the first practical functionally-equivalent extraction attack for direct extraction (i.e., without training) of a model's weights. We perform experiments both on academic datasets and a state-of-the-art image classifier trained with 1 billion proprietary images. In addition to broadening the scope of model extraction research, our work demonstrates the practicality of model extraction attacks against production-grade systems.

Figures

Figures reproduced from arXiv: 1909.01838 by the authors.

Figure 1
Figure 1. Illustrating fidelity vs. accuracy. The solid blue [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. 2-dimension intuition for the functionally equivalent [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. An example sweep for critical point search. Here [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Efficient and accurate 2-linear testing subroutine in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Fidelity is easier on more prototypical examples. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: For a 16-neuron MNIST model the attack works. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 41 canonical work pages

  1. [1]

    Energy and policy considerations for deep learning in nlp,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” arXiv preprint arXiv:1906.02243, 2019

    arXiv 1906

  2. [2]

    Xlnet: Generalized autoregressive pretraining for language understanding,

    Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. R. Salakhut- dinov, and Q. V . Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in neural information processing systems, 2019, pp. 5754– 5764

    work page 2019

  3. [3]

    The unreasonable effectiveness of data,

    A. Halevy, P. Norvig, and F. Pereira, “The unreasonable effectiveness of data,” 2009

    work page 2009

  4. [4]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  5. [5]

    Sequence to sequence learning with neural networks,

    I. Sutskever, O. Vinyals, and Q. V . Le, “Sequence to sequence learning with neural networks,” inNeural in- formation processing systems, 2014, pp. 3104–3112

    work page 2014

  6. [6]

    Wavenet: A generative model for raw audio

    A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio.”SSW, vol. 125, 2016

    work page 2016

  7. [7]

    Practical black-box attacks against machine learning,

    N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” inProceedings of the 2017 ACM on Asia conference on computer and communica- tions security. ACM, 2017, pp. 506–519

    work page 2017

  8. [8]

    Adversarial learning,

    D. Lowd and C. Meek, “Adversarial learning,” inPro- ceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining . ACM, 2005, pp. 641–647

    work page 2005

Show all 62 references
  1. [9]

    Membership inference attacks against machine learn- ing models,

    R. Shokri, M. Stronati, C. Song, and V . Shmatikov, “Membership inference attacks against machine learn- ing models,” in 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017, pp. 3–18

    2017

  2. [10]

    Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models,

    A. Salem, Y . Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes, “Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models,” arXiv preprint arXiv:1806.01246 , 2018

    2018 arXiv

  3. [11]

    Stealing machine learning models via pre- diction apis,

    F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ris- tenpart, “Stealing machine learning models via pre- diction apis,” in 25th{USENIX} Security Symposium ({USENIX} Security 16), 2016, pp. 601–618

    2016

  4. [12]

    Knockoff nets: Stealing functionality of black-box models,

    T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing functionality of black-box models,” inProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4954–4963

    2019

  5. [13]

    Model extraction and active learning,

    V . Chandrasekaran, K. Chaudhuri, I. Giacomelli, S. Jha, and S. Yan, “Model extraction and active learning,” CoRR, vol. abs/1811.02054, 2018. [Online]. Available: http://arxiv.org/abs/1811.02054

    2018 arXiv

  6. [14]

    To- wards reverse-engineering black-box neural networks,

    S. J. Oh, M. Augustin, B. Schiele, and M. Fritz, “To- wards reverse-engineering black-box neural networks,” arXiv preprint arXiv:1711.01768, 2017

    2017 arXiv

  7. [15]

    A framework for the extraction of deep neural networks by leveraging public data,

    S. Pal, Y . Gupta, A. Shukla, A. Kanade, S. K. Shevade, and V . Ganapathy, “A framework for the extraction of deep neural networks by leveraging public data,” CoRR, vol. abs/1905.09165, 2019. [Online]. Available: http://arxiv.org/abs/1905.09165

    1905 arXiv

  8. [16]

    Copycat cnn: Steal- ing knowledge by persuading confession with random non-labeled data,

    J. R. Correia-Silva, R. F. Berriel, C. Badue, A. F. de Souza, and T. Oliveira-Santos, “Copycat cnn: Steal- ing knowledge by persuading confession with random non-labeled data,” in 2018 International Joint Confer- ence on Neural Networks (IJCNN). IEEE, 2018

    2018

  9. [17]

    Overlearning reveals sensi- tive attributes,

    C. Song and V . Shmatikov, “Overlearning reveals sensi- tive attributes,”arXiv preprint arXiv:1905.11742, 2019

    1905 arXiv

  10. [18]

    Secu- rity analysis of deep neural networks operating in the presence of cache side-channel attacks,

    S. Hong, M. Davinroy, Y . Kaya, S. N. Locke, I. Rackow, K. Kulda, D. Dachman-Soled, and T. Dumitra¸ s, “Secu- rity analysis of deep neural networks operating in the presence of cache side-channel attacks,”arXiv preprint arXiv:1810.03487, 2018

    arXiv

  11. [19]

    Model reconstruction from model explanations,

    S. Milli, L. Schmidt, A. D. Dragan, and M. Hardt, “Model reconstruction from model explanations,”arXiv preprint arXiv:1807.05185, 2018

    2018 arXiv

  12. [20]

    Rectified linear units im- prove restricted boltzmann machines,

    V . Nair and G. E. Hinton, “Rectified linear units im- prove restricted boltzmann machines,” inProceedings of the 27th international conference on machine learn- ing (ICML-10), 2010, pp. 807–814

    2010

  13. [21]

    A method for solving the convex pro- gramming problem with convergence rate o (1/kˆ 2),

    Y . E. Nesterov, “A method for solving the convex pro- gramming problem with convergence rate o (1/kˆ 2),” in Dokl. akad. nauk Sssr, vol. 269, 1983, pp. 543–547

    1983

  14. [22]

    Adaptive subgradient methods for online learning and stochastic optimization,

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research, vol. 12, no. Jul, pp. 2121–2159, 2011

    2011

  15. [23]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  16. [24]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  17. [25]

    Csi neu- ral network: Using side-channels to recover your ar- tificial neural network information,

    L. Batina, S. Bhasin, D. Jap, and S. Picek, “Csi neu- ral network: Using side-channels to recover your ar- tificial neural network information,” arXiv preprint arXiv:1810.09076, 2018

    2018 arXiv

  18. [26]

    Differential power anal- ysis,

    P. Kocher, J. Jaffe, and B. Jun, “Differential power anal- ysis,” in Annual International Cryptology Conference. Springer, 1999, pp. 388–397

    1999

  19. [27]

    On the learnability of deep random networks,

    A. Das, S. Gollapudi, R. Kumar, and R. Panigrahy, “On the learnability of deep random networks,”CoRR, vol. abs/1904.03866, 2019

    1904 arXiv

  20. [28]

    Exploring the limits of weakly supervised pretraining,

    D. Mahajan, R. Girshick, V . Ramanathan, K. He, M. Paluri, Y . Li, A. Bharambe, and L. van der Maaten, “Exploring the limits of weakly supervised pretraining,” in Proceedings of the European Conference on Com- puter Vision (ECCV), 2018, pp. 181–196

    2018

  21. [29]

    Zero-shot knowledge trans- fer via adversarial belief matching,

    P. Micaelli and A. Storkey, “Zero-shot knowledge trans- fer via adversarial belief matching,” arXiv preprint arXiv:1905.09768, 2019

    1905 arXiv

  22. [30]

    Language models are unsupervised multi- task learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multi- task learners,”OpenAI Blog, vol. 1, no. 8, 2019

    2019

  23. [31]

    Cnn features off-the-shelf: an astounding baseline for recognition,

    A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn features off-the-shelf: an astounding baseline for recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2014, pp. 806–813

    2014

  24. [32]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transform- ers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  25. [33]

    Queries and concept learning,

    D. Angluin, “Queries and concept learning,”Machine learning, vol. 2, no. 4, pp. 319–342, 1988

    1988

  26. [34]

    Combining labeled and un- labeled data with co-training,

    A. Blum and T. Mitchell, “Combining labeled and un- labeled data with co-training,” in Proceedings of the eleventh annual conference on Computational learning theory. Citeseer, 1998, pp. 92–100

    1998

  27. [35]

    Com- bining mixmatch and active learning for better accuracy with fewer labels,

    S. Song, D. Berthelot, and A. Rostamizadeh, “Com- bining mixmatch and active learning for better accuracy with fewer labels,” 2020. [Online]. Available: https://openreview.net/forum?id=HJxWl0NKPB

    2020

  28. [36]

    Rethinking deep active learning: Using unlabeled data at model training,

    O. Siméoni, M. Budnik, Y . Avrithis, and G. Gravier, “Rethinking deep active learning: Using unlabeled data at model training,” 2020. [Online]. Available: https://openreview.net/forum?id=rJehllrtDS

    2020

  29. [37]

    S4l: Self-supervised semi-supervised learning,

    X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semi-supervised learning,”arXiv preprint arXiv:1905.03670, 2019

    1905 arXiv

  30. [38]

    Mixmatch: A holistic ap- proach to semi-supervised learning,

    D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, “Mixmatch: A holistic ap- proach to semi-supervised learning,” arXiv preprint arXiv:1905.02249, 2019

    1905 arXiv

  31. [39]

    Reading digits in natural images with unsu- pervised feature learning,

    Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y . Ng, “Reading digits in natural images with unsu- pervised feature learning,” 2011

    2011

  32. [40]

    Learning multiple layers of fea- tures from tiny images,

    A. Krizhevsky et al., “Learning multiple layers of fea- tures from tiny images,” Citeseer, Tech. Rep., 2009

    2009

  33. [41]

    Hidden technical debt in machine learn- ing systems,

    D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V . Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learn- ing systems,” in Advances in neural information pro- cessing systems, 2015, pp. 2503–2511

    2015

  34. [42]

    Sim- ple and scalable predictive uncertainty estimation using deep ensembles,

    B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Sim- ple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems, 2017, pp. 6402–6413

    2017

  35. [43]

    H. Xiao, K. Rasul, and R. V ollgraf. (2017) Fashion- mnist: a novel image dataset for benchmarking machine learning algorithms

    2017

  36. [44]

    Prototypical examples in deep learning: Metrics, characteristics, and utility,

    N. Carlini, U. Erlingsson, and N. Papernot, “Prototypical examples in deep learning: Metrics, characteristics, and utility,” 2019. [Online]. Available: https://openreview. net/forum?id=r1xyx3R9tQ

    2019

  37. [45]

    Gradient-based learning applied to document recog- nition,

    Y . LeCun, L. Bottou, Y . Bengio, P. Haffner et al. , “Gradient-based learning applied to document recog- nition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  38. [46]

    Google, “Jax,” https://github.com/google/jax, 2019

    2019

  39. [47]

    Intriguing properties of neural networks,

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Er- han, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  40. [48]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  41. [49]

    Defending against model stealing attacks using deceptive perturba- tions,

    T. Lee, B. Edwards, I. Molloy, and D. Su, “Defending against model stealing attacks using deceptive perturba- tions,”arXiv preprint arXiv:1806.00054, 2018

    2018 arXiv

  42. [50]

    Adding robustness to support vector machines against adver- sarial reverse engineering,

    I. M. Alabdulmohsin, X. Gao, and X. Zhang, “Adding robustness to support vector machines against adver- sarial reverse engineering,” inProceedings of the 23rd ACM International Conference on Conference on Infor- mation and Knowledge Management. ACM, 2014, pp. 231–240

    2014

  43. [51]

    Prada: protecting against dnn model steal- ing attacks,

    M. Juuti, S. Szyller, A. Dmitrenko, S. Marchal, and N. Asokan, “Prada: protecting against dnn model steal- ing attacks,”arXiv preprint arXiv:1805.02628, 2018

    2018 arXiv

  44. [52]

    Model extraction warning in mlaas paradigm,

    M. Kesarwani, B. Mukhoty, V . Arya, and S. Mehta, “Model extraction warning in mlaas paradigm,” inPro- ceedings of the 34th Annual Computer Security Appli- cations Conference. ACM, 2018, pp. 371–380

    2018

  45. [53]

    Stealing hyperparameters in machine learning,

    B. Wang and N. Z. Gong, “Stealing hyperparameters in machine learning,” in2018 IEEE Symposium on Secu- rity and Privacy (SP). IEEE, 2018, pp. 36–52

    2018

  46. [54]

    Protecting intellectual prop- erty of deep neural networks with watermarking,

    J. Zhang, Z. Gu, J. Jang, H. Wu, M. P. Stoecklin, H. Huang, and I. Molloy, “Protecting intellectual prop- erty of deep neural networks with watermarking,” in Proceedings of the 2018 on Asia Conference on Com- puter and Communications Security. ACM, 2018, pp. 159–172

    2018

  47. [55]

    Em- bedding watermarks into deep neural networks,

    Y . Uchida, Y . Nagai, S. Sakazawa, and S. Satoh, “Em- bedding watermarks into deep neural networks,” in Pro- ceedings of the 2017 ACM on International Conference on Multimedia Retrieval. ACM, 2017, pp. 269–277

    2017

  48. [56]

    On the (im) possi- bility of obfuscating programs,

    B. Barak, O. Goldreich, R. Impagliazzo, S. Rudich, A. Sahai, S. Vadhan, and K. Yang, “On the (im) possi- bility of obfuscating programs,” inAnnual international cryptology conference. Springer, 2001, pp. 1–18

    2001

  49. [57]

    A privacy-preserving protocol for neural-network-based computation,

    M. Barni, C. Orlandi, and A. Piva, “A privacy-preserving protocol for neural-network-based computation,” inPro- ceedings of the 8th workshop on Multimedia and secu- rity. ACM, 2006, pp. 146–151

    2006

  50. [58]

    Reluplex: An efficient smt solver for verifying deep neural networks,

    G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient smt solver for verifying deep neural networks,” inInternational Con- ference on Computer Aided Verification . Springer, 2017, pp. 97–117. A Formal Statements for Section 3.3 Here, we give...

    2017

  51. [59]

    This step is the most nontrivial to analyze, but fortunately this was addressed in [19]

    Critical Point Search. This step is the most nontrivial to analyze, but fortunately this was addressed in [19]. They found this step requires O(hlog(h)) gradient queries, which we simulate with O(hlog(h)) model queries

  52. [60]

    This piece is significantly compli- cated by not having access to gradient queries

    Weight Recovery. This piece is significantly compli- cated by not having access to gradient queries. For each 5 10 15 Bits of precision in logits 0 100 200 300 400 500Frequency Figure 6: For a 16-neuron MNIST model the attack works. Plotted here is number of bits of precision o...

  53. [61]

    For each ReLU, we require only three queries

    Global Sign Recovery. For each ReLU, we require only three queries. Then this step is O(h)

  54. [62]

    This step requires h queries to make the system of linear equations full rank (although in practice we reuse previous queries here, making this step require 0 queries)

    Last Layer Extraction. This step requires h queries to make the system of linear equations full rank (although in practice we reuse previous queries here, making this step require 0 queries). Overall, the algorithm requires O(hlog(h) + dh + h) = O(dh) queries. Extraction requi...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.