Breaking TinyML: Why Quantized Neural Networks Need Domain-Specific Security Analysis

Andrea Mattia Garavagno; Jacob Huckelberry; James Mickens; Peter A. Beerel; Vijay Janapa Reddi; Yuke Zhang

REVIEW 3 major objections 5 minor 12 references

A two-step extraction-plus-white-box pipeline breaks quantized TinyML models more effectively than gray-box baselines, and only on hardware.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 14:05 UTC pith:2MFICJ7T

load-bearing objection Solid first empirical demo that a known two-step pipeline beats gray-box attacks on real TinyML QNN hardware; the "hardware-specific" framing is overstated relative to the absolute numbers. the 3 major comments →

arxiv 2606.14427 v1 pith:2MFICJ7T submitted 2026-06-12 cs.CR

Breaking TinyML: Why Quantized Neural Networks Need Domain-Specific Security Analysis

Jacob Huckelberry , Andrea Mattia Garavagno , Yuke Zhang , Peter A. Beerel , James Mickens , Vijay Janapa Reddi This is my paper

classification cs.CR

keywords TinyMLquantized neural networksmodel extractionadversarial exampleshardware securitygradient discontinuityMLPerf Tiny

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TinyML accelerators almost exclusively run 8-bit quantized networks, yet security evaluations have simply reused attacks designed for full-precision models. Those attacks often look weaker on quantized hardware, which has led people to conclude that quantization itself improves robustness. This paper shows the opposite: the weakness is in the attacks, not the models. By first extracting a high-fidelity surrogate with a one-time 50,000-query budget and then generating white-box adversarial examples on that continuous surrogate, the authors circumvent the gradient discontinuities that quantization creates. On the MLPerf Tiny CIFAR-10 ResNet the pipeline drops accuracy from 85.3% to 38%—more than twice the reduction of the best gray-box baseline—while remaining practical on real microcontrollers. The same pipeline underperforms relative to existing attacks once quantization is removed, proving the vulnerability is hardware-specific. The practical stakes are high: a single extracted model can yield reusable physical stickers that fool millions of devices that cannot be updated.

Core claim

A two-step pipeline—Copycat model extraction to an 82–84% fidelity continuous surrogate followed by PGD or FGSM—circumvents quantization-induced gradient discontinuities and produces adversarial examples that transfer to int-8 TinyML hardware, cutting CIFAR-10 accuracy by 47.3% with a one-time 50k-query cost and outperforming gray-box baselines by more than 20 points; the same pipeline is weaker than existing attacks on full-precision counterparts, showing that the vulnerability is hardware-specific rather than a general model weakness.

What carries the argument

The two-step attack pipeline: first, Copycat CNN extraction of a continuous surrogate that approximates the quantized decision boundary (82–84% fidelity on hard labels); second, white-box PGD/FGSM on that surrogate whose adversarial examples transfer to the real int-8 hardware target.

Load-bearing premise

The attacker already possesses tens of thousands of unlabeled images drawn from the same distribution the target was trained on, plus temporary physical access that lets them inject digital inputs and read only hard labels.

What would settle it

Repeat the CIFAR-10 and VWW experiments with a surrogate trained only on a different distribution (or far fewer than 50k ODD samples) and check whether accuracy reduction on the int-8 hardware still exceeds the best gray-box baseline; if not, the claimed practical advantage collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid first empirical demo that a known two-step pipeline beats gray-box attacks on real TinyML QNN hardware; the "hardware-specific" framing is overstated relative to the absolute numbers.

read the letter

The useful core of this paper is straightforward: they take the familiar Copycat-extraction + FGSM/PGD pipeline, run it against the actual MLPerf Tiny int-8 models on commercial MCUs, and show it produces a 47% accuracy drop on CIFAR-10 ResNet with a one-time 50k-query budget while Square and ZOO stay under 20%. They also measure wall-clock interaction times on three Nucleo boards. That combination of numbers and real-hardware timing is new for this niche and worth having.

What they do well is keep the experimental surface clean. Same models and datasets as Costa et al., clear fidelity table (82–84%), side-by-side int-8 vs FP32 transfer plots, and honest reporting that VWW is much weaker (only ~12% reduction). The threat model is explicit about needing ODD samples and hard-label GPIO/power observation; no hand-waving there.

The soft spot is the central claim that the attack is “hardware-specific.” Absolute accuracy reductions for their own pipeline are almost identical on int-8 and FP32 (47% vs 45% for PGD on CIFAR-10). The differential success comes almost entirely from the baselines getting dramatically stronger once quantization is removed. Section V-D’s explanation (surrogate stays full-precision, so the method is suboptimal on DNNs) is plausible, but they never run the controlled ablation that would isolate quantization/gradient-masking while holding the attack fixed. So the novelty is more “this pipeline is relatively better under the constraints of real TinyML hardware” than “it uniquely exploits a quantization vulnerability.” That is still a useful result; it just does not quite carry the weight the title and abstract put on it.

No code or data release is a real but secondary reproducibility hit. Free parameters (ε, query budget, epochs, surrogate architecture) are standard for the genre and not hidden.

This is for people working on edge-AI security or TinyML deployment who need concrete attack numbers and timing data rather than new theory. It is serious, empirically grounded work that a referee should see. I would accept it for peer review and expect the usual request for artifacts plus a tighter statement of what “hardware-specific” actually means.

Referee Report

3 major / 5 minor

Summary. The paper argues that TinyML QNN security cannot be adequately assessed by simply porting DNN evasion attacks, because quantization-induced gradient discontinuities mask those attacks and create an artificial impression of robustness. It applies a two-step pipeline (Copycat CNN extraction of a full-precision surrogate from hard-label queries on Original Domain Data, followed by FGSM/PGD transfer) to MLPerf Tiny models (ResNetv1 on CIFAR-10, MobileNetV1-0.25 on VWW) running on commercial MCUs. On the int-8 CIFAR-10 target the pipeline reduces accuracy from 85.30% to 38.00% (47.30% reduction) with a one-time 50k-query budget, outperforming the best gray-box baseline (Square-L∞, 19.70%) while remaining more query-efficient than black-box methods; effectiveness is weaker on VWW and, relative to SOTA, weaker on full-precision counterparts. The authors conclude that domain-specific, hardware-aware analysis is required.

Significance. If the hardware-specificity claim holds, the work usefully reframes TinyML security: quantization is not an inherent robustness feature but a source of gradient masking that conventional attacks fail to exploit, and a practical extraction-plus-transfer pipeline can close that gap under realistic hard-label, temporary-access constraints. The empirical package is concrete—fidelity numbers (Table I), accuracy reductions against the same models/datasets used by Costa et al. (Table II), and wall-clock query times on three Nucleo boards (Table III)—and therefore falsifiable and useful for subsequent defense work. The contribution is primarily empirical and comparative rather than a new algorithmic primitive, but it is timely for the expanding TinyML hardware ecosystem.

major comments (3)

The central claim that the attack is hardware-specific (Abstract, §I contribution 4, §V-D, Conclusion) rests on a mismatched baseline comparison rather than a controlled isolation of quantization. Table II shows the pipeline’s absolute accuracy reductions are nearly identical across domains (CIFAR-10 PGD: 47.3% int-8 vs 44.6% FP32; VWW similarly close). The claimed specificity is driven almost entirely by the fact that gray-box/black-box baselines become far stronger on FP32 (e.g., Square-L2 jumps from 5.2% to 82.2%). §V-D attributes this to the surrogate remaining full-precision and therefore suboptimal once quantization is removed, but the paper never holds the attack method fixed while varying only the presence of quantization (or gradient masking) on the same architecture. Without that ablation, the differential success could simply reflect the known weakness of gradient-free attacks
Threat-model premise in §III-C (and the extraction setup in §IV-D) assumes the attacker already possesses 50k unlabeled samples drawn from the Original Domain Data distribution of the target. Table I fidelities and the “one-time 50k-query” efficiency claim depend on this. For many real TinyML deployments the training distribution is proprietary or only partially known; the paper does not evaluate extraction under a mismatched or reduced ODD budget. At minimum the manuscript should quantify sensitivity of fidelity and transfer success to ODD size/distribution, or explicitly bound the claim to the ODD-available setting.
VWW results (Table II, Fig. 2b) show only 10–13% accuracy reduction and are weaker than Square-L2 (26.4%), which the authors correctly note requires confidence scores. The binary-boundary explanation in §V-C is plausible but untested; without an ablation that varies number of classes or boundary complexity on a controlled architecture, it remains post-hoc. Given that the paper markets the pipeline as generally useful for TinyML hardware, the limited VWW effectiveness should be more carefully scoped rather than presented as still demonstrating hardware-specific vulnerability.

minor comments (5)

No error bars, multiple random seeds, or statistical tests accompany the accuracy-reduction or fidelity numbers in Tables I–II; even a short note on run-to-run variance would strengthen the empirical claims.
Fig. 2 caption and body text refer to “solids vs dotted” lines for int-8 vs FP32; ensure the figure itself is labeled consistently and that the overlapping-line claim for CIFAR-10 is visually unambiguous.
Table III footnote 1 describes the timing methodology; it would help to state the measured average query latency per board explicitly so readers can reproduce the hour figures.
Minor typos and formatting: “T arget” with a space in Fig. 2 captions; “F . Hardware Selection” spacing; occasional missing spaces after periods in the introduction.
The abstract and introduction repeatedly claim “no attack strategies tailored specifically for TinyML hardware have been proposed yet”; a short related-work paragraph acknowledging any concurrent physical or side-channel TinyML attacks would avoid overstatement.

Circularity Check

0 steps flagged

No significant circularity; attack success, fidelity, and timing results are direct experimental measurements against external models, hardware, and independent baselines.

full rationale

The paper's central claims (47.30% accuracy reduction on CIFAR-10 int-8 via Copycat extraction + PGD, outperformance of gray-box methods, hardware-specificity via relative underperformance on FP32, and query-time advantages) are obtained by running the two-step pipeline on MLPerf Tiny models and commercial NUCLEO boards, then comparing numbers to gray-box/black-box results taken verbatim from the independent prior work of Costa et al. [4]. Table I fidelities, Figure 2 transfer curves, Table II accuracy reductions, and Table III wall-clock times are all measured quantities, not quantities defined in terms of themselves or fitted parameters re-labeled as predictions. There are no self-definitional equations, no uniqueness theorems imported from overlapping authors, no ansatz smuggled via self-citation, and no renaming of a known pattern. The ODD-access and hard-label assumptions are explicit threat-model premises, not circular reductions. Minor ecosystem citations (e.g., MLPerf Tiny) are ordinary and non-load-bearing. The derivation chain is therefore ordinary experimental security research with no circular steps.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on standard adversarial-ML assumptions plus a realistic but non-trivial threat model (ODD access + temporary physical injection + hard-label observation). No new physical entities are postulated; free parameters are the usual attack hyper-parameters and the chosen surrogate architectures/query budget. All are either conventional in the literature or explicitly stated.

free parameters (4)

epsilon (L∞ perturbation budget) = 8/255
Fixed at the conventional RobustBench value 8/255; controls attack strength and is not derived.
query budget for extraction = 50000
Chosen as 50 000 ODD samples (half the VWW training set); directly determines reported fidelity and total cost.
surrogate fine-tuning epochs = 10
Set to 10; affects fidelity numbers in Table I.
surrogate architecture choice = ResNet-50 / MobileNetV1
ResNet-50 or MobileNetV1 selected by hand for each target; fidelity and transferability depend on this choice.

axioms (4)

domain assumption Quantization maps continuous activations to discrete levels, producing zero or undefined gradients that block direct white-box attacks.
Stated in §II-C as the barrier that the surrogate pipeline is designed to circumvent; standard in QNN literature.
domain assumption An attacker can obtain a large unlabeled set drawn from the Original Domain Data distribution of the target application.
Threat-model premise in §III-C; required for the Copycat extraction phase that yields 82–84% fidelity.
domain assumption Temporary physical access permits digital injection of inputs and observation of hard labels only (GPIO / power / network side-channels).
§III-B; defines the realistic black-box interface used for both extraction and evaluation.
domain assumption Adversarial examples transfer from a high-fidelity surrogate to the target when decision-boundary geometry is approximately preserved.
Classic transferability assumption invoked in §IV-A/B; fidelity of 82–84% is claimed to be sufficient.

pith-pipeline@v1.1.0-grok45 · 16397 in / 3107 out tokens · 32313 ms · 2026-07-12T14:05:53.962817+00:00 · methodology

0 comments

read the original abstract

Most TinyML hardware accelerators focus on supporting Quantized Neural Networks (QNNs) to meet stringent constraints on power consumption and size. Despite this, the security aspects of quantization within TinyML hardware remain largely unexplored. Although previous studies indicate that QNNs demonstrate similar or enhanced robustness when compared to full-precision Deep Neural Networks (DNNs) against typical evasion attacks, no attack strategies tailored specifically for TinyML hardware have been proposed yet. This paper addresses this shortfall by demonstrating how a two-step attack pipeline can surpass the current state-of-the-art in the QNN context and shows the need for more hardware-aware security research.

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 6 linked inside Pith

[1]

An evaluation of edge tpu accelerators for convolutional neural networks,

K. Seshadri, B. Akin, J. Laudon, R. Narayanaswami, and A. Yaz- danbakhsh, “An evaluation of edge tpu accelerators for convolutional neural networks,” in2022 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2022, pp. 79–91

2022
[2]

Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference,

F. Conti, P. D. Schiavone, and L. Benini, “Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2940–2951, 2018

2018
[3]

Adversarial fine-tuning of compressed neural networks for joint improvement of robustness and efficiency,

H. Thorsteinsson, V . J. Henriksen, T. Chen, and R. Selvan, “Adversarial fine-tuning of compressed neural networks for joint improvement of robustness and efficiency,”arXiv preprint arXiv:2403.09441, 2024

Pith/arXiv arXiv 2024
[4]

David and goliath: An empirical evaluation of attacks and defenses for qnns at the deep edge,

M. Costa and S. Pinto, “David and goliath: An empirical evaluation of attacks and defenses for qnns at the deep edge,” in2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P). IEEE, 2024, pp. 524–541

2024
[5]

Transferability in ma- chine learning: from phenomena to black-box attacks using adversarial samples,

N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in ma- chine learning: from phenomena to black-box attacks using adversarial samples,”arXiv preprint arXiv:1605.07277, 2016

Pith/arXiv arXiv 2016
[6]

Prada: protecting against dnn model stealing attacks,

M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “Prada: protecting against dnn model stealing attacks,” in2019 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2019, pp. 512–527

2019
[7]

Knockoff nets: Stealing function- ality of black-box models,

T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing function- ality of black-box models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4954–4963

2019
[8]

Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data,

J. R. Correia-Silva, R. F. Berriel, C. Badue, A. F. De Souza, and T. Oliveira-Santos, “Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data,” in2018 International joint conference on neural networks (IJCNN). IEEE, 2018, pp. 1–8

2018
[9]

Mlperf tiny bench- mark,

C. Banbury, V . J. Reddi, P. Torelli, J. Holleman, N. Jeffries, C. Kiraly, P. Montino, D. Kanter, S. Ahmed, D. Pauet al., “Mlperf tiny bench- mark,”arXiv preprint arXiv:2106.07597, 2021

Pith/arXiv arXiv 2021
[10]

Explaining and harnessing adversarial examples,

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014

Pith/arXiv arXiv 2014
[11]

Towards deep learning models resistant to adversarial attacks,

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017

Pith/arXiv arXiv 2017
[12]

Robustbench: a standardized adversarial robustness benchmark,

F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammar- ion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,”arXiv preprint arXiv:2010.09670, 2020. Jacob Huckelberryreceived his B.Sc. in Computer Science from the United States Military Academy in 2023 and his M.Sc. in Data Science from Harvard...

Pith/arXiv arXiv 2010

[1] [1]

An evaluation of edge tpu accelerators for convolutional neural networks,

K. Seshadri, B. Akin, J. Laudon, R. Narayanaswami, and A. Yaz- danbakhsh, “An evaluation of edge tpu accelerators for convolutional neural networks,” in2022 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2022, pp. 79–91

2022

[2] [2]

Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference,

F. Conti, P. D. Schiavone, and L. Benini, “Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2940–2951, 2018

2018

[3] [3]

Adversarial fine-tuning of compressed neural networks for joint improvement of robustness and efficiency,

H. Thorsteinsson, V . J. Henriksen, T. Chen, and R. Selvan, “Adversarial fine-tuning of compressed neural networks for joint improvement of robustness and efficiency,”arXiv preprint arXiv:2403.09441, 2024

Pith/arXiv arXiv 2024

[4] [4]

David and goliath: An empirical evaluation of attacks and defenses for qnns at the deep edge,

M. Costa and S. Pinto, “David and goliath: An empirical evaluation of attacks and defenses for qnns at the deep edge,” in2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P). IEEE, 2024, pp. 524–541

2024

[5] [5]

Transferability in ma- chine learning: from phenomena to black-box attacks using adversarial samples,

N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in ma- chine learning: from phenomena to black-box attacks using adversarial samples,”arXiv preprint arXiv:1605.07277, 2016

Pith/arXiv arXiv 2016

[6] [6]

Prada: protecting against dnn model stealing attacks,

M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “Prada: protecting against dnn model stealing attacks,” in2019 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2019, pp. 512–527

2019

[7] [7]

Knockoff nets: Stealing function- ality of black-box models,

T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing function- ality of black-box models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4954–4963

2019

[8] [8]

Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data,

J. R. Correia-Silva, R. F. Berriel, C. Badue, A. F. De Souza, and T. Oliveira-Santos, “Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data,” in2018 International joint conference on neural networks (IJCNN). IEEE, 2018, pp. 1–8

2018

[9] [9]

Mlperf tiny bench- mark,

C. Banbury, V . J. Reddi, P. Torelli, J. Holleman, N. Jeffries, C. Kiraly, P. Montino, D. Kanter, S. Ahmed, D. Pauet al., “Mlperf tiny bench- mark,”arXiv preprint arXiv:2106.07597, 2021

Pith/arXiv arXiv 2021

[10] [10]

Explaining and harnessing adversarial examples,

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014

Pith/arXiv arXiv 2014

[11] [11]

Towards deep learning models resistant to adversarial attacks,

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017

Pith/arXiv arXiv 2017

[12] [12]

Robustbench: a standardized adversarial robustness benchmark,

F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammar- ion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,”arXiv preprint arXiv:2010.09670, 2020. Jacob Huckelberryreceived his B.Sc. in Computer Science from the United States Military Academy in 2023 and his M.Sc. in Data Science from Harvard...

Pith/arXiv arXiv 2010