pith. sign in

arxiv: 2606.31653 · v1 · pith:QZB2HNHJnew · submitted 2026-06-30 · 💻 cs.LG · cs.AI

Improving Certified Robustness via Adversarial Distillation

Matteo Melis , Jesus Martinez Del Rincon , Vishal Sharma This is my paper

Pith reviewed 2026-07-01 06:09 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords certified robustnessadversarial distillationinterval bound propagationneural network certificationcertified trainingrobustness benchmarks
0
0 comments X

The pith

Combining adversarial distillation with an interval bound upper bound produces models with higher certified accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that a training objective called AD-CERT merges logit-level distillation from an empirically robust teacher with an upper bound computed via interval bound propagation. This supplies a surrogate lower bound on worst-case loss that complements the upper bound, yielding better certified accuracy than earlier certified training approaches. The method records state-of-the-art certified numbers on several robustness benchmarks and improves certified accuracy by as much as 5.40 percentage points when distillation occurs at the logit level rather than in feature space. A reader would care because the approach narrows the usual gap between high standard accuracy from adversarial training and the formal guarantees required for certified models.

Core claim

AD-CERT achieves state-of-the-art certified performance on several robustness benchmarks by combining adversarial distillation with an IBP upper bound, with logit-level distillation improving certified accuracy over feature-space distillation by up to 5.40 percentage points. Distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training.

What carries the argument

The AD-CERT objective, which pairs logit-level adversarial distillation from a robust teacher with an Interval Bound Propagation (IBP) upper bound on the worst-case loss.

If this is right

  • The combined objective produces models whose certified accuracy exceeds that of prior certified training methods on multiple benchmarks.
  • Performing distillation at the logit level rather than in feature space raises certified accuracy by up to 5.40 percentage points under identical conditions.
  • The approach interpolates between lower and upper bounds on worst-case loss and thereby improves the standard-versus-certified accuracy trade-off.
  • The same logit distillation step extends the earlier practice of mixing adversarial objectives with loose IBP over-approximations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Empirical robustness signals from a teacher model can be transferred directly into the certified training loop without needing tighter formal relaxations.
  • The logit-space benefit may appear in other certified training pipelines that currently rely only on bound propagation.
  • Testing the same distillation step on larger architectures or different perturbation radii would show whether the observed gains remain consistent.

Load-bearing premise

Distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate when combined with an IBP upper bound.

What would settle it

A controlled retraining experiment on a standard benchmark such as CIFAR-10 at epsilon 8/255 that produces certified accuracy no higher than the strongest prior IBP-only certified training baseline.

Figures

Figures reproduced from arXiv: 2606.31653 by Jesus Martinez del Rincon, Matteo Melis, Vishal Sharma.

Figure 1
Figure 1. Figure 1: Overview of AD-CERT (§3.1) loss formulation, which combines adversarial distillation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity analysis of AD-CERT with respect to the IBP coefficient, [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of IBP relaxation and the tighter convex hull relaxation. The shaded regions [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
read the original abstract

Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript introduces AD-CERT, a certified training objective that combines adversarial distillation from an empirically robust teacher with an IBP upper bound on the worst-case loss. It claims that logit-level distillation serves as an effective lower-bound surrogate, yielding state-of-the-art certified accuracy on robustness benchmarks and improving certified accuracy by up to 5.40 percentage points over a robust feature-space distillation baseline in a unified experimental setup.

Significance. If the empirical results hold, the work offers a practical interpolation between adversarial lower bounds and IBP upper bounds that improves the certified-standard accuracy trade-off. The logit-versus-feature distillation comparison supplies a concrete, falsifiable empirical finding that can guide subsequent certified training research.

minor comments (2)
  1. [Abstract] Abstract: the SOTA claim and the 5.40 pp figure are presented without naming the specific benchmarks, teacher models, or competing certified methods, making immediate assessment of the result difficult.
  2. The manuscript should clarify whether the reported improvements include error bars or statistical significance tests across multiple random seeds.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, significance assessment, and recommendation of minor revision. No major comments were raised in the report.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper describes a hybrid training objective that interpolates between an adversarial distillation lower-bound surrogate and an IBP upper bound. The central claim is an empirical performance improvement (logit-level distillation outperforming feature-space by up to 5.40 pp on certified accuracy), presented as a measured result rather than a quantity derived by definition from the inputs. No equations reduce a prediction to a fitted parameter by construction, no uniqueness theorem is imported from self-citations, and no ansatz is smuggled via prior work. The derivation chain remains self-contained against external benchmarks and does not collapse to tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract provides no information on free parameters, axioms, or invented entities; the approach relies on standard assumptions in certified and adversarial training.

pith-pipeline@v0.9.1-grok · 5745 in / 1102 out tokens · 42774 ms · 2026-07-01T06:09:25.425347+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages

  1. [1]

    Adversarial training and provable defenses: Bridging the gap

    Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. InInternational Conference on Learning Representations, 2020

    work page 2020

  2. [2]

    Evasion attacks against machine learning at test time

    Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndi´c, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Advanced Information Systems Engineering, 2013

    work page 2013

  3. [3]

    Rudy Bunel, Jingyue Lu, Ilker Turkaslan, Philip H. S. Torr, Pushmeet Kohli, and M. Pawan Kumar. Branch and bound for piecewise linear neural network verification.J. Mach. Learn. Res., 2020

    work page 2020

  4. [4]

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

    Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. InInternational Conference on Machine Learning, 2020

    work page 2020

  5. [5]

    Decoupled kullback-leibler divergence loss

    Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  6. [6]

    Learning better certified models from empirically-robust teachers, 2026

    Alessandro De Palma. Learning better certified models from empirically-robust teachers, 2026. URLhttps://arxiv.org/abs/2602.02626

    work page arXiv 2026

  7. [7]

    Pawan Kumar, and Robert Stanforth

    Alessandro De Palma, Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar, and Robert Stanforth. Ibp regularization for verified adversarial robustness via branch-and-bound. In International Conference on Machine Learning, 2022

    work page 2022

  8. [8]

    Pawan Kumar, Robert Stan- forth, and Alessio Lomuscio

    Alessandro De Palma, Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar, Robert Stan- forth, and Alessio Lomuscio. Expressive losses for verified robustness via convex combinations. InInternational Conference on Machine Learning, 2024

    work page 2024

  9. [9]

    Formal verification of piece-wise linear feed-forward neural networks.Auto- mated Technology for Verification and Analysis, 2017

    Ruediger Ehlers. Formal verification of piece-wise linear feed-forward neural networks.Auto- mated Technology for Verification and Analysis, 2017

    work page 2017

  10. [10]

    Complete verification via multi-neuron relaxation guided branch-and-bound

    Claudio Ferrari, Mark Niklas Mueller, Nikola Jovanovi ´c, and Martin Vechev. Complete verification via multi-neuron relaxation guided branch-and-bound. InInternational Conference on Learning Representations, 2022

    work page 2022

  11. [11]

    Adversarially robust distillation

    Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. InAssociation for the Advancement of Artificial Intelligence (AAAI) Conference on Artificial Intelligence, 2020

    work page 2020

  12. [12]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. InInternational Conference on Learning Representations, 2015

    work page 2015

  13. [13]

    On the effectiveness of interval bound propagation for training verifiably robust models

    Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. InAdvances in Neural Information Processing Systems, 2018. 12

    work page 2018

  14. [14]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InInternational Conference on Computer Vision, 2015

    work page 2015

  15. [15]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems, 2015

    work page 2015

  16. [16]

    On the paradox of certified training.Transactions on Machine Learning Research, 2022

    Nikola Jovanovi´c, Mislav Balunovi´c, Maximilian Baader, and Martin Vechev. On the paradox of certified training.Transactions on Machine Learning Research, 2022

    work page 2022

  17. [17]

    Reluplex: An efficient SMT solver for verifying deep neural networks

    Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. InComputer Aided Verification, volume 10426 ofLecture Notes in Computer Science, pages 97–117. Springer, 2017

    work page 2017

  18. [18]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations, 2015

    work page 2015

  19. [19]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    work page 2009

  20. [20]

    Tiny imagenet visual recognition challenge.CS 231N, 7(7):3, 2015

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge.CS 231N, 7(7):3, 2015

    work page 2015

  21. [21]

    Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. MNIST handwritten digit database,

    work page

  22. [22]

    URLhttp://yann.lecun.com/exdb/mnist

    work page

  23. [23]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. InInternational Conference on Learning Representations, 2018

    work page 2018

  24. [24]

    Connecting certified and adversarial training

    Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Connecting certified and adversarial training. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  25. [25]

    Understanding certified training with interval bound propagation

    Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Understanding certified training with interval bound propagation. InInternational Conference on Learning Representa- tions, 2024

    work page 2024

  26. [26]

    Ctbench: A library and benchmark for certified training

    Yuhao Mao, Stefan Balauca, and Martin Vechev. Ctbench: A library and benchmark for certified training. InInternational Conference on Machine Learning, 2025

    work page 2025

  27. [27]

    Differentiable abstract interpretation for provably robust neural networks

    Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. InInternational Conference on Machine Learning, 2018

    work page 2018

  28. [28]

    Certified training: Small boxes are all you need

    Mark Niklas Müller, Franziska Eckert, Marc Fischer, and Martin Vechev. Certified training: Small boxes are all you need. InInternational Conference on Learning Representations, 2023

    work page 2023

  29. [29]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performa...

    work page 2019

  30. [30]

    Fast certified robust training with short warmup

    Zhouxing Shi, Yihan Wang, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Fast certified robust training with short warmup. InAdvances in Neural Information Processing Systems, 2021

    work page 2021

  31. [31]

    An abstract domain for certifying neural networks.Proc

    Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying neural networks.Proc. ACM Program. Lang., 3(POPL), 2019

    work page 2019

  32. [32]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. InInternational Conference on Learning Representations, 2014

    work page 2014

  33. [33]

    Evaluating robustness of neural networks with mixed integer programming

    Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. InInternational Conference on Learning Representations, 2019

    work page 2019

  34. [34]

    On adaptive attacks to adversarial example defenses

    Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  35. [35]

    Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification

    Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. InAdvances in Neural Information Processing Systems, 2021. 13

    work page 2021

  36. [36]

    Zico Kolter

    Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. InInternational Conference of Machine Learning, 2018

    work page 2018

  37. [37]

    Automatic perturbation analysis for scalable certified robustness and beyond.Advances in Neural Information Processing Systems, 2020

    Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond.Advances in Neural Information Processing Systems, 2020

    work page 2020

  38. [38]

    Fast and Complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers

    Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and Complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. InInternational Conference on Learning Representations, 2021

    work page 2021

  39. [39]

    Rethinking lipschitz neural networks and certified robustness: A boolean function perspective

    Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  40. [40]

    Boosting the certified robustness of l-infinity distance nets

    Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Boosting the certified robustness of l-infinity distance nets. InInternational Conference on Learning Representations, 2022

    work page 2022

  41. [41]

    Efficient neural network robustness certification with general activation functions

    Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. InAdvances in Neural Information Processing Systems, 2018

    work page 2018

  42. [42]

    Towards stable and efficient training of verifiably robust neural networks

    Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. InInternational Conference on Learning Representations, 2020

    work page 2020

  43. [43]

    General cutting planes for bound-propagation-based neural network verification

    Huan Zhang, Shiqi Wang, Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. General cutting planes for bound-propagation-based neural network verification. In Advances in Neural Information Processing Systems, 2022

    work page 2022

  44. [44]

    Generating less certain adversarial examples improves robust generalization.Transactions on Machine Learning Research, 2025

    Minxing Zhang, Michael Backes, and Xiao Zhang. Generating less certain adversarial examples improves robust generalization.Transactions on Machine Learning Research, 2025

    work page 2025

  45. [45]

    Reliable adversarial distillation with unreliable teachers

    Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. In International Conference on Learning Representations, 2022

    work page 2022

  46. [46]

    Revisiting adversarial robustness distillation: Robust soft labels make student better

    Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. InInternational Conference on Computer Vision, 2021. 14 A Interval Bound Propagation (IBP) IBP has been at the core ofallrecent state-of-the-art certified training methods [ 6, 7, 8, 23, 27, 29]. Here, we formalise ...

    work page 2021

  47. [47]

    [29], with weight 0.5 on MNIST and CIFAR-10, and 0.2 on TinyImageNet, during warmup

    We use the IBP regularisation proposed by Shi et al. [29], with weight 0.5 on MNIST and CIFAR-10, and 0.2 on TinyImageNet, during warmup. In total, we train for 70 epochs on MNIST, 160 epochs on CIFAR-10 with ε= 2 255, 240 epochs on CIFAR-10 with ε= 8 255, and 160 epochs on TinyImageNet. OptimisationIn general, we follow the exact optimisation setup from ...

    work page

  48. [48]

    with a learning rate of 5×10 −4. The learning rate is decayed by a factor of 0.2 at epochs 50 and 20 60 for MNIST, epochs 120 and 140 for CIFAR-10 with ε= 2 255, epochs 200 and 220 for CIFAR-10 with ε= 8 255, and epochs 120 and 140 for TinyImageNet. We use a batch size of 256 for MNIST and 128 for CIFAR-10 and TinyImageNet. Gradients are clipped to 10 in ...

    work page

  49. [49]

    The final hyperparameters used for AD-CERT across all settings are reported in Table 6

    For TinyImageNet, we also observe improvements from increasing wrob and enlarging the PGD attack region used in the empirical branch. The final hyperparameters used for AD-CERT across all settings are reported in Table 6. Table 6: Best hyperparameters for AD-CERT across dataset settings. MNIST CIFAR-10 TinyImageNet Hyperparameter0.1 0.3 2 255 8 255 1 255 ...

    work page