Pith. sign in

REVIEW 3 major objections 5 minor 43 references

A teacher-only score ranks data augmentations for low-bit distillation, lifting accuracy by up to 9 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:36 UTC pith:KRTNQISF

load-bearing objection Cheap teacher-side DA ranking for QAT+KD is a good idea, but the GCMI extension for mixed-label augmentations is mathematically undefined as written, and the validation needs seeds and a code release. the 3 major comments →

arxiv 2509.03850 v1 pith:KRTNQISF submitted 2025-09-04 cs.LG cs.CV

Data-Augmented Quantization-Aware Knowledge Distillation

Justin Kur , Kaiqi Zhao This is my paper
classification cs.LG cs.CV
keywords data augmentation selectionquantization-aware trainingknowledge distillationcontextual mutual informationlow-bit quantizationteacher-student distillationdata augmentation search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that for a fixed full-precision teacher, the best data augmentation to use when distilling into a low-bit quantized student can be chosen before any student is trained. It proposes the score M(f) = DEV(f) − CMI(f), where CMI is the contextual mutual information in the teacher's soft predictions about each image beyond its class label, and DEV penalizes the average distance between each class's mean teacher prediction and the one-hot label. Ranking candidate augmentations by this score and using the lowest-scoring one improved top-1 accuracy of established QAT and KD pipelines in the paper's experiments by up to 9 percentage points. Because the score costs only two forward passes through the training set and never touches the quantizer or the student, it gives practitioners a cheap way to prune an otherwise expensive augmentation search.

Core claim

The paper's central claim is that Eq. 4, M(f)=DEV(f)-CMI_emp(f), is a reliable teacher-side ranking of data augmentations for KD-assisted QAT, and that the augmentation minimizing M is the right one to use. CMI_emp is the empirical contextual mutual information of the teacher's predictions: information each augmented image carries beyond what its class label already determines. DEV is the average divergence between each class's mean teacher prediction (its empirical prototype) and the one-hot ground-truth vector. To handle augmentations such as CutMix and MixUp that assign fractional labels, the paper replaces CMI_emp with GCMI and rebuilds the prototypes from label-weighted teacher predicti

What carries the argument

The load-bearing object is the teacher-only score M(f) = DEV(f) − CMI_emp(f), computed over the training set in two forward passes of the full-precision teacher. The first pass builds each class's average soft prediction (prototype); the second pass averages the per-sample divergence between each augmented image's prediction and its class prototype, giving CMI. DEV punishes prototypes that drift from one-hot labels, so the minimum of M selects an augmentation that gives the teacher rich non-label information while staying faithful to ground truth. For mixed-label augmentations such as CutMix and MixUp, the generalized GCMI term and normalized label-weighted prototypes extend the same recipe

Load-bearing premise

The claim rests on the assumption that the teacher's full-precision prediction statistics on augmented images predict how accurate a quantized student will finally be when trained with that augmentation under KD, even though the metric never trains the student.

What would settle it

Take a dataset, teacher, and bit-width not in the paper (for example, ImageNet-1K with a W4A4 student), rank the seven augmentations by M(f), train a quantized student under each, and compare the M ranking with final top-1 accuracies. If the top-ranked augmentations do not land among the best students, or the Spearman correlation is near zero or positive, the teacher-only transfer claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A fixed teacher lets a practitioner rank many augmentation policies for roughly the cost of two forward passes, replacing a full grid search over DA choices in QAT/KD.
  • The metric is independent of the quantizer and student bit-width, so the same ranking can be reused across different students and precision settings without recomputation.
  • In the paper's reported comparisons, the selected augmentation improves existing QAT methods (EWGS, PACT, LSQ, DoReFa) and KD methods (CRD, NST, RKD, SP, AT, CC, traditional KD), and beats the KD+QAT baselines QKD, SPEQ, and SQAKD.
  • At W4A4 and above the quantized student trained with the selected DA often exceeds the full-precision teacher's accuracy in the paper's experiments.
  • Initializing the student with the teacher's weights adds roughly 5 percentage points on top of the selected DA in the paper's ablation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the teacher-only transfer holds beyond quantization, the same two-pass score could rank augmentations for ordinary KD into smaller, pruned, or width-reduced students, making DA search nearly free for a much broader class of distillation problems.
  • The paper applies the metric once before training and keeps the chosen DA fixed; a natural extension is to recompute M periodically and schedule or mix augmentations over training, which could capture epoch-dependent DA effects the static ranking misses.
  • The DEV term uses one-hot labels, so the metric may need reweighting under label noise or label smoothing; the GCMI machinery already anticipates non-one-hot targets, so this extension is small.
  • The large gap between M and variance-based ranking on Tiny ImageNet suggests the teacher's predictive entropy is most informative in low-bit regimes; testing the same comparison at W8A8 or full precision would show where the teacher-only signal degrades.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a teacher-only metric M(f)=DEV(f)-CMI_emp(f) for ranking data augmentations (DAs) in knowledge-distillation-assisted quantization-aware training (QAT). The metric is computed from the full-precision teacher's predictions over augmented training images: DEV measures the KL divergence between each class's average prediction and the one-hot label, and CMI measures the average KL divergence between individual predictions and the class prototype. A generalized version (GCMI, Eq. 5-6) is introduced to handle mixed-label augmentations such as CutMix and MixUp. The authors report that the DA selected by minimizing M improves top-1 accuracy of several QAT and KD baselines on CIFAR-10, CIFAR-100, and Tiny ImageNet, with gains up to about 9 percentage points, and that the metric has better Spearman correlation with quantized student accuracy than the variance-based baseline of Wang et al.

Significance. If the central claim holds, the paper offers a practical and cheap way to select DAs for KD-assisted QAT: two forward passes of a trained teacher, no student training, and no modification to the QAT/KD algorithm. This is a useful contribution because DA choice is often treated as an expensive grid-search hyperparameter, and the paper's compatibility with any KD/QAT method makes the result broadly applicable. The empirical evaluation is reasonably broad, covering multiple architectures, datasets, quantizers, bit-widths, and KD methods. However, the validity of the metric for mixed-label augmentations is mathematically questionable as written, and the empirical evidence is weakened by the absence of uncertainty quantification and by at least one contrary result in the ablation study. These issues need to be resolved before the paper's claims can be accepted.

major comments (3)
  1. [§3.3, Eq. (5)–(6)] GCMI is ill-defined. In Eq. (6), Z_j = Σ_k w_jk P_{x_k} is a sum of probability vectors, and dividing by the L2 norm ||Z_j|| yields a vector whose entries do not sum to 1. Therefore Q_i^emp = Σ_j P_{x_ij} Z_j/||Z_j|| is not a probability distribution, and the KL divergences in Eq. (5) are undefined. The metric as written cannot evaluate CutMix/MixUp, yet Table 1 includes CutMix and §3.3 claims to handle 'arbitrarily complex augmentations.' Moreover, for single-label data GCMI does not reduce to Eq. (1)–(2), because the L2-normalized prototype differs from the class average Q_y^emp. Please use L1 normalization (or another explicit construction that yields a probability vector), state it clearly, and recompute the affected ranking correlations.
  2. [Eq. (4)] The metric M(f) = DEV(f) - GCMI_emp(f) is a plain difference with no justification for equal weighting or scale alignment. DEV is an average over C classes of KL(1_y || Q_y), while CMI is an average over N samples; the two terms have different magnitudes and dependencies, so the ranking induced by Eq. (4) may be dominated by whichever term happens to be larger. No ablation is reported for a weighted version M = DEV - λ·CMI, and the two terms are never reported separately. Since Eq. (4) is the core methodological contribution, the equal-weight choice is load-bearing. Please provide per-DA values of DEV and CMI and a sensitivity analysis over λ.
  3. [§4.1, Table 6] The central empirical claim that selecting the lowest-M DA 'significantly improves' QAT/KD performance is not uniformly supported. In Table 6, on CIFAR-100 ResNet-32 at W2A2 the selected Trivial Augment achieves 70.20%, which is 0.42 pp lower than the limited-DA baseline (70.62%). Figure 1b also shows the selected DA is not the best (by 0.3 pp). All accuracy numbers appear to be single runs; no standard deviations or significance tests are given. Several headline gaps are in the 0.1–0.3 pp range, which may be within run-to-run noise for these settings. In addition, the Spearman correlations in Table 1 are computed over only 7 DAs, giving wide confidence intervals. Please report repeated-seed results and statistically justify the strength of the claims.
minor comments (5)
  1. [§4.1] Typo: 'perfomance' should be 'performance'.
  2. [References] References [34] and [35] are the same paper (Wang et al., NeurIPS 2022) and should be consolidated.
  3. [§3.3, Eq. (6)] The notation w_jk is not precisely defined. State that w_jk is the label weight for class j of image k under the augmentation and clarify the normalization constraints (e.g., sum_j w_jk = 1).
  4. [Fig. 1 caption] The caption should indicate which horizontal line corresponds to the DA selected by 'Ours' and which to the DA selected by 'Var', since the figure text only says they are 'marked with horizontal lines'.
  5. [§4.2] In Table 2, the row 'Ours (EWGS)' is clear, but in Table 3 the term 'QAT' is used both for the generic method and for the No-KD baseline. Consider renaming the baseline column 'QAT-only' to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the augmentation-ranking metric is a fixed teacher-only heuristic validated against independently trained quantized students; the only self-citation is a non-load-bearing baseline.

full rationale

The paper's central claim is that the teacher-only metric M(f)=DEV(f)-CMI_emp(f) ranks data augmentations for KD-assisted QAT. The components come from prior work (CMI, Eq. 1-2) and a simple centroid-deviation term (Eq. 3), combined with equal weights in Eq. 4; no coefficient is fitted to the student accuracies it later predicts. The validation in Table 1 and Fig. 1 compares the metric's ranking to independently obtained quantized-student top-1 accuracies, so the prediction target is not an input to the metric. There is no fitted parameter renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled in via citation. The only self-citation is SQAKD [41], by the second author, which is used for teacher-weight initialization and as a comparison baseline; the DA-ranking claim does not reduce to it. A separate mathematical concern exists in Sec. 3.3, Eqs. 5-6: L2-normalizing Z_j yields a Q_i^emp that is not a probability vector, so the KL divergence in Eq. 5 is formally undefined for CutMix/MixUp. This is a correctness/omitted-proof issue, not a circular reduction, and it does not change the circularity verdict. Overall, the derivation is self-contained with respect to the stated claim, and the self-citation is minor and non-load-bearing.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central metric contains no constants fitted to accuracy, so the main contributions rest on assumptions about teacher-student transfer, the equal-weight combination of CMI and DEV, and the correctness of GCMI. The only tuned numbers are training-loss weights from an ablation (Table 7), which are peripheral to the metric itself.

free parameters (2)
  • DEV/CMI combination weights = 1 and -1
    Eq 4 combines DEV and CMI with unit weights by fiat; no sensitivity analysis or learned coefficients.
  • CE/KD loss weights = gamma=1, alpha=2 (3 and 6 on MobileNetV2)
    Tuned via ablation in Table 7 and reused for most experiments; not part of the metric itself.
axioms (4)
  • domain assumption CMI (Eq 1-2) measures useful dark knowledge for distillation.
    Adopted from Ye et al. [36]; the entire metric assumes maximizing CMI while keeping class prototypes near one-hot improves distillation.
  • ad hoc to paper M=DEV-CMI with equal weighting is the right objective for ranking DAs.
    Eq 4 combines two differently-sized KL averages with no derivation or sensitivity analysis; this is a hand-specified criterion.
  • domain assumption Teacher model statistics predict quantized student performance under the same DA.
    Stated in Sec 3.2-3.3 and validated only via three Spearman correlation tables; this is the paper's core proxy.
  • ad hoc to paper Generalized CMI (Eq 5-6) correctly handles mixed-label augmentations.
    The printed equation uses P_{xij} instead of label weights w_{ij}, so this axiom currently has an internal inconsistency.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 13036 in / 14108 out tokens · 135235 ms · 2026-08-05T10:36:32.298088+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Data-Augmented Quantization-Aware Knowledge Distillation." pith.science (2026). https://pith.science/paper/KRTNQISF

@misc{pith2026250903850,
  author       = {Pith},
  title        = {Pith review of: Data-Augmented Quantization-Aware Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KRTNQISF}},
  note         = {Machine review of arXiv:2509.03850}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Quantization-aware training (QAT) and Knowledge Distillation (KD) are combined to achieve competitive performance in creating low-bit deep learning models. Existing KD and QAT works focus on improving the accuracy of quantized models from the network output perspective by designing better KD loss functions or optimizing QAT's forward and backward propagation. However, limited attention has been given to understanding the impact of input transformations, such as data augmentation (DA). The relationship between quantization-aware KD and DA remains unexplored. In this paper, we address the question: how to select a good DA in quantization-aware KD, especially for the models with low precisions? We propose a novel metric which evaluates DAs according to their capacity to maximize the Contextual Mutual Information--the information not directly related to an image's label--while also ensuring the predictions for each class are close to the ground truth labels on average. The proposed method automatically ranks and selects DAs, requiring minimal training overhead, and it is compatible with any KD or QAT algorithm. Extensive evaluations demonstrate that selecting DA strategies using our metric significantly improves state-of-the-art QAT and KD works across various model architectures and datasets.

Figures

Figures reproduced from arXiv: 2509.03850 by Justin Kur, Kaiqi Zhao.

Figure 1
Figure 1. Figure 1: Scatter visualization of the quantized model’s top-1 test [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The LayerCam [14] heatmaps for the teacher network (left), the quantized model with data augmentations (center), and the quantized model without data augmentations (right). The color indicates the area of focus of the network, on a scale of red (low focus) to violet (high focus). Loss Terms. Tab. 7 shows our results with different hyper￾parameter settings for loss terms trained under minimal data augmentat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 38 canonical work pages

  1. [1]

    Stochastic precision ensemble: Self-knowledge distil- lation for quantized deep neural networks

    Yoonho Boo, Sungho Shin, Jungwook Choi, and Wonyong Sung. Stochastic precision ensemble: Self-knowledge distil- lation for quantized deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8):6794–6802,

    work page

  2. [2]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4794– 4802, 2019. 3

    work page 2019

  3. [3]

    Pact: Parameterized clipping activation for quantized neural networks, 2018

    Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018. 2, 4, 5, 7

    work page 2018

  4. [4]

    Binarized neural networks: Train- ing deep neural networks with weights and activations con- strained to +1 or -1, 2016

    Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Train- ing deep neural networks with weights and activations con- strained to +1 or -1, 2016. 2

    work page 2016

  5. [5]

    Autoaugment: Learning augmentation policies from data

    Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasude- van, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018. 2, 4

    Pith/arXiv arXiv 2018

  6. [6]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition work- shops, pages 702–703, 2020. 2, 4

    work page 2020

  7. [7]

    Esser, Jeffrey L

    Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization, 2020. 2, 4, 5, 7

    work page 2020

  8. [8]

    A survey of quan- tization methods for efficient neural network inference

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quan- tization methods for efficient neural network inference. In Low-power computer vision, pages 291–326. Chapman and Hall/CRC, 2022. 2

    work page 2022

  9. [9]

    Augmix: A simple data processing method to improve robustness and uncertainty

    Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019. 2, 4

    Pith/arXiv arXiv 1912

  10. [10]

    Distilling the knowledge in a neural network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. 2

    2015

  11. [11]

    Population based augmentation: Efficient learning of aug- mentation policy schedules

    Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of aug- mentation policy schedules. In International conference on machine learning, pages 2731–2741. PMLR, 2019. 2

    work page 2019

  12. [12]

    Relation networks for object detection

    Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3588–3597, 2018. 1

    work page 2018

  13. [13]

    Like what you like: Knowl- edge distill via neuron selectivity transfer, 2017

    Zehao Huang and Naiyan Wang. Like what you like: Knowl- edge distill via neuron selectivity transfer, 2017. 2, 4, 7

    work page 2017

  14. [14]

    Layercam: Exploring hierarchical class activation maps

    Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps. IEEE Transactions on Image Processing, PP:1–1, 2021. 8

    work page 2021

  15. [15]

    Learning to quantize deep networks by optimizing quantization intervals with task loss, 2018

    Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss, 2018. 2

    work page 2018

  16. [16]

    Qkd: Quantization-aware knowledge distillation,

    Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and No- jun Kwak. Qkd: Quantization-aware knowledge distillation,

    work page

  17. [17]

    Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018

    Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018. 1

    work page 2018

  18. [18]

    Learning multiple layers of features from tiny images.(2009), 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.(2009), 2009. 4

    work page 2009

  19. [19]

    Tiny imagenet visual recognition challenge

    Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 4

    work page 2015

  20. [20]

    Network quantization with element-wise gradient scaling, 2021

    Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradient scaling, 2021. 2, 4, 5, 7

    work page 2021

  21. [21]

    Mqbench: Towards reproducible and deployable model quan- tization benchmark, 2022

    Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan. Mqbench: Towards reproducible and deployable model quan- tization benchmark, 2022. 5

    work page 2022

  22. [22]

    Fast autoaugment, 2019

    Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment, 2019. 2

    work page 2019

  23. [23]

    Trivialaugment: Tuning- free yet state-of-the-art data augmentation

    Samuel G M¨uller and Frank Hutter. Trivialaugment: Tuning- free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774–782, 2021. 2, 4

    work page 2021

  24. [24]

    Re- lational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Re- lational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 4, 7

    work page 2019

  25. [25]

    Correla- tion congruence for knowledge distillation

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correla- tion congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 4, 7

    work page 2019

  26. [26]

    Correlation congruence for knowledge distillation, 2019

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation, 2019. 1

    work page 2019

  27. [27]

    Collabo- rative multi-teacher knowledge distillation for learning low bit-width deep neural networks

    Cuong Pham, Tuan Hoang, and Thanh-Toan Do. Collabo- rative multi-teacher knowledge distillation for learning low bit-width deep neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6435–6443, 2023. 3

    work page 2023

  28. [28]

    Xnor-net: Imagenet classification using binary convolutional neural networks, 2016

    Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, 2016. 2

    work page 2016

  29. [29]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge,

  30. [30]

    Teachaugment: Data augmentation optimiza- tion using teacher knowledge

    Teppei Suzuki. Teachaugment: Data augmentation optimiza- tion using teacher knowledge. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10894–10904, 2022. 2

    work page 2022

  31. [31]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015. 1 9

    work page 2015

  32. [32]

    Contrastive representation distillation, 2022

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation, 2022. 2, 4, 7

    work page 2022

  33. [33]

    Similarity-preserving knowl- edge distillation, 2019

    Frederick Tung and Greg Mori. Similarity-preserving knowl- edge distillation, 2019. 4, 7

    work page 2019

  34. [34]

    What makes a ”good” data augmentation in knowledge distillation - a statistical perspective

    Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes a ”good” data augmentation in knowledge distillation - a statistical perspective. In Advances in Neural Information Processing Systems, pages 13456–13469. Curran Associates, Inc., 2022. 2, 4, 5, 6

    work page 2022

  35. [35]

    What makes a” good” data augmentation in knowledge distillation- a statistical perspective

    Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes a” good” data augmentation in knowledge distillation- a statistical perspective. Advances in Neural Information Processing Systems, 35:13456–13469, 2022. 1

    work page 2022

  36. [36]

    Bayes conditional distribution estimation for knowledge distillation based on conditional mutual informa- tion

    Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and En- Hui Yang. Bayes conditional distribution estimation for knowledge distillation based on conditional mutual informa- tion. arXiv preprint arXiv:2401.08732, 2024. 3

    Pith/arXiv arXiv 2024

  37. [37]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features, 2019

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features, 2019. 4

    work page 2019

  38. [38]

    Paying more atten- tion to attention: Improving the performance of convolutional neural networks via attention transfer, 2017

    Sergey Zagoruyko and Nikos Komodakis. Paying more atten- tion to attention: Improving the performance of convolutional neural networks via attention transfer, 2017. 2, 4, 7

    work page 2017

  39. [39]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion, 2018. 4

    work page 2018

  40. [40]

    Adversarial autoaugment, 2019

    Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment, 2019. 2

    work page 2019

  41. [41]

    Self-supervised quantization- aware knowledge distillation

    Kaiqi Zhao and Ming Zhao. Self-supervised quantization- aware knowledge distillation. In Proceedings of The 27th International Conference on Artificial Intelligence and Statis- tics, pages 4375–4383. PMLR, 2024. 1, 2, 3, 4, 6, 7

    work page 2024

  42. [42]

    Dorefa-net: Training low bitwidth convolu- tional neural networks with low bitwidth gradients, 2018

    Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolu- tional neural networks with low bitwidth gradients, 2018. 2, 4, 5, 7

    work page 2018

  43. [43]

    Towards effective low-bitwidth convolutional neural networks

    Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low-bitwidth convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7920–7928,

    work page