Pith. sign in

REVIEW 4 major objections 4 minor 76 references

Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single student network trained on teacher soft labels with an adversarial loss beats both its teachers and the traditional ensemble at one model's inference cost.

desk verdict The ImageNet flagship number is not an apples-to-apples comparison: teacher-matched, the distilled student loses to the traditional ensemble, so the headline overclaims without a matched averaging baseline over the Plus teacher set. read the letter →

arxiv 1908.08520 v1 pith:L4DPGDLJ submitted 2019-08-22 cs.CV cs.LG

Zhiqiang Shen , Zhankui He , Wanyun Cui , Jiahui Yu , Yutong Zheng , Chenchen Zhu , Marios Savvides This is my paper · ORCID
classification cs.CVcs.LG
keywords adversariallearningknowledgedistillationmulti-modelensemblenoisydatarefinementteacher-studentimageclassificationiterative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an ensemble of pretrained image-classification networks can be compressed into a single student network without losing accuracy, and in fact while gaining accuracy. The student is trained to reproduce, block by block, the soft output distributions of randomly selected teachers, while a set of discriminators tries to tell teacher features from student features and the student tries to fool them. The same scheme is applied to noisy labels by treating teacher soft labels as cleaner supervision than the noisy ground truth and repeating the procedure iteratively. On ImageNet the resulting ResNet-50 student reaches a 21.79% top-1 validation error, beating the 23.85% baseline and the 22.76% traditional ensemble; on the noisy iMaterialist dataset it improves a ResNet-101 baseline from 8.96% to 7.81% top-3 error.

What carries the argument

The mechanism is the joint objective $L = \alpha L_{\text{Sim}} + \beta L_{\text{GAN}}$. $L_{\text{Sim}}$ sums similarity distances, typically cross-entropy between softmax outputs, computed block-wise after an adaptive pooling layer aligns teacher and student feature maps of possibly different sizes; $L_{\text{GAN}}$ sums binary cross-entropy losses from several three-layer fully-connected discriminators, one per aligned block, that try to tell teacher features from student features while the student tries to fool them. A teacher selection module randomly picks one teacher per iteration from a pretrained model zoo, and the teacher parameters stay frozen. For noisy data, the same loss is reused in an iterative loop where the student from round $t$ becomes the teacher for round $t+1$.

What would settle it

Take a dataset with clean labels, corrupt a known fraction of labels to create a noisy training set, train the teacher zoo on the corrupted labels, and compare the teachers' soft-label predictions against the clean labels; if the teachers are not more accurate than the corrupted labels on the clean test set, the iterative-refinement mechanism cannot work as claimed.

Watch

Extended reading notes

Core claim

The central claim is that a student network trained with the objective $L = \alpha L_{\text{Sim}} + \beta L_{\text{GAN}}$, where $L_{\text{Sim}}$ matches the student's block-wise pooled features and final soft outputs to those of a randomly selected teacher and $L_{\text{GAN}}$ is a sum of adversarial losses from stacked discriminators, distills the diverse knowledge of a teacher zoo into a single network. Trained this way, the student attains lower classification error than both the average of the individual teachers and the conventional prediction-averaging ensemble, at the same forward-pass cost as a single network. For noisy data, the paper argues that teacher soft labels are more accurate than the noisy ground-truth labels and introduces an iterative refinement loop in which the previous round's student becomes the next round's teacher, progressively improving validation accuracy on the iMaterialist product dataset.

Load-bearing premise

The method's noisy-data branch assumes that teachers trained on the noisy training set emit soft labels that are more accurate than the original noisy labels; the paper states this without clean-label evidence, so if the teachers have memorized the noise, iterative refinement will propagate errors instead of correcting them.

Editorial extensions

If this is right

  • MEAL delivers a true ensemble's accuracy at a single network's inference cost: test-time FLOPs stay constant as the number of teachers grows, unlike snapshot ensembles.
  • The student can distill from teachers of different architectures and depths, since the adaptive pooling layer maps intermediate feature maps of different sizes to a common length.
  • On the noisy iMaterialist dataset, re-training the student with the previous round's student as teacher improves validation accuracy, and adding data augmentation (MixUp, CutMix, cosine learning-rate schedule) further lowers top-3 error from 7.81% to 6.89%.
  • The benefit of adding more teachers plateaus around four models on CIFAR-10; beyond that, accuracy can slightly drop for VGG-19 and DenseNet, but never below the single-model baseline.
  • A single-model 'ensemble' with adversarial learning alone already improves over the baseline, indicating the adversarial term contributes independently of the teacher diversity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the adversarial feature-matching loss may be acting primarily as a regularizer, so ablating the number of discriminators on a fixed teacher set would reveal how much of the gain comes from distribution matching rather than from teacher diversity.
  • Inference: if teacher soft labels really are more accurate than noisy labels, the iterative refinement loop could be applied to web-scraped data with unknown noise rates, stopping when held-out validation accuracy stops improving.
  • Inference: comparing against a weighted or geometric-mean ensemble of the same teachers would test whether MEAL's edge over the traditional ensemble persists against a stronger averaging baseline.
  • Inference: because the discriminators operate on block-wise aligned features, the same distillation scheme should transfer to dense-prediction tasks such as semantic segmentation or object detection, where logit-level distillation loses spatial information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MEAL, a knowledge-distillation framework that trains a student network using soft labels and block-wise intermediate features from multiple pretrained teacher networks, augmented by adversarial discriminators that try to distinguish teacher features from student features. The method is designed to yield an implicit ensemble with single-network inference cost. The authors also propose an iterative refinement variant for learning with noisy labels, applied to the iMaterialist product dataset. Experiments are reported on CIFAR-10/100, SVHN, ImageNet, and iMaterialist, showing improvements over single models and, in several settings, over traditional prediction averaging.

Significance. If the central claims are substantiated, the paper would offer a practically useful way to compress an ensemble into a single network at no extra inference cost, with evidence across multiple architectures and datasets. The paper is thorough in its experimental scope, includes an ablation of loss components, and provides an explicit FLOPs comparison against traditional ensembles. However, several load-bearing issues remain: the ImageNet comparison that supports the headline number is not teacher-matched, the ablation table appears to contradict the text about the benefit of the adversarial term, and the adversarial joint-update procedure is not written as a valid minimax update. The noisy-data contribution also lacks comparisons to standard noisy-label methods. These issues need to be resolved before the claims can be fully credited.

major comments (4)
  1. [8.4, Table 5, Fig. 9] The ImageNet evidence does not support the claim that MEAL outperforms a traditional ensemble with the same teacher capacity. In Table 5, with the same teacher set (VGG-19 w/BN and ResNet-50), the MEAL student obtains 23.58% top-1 error while the traditional ensemble over those teachers obtains 22.76%, so the matched-teacher comparison goes against the central claim. The headline 21.79% result is from 'Ours PlusJ', which uses more powerful teachers (ResNet-101/152) according to footnote 5, but no traditional ensemble over that Plus teacher set is reported. The abstract, introduction, and Fig. 9 present 21.79% as the MEAL result and compare it with 22.76%, which conflates teacher capacity with the MEAL objective. Please report a traditional ensemble baseline over the Plus teacher set, or revise the claim to the weaker form that is actually supported. Additionally, Fig. 9 reports 21.69% for 'Our Ens.' on ImageNet while Table 5 and the abstract state 21.79%; this inconsistency must be corrected.
  2. [8.3, Table 1] Table 1 contradicts the text about the benefit of adversarial learning. The row with cross-entropy plus intermediate layers reports 6.10% test error, while the row that additionally includes adversarial learning reports 6.17%, and the row using all losses reports 7.57%. The text states that 'Using cross-entropy, intermediate layers and adversarial learning achieve the best result,' but in the table the best result is the configuration without the adversarial term. Since the adversarial discriminator is a central contribution of the paper, please correct the numbers or the text, and explain how the adversarial component helps if the reported ablation shows it hurting.
  3. [5.1, 5.2, Algorithm 1] The sign conventions in the adversarial update are internally inconsistent and as written are not a valid minimax procedure. Equation (8) defines L^j_GAN as an objective that the discriminator should maximize, following Goodfellow et al. However, Algorithm 1 Strategy 2 updates both the discriminator and the student by descending the gradient of alpha*L_Sim + beta*L_GAN, which would minimize rather than maximize the discriminator objective. In contrast, Algorithm 1 Strategy 1 correctly ascends the discriminator gradient and uses +beta*log(1-D(S)) for the student. Please clarify the exact loss definitions and update directions for both strategies; as written, the joint-training variant is not reproducible and cannot be interpreted as adversarial training.
  4. [7, 8.5, Table 6] The noisy-data contribution is not supported by the necessary evidence. Section 7 asserts that 'the soft labels predicted from teacher models usually are more accurate than the noisy labels provided by the noisy dataset,' but no accuracy of the teachers on clean labels is reported to justify this assertion. Table 6 compares MEAL only against its own base models and data-augmentation variants; there are no comparisons with standard noisy-label methods such as MentorNet, CleanNet, loss-correction approaches, or importance reweighting. Without such baselines, the iterative refinement results do not establish that MEAL is competitive for learning with noisy labels. Please add these comparisons or narrow the claim accordingly.
minor comments (4)
  1. [4.3, Eq. (9)] The sentence 'In Eq. 9, x is the concatenation of teacher and student outputs' appears to refer to Eq. (8) or the discriminator input in Fig. 6, not to Eq. (9), which is the summed loss.
  2. [8.2] There is a typo in 'ImagNet' in the paragraph describing the ImageNet setup.
  3. [8.1] The paper says 'We also provide top-1 results in our experiments' for iMaterialist, but Table 6 reports only top-3 error; please either add top-1 numbers or remove the statement.
  4. [8.5] The claim that about 30% of iMaterialist training images have incorrect labels is not accompanied by a reference or a direct measurement; please provide a source or clarify the provenance of this statistic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy gains are empirical results measured against human labels and are not entailed by the MEAL training objective by construction.

full rationale

The derivation chain is: pretrain a teacher zoo on one-hot labels; train a student on the same images using soft labels from randomly selected teachers by minimizing L = αL_Sim + βL_GAN; evaluate on human-annotated validation/test sets. No equation or definition ties the reported test error to the fitted loss weights, block coefficients, or teacher soft labels; the student is not trained on test labels, and no fitted parameter is renamed as a prediction. The citation to the authors' prior AAAI paper [9] is provenance for the preliminary version, not a load-bearing theorem. Two weaknesses were checked and are not circular: (1) Section 7 asserts that 'soft labels predicted from teacher models usually are more accurate than the noisy labels provided by the noisy dataset' without clean-label evidence; this is an empirical premise about teacher reliability, not a reduction of the test result to the training objective. (2) Table 5's 'Ours Plus' result uses stronger teachers (ResNet-101/152, footnote 5) without a matching traditional averaging ensemble, so the headline comparison with 22.76% is not teacher-matched; that is a benchmark-validity limitation, not a circularity. The central claims remain independently testable against external human labels, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces hand-tuned loss trade-offs and block weights, and relies on domain assumptions about teacher quality, most critically that teachers trained on noisy labels provide more accurate soft supervision than the noisy ground truth. No new latent entities are postulated; the discriminators are standard components. The core ensemble claim rests on the well-established assumption that soft labels carry information beyond one-hot labels, while the noisy-data claim rests on an unverified premise about teacher reliability.

free parameters (3)
  • loss trade-off weights alpha and beta = 1 (both)
    Set to 1 by cross-validation in Section 5.1; they balance similarity and adversarial losses.
  • block-wise loss weights = [0.01, 0.05, 1] for 3-block, [0.001, 0.01, 0.05, 0.1, 1] for 5-block
    Hand-chosen coefficients in Section 5.1 to balance contributions of different blocks.
  • discriminator update steps k = 1
    In Section 5.2, k=1 is chosen for the alternate update strategy.
assumptions (4)
  • domain assumption Soft labels from teacher networks are more informative than one-hot labels for training the student.
    Foundation of knowledge distillation; central to the ensemble and noisy data claims in Sections 1 and 7.
  • domain assumption Teacher predictions on noisy data are more accurate than the provided noisy labels.
    Section 7 states this premise as the basis for iterative refinement, but no clean-label evidence for teacher accuracy is provided.
  • domain assumption Teacher and student networks must have the same number of blocks for feature alignment.
    Section 3 requires identical block counts so intermediate outputs can be aligned, limiting applicability to different architectures.
  • domain assumption Adversarial feature matching improves generalization without destabilizing training.
    The paper tests this empirically, but it remains an unproven premise motivating the discriminator design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement." pith.science (2026). https://pith.science/paper/L4DPGDLJ

@misc{pith2026190808520,
  author       = {Pith},
  title        = {Pith review of: Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4DPGDLJ}},
  note         = {Machine review of arXiv:1908.08520}
}
read the original abstract

Generic Image recognition is a fundamental and fairly important visual problem in computer vision. One of the major challenges of this task lies in the fact that single image usually has multiple objects inside while the labels are still one-hot, another one is noisy and sometimes missing labels when annotated by humans. In this paper, we focus on tackling these challenges accompanying with two different image recognition problems: multi-model ensemble and noisy data recognition with a unified framework. As is well-known, usually the best performing deep neural models are ensembles of multiple base-level networks, as it can mitigate the variation or noise containing in the dataset. Unfortunately, the space required to store these many networks, and the time required to execute them at runtime, prohibit their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where the knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. Extensive experiments on CIFAR-10/100, SVHN, ImageNet and iMaterialist Challenge Dataset demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%. On iMaterialist Challenge Dataset, our MEAL obtains a remarkable improvement of top-3 1.15% (official evaluation metric) on a strong baseline model of ResNet-101.

Figures

Figures reproduced from arXiv: 1908.08520 by the authors.

Figure 1
Figure 1. Visualizations of validation images from the Ima [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Left is a training example of class “tobacco shop” [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Overview of our proposed architecture. We input the same image into the teacher and student networks to generate [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: The process of adaptive pooling in forward and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Illustration of our two gradient update strategies. Red line indicates [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Image samples from two benchmarks grouped by clean and noisy/multi-label categories. In each group, black box [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Error rates (%) on CIFAR-10 and CIFAR-100, SVHN and ImageNet datasets. In each figure, the results from left [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Top-1 error rates (%) of training and validation with [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Accuracy curves of our Iterative Refinement method during training under different re-training budgets on iMa￾terialist products dataset. the traditional ensemble 22.76%/6.49%. This shows great potential on large-scale real-size datasets. 8.5 Results of Noisy Data Ref…
Figure 12
Figure 12. Figure 12: Error rate (%) on CIFAR-10 with MobileNet, VGG-19 w/BN and DenseNet. [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Error rates (%) of different re-training time with [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Accuracy of our ensemble method under different [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 15
Figure 15. Figure 15: Probability Distributions between five networks. [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 54 canonical work pages

  1. [1]

    Im- agenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Im- agenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  2. [2]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification

    I. Krasin, T. Duerig, N. Alldrin, A. Veit et al. , “Openimages: A public dataset for large-scale multi-label and multi-class image classification.” https://github.com/openimages, 2016

    work page 2016

  3. [3]

    Dropout: a simple way to prevent neural networks from overfitting

    N. Srivastava, G. E. Hinton, A. Krizhevsky et al. , “Dropout: a simple way to prevent neural networks from overfitting.” JMLR, 2014

    work page 2014

  4. [4]

    Regu- larization of neural networks using dropconnect,

    L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, “Regu- larization of neural networks using dropconnect,” in International conference on machine learning, 2013, pp. 1058–1066

    work page 2013

  5. [5]

    Deep networks with stochastic depth,

    G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep networks with stochastic depth,” inEuropean conference on computer vision. Springer, 2016, pp. 646–661

    work page 2016

  6. [6]

    Swapout: Learning an ensemble of deep architectures,

    S. Singh, D. Hoiem, and D. Forsyth, “Swapout: Learning an ensemble of deep architectures,” in Advances in neural information processing systems, 2016, pp. 28–36

    work page 2016

  7. [7]

    Visualizing data using t-sne,

    L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of machine learning research , vol. 9, no. Nov, pp. 2579–2605, 2008

    2008

  8. [8]

    Shake-shake regularization,

    X. Gastaldi, “Shake-shake regularization,” arXiv preprint arXiv:1705.07485, 2017

    arXiv 2017

Show all 76 references
  1. [9]

    Meal: Multi-model ensemble via adversarial learning,

    Z. Shen, Z. He, and X. Xue, “Meal: Multi-model ensemble via adversarial learning,” in AAAI, 2019

    2019

  2. [10]

    Snapshot ensembles: Train 1, get m for free,

    G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” inICLR, 2017

    2017

  3. [11]

    Neural network ensembles,

    L. K. Hansen and P . Salamon, “Neural network ensembles,” IEEE transactions on pattern analysis and machine intelligence , vol. 12, no. 10, pp. 993–1001, 1990

    1990

  4. [12]

    When networks disagree: En- semble methods for hybrid neural networks,

    M. P . Perrone and L. N. Cooper, “When networks disagree: En- semble methods for hybrid neural networks,” in How We Learn; How We Remember: Toward an Understanding of Brain and Neural Systems: Selected Papers of Leon N Cooper . World Scientific, 1995, pp. 342–358

    1995

  5. [13]

    Neural network ensembles, cross validation, and active learning,

    A. Krogh and J. Vedelsby, “Neural network ensembles, cross validation, and active learning,” in Advances in neural information processing systems, 1995, pp. 231–238

    1995

  6. [14]

    Ensemble methods in machine learning,

    T. G. Dietterich, “Ensemble methods in machine learning,” in International workshop on multiple classifier systems , 2000, pp. 1–15

    2000

  7. [15]

    Simple and scalable predictive uncertainty estimation using deep ensembles,

    B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems , 2017, pp. 6402–6413

    2017

  8. [16]

    Knowledge distillation by on-the-fly native ensemble,

    X. Zhu, S. Gong et al., “Knowledge distillation by on-the-fly native ensemble,” in Advances in Neural Information Processing Systems , 2018, pp. 7517–7527

    2018

  9. [17]

    Diverse ensemble evolution: Curriculum data-model marriage,

    T. Zhou, S. Wang, and J. A. Bilmes, “Diverse ensemble evolution: Curriculum data-model marriage,” in Advances in Neural Informa- tion Processing Systems, 2018, pp. 5905–5916

    2018

  10. [18]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  11. [19]

    Semi-supervised knowledge transfer for deep learning from private training data,

    N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Tal- war, “Semi-supervised knowledge transfer for deep learning from private training data,” in ICLR, 2017

    2017

  12. [20]

    Learning from noisy labels with distillation,

    Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1910–1918

    2017

  13. [21]

    A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,

    J. Yim, D. Joo, J. Bae, and J. Kim, “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4133–4141

    2017

  14. [22]

    Generative adversarial nets,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680

    2014

  15. [23]

    Label refinery: Improving imagenet classification through label progression,

    H. Bagherinezhad, M. Horton, M. Rastegari, and A. Farhadi, “Label refinery: Improving imagenet classification through label progression,” in ECCV, 2018. 12

    2018

  16. [24]

    Wasserstein generative adversarial networks,

    M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International conference on machine learn- ing, 2017, pp. 214–223

    2017

  17. [25]

    Improved training of wasserstein gans,

    I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in Advances in neural information processing systems, 2017, pp. 5767–5777

    2017

  18. [26]

    On convergence and stability of gans,

    N. Kodali, J. Abernethy, J. Hays, and Z. Kira, “On convergence and stability of gans,” arXiv preprint arXiv:1705.07215, 2017

    2017 arXiv

  19. [27]

    Many paths to equilibrium: Gans do not need to decrease a divergence at every step,

    W. Fedus, M. Rosca, B. Lakshminarayanan, A. M. Dai, S. Mo- hamed, and I. Goodfellow, “Many paths to equilibrium: Gans do not need to decrease a divergence at every step,” arXiv preprint arXiv:1710.08446, 2017

    2017 arXiv

  20. [28]

    Least squares generative adversarial networks,

    X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2794– 2802

    2017

  21. [29]

    Image-to-image translation with conditional adversarial networks,

    P . Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134

    2017

  22. [30]

    Unpaired image-to- image translation using cycle-consistent adversarial networks,

    J.-Y. Zhu, T. Park, P . Isola, and A. A. Efros, “Unpaired image-to- image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2223–2232

    2017

  23. [31]

    Toward multimodal image-to-image translation,

    J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman, “Toward multimodal image-to-image translation,” in Advances in Neural Information Processing Systems, 2017, pp. 465– 476

    2017

  24. [32]

    Unsupervised image-to-image translation networks,

    M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in neural information processing systems, 2017, pp. 700–708

    2017

  25. [33]

    Multimodal unsupervised image-to-image translation,

    X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 172–189

    2018

  26. [34]

    Towards instance-level image-to-image translation,

    Z. Shen, M. Huang, J. Shi, X. Xue, and T. Huang, “Towards instance-level image-to-image translation,” in CVPR, 2019

    2019

  27. [35]

    Image generation from scene graphs,

    J. Johnson, A. Gupta, and L. Fei-Fei, “Image generation from scene graphs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1219–1228

    2018

  28. [36]

    Finding tiny faces in the wild with generative adversarial network,

    Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Finding tiny faces in the wild with generative adversarial network,” pp. 21–30, 2018

    2018

  29. [37]

    Net2net: Accelerating learning via knowledge transfer,

    T. Chen, I. Goodfellow, and J. Shlens, “Net2net: Accelerating learning via knowledge transfer,” in ICLR, 2016

    2016

  30. [38]

    Training shallow and thin net- works for acceleration via knowledge distillation with conditional adversarial networks,

    Z. Xu, Y.-C. Hsu, and J. Huang, “Training shallow and thin net- works for acceleration via knowledge distillation with conditional adversarial networks,” arXiv preprint arXiv:1709.00513, 2017

    2017 arXiv

  31. [39]

    Mean teachers are better role mod- els: Weight-averaged consistency targets improve semi-supervised deep learning results,

    A. Tarvainen and H. Valpola, “Mean teachers are better role mod- els: Weight-averaged consistency targets improve semi-supervised deep learning results,” in Advances in neural information processing systems, 2017, pp. 1195–1204

    2017

  32. [40]

    Large scale distributed neural network training through online distillation,

    R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,” in ICLR, 2018

    2018

  33. [41]

    Darkrank: Accelerating deep metric learning via cross sample similarities transfer,

    Y. Chen, N. Wang, and Z. Zhang, “Darkrank: Accelerating deep metric learning via cross sample similarities transfer,” in Thirty- Second AAAI Conference on Artificial Intelligence , 2018

    2018

  34. [42]

    Relational knowledge distillation,

    W. Park, D. Kim, Y. Lu, and M. Cho, “Relational knowledge distillation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3967–3976

    2019

  35. [43]

    Knowledge transfer via distillation of activation boundaries formed by hidden neurons,

    B. Heo, M. Lee, S. Yun, and J. Y. Choi, “Knowledge transfer via distillation of activation boundaries formed by hidden neurons,” arXiv preprint arXiv:1811.03233, 2018

    2018 arXiv

  36. [44]

    Born again neural networks,

    T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” arXiv preprint arXiv:1805.04770, 2018

    2018 arXiv

  37. [45]

    Revisiting knowledge transfer for training object class detectors,

    J. Uijlings, S. Popov, and V . Ferrari, “Revisiting knowledge transfer for training object class detectors,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 1101–1110

    2018

  38. [46]

    Moonshine: Distilling with cheap convolutions,

    E. J. Crowley, G. Gray, and A. J. Storkey, “Moonshine: Distilling with cheap convolutions,” in Advances in Neural Information Pro- cessing Systems, 2018, pp. 2888–2898

    2018

  39. [47]

    Snapshot distillation: Teacher-student optimization in one generation,

    C. Yang, L. Xie, C. Su, and A. L. Yuille, “Snapshot distillation: Teacher-student optimization in one generation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 2859–2868

    2019

  40. [48]

    Learning to learn from noisy labeled data,

    J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli, “Learning to learn from noisy labeled data,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5051–5059

    2019

  41. [49]

    Toward robustness against label noise in training deep discriminative neural networks,

    A. Vahdat, “Toward robustness against label noise in training deep discriminative neural networks,” in Advances in Neural Information Processing Systems, 2017, pp. 5596–5605

    2017

  42. [50]

    Learning from massive noisy labeled data for image classification,

    T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2691–2699

    2015

  43. [51]

    Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,

    L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” arXiv preprint arXiv:1712.05055, 2017

    2017 arXiv

  44. [52]

    Training deep neural-networks using a noise adaptation layer,

    J. Goldberger and E. Ben-Reuven, “Training deep neural-networks using a noise adaptation layer,” in ICLR, 2016

    2016

  45. [53]

    Learning from noisy large-scale datasets with minimal supervision,

    A. Veit, N. Alldrin, G. Chechik, I. Krasin, A. Gupta, and S. Be- longie, “Learning from noisy large-scale datasets with minimal supervision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 839–847

    2017

  46. [54]

    Training convolutional networks with noisy labels,

    S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fer- gus, “Training convolutional networks with noisy labels,” arXiv preprint arXiv:1406.2080, 2014

    2014 arXiv

  47. [55]

    Making deep neural networks robust to label noise: A loss correc- tion approach,

    G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correc- tion approach,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1944–1952

    2017

  48. [56]

    Learning to reweight examples for robust deep learning,

    M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight examples for robust deep learning,” arXiv preprint arXiv:1803.09050, 2018

    2018 arXiv

  49. [57]

    Cleannet: Transfer learning for scalable image classifier training with label noise,

    K.-H. Lee, X. He, L. Zhang, and L. Yang, “Cleannet: Transfer learning for scalable image classifier training with label noise,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5447–5456

    2018

  50. [58]

    Classification with noisy labels by importance reweighting,

    T. Liu and D. Tao, “Classification with noisy labels by importance reweighting,” IEEE Transactions on pattern analysis and machine intelligence, vol. 38, no. 3, pp. 447–461, 2015

    2015

  51. [59]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009

    2009

  52. [60]

    Reading digits in natural images with unsupervised feature learning,

    Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in NIPS workshop on deep learning and unsupervised feature learning, vol. 2011, no. 2, 2011, p. 5

    2011

  53. [61]

    Very deep convolutional net- works for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional net- works for large-scale image recognition,” in ICLR, 2015

    2015

  54. [62]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016

    2016

  55. [63]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4700–4708

    2017

  56. [64]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications,

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  57. [65]

    Automatic differ- entiation in pytorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differ- entiation in pytorch,” 2017

    2017

  58. [66]

    Deeply-supervised nets

    C.-Y. Lee, S. Xie, P . W. Gallagher et al., “Deeply-supervised nets.” in AISTATS, 2015

    2015

  59. [67]

    Fitnets: Hints for thin deep nets,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” in International Conference on Learning Representations, 2015

    2015

  60. [68]

    Fractalnet: Ultra-deep neural networks without residuals,

    G. Larsson, M. Maire, and G. Shakhnarovich, “Fractalnet: Ultra-deep neural networks without residuals,” arXiv preprint arXiv:1605.07648, 2016

    2016 arXiv

  61. [69]

    Learning efficient convolutional networks through network slimming,

    Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2736–2744

    2017

  62. [70]

    Maxout networks,

    I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” in ICML, 2013

    2013

  63. [71]

    Imagenet classifi- cation with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifi- cation with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105

    2012

  64. [72]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015. 13

    2015 arXiv

  65. [73]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y. Jia, P . Sermanet et al., “Going deeper with convolutions,” in CVPR, 2015

    2015

  66. [74]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in International Conference on Learning Representations, 2018

    2018

  67. [75]

    Cutmix: Regularization strategy to train strong classifiers with localizable features,

    S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” arXiv preprint arXiv:1905.04899, 2019

    1905 arXiv

  68. [76]

    Sgdr: Stochastic gradient descent with warm restarts,

    I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.