Expandable Residual Approximation for Knowledge Distillation

Binghui Chen; Qixiang Ye; Yunfan Liu; Zhaoyi Yan

REVIEW 4 major objections 4 minor 86 references

Expandable Residual Approximation for Knowledge Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A stepwise residual approximation of the teacher–student feature gap reduces distillation difficulty and yields gains on ImageNet and COCO.

desk verdict A plausible KD method with a shaky theoretical wrapper and unverifiable experiments in the text I saw; the stress-test concern about the multi-step loss is real. read the letter →

arxiv 2508.16050 v1 pith:FLXUEI7O submitted 2025-08-22 cs.CV

Zhaoyi Yan , Binghui Chen , Yunfan Liu , Qixiang Ye This is my paper

classification cs.CV

keywords knowledgedistillationresidualapproximationfeaturealignmentteacher-studentcapacitygapmulti-branchnetworkImageNetclassificationobjectdetectionStone-Weierstrassanalogy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Expandable Residual Approximation (ERA), a knowledge-distillation method that breaks the teacher–student feature gap into K sequential residual terms instead of forcing a single projection. Each residual is learned by a separate branch of a Multi-Branched Residual Network, effectively dividing a hard approximation problem into smaller steps. The method reports a 1.41% Top-1 accuracy improvement on ImageNet and a 1.40 AP improvement on MS COCO object detection. A reader should care because the approach offers a direct, plug-in-friendly way to ease the capacity gap that often limits knowledge transfer from large teachers to small students.

What carries the argument

The central mechanism is the Multi-Branched Residual Network (MBRNet): a cascade of K small FC–BN–ReLU stacks, where branch k learns the residual Δfk = ft − f̂k−1 after summing the first k−1 branch outputs. The mathematical identity is the expansion ft ≈ fs + Σ_{i=1}^K P_i Δf̂i, with stepwise losses LFD_k = ||Δf_k − P_k Δf̂_k||^2. A second component, Teacher Weight Integration (TWI), reuses the teacher's classifier head weights to compute logits for each branch, reducing capacity mismatch without adding a trainable head.

What would settle it

Train ERA with all K branches optimized jointly by a single loss on the final approximation f̂K instead of the per-step residual losses, and compare test accuracy on ImageNet. If the joint version matches or exceeds the sequential version, then the sequential residual decomposition is not the source of the gain, and the claim that multiple steps reduce approximation difficulty would be undercut.

Watch

Extended reading notes

Core claim

The central discovery is that the residual knowledge between teacher and student features can be approximated progressively in K steps, with each step fitting a smaller residual, rather than all at once. Formally, the paper treats the student output fs as a zero-order approximation and seeks to approximate ft − P0fs by a sum of K network outputs. Each branch k is trained to predict the residual left by the previous branches, with a per-step loss on that residual. The paper implements this idea as Multi-Branched Residual Network (MBRNet) plus Teacher Weight Integration (TWI), and the experimental gains suggest that the divide-and-conquer residual decomposition reduces the difficulty of mimick

Load-bearing premise

The method assumes that fitting K residual terms one after another yields a better final approximation of the teacher's features than fitting one projection, and that the small branch networks can actually represent those residuals; the Stone-Weierstrass analogy does not guarantee this for deep-network features.

Editorial extensions

If this is right

If the residual decomposition is as effective as reported, distillation gains no longer require a large, expressive projection head; small stepwise branches can transfer more knowledge from a high-capacity teacher.
The stepwise residual losses act as a form of curriculum on feature alignment, which could be combined with existing logit-based or attention-based distillation methods to push accuracies further.
The number of branches K becomes a tunable knob trading approximation fidelity against computational cost at training and inference.
Teacher weight integration suggests that borrowing teacher classifier weights is a cheap and effective way to reduce the capacity gap in feature distillation, potentially applicable to other distillation frameworks.
The reported ImageNet and COCO improvements indicate the method transfers beyond classification, at least to dense prediction tasks like detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's Stone-Weierstrass grounding is analogical rather than a verified theorem for deep features: the conditions of compactness and continuity are not checked, so the practical gain may come less from function-approximation guarantees and more from the curriculum effect of progressively simpler regression targets.
The residual decomposition could be applied to any feature-alignment objective, including segmentation or 3D tasks, where the per-pixel or per-point residual between teacher and student is large; the COCO detection result is a first hint that the idea generalizes.
One could extend the framework by letting the branches specialize adaptively, e.g., by learning the order in which residuals are fitted, rather than a fixed cascade, to see whether the best ordering is dataset-dependent.
A direct diagnostic is to record the magnitude of each residual term during training; if the residuals shrink monotonically, that supports the 'easier steps' interpretation, and if not, the mechanism is more subtle.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper proposes a concrete KD method: approximate the teacher-student feature residual with a K-branch residual network, plus a teacher-weight-integration trick that reuses the teacher's classifier head. That combination is new to me, and the architecture is simple enough to be worth testing. The writing is clear, and the idea of breaking residual fitting into stages is a reasonable heuristic that could help in practice.

That said, the theoretical framing is mostly decoration. The Stone-Weierstrass theorem is invoked as if it grounds the multi-step decomposition, but the paper never checks its conditions on deep-network features, and universal approximation by DNNs already covers the existence claim. The theorem doesn't say anything about whether this particular loss finds a decomposed solution. I'm not convinced it does: in Eq. (11), the total objective is a sum of prefix errors. With sufficient capacity, branch 1 can fit the entire residual and make the later branches irrelevant. Nothing in the loss or the architecture as described forces a progressive decomposition. So the stress-test concern lands squarely.

The larger consequence is that the reported +1.41% ImageNet and +1.40 COCO gains could come from the added parameters of MBRNet or from TWI, not from the multi-step mechanism. The experimental section is absent from the text I saw, so I can't check whether the authors ran ablations that control capacity or freeze earlier branches. That's the load-bearing gap. If the experiments include such ablations, the paper could be solid; if not, the central claim is unsubstantiated.

There's also a minor issue: the paper treats the student's feature as a zero-order approximation without discussing what happens when the teacher and student feature spaces differ in dimensionality; the projection P handles that, but the residual decomposition's interaction with that projection is left vague.

Verdict: worth a serious referee, because the method is plausible and the gains, if real, matter for KD. But the referee should demand ablations that isolate the multi-step contribution from capacity effects, and the authors should soften the Stone-Weierstrass claim to an analogy rather than a proof. I'd probably not cite it in my own work until those ablations are public.

Yours, [Your name]

Referee Report

4 major / 4 minor

Summary. The paper proposes Expandable Residual Approximation (ERA), a knowledge-distillation method that decomposes the teacher-student feature residual into K additive branches implemented by a Multi-Branched Residual Network (MBRNet), motivated by the Stone-Weierstrass approximation theorem. A Teacher Weight Integration (TWI) strategy reuses the teacher classifier head weights. The abstract claims +1.41% ImageNet Top-1 and +1.40 COCO AP improvements, state-of-the-art results, and code/model release. The main theoretical argument is that approximating the residual in K steps reduces approximation difficulty. The central claims are empirical, but the provided full text contains no experimental section.

Significance. If the claimed gains are real and attributable to the multi-step residual decomposition, ERA would be a practical contribution to knowledge distillation, and the code/model release would aid reproducibility. The MBRNet architecture is clearly described and TWI is an interesting capacity-mitigation idea. However, in the submitted manuscript the empirical evidence is absent, and the theoretical motivation does not establish that the proposed objective implements the claimed progressive decomposition. The significance of the method therefore cannot currently be assessed.

major comments (4)

[Abstract / Full Text (Experiments missing)] The manuscript as provided contains no experiments section. The abstract claims extensive experiments and specific numbers (+1.41% ImageNet Top-1, +1.40 COCO AP), but there are no datasets, baselines, implementation details, ablation studies, or comparisons. Since the central contribution is empirical, this is a load-bearing omission. The claims are not verifiable from the submitted text.
[Section IV-B, Eq. (8)-(11)] The objective in Eq. (11) does not by itself enforce progressive decomposition. Define hat f_k = f_s + sum_{i=1}^k P_i Delta hat f_i. Then L_FD_k = || f_t - hat f_k ||^2, so jointly minimizing sum_k L_FD_k is a sum of prefix errors. At a global optimum with sufficient capacity, the first branch can fit the entire residual f_t - f_s, making all later L_FD_k zero. The optimum does not require a residual decomposition at all. The claimed divide-and-conquer effect requires freezing early branches, a curriculum, or sequential greedy training; such a schedule is not stated in Section IV-B. Without it, reported gains could be capacity effects rather than effects of multi-step approximation.
[Section IV-A, Eqs. (6)-(8)] The Stone-Weierstrass justification is heuristic and the theorem's hypotheses are not checked. The theorem requires a continuous function on a compact space and a function family that is an algebra separating points. The paper does not show that DNN branches form such an algebra, that teacher/student feature maps are continuous functions on a compact domain, or that Eq. (8) uses a norm compatible with the theorem. Moreover, Stone-Weierstrass guarantees existence of an approximating sum, not that the particular empirical objective in Eq. (11) finds a progressive decomposition or is easier to optimize than the single-step objective Eq. (5).
[Section IV-B (ablations missing)] The paper does not provide a capacity-controlled ablation. The reported improvements could be explained by the added parameters of MBRNet (K branches of m FC-BN-ReLU blocks) or by the TWI head re-use, rather than by residual decomposition. To support the central mechanism, the authors would need a comparison with a single-branch residual module of equivalent total capacity, and an ablation with/without TWI. This is a load-bearing point because the theoretical argument does not establish the mechanism, and the experiments are absent.

minor comments (4)

[Eq. (11)] The branch index in Eq. (11) appears as P_i Delta hat f_i, but within the k-th loss it should be P_k Delta hat f_k. Also, the norm expression should be squared to match Eq. (5) and the text; the formatting currently writes \| \cdot \|_2 without an explicit square.
[Eq. (6)] The Stone-Weierstrass statement is written as f approximated by sum p_i g_i. This product form is nonstandard; the theorem usually concerns the algebra generated by certain functions. Please clarify the precise family of functions and why the product form is appropriate.
[Section IV-A, Eq. (10)-(11)] The notation f^n_k is confusingly described as 'the n-th sample of the student feature'. In Eq. (10), Delta f_k is f_t - hat f_{k-1}, so Eq. (11) should explicitly define the per-sample target. Please update the notation for clarity.
[Figures] The text references Fig. 2 and Fig. 3, but these figures are not included in the provided full text. Please ensure all referenced figures are present and captioned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ERA objective is defined against the external teacher feature, and reported gains are empirical claims rather than consequences of the definition.

full rationale

The paper's derivation chain is self-contained with respect to circularity. Eq. (8) rewrites the Stone-Weierstrass-inspired sum by treating the student output as a zero-order term; this is a definition, not a fitted result. Eqs. (9) and (10) define the k-th approximation and its residual target as ft minus the previous approximation, which is the standard residual-learning formulation. Eq. (11) then trains each branch against that residual. The final alignment ft ≈ fhat_K follows by construction only if every branch actually minimizes its loss, but the loss itself is evaluated against the external teacher feature ft, so the method is not using the target result to fit constants. The paper does not claim to derive the empirical improvements from the theorem; it uses Stone-Weierstrass only as motivation ('Inspired by'), and the cited theorem is external standard mathematics, not a self-citation. The skeptic's concern that the joint objective in Eq. (11) does not enforce progressive decomposition is a validity/effectiveness limitation, not a circularity: the method could fail for that reason, but the prediction is not equivalent to its input by construction. No fitted parameter is renamed as a prediction, and no load-bearing self-citation chain is present. Therefore, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new physical entities; it proposes a new architecture and loss for knowledge distillation. The main assumptions are about the applicability of the Stone-Weierstrass theorem and the effectiveness of the multi-step training.

free parameters (2)

K (number of residual branches)
The number of branches is a hyperparameter that controls the granularity of the residual decomposition; it is chosen via experiments, but the value is not given in the provided text.
m (number of FC-BN-ReLU blocks per branch)
The depth of each branch is a hyperparameter that affects capacity; it is chosen empirically, but the value is not specified in the supplied section.

assumptions (3)

domain assumption The feature functions of the teacher and student are continuous on a compact space, satisfying the hypotheses of the Stone-Weierstrass theorem.
Invoked in Section IV.A, Eq. (6) and Eq. (7), where the theorem is applied to feature maps. No evidence is given that deep network features meet these conditions.
domain assumption The student output fs can be treated as a zero-order approximation of the teacher's feature ft.
Stated in Section IV.A after Eq. (7), where the reformulation defines fs as the initial approximation. This is a modeling choice that is not justified beyond intuition.
domain assumption Optimizing the per-branch squared error in Eq. (11) produces a progressive reduction of the residual and improves the final student representation.
The paper relies on this to claim the method reduces approximation difficulty. It is not proven and is the central empirical assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Expandable Residual Approximation for Knowledge Distillation." pith.science (2026). https://pith.science/paper/FLXUEI7O

@misc{pith2026250816050,
  author       = {Pith},
  title        = {Pith review of: Expandable Residual Approximation for Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FLXUEI7O}},
  note         = {Machine review of arXiv:2508.16050}
}

read the original abstract

Knowledge distillation (KD) aims to transfer knowledge from a large-scale teacher model to a lightweight one, significantly reducing computational and storage requirements. However, the inherent learning capacity gap between the teacher and student often hinders the sufficient transfer of knowledge, motivating numerous studies to address this challenge. Inspired by the progressive approximation principle in the Stone-Weierstrass theorem, we propose Expandable Residual Approximation (ERA), a novel KD method that decomposes the approximation of residual knowledge into multiple steps, reducing the difficulty of mimicking the teacher's representation through a divide-and-conquer approach. Specifically, ERA employs a Multi-Branched Residual Network (MBRNet) to implement this residual knowledge decomposition. Additionally, a Teacher Weight Integration (TWI) strategy is introduced to mitigate the capacity disparity by reusing the teacher's head weights. Extensive experiments show that ERA improves the Top-1 accuracy on the ImageNet classification benchmark by 1.41% and the AP on the MS COCO object detection benchmark by 1.40, as well as achieving leading performance across computer vision tasks. Codes and models are available at https://github.com/Zhaoyi-Yan/ERA.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 74 canonical work pages

[1]

Simonyan and A

K. Simonyan and A. Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' in ICLR, 2015

2015
[2]

K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in CVPR, 2016, pp. 770--778

2016
[3]

Huang, Z

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, ``Densely connected convolutional networks,'' in CVPR, 2017, pp. 4700--4708

2017
[4]

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, ``A convnet for the 2020s,'' in CVPR, 2022, pp. 11\,976--11\,986

2022
[5]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in ICLR, 2021

2021
[6]

Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong et al., ``Swin transformer v2: Scaling up capacity and resolution,'' in CVPR, 2022, pp. 12\,009--12\,019

2022
[7]

Hinton, O

G. Hinton, O. Vinyals, and J. Dean, ``Distilling the knowledge in a neural network,'' arXiv preprint arXiv:1503.02531, 2015

arXiv 2015
[8]

G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, ``Learning efficient object detection models with knowledge distillation,'' NeurIPS, vol. 30, 2017

work page 2017

Show all 86 references

[9]

Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, ``Structured knowledge distillation for semantic segmentation,'' in CVPR, 2019, pp. 2604--2613

2019
[10]

Touvron, M

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J \' e gou, ``Training data-efficient image transformers & distillation through attention,'' in ICML, 2021, pp. 10\,347--10\,357

2021
[11]

Ba and R

J. Ba and R. Caruana, ``Do deep nets really need to be deep?'' NeurIPS, vol. 27, 2014

2014
[12]

Chen, J.-P

D. Chen, J.-P. Mei, C. Wang, Y. Feng, and C. Chen, ``Online knowledge distillation with diverse peers,'' in AAAI, vol. 34, no. 04, 2020, pp. 3430--3437

2020
[13]

L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, ``Revisiting knowledge distillation via label smoothing regularization,'' in CVPR, 2020, pp. 3903--3911

2020
[14]

J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos, ``Knowledge distillation via softmax regression representation learning,'' in ICLR, 2021

2021
[15]

Komodakis and S

N. Komodakis and S. Zagoruyko, ``Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer,'' ICLR, 2017

2017
[16]

Tung and G

F. Tung and G. Mori, ``Similarity-preserving knowledge distillation,'' in ICCV, 2019, pp. 1365--1374

2019
[17]

B. Heo, M. Lee, S. Yun, and J. Y. Choi, ``Knowledge transfer via distillation of activation boundaries formed by hidden neurons,'' in AAAI, vol. 33, no. 01, 2019, pp. 3779--3787

2019
[18]

P. Chen, S. Liu, H. Zhao, and J. Jia, ``Distilling knowledge via knowledge review,'' in CVPR, 2021, pp. 5008--5017

2021
[19]

Y. Jang, H. Lee, S. J. Hwang, and J. Shin, ``Learning what and where to transfer,'' in ICML, 2019, pp. 3030--3039

2019
[20]

Y. Chen, N. Wang, and Z. Zhang, ``Darkrank: Accelerating deep metric learning via cross sample similarities transfer,'' AAAI, pp. 2852--2859, 2018

2018
[21]

W. Park, D. Kim, Y. Lu, and M. Cho, ``Relational knowledge distillation,'' in CVPR, 2019, pp. 3967--3976

2019
[22]

B. Peng, X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang, ``Correlation congruence for knowledge distillation,'' in ICCV, 2019, pp. 5007--5016

2019
[23]

H. Wu, Y. Gao, Y. Zhang, S. Lin, Y. Xie, X. Sun, and K. Li, ``Self-supervised models are good teaching assistants for vision transformers,'' in ICML, 2022, pp. 24\,031--24\,042

2022
[24]

J. Yim, D. Joo, J. Bae, and J. Kim, ``A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,'' in CVPR, 2017, pp. 4133--4141

2017
[25]

S. H. Lee, D. H. Kim, and B. C. Song, ``Self-supervised knowledge distillation using singular value decomposition,'' in ECCV, 2018, pp. 335--350

2018
[26]

Zhang and Y

C. Zhang and Y. Peng, ``Better and faster: knowledge transfer from multiple self-supervised learning tasks via graph distillation for video classification,'' in IJCAI, 2018, pp. 1135--1141

2018
[27]

Lee and B

S. Lee and B. C. Song, ``Graph-based knowledge distillation by multi-head attention network,'' in BMVC, 2020

2020
[28]

Passalis, M

N. Passalis, M. Tzelepi, and A. Tefas, ``Heterogeneous knowledge distillation using information flow modeling,'' in CVPR, 2020, pp. 2339--2348

2020
[29]

C. K. Joshi, F. Liu, X. Xun, J. Lin, and C. S. Foo, ``On representation knowledge distillation for graph neural networks,'' IEEE TNNLS, vol. 35, no. 4, pp. 4656--4667, 2024

2024
[30]

F. Ding, Y. Yang, H. Hu, V. Krovi, and F. Luo, ``Dual-level knowledge distillation via knowledge alignment and correlation,'' IEEE TNNLS, vol. 35, no. 2, pp. 2425--2435, 2024

2024
[31]

X. Wang, R. Zhang, Y. Sun, and J. Qi, ``Kdgan: Knowledge distillation with generative adversarial networks,'' NeurIPS, vol. 31, 2018

2018
[32]

Z. Shen, Z. He, and X. Xue, ``Meal: Multi-model ensemble via adversarial learning,'' in AAAI, vol. 33, no. 01, 2019, pp. 4886--4893

2019
[33]

Y. Wang, C. Xu, C. Xu, and D. Tao, ``Adversarial learning of portable student networks,'' in AAAI, vol. 32, no. 1, 2018, pp. 4260--4267

2018
[34]

T. Chen, I. Goodfellow, and J. Shlens, ``Net2net: Accelerating learning via knowledge transfer,'' arXiv preprint arXiv:1511.05641, 2015

2015 arXiv
[35]

J. Gou, L. Sun, B. Yu, L. Du, K. Ramamohanarao, and D. Tao, ``Collaborative knowledge distillation via multiknowledge transfer,'' IEEE TNNLS, vol. 35, no. 5, pp. 6718--6730, 2024

2024
[36]

Vongkulbhisal, P

J. Vongkulbhisal, P. Vinayavekhin, and M. Visentini-Scarzanella, ``Unifying heterogeneous classifiers with distillation,'' in CVPR, 2019, pp. 3175--3184

2019
[37]

F. Yuan, L. Shou, J. Pei, W. Lin, M. Gong, Y. Fu, and D. Jiang, ``Reinforced multi-teacher selection for knowledge distillation,'' in AAAI, vol. 35, no. 16, 2021, pp. 14\,284--14\,291

2021
[38]

S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, ``Improved knowledge distillation via teacher assistant,'' in AAAI, vol. 34, 2020, pp. 5191--5198

2020
[39]

M. Gao, Y. Wang, and L. Wan, ``Residual error based knowledge distillation,'' Neurocomputing, vol. 433, pp. 154--161, 2021

2021
[40]

Romero, N

A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, ``Fitnets: Hints for thin deep nets,'' in ICLR, 2015

2015
[41]

Huang and N

Z. Huang and N. Wang, ``Like what you like: Knowledge distill via neuron selectivity transfer,'' arXiv preprint arXiv:1707.01219, 2017

2017 arXiv
[42]

Zagoruyko and N

S. Zagoruyko and N. Komodakis, ``Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,'' in ICLR, 2022

2022
[43]

Passalis and A

N. Passalis and A. Tefas, ``Learning deep representations with probabilistic knowledge transfer,'' in ECCV, 2018, pp. 268--284

2018
[44]

S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai, ``Variational information distillation for knowledge transfer,'' in CVPR, 2019, pp. 9163--9171

2019
[45]

Passalis, M

N. Passalis, M. Tzelepi, and A. Tefas, ``Probabilistic knowledge transfer for lightweight deep representation learning,'' IEEE TNNLS, vol. 32, no. 5, pp. 2030--2039, 2020
[46]

Huang, S

T. Huang, S. You, F. Wang, C. Qian, and C. Xu, ``Knowledge distillation from a stronger teacher,'' NeurIPS, vol. 35, pp. 33\,716--33\,727, 2022

2022
[47]

W. Son, J. Na, J. Choi, and W. Hwang, ``Densely guided knowledge distillation using multiple teacher assistants,'' in ICCV, 2021, pp. 9395--9404

2021
[48]

Chen, J.-P

D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y. Feng, and C. Chen, ``Knowledge distillation with the reused teacher classifier,'' in CVPR, 2022, pp. 11\,933--11\,942

2022
[49]

Z. Yang, Z. Li, Y. Gong, T. Zhang, S. Lao, C. Yuan, and Y. Li, ``Rethinking knowledge distillation via cross-entropy,'' arXiv preprint arXiv:2208.10139, 2022

2022 arXiv
[50]

Y. Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan, ``Knowledge distillation via instance relationship graph,'' in CVPR, 2019, pp. 7096--7104

2019
[51]

H. Chen, Y. Wang, C. Xu, C. Xu, and D. Tao, ``Learning student networks via feature embedding,'' IEEE TNNLS, vol. 32, no. 1, pp. 25--35, 2020

2020
[52]

Goodfellow, J

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, ``Generative adversarial nets,'' NeurIPS, pp. 2672--2680, 2014

2014
[53]

Radford, L

A. Radford, L. Metz, and S. Chintala, ``Unsupervised representation learning with deep convolutional generative adversarial networks,'' in ICLR, 2016

2016
[54]

Karras, T

T. Karras, T. Aila, S. Laine, and J. Lehtinen, ``Progressive growing of GAN s for improved quality, stability, and variation,'' in ICLR, 2018

2018
[55]

Karras, S

T. Karras, S. Laine, and T. Aila, ``A style-based generator architecture for generative adversarial networks,'' in CVPR, 2019, pp. 4401--4410

2019
[56]

Karras, S

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, ``Analyzing and improving the image quality of StyleGAN ,'' in CVPR, 2020, pp. 8110--8119

2020
[57]

Karras, M

T. Karras, M. Aittala, S. Laine, E. H \"a rk \"o nen, J. Hellsten, J. Lehtinen, and T. Aila, ``Alias-free generative adversarial networks,'' in NeurIPS, 2021, pp. 852--863

2021
[58]

H. Chen, Y. Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian, ``Data-free learning of student networks,'' in ICCV, 2019, pp. 3514--3522

2019
[59]

Micaelli and A

P. Micaelli and A. J. Storkey, ``Zero-shot knowledge transfer via adversarial belief matching,'' NeurIPS, vol. 32, 2019

2019
[60]

X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, ``Tinybert: Distilling bert for natural language understanding,'' arXiv preprint arXiv:1909.10351, 2019

1909 arXiv
[61]

X. Li, S. Li, B. Omar, F. Wu, and X. Li, ``Reskd: Residual-guided knowledge distillation,'' IEEE TIP, vol. 30, pp. 4735--4746, 2021

2021
[62]

Huang, Z

W. Huang, Z. Peng, L. Dong, F. Wei, J. Jiao, and Q. Ye, ``Generic-to-specific distillation of masked autoencoders,'' in CVPR, 2023, pp. 15\,996--16\,005

2023
[63]

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, ``Mobilenets: Efficient convolutional neural networks for mobile vision applications,'' arXiv preprint arXiv:1704.04861, 2017

2017 arXiv
[64]

Zhang, X

X. Zhang, X. Zhou, M. Lin, and J. Sun, ``Shufflenet: An extremely efficient convolutional neural network for mobile devices,'' in CVPR, 2018, pp. 6848--6856

2018
[65]

E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, ``Randaugment: Practical automated data augmentation with a reduced search space,'' in CVPRW, 2020, pp. 702--703

2020
[66]

Y. Tian, D. Krishnan, and P. Isola, ``Contrastive representation distillation,'' in ICLR, 2019

2019
[67]

Krizhevsky, G

A. Krizhevsky, G. Hinton et al., ``Learning multiple layers of features from tiny images,'' 2009

2009
[68]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ``Imagenet: A large-scale hierarchical image database,'' in CVPR, 2009, pp. 248--255

2009
[69]

B. Zhao, Q. Cui, R. Song, Y. Qiu, and J. Liang, ``Decoupled knowledge distillation,'' in CVPR, 2022, pp. 11\,953--11\,962

2022
[70]

Marcel and Y

S. Marcel and Y. Rodriguez, ``Torchvision the machine-vision package of torch,'' in ACM MM, 2010, pp. 1485--1488

2010
[71]

Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang, ``Curriculum temperature for knowledge distillation,'' in AAAI, vol. 37, no. 2, 2023, pp. 1504--1512

2023
[72]

S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, ``Logit standardization in knowledge distillation,'' in CVPR, 2024, pp. 15\,731--15\,740

2024
[73]

Wightman, H

R. Wightman, H. Touvron, and H. J \'e gou, ``Resnet strikes back: An improved training procedure in timm,'' arXiv preprint arXiv:2110.00476, 2021

2021 arXiv
[74]

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \'a r, and C. L. Zitnick, ``Microsoft coco: Common objects in context,'' in ECCV, 2014, pp. 740--755

2014
[75]

Cai and N

Z. Cai and N. Vasconcelos, ``Cascade r-cnn: High quality object detection and instance segmentation,'' IEEE TPAMI, vol. 43, no. 5, pp. 1483--1498, 2019

2019
[76]

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll \'a r, ``Focal loss for dense object detection,'' in CVPR, 2017, pp. 2980--2988

2017
[77]

S. Xie, R. Girshick, P. Doll \'a r, Z. Tu, and K. He, ``Aggregated residual transformations for deep neural networks,'' in CVPR, 2017, pp. 1492--1500

2017
[78]

Z. Yang, Z. Li, X. Jiang, Y. Gong, Z. Yuan, D. Zhao, and C. Yuan, ``Focal and global knowledge distillation for detectors,'' in CVPR, 2022, pp. 4643--4652

2022
[79]

T.-Y. Lin, P. Doll \'a r, R. Girshick, K. He, B. Hariharan, and S. Belongie, ``Feature pyramid networks for object detection,'' in CVPR, 2017, pp. 2117--2125

2017
[80]

Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu, ``Intra-class feature variation distillation for semantic segmentation,'' in ECCV, 2020, pp. 346--362

2020
[81]

C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen, ``Channel-wise knowledge distillation for dense prediction,'' in ICCV, 2021, pp. 5311--5320

2021
[82]

C. Yang, H. Zhou, Z. An, X. Jiang, Y. Xu, and Q. Zhang, ``Cross-image relational knowledge distillation for semantic segmentation,'' in CVPR, 2022, pp. 12\,319--12\,328

2022
[83]

L. Li, P. Dong, Z. Wei, and Y. Yang, ``Automated knowledge distillation via monte carlo tree search,'' in ICCV, 2023, pp. 17\,413--17\,424

2023
[84]

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, ``Encoder-decoder with atrous separable convolution for semantic image segmentation,'' in ECCV, 2018, pp. 801--818

2018
[85]

Cordts, M

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, ``The cityscapes dataset for semantic urban scene understanding,'' in CVPR, 2016, pp. 3213--3223

2016
[86]

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, ``Pyramid scene parsing network,'' in CVPR, 2017, pp. 2881--2890

2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Simonyan and A

K. Simonyan and A. Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' in ICLR, 2015

2015

[2] [2]

K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in CVPR, 2016, pp. 770--778

2016

[3] [3]

Huang, Z

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, ``Densely connected convolutional networks,'' in CVPR, 2017, pp. 4700--4708

2017

[4] [4]

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, ``A convnet for the 2020s,'' in CVPR, 2022, pp. 11\,976--11\,986

2022

[5] [5]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in ICLR, 2021

2021

[6] [6]

Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong et al., ``Swin transformer v2: Scaling up capacity and resolution,'' in CVPR, 2022, pp. 12\,009--12\,019

2022

[7] [7]

Hinton, O

G. Hinton, O. Vinyals, and J. Dean, ``Distilling the knowledge in a neural network,'' arXiv preprint arXiv:1503.02531, 2015

arXiv 2015

[8] [8]

G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, ``Learning efficient object detection models with knowledge distillation,'' NeurIPS, vol. 30, 2017

work page 2017