Pith. sign in

REVIEW 3 major objections 6 minor 49 references

Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that large-capacity-gap knowledge distillation fails because teacher and student feature manifolds have mismatched intrinsic dimensions, and that progressively co-evolving a stronger teacher with a smaller student fixes th

desk verdict A plausible empirical KD contribution that is held back by unresolved reproducibility gaps and an overclaimed theoretical wrapper. read the letter →

arxiv 2608.00129 v1 pith:DJW77UQI submitted 2026-07-31 cs.LG

Tiancong Cheng , Ying Zhang , Zhiwen Yu , Yifang Yin , Bin Guo This is my paper
classification cs.LG
keywords knowledgedistillationmodelcompressionprogressivecurriculumintrinsicdimensionteacher-studentco-evolutionfeatureadapterLipschitzcontinuitysemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation normally degrades when the student is much smaller than the teacher. The paper argues the cause is an intrinsic-dimension gap: the teacher's representations live on a lower-dimensional manifold, and forcing a high-dimensional student to collapse onto them injects noisy, conflicting gradients. It proposes Progressive^2, which on the teacher side feeds layers from shallow to deep over time (a raw-to-rich curriculum) and fuses the selected layers through a teacher-side adapter, and on the student side iteratively compresses the student while using the previous round's student as the next teacher. On Cityscapes, the co-evolved version beats the strongest multi-stage baseline by 2.6 and 2.8 mIoU at 25% and 6.3% student sizes; on CIFAR-100 it gains 1.8% top-1 accuracy. The teacher-side adapter is supported by a Lipschitz-continuity proxy that the authors explicitly stop short of calling a formal proof.

What carries the argument

Two coordinated curricula plus a fusion adapter. Intrinsic dimension (ID), the effective number of degrees of freedom in a model's feature manifold, is the paper's diagnostic: a large ID gap between teacher and student predicts distillation difficulty. The raw-to-rich schedule gradually includes deeper teacher layers over time windows. The teacher-side adapter, a compact convolution-based merge block with channel alignment, downsampling, and normalization, fuses all selected teacher layers into one feature map so the student faces a single knowledge target rather than many competing gradient directions. The student-side schedule compresses the model by channel pruning each round and re-uses

What would settle it

A decisive test: take one teacher-student pair and artificially change the student's feature-manifold structure to vary the measured intrinsic-dimension gap while keeping semantic content fixed; if final distillation accuracy does not track the gap, the claim that the gap is the active mechanism fails. A simpler check is to find any configuration where a larger measured ID gap distills more successfully than a smaller one under the same compute budget.

Watch

Extended reading notes

Core claim

The paper's central claim is that the teacher-student gap is not just a capacity mismatch but an optimization problem: final teacher features occupy a lower intrinsic dimension than the student's, and forcing a direct collapse onto that manifold creates gradient noise and curvature spikes. Progressive^2 attacks this on both sides. The teacher side introduces a curriculum over representation depth, starting with shallow, semantically simple layers and adding deeper layers over time, then condenses the selected layers through a teacher-side multi-feature adapter so the student sees exactly one fused distillation target. The student side runs multiple rounds: each round compresses the student f

Load-bearing premise

The theoretical story rests on the assumptions that distillation gradient noise is bounded by the product of parameter norm and noise norm, that the noise grows monotonically with the intrinsic-dimension gap, and that the loss is smooth enough for a Lipschitz analysis; the authors explicitly label these as proxies, and if they fail, the method stands only on its empirical results.

Editorial extensions

If this is right

  • If the central claim holds, distillation no longer needs to be abandoned at extreme compression: a 6.3%-size student can outperform larger previously distilled students.
  • The progressive teacher alone provides a tunable accuracy-versus-training-time knob, which suits quality-of-service-aware deployment where training cost is amortized offline.
  • The teacher-side adapter reduces the number of competing gradient directions and the parameter overhead of multi-layer distillation, making training both cheaper and more stable.
  • The recipe transfers across tasks, backbones, and model families, so the gains are not confined to one dataset or architecture.
  • Because the student becomes the teacher in the next round, the method needs no separately pretrained assistant model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: if the intrinsic-dimension gap is genuinely causal, ID estimation could become a pre-training diagnostic: measure the gap first and predict which teacher-student pairs will fail, then apply a progressive schedule only where needed.
  • Editorial: the teacher-side fusion principle may generalize beyond distillation to any alignment problem where a small-capacity learner must match a high-capacity teacher, such as transfer in reinforcement learning or representation alignment across modalities.
  • Editorial: the fixed time-window curriculum could be made adaptive, advancing the layer schedule when the student's measured ID approaches the teacher's; the paper does not explore this dynamic-stopping variant.
  • Editorial: the reverse-progression result suggests that semantic depth per se is not essential; the benefit may come from gradually increasing target complexity, so the same schedule could apply to shuffled or non-visual layer orders.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Progressive^2, a knowledge-distillation framework for large teacher-to-small student compression. It combines (i) a progressively stronger teacher (P rog^2-t) that adds teacher layers over time in a claimed 'raw-to-rich' front-to-back order and fuses them with a teacher-side multi-feature adapter, and (ii) an optional progressively smaller student (P rog^2-ts) that compresses the student over rounds and uses the previous-round student as the next teacher. The authors claim consistent state-of-the-art performance across Cityscapes, CIFAR-100, Tiny-ImageNet, NYU-V2, and GLUE, and provide a Lipschitz-continuity-based proxy analysis of the adapter's stability and convergence benefits. The paper includes extensive experiments, mobile-device evaluations, and QoS-oriented analysis.

Significance. If the empirical results are reproducible, the paper makes a practical contribution: the progressive-teacher idea is simple, can be deployed independently, and the reported gains in substantial-compression regimes (e.g., 2.6/2.8 mIoU over TAKD+DIST on Cityscapes at 25%/6.3% student sizes, 1.8% top-1 on CIFAR-100) are meaningful. Strengths include evaluation across multiple tasks and backbones, three-run standard deviations in the main tables, FLOPs/memory measurements, and on-device inference results. The authors are also transparent that the theory is a proxy, not a formal guarantee, and that the intrinsic-dimension relationship is correlational. However, the central claim is empirical, and the current reproducibility package is insufficient: code is withheld, exact seeds are not recorded, and the main text and appendix disagree on training schedules. The theoretical analysis, as written, does not provide independent support for the method and contains technical inconsistencies.

major comments (3)
  1. [Sec. IV; Appendix Table XII; Appendix Table XI / Appendix G] The headline claim 'consistently outperforms state-of-the-art distillation methods' is an empirical claim supported by Tables I, II, VI, and the appendix tables. The current protocol does not allow independent verification: Appendix Table XII states that seeds are randomly drawn from randint(0,10000) and that 'The exact sampled seed values are None', and code is promised only after acceptance. Moreover, Sec. IV says Cityscapes is trained with 'no learning rate decay', while Appendix Table XI says the learning rate is reduced to 5e-5 at epoch 100. Appendix G sets the Tiny-ImageNet learning rate to 0.1, while Table XI sets it to 0.01. These contradictions make the exact training protocol ambiguous. Because the theoretical sections are explicitly proxy analyses, the tables must carry the central claim. I request release of code and exact seeds, and reconciliation of all schedule discrepanci
  2. [Table IV; Sec. III-A1] The 'raw-to-rich' front-to-back teacher-layer ordering is a stated contribution, but Table IV does not consistently support it. In the full configuration (P-ss and P-st both active), the reverse order Dec1->5(R) reaches 62.00 mIoU, slightly above the proposed Enc1->5 at 61.77 and Enc1->5(R) at 61.43. Thus, when both components are used, back-to-front progress is at least as good as the proposed front-to-back progress. The text says the method 'remains effective' under reversal, which is too weak; it does not establish the semantic curriculum claimed. Table IV also lacks the means/standard deviations reported elsewhere (e.g., Tables I and V). Please provide repeated runs with std and a significance test, or weaken the raw-to-rich claim accordingly.
  3. [Sec. III-D, Theorem 3.1, Eqs. (10)-(21); Appendix O] The theoretical support for the teacher-side adapter is presented as a Lipschitz-continuity analysis, but as written it is heuristic. Eq. (15) applies a second-order Taylor expansion and an H-Lipschitz assumption to a non-convex deep-network loss without justification. Eq. (17) derives H*=(R+1)H_max, yet the main text then restricts the step size to alpha <= 1/(R*Hmax), while Appendix B uses alpha <= 1/Hmax; these are inconsistent. Eq. (11) takes the feature-perturbation bound ||Phi(theta; f+r)-Phi(theta; f)|| <= ||theta|| ||r|| from Eq. (10) and applies it to distillation gradient noise, whose boundedness and monotone dependence on the ID gap are assumed, not derived. Appendix O similarly relies on assumed beta/epsilon inequalities and is explicitly non-formal. Since the paper itself labels these as proxies, the contribution statement 'theoretically supported by the framework of Lipschi
minor comments (6)
  1. [Fig. 2 and Sec. I] The text references Fig. 2(c) ('e.g., as observed with BAM [35] in Fig. 2 (c)'), but the figure caption lists only panels (a), (b), and (d). Either add the missing panel or fix the reference.
  2. [Eq. (15)] The second-order term is written as [(-alpha*g)]^2, which is not defined for a vector. It should be (alpha^2/2) g^T Hessian g, and the inequality leading to alpha/2 ||g||^2 requires explicitly using alpha <= 1/H.
  3. [Table VII] The lambda-sweep table reports single values without standard deviations, unlike the main tables. Given the sensitivity of distillation loss weights, please report mean and std (or at least the number of runs) for these sweeps.
  4. [Algorithm 2] The compression function g(r)=r/n is under-specified: clarify whether r is the total target compression ratio or the per-round ratio, and specify how Phi(s_{k-1}, g(r)) changes channel counts at each round.
  5. [Tables IX and X] The depth-estimation errors in Table IX (e.g., Abs Err 0.4810) and Table X (e.g., Abs Err 0.0161) appear to be on different scales or use different normalization. Please specify the metric definition/units for each table.
  6. [Appendix O] The VC-type bound is applied in an asymptotic regime with assumed inequalities (beta_st <= beta_sm, epsilon_mt+epsilon_sm <= epsilon_st). These assumptions should be stated as such in the main text when referencing this appendix, rather than as established relationships.

Circularity Check

1 steps flagged · score 2.0 of 10

Empirical comparisons are independent, but the claimed Lipschitz 'theoretical support' for the teacher-side adapter partly reduces to restating the adapter's smaller parameter/direction count.

  1. self definitional [Section III-D, Eqs. 11-13; Appendix C (Table III and equal-noise assumption)]
    "U(θ_Net, r) ∝ ∥θ_Net∥ · ∥r∥ ... Observations indicate the relative magnitudes of the parameters ... ∥θ^{m−to−n}_{Net}∥ ≥ ∥θ^{1−to−n}_{Net}∥ ≥ ∥θ^{1−to−1}_{Net}∥, and the comparison of their training is as follows where our teacher-side adapter-based model has the least uncertainty: U^{1−to−1} ≤ U^{1−to−n} ≤ U^{m−to−n}."

    Under the paper's own proxy, U is defined as ∥θ_Net∥·∥r∥ (Eq. 11), and Appendix C assumes an 'equivalent noise level' r for all methods. The claimed uncertainty ordering (Eq. 13) is therefore exactly the parameter-count ordering (Eq. 12) restated: fewer adapter parameters imply lower U by construction, not by an independent Lipschitz argument. The convergence-complexity conclusion has the same structure: R (number of distillation directions) is larger for m-to-n and 1-to-n, and the paper defines convergence difficulty through the R+1 constraints, so C_{m-to-n} > C_{1-to-n} > C_{1-to-1} is a restatement of the direction-count ordering. The empirical tables are independent, but this particular 'theoretically supported' benefit of the 1-to-1 teacher-side adapter is built into the proxy defini

full rationale

The paper's central empirical claims (Tables I, II, VI and appendix tables) are obtained by training against fixed external baselines; no experimental number is produced by plugging the claimed result back into the derivation, and the ablations (Tables IV, V) test the design choices independently. The only circularity-like reduction I can exhibit is in the theoretical proxy for the teacher-side adapter: the lower uncertainty and faster convergence are largely a monotone restatement of the adapter's smaller number of distillation directions/parameters under the paper's own U ∝ ∥θ∥·∥r∥ and equal-noise assumptions (Eqs. 10-13, 20-21). This is a real reduction but it concerns the explanatory 'theoretical support,' not the empirical comparisons. The paper itself repeatedly disclaims formal guarantees and notes that the intrinsic-dimension-gap relationship is correlational rather than causal (Sec. III-D, Conclusion, Appendix O), which further lowers the severity. Reproducibility concerns about withheld seeds and inconsistent learning-rate statements are correctness risks, not circularity. I therefore assign a low score of 2 rather than 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The paper's central method depends on several hand-chosen hyperparameters (lambda, alpha_j, stage counts, adapter LR) and on proxy assumptions about Lipschitz smoothness, intrinsic-dimension causality, equal noise across adapter designs, and the reliability of a self-distilled previous student as teacher. The only genuinely new postulated construct is the teacher-side adapter, whose independent evidence is limited to internal ablations; the instability metric has some external calibration via gradient-variance correlation.

free parameters (5)
  • distillation weight lambda = 1 (Cityscapes), 10 (NYU-V2), 100 (CIFAR-100); Table VII
    Global weight on the feature-alignment loss in Eqns 4 and 6; tuned per dataset by validation sweep, directly controls distillation strength.
  • per-layer loss weights alpha_j (Eq. 2) = not reported ('determined empirically')
    Used in the pre-adapter loss; the authors state alpha_j is determined empirically, introducing hand-chosen weights not fixed in the paper.
  • number of progressive stages/time-windows (K, n) = K windows within a stage; n=5 stages used for Cityscapes multi-stage (Table VI)
    The schedule of layer inclusion and student compression is a hand-selected curriculum; stage count trades accuracy against training time.
  • teacher-side adapter learning rate = 1e-4 chosen by ablation (Appendix Table V)
    The adapter's learning rate is selected by a sweep from 1e-1 to 1e-6; 1e-4 is chosen.
  • ID estimator sampling settings (rho=0.9, class-balanced truncation, central quantile) = rho=0.9; class-balanced subset
    These protocol choices affect the reported ID gaps that motivate the method; they are estimation decisions rather than fits to target accuracies.
assumptions (6)
  • ad hoc to paper Gradients of task and distillation losses are H-Lipschitz and the second-order Taylor expansion is valid, so Theorem 3.1 applies to non-convex deep networks.
    Invoked in Sec. III-D for the convergence proof; the authors explicitly call this a proxy, not a formal guarantee.
  • domain assumption The intrinsic-dimension gap between teacher and student manifolds is a valid proxy or causal driver of distillation difficulty, and progressive layer inclusion monotonically reduces teacher feature ID.
    Used throughout Sec. I and III to motivate the curriculum; the Conclusion admits the relationship is correlational and lacks formal causal validation.
  • ad hoc to paper Distillation gradient noise r has a bound monotonically increasing with the ID gap, and the noise magnitude is equal across adapter configurations being compared.
    Assumed in Sec. III-D to derive U1-to-1 <= U1-to-n <= Um-to-n; Appendix Table III footnote admits equal-noise simplification.
  • domain assumption The previous student can serve as a valid teacher for the next smaller student (t_k = s_{k-1}), and partial-layer distillation from this self-teacher transfers meaningful knowledge.
    Core design of P rog2-ts in Algorithm 2; no formal convergence proof is provided for this multi-stage self-distillation process.
  • domain assumption A teacher-side adapter that fuses multiple teacher layers through 1x1 conv, downsampling, and 3x3 conv preserves the comprehensive teacher knowledge and is sufficient as a single alignment target for the last student layer.
    Central to the teacher-side adapter design in Sec. III-A2 and Fig. 5; supported only by internal ablations.
  • ad hoc to paper The TwoNN intrinsic-dimension estimator's local-uniformity assumption and the VC-bound inequalities in Appendix O (beta_st<=beta_sm, epsilon_mt+epsilon_sm<=epsilon_st, etc.) hold.
    Appendix A uses TwoNN with a local-uniformity assumption; Appendix O's multi-stage error-bound argument relies on asymptotic inequalities that are hypotheses, not proven facts.
invented entities (2)
  • Teacher-side multi-layer knowledge adapter
    purpose: Fuses progressively selected teacher features into one target feature so the student aligns only its last layer with one fused feature, reducing gradient conflicts and training uncertainty.
    Validated only through internal ablations (Table V) and the paper's own experiments; no external falsifiable prediction outside this paper.
  • Instability metric S_instability independent evidence
    purpose: Quantifies optimization instability as a product of a gradient-alignment penalty and normalized curvature, used to show the method stabilizes training.
    Benchmarked against total gradient-norm variance with Pearson r=0.86 in Appendix P, giving an external optimization diagnostic, though the metric itself is newly defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression." pith.science (2026). https://pith.science/paper/DJW77UQI

@misc{pith2026260800129,
  author       = {Pith},
  title        = {Pith review of: Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJW77UQI}},
  note         = {Machine review of arXiv:2608.00129}
}
abstract

Knowledge distillation (KD) is a widely utilized technique for transferring knowledge from a large model (the teacher) to a smaller model (the student). Owing to its flexibility and broad applicability, KD has been extensively applied in the compression of server-side models to meet the Quality of Service (QoS) requirements of client users. Despite significant advancements, the performance of distillation is substantially compromised when a large disparity exists between the capabilities of the server and the requirements of the client. To alleviate this problem, we propose a novel distillation approach, named Progressive$^2$, which operates through the combination of a progressively stronger teacher and a progressively smaller student. On the side of the teacher, rather than involving all layers simultaneously, we progressively select additional layers for distillation following a raw-to-rich semantic progression, establishing a systematic learning curriculum. Furthermore, we design a teacher-side multi-feature fusion adapter for the teacher to improve training stability, which is theoretically supported by the framework of Lipschitz continuity. On the side of the student, rather than directly training a tiny model, we gradually reduce the size of the network to facilitate an iterative co-evolution with the teacher. Progressive$^2$ serves as a flexible framework; the progressive strategy of the teacher can be deployed independently to achieve an optimal balance between accuracy and training efficiency, while the joint integration of the teacher and the student yields further improvements in overall performance.

Figures

Figures reproduced from arXiv: 2608.00129 by the authors.

Figure 1
Figure 1. (a) Substantial compression of pre-trained models for mobile scenarios, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Results of several classic distillation methods. For each method, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overview of two versions of our proposed progressive [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: The backpropagation gradient conflict between teacher-student dis [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The schematic diagram of our teacher-side multi-layer adapter with [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Illustration of why getting a suitable gradient direction from knowledge distillation loss is hard. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: A flowchart illustrating the proof process of model training uncertainty [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Figure of hardware devices. using top-1 and top-5 accuracy metrics. QoS is the service’s consistent capability to be effective, available, and reliable. Following the fundamental definitions outlined by [54], we characterize the QoS value of KD from server to client ba…
Figure 9
Figure 9. Figure 9: Performance evaluation on mobile devices. Six distinct ResNet model [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Training instability on the CIFAR-100 dataset under an experimental [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: The training mIoU and loss curves for models of different sizes [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Quality-of-Services values with different user preferences of knowl [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: The Top-1 accuracy curves for different-sized student models, using [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 1
Figure 1. Figure 1: Teacher, student intrinsic dimension and mIoU at typical epochs. [PITH_FULL_IMAGE:figures/full_fig_p018_1.png]
Figure 2
Figure 2. Figure 2: The backpropagation gradient conflict between teacher [PITH_FULL_IMAGE:figures/full_fig_p019_2.png]
Figure 3
Figure 3. Figure 3: Intuitive visualization about the teacher-side multi-layer [PITH_FULL_IMAGE:figures/full_fig_p020_3.png]
Figure 4
Figure 4. Figure 4: The conflict between teacher-student distillation and [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Illustration of why getting a suitable knowledge distillation direction is hard. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Training instability on the CIFAR-100 dataset under an [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: The standardized mIoU results of models under [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 9
Figure 9. Figure 9: Correlation between the proposed instability metric [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 8
Figure 8. Figure 8: Visualization of the proposed gradient norm variance [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 8 linked inside Pith

  1. [1]

    Maximum likelihood estimation of intrinsic dimension,

    E. Levina and P. Bickel, “Maximum likelihood estimation of intrinsic dimension,”Advances in neural information processing systems, vol. 17, 2004

    2004

  2. [2]

    Estimating the intrinsic dimension of datasets by a minimal neighborhood information,

    E. Facco, M. d’Errico, A. Rodriguez, and A. Laio, “Estimating the intrinsic dimension of datasets by a minimal neighborhood information,” Scientific reports, vol. 7, no. 1, p. 12140, 2017

    2017

  3. [3]

    Intrinsic dimension of data representations in deep neural networks,

    A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan, “Intrinsic dimension of data representations in deep neural networks,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  4. [4]

    Rethinking feature-based knowledge distillation for face recognition,

    J. Li, Z. Guo, H. Li, S. Han, J.-W. Baek, M. Yang, R. Yang, and S. Suh, “Rethinking feature-based knowledge distillation for face recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 20 156–20 165

    2023

  5. [5]

    Understand- ing deep learning requires rethinking generalization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understand- ing deep learning requires rethinking generalization,” inInternational Conference on Learning Representations, 2017

    2017

  6. [6]

    Reduc- ing overfitting in deep networks by decorrelating representations,

    M. Cogswell, F. Ahmed, R. Girshick, L. Zitnick, and D. Batra, “Reduc- ing overfitting in deep networks by decorrelating representations,”arXiv preprint arXiv:1511.06068, 2015

    arXiv 2015

  7. [7]

    Rethinking the value of network pruning,

    Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” inInternational Conference on Learning Representations, 2019

    2019

  8. [8]

    Separability and geometry of object manifolds in deep neural networks,

    U. Cohen, S. Chung, D. D. Lee, and H. Sompolinsky, “Separability and geometry of object manifolds in deep neural networks,”Nature communications, vol. 11, no. 1, p. 746, 2020

    2020

Show all 49 references
  1. [9]

    The cityscapes dataset for semantic urban scene understanding,

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” inCVPR, 2016. 15

    2016

  2. [10]

    Segnet: A deep con- volutional encoder-decoder architecture for image segmentation,

    V . Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep con- volutional encoder-decoder architecture for image segmentation,”IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481–2495, 2017

    2017

  3. [11]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  4. [12]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, University of Tront, 2009

    2009

  5. [13]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” inInternational Conference on Learning Representations, 2017

    2017

  6. [14]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”stat, vol. 1050, p. 9, 2015

    2015

  7. [15]

    Improved knowledge distillation via teacher assis- tant,

    S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, “Improved knowledge distillation via teacher assis- tant,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, 2020, pp. 5191–5198

    2020

  8. [16]

    Knowledge distillation from a stronger teacher,

    T. Huang, S. You, F. Wang, C. Qian, and C. Xu, “Knowledge distillation from a stronger teacher,”Advances in Neural Information Processing Systems, vol. 35, pp. 33 716–33 727, 2022

    2022

  9. [17]

    Conflict-averse gradi- ent descent for multi-task learning,

    B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, “Conflict-averse gradi- ent descent for multi-task learning,”Advances in Neural Information Processing Systems, vol. 34, pp. 18 878–18 890, 2021

    2021

  10. [18]

    Patient knowledge distillation for bert model compression,

    S. Sun, Y . Cheng, Z. Gan, and J. Liu, “Patient knowledge distillation for bert model compression,”arXiv preprint arXiv:1908.09355, 2019

    1908 arXiv

  11. [19]

    Curriculum temperature for knowledge distillation,

    Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang, “Curriculum temperature for knowledge distillation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2, 2023, pp. 1504–1512

    2023

  12. [20]

    Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,

    H. Zhou and L. Song, “Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,” inProceedings of International Conference on Learning Representations (ICLR), 2021

    2021

  13. [21]

    Controllable dynamic multi- task architectures,

    D. S. Raychaudhuri, Y . Suh, S. Schulter, X. Yu, M. Faraki, A. K. Roy-Chowdhury, and M. Chandraker, “Controllable dynamic multi- task architectures,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10 955–10 964

    2022

  14. [22]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  15. [23]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  16. [24]

    Student customized knowledge distillation: Bridging the gap between student and teacher,

    Y . Zhu and Y . Wang, “Student customized knowledge distillation: Bridging the gap between student and teacher,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5057–5066

    2021

  17. [25]

    Gra- dient surgery for multi-task learning,

    T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Gra- dient surgery for multi-task learning,”Advances in Neural Information Processing Systems, vol. 33, pp. 5824–5836, 2020

    2020

  18. [26]

    Famo: Fast adaptive multitask optimization,

    B. Liu, Y . Feng, P. Stone, and Q. Liu, “Famo: Fast adaptive multitask optimization,”Advances in Neural Information Processing Systems, vol. 36, pp. 57 226–57 243, 2023

    2023

  19. [27]

    Knowledge distillation for multi-task learning,

    W.-H. Li and H. Bilen, “Knowledge distillation for multi-task learning,” inEuropean Conference on Computer Vision. Springer, 2020, pp. 163– 176

    2020

  20. [28]

    Contrastive representation distilla- tion,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive representation distilla- tion,” inInternational Conference on Learning Representations, 2020

    2020

  21. [29]

    Cross-layer distillation with semantic calibration,

    D. Chen, J.-P. Mei, Y . Zhang, C. Wang, Z. Wang, Y . Feng, and C. Chen, “Cross-layer distillation with semantic calibration,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, 2021, pp. 7028–7036

    2021

  22. [30]

    On the efficacy of knowledge distillation,

    J. H. Cho and B. Hariharan, “On the efficacy of knowledge distillation,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4794–4802

    2019

  23. [31]

    Segformer: Simple and efficient design for semantic segmentation with transformers,

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,”Advances in neural information processing systems, vol. 34, pp. 12 077–12 090, 2021

    2021

  24. [32]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  25. [33]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    V . Sanh, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.” inProceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019), 2019

    2019

  26. [34]

    Tinybert: Distilling bert for natural language understanding,

    X. Jiao, Y . Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling bert for natural language understanding,” inFindings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 4163–4174

    2020

  27. [35]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  28. [36]

    Learning multiple dense prediction tasks from partially annotated data,

    W.-H. Li, X. Liu, and H. Bilen, “Learning multiple dense prediction tasks from partially annotated data,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 879–18 889

    2022

  29. [37]

    Does knowledge distillation really work?

    S. Stanton, P. Izmailov, P. Kirichenko, A. A. Alemi, and A. G. Wilson, “Does knowledge distillation really work?”Advances in neural infor- mation processing systems, vol. 34, pp. 6906–6919, 2021

    2021

  30. [38]

    The platonic representation hypothesis,

    M. Huh, B. Cheung, T. Wang, and P. Isola, “The platonic representation hypothesis,”CoRR, 2024

    2024

  31. [39]

    Densely guided knowledge distillation using multiple teacher assistants,

    W. Son, J. Na, J. Choi, and W. Hwang, “Densely guided knowledge distillation using multiple teacher assistants,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9395–9404

    2021

  32. [40]

    Monotakd: Teaching assistant knowledge distillation for monocular 3d object detection,

    H.-I. Liu, C. Wu, J.-H. Cheng, W. Chai, S.-Y . Wang, G. Liu, H. Latapie, J.-C. Wu, J.-N. Hwang, H.-H. Shuaiet al., “Monotakd: Teaching assistant knowledge distillation for monocular 3d object detection,” in Proceedings of the Computer Vision and Pattern Recognition Confer- enc...

    2025

  33. [41]

    An overview of statistical learning theory,

    V . N. Vapnik, “An overview of statistical learning theory,”IEEE trans- actions on neural networks, vol. 10, no. 5, pp. 988–999, 1999

    1999

  34. [42]

    Unifying dis- tillation and privileged information,

    D. Lopez-Paz, L. Bottou, B. Sch ¨olkopf, and V . Vapnik, “Unifying dis- tillation and privileged information,”arXiv preprint arXiv:1511.03643, 2015

    2015 arXiv

  35. [43]

    On the difficulty of training recurrent neural networks,

    R. Pascanu, T. Mikolov, and Y . Bengio, “On the difficulty of training recurrent neural networks,” inInternational conference on machine learning. Pmlr, 2013, pp. 1310–1318

    2013

  36. [44]

    An empirical model of large-batch training,

    S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team, “An empirical model of large-batch training,”arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  37. [45]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,

    Z. Chen, V . Badrinarayanan, C.-Y . Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” inInternational conference on machine learning. PMLR, 2018, pp. 794–803

    2018

  38. [46]

    A study of gradient variance in deep learning,

    F. Faghri, D. Duvenaud, D. J. Fleet, and J. Ba, “A study of gradient variance in deep learning,”arXiv preprint arXiv:2007.04532, 2020

    2007 arXiv

  39. [47]

    Decoupled knowledge distillation,

    B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang, “Decoupled knowledge distillation,” inProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2022, pp. 11 953–11 962

    2022

  40. [48]

    End-to-end multi-task learning with attention,

    S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1871–1880

    2019

  41. [49]

    Inverted pyramid multi-task transformer for dense scene understanding,

    H. Ye and D. Xu, “Inverted pyramid multi-task transformer for dense scene understanding,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 514–530. 16 TABLE XI: Summary of dataset-level experimental configurations. Dataset Task Data split / scale Architecture Opti...

    2022

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.