Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Soft-label training from teacher ensembles sets a new respiratory sound classification record on ICBHI, reaching 64.39.

desk verdict Useful, reproducible KD study for respiratory audio with a SOTA claim that needs a validation split before you trust the number. read the letter →

arxiv 2505.22027 v1 pith:3JS6M2OS submitted 2025-05-28 cs.SD cs.AIeess.AS

Miika Toikkanen , June-Woo Kim This is my paper
classification cs.SDcs.AIeess.AS
keywords respiratorysoundclassificationknowledgedistillationsoftlabelsensembleICBHIbenchmarkarchitecture-agnosticstate-of-the-art
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that replacing one-hot training labels with soft, probabilistic labels produced by an ensemble of teacher models improves respiratory sound classification across many architectures, with no extra cost at inference time. It reports a new state-of-the-art ICBHI Score of 64.39, up from the previous best of 63.54, and an average gain of 1.16 across six architectures. This matters because respiratory sound datasets are small and noisy, so a cheap training-time boost to single-model accuracy could make diagnostic support more practical on lightweight devices.

What carries the argument

The central object is the soft label: a probability distribution over classes generated by a teacher ensemble. Two variants are used: the mean teacher takes the softmax of the average of $k$ teachers' logits, while the random teacher uniformly samples one teacher's logits at each training step. The student model is trained with cross-entropy loss against these soft labels, and the hard labels are discarded entirely. This machinery transfers ensemble knowledge into a single network, allowing a 'self-distilled' student to outperform its own hard-label training even when teacher and student share the same architecture.

What would settle it

Retrain the BTS student with the teacher count and curated teacher set chosen on a held-out validation split, leaving the ICBHI test set untouched for final evaluation; if the resulting score fails to beat 63.54, the state-of-the-art claim is not robust.

Watch

Extended reading notes

Core claim

On the authors' own terms, the central discovery is that response-based knowledge distillation—training a student on the softmax of mean or randomly sampled teacher logits instead of hard labels—consistently boosts respiratory sound classifiers, even when the teacher is a single instance of the same architecture as the student. The best distilled single model, BTS-d with a random teacher ensemble of size 15, reaches 64.39 on ICBHI, surpassing the previous best by 0.85. A second-generation ensemble distilled from already-distilled students reaches 65.45, and the gains hold for lightweight architectures such as ResNet18, EfficientNet, and CNN6. The ablation shows that the improvement comes from the distribution of the soft labels, not from additive label noise, since noised-label controls perform worse than the baseline.

Load-bearing premise

The paper selects the teacher count and the curated teacher set using the test set's empirical scores, so if those choices were fixed on a separate validation set, the headline score could be lower.

Editorial extensions

If this is right

  • Any existing respiratory sound classifier can be improved by training it against soft labels from an ensemble of itself, without changing its architecture or test-time latency.
  • The second-generation ensemble result suggests that iterative distillation—distilling an already-distilled ensemble—can yield stronger predictors at the same inference cost.
  • Because the method is architecture-agnostic and works on small models, it offers a practical path to deploy accurate respiratory screening on edge devices with limited compute.
  • The observed trade-off (specificity up, sensitivity slightly down) implies that distillation shifts the classifier's operating point, which must be accounted for when setting clinical decision thresholds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter validation protocol that fixes the teacher count $k$ and the curated teacher set on a held-out split rather than on the test set might lower the reported 64.39; the paper's own model selection appears to use the test set, so the true out-of-sample gain is plausibly somewhat smaller.
  • The same soft-label recipe could transfer to other small medical audio datasets, such as heart sound or pathological speech, where data scarcity and inference constraints are the dominant bottlenecks.
  • The analysis of teacher validation loss suggests that students overfit to the teacher's label distribution beyond a certain $k$; a testable extension is that temperature-scaled softmax or label smoothing would push the optimal teacher count higher.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an architecture-agnostic knowledge-distillation method for respiratory sound classification. A teacher ensemble of BTS models generates soft labels, either by averaging logits over k teachers ('mean teacher') or by sampling a random teacher's logits per iteration ('random teacher'); students are trained with cross-entropy on these soft labels in place of hard labels. Experiments on the ICBHI benchmark report a state-of-the-art single-model Score of 64.39 (random teacher, k=15), gains across six architectures averaging 1.16, and a second-generation ensemble BTS-d++ score of 65.45. The authors argue the method adds no inference-time cost and release code.

Significance. If the headline result holds, the paper makes a useful, simple contribution: it systematically studies soft-label distillation from ensembles for respiratory sound classification, demonstrates that even a single self-teacher helps, and shows architecture-agnostic gains. Strengths include multi-seed evaluation for all student models (5 seeds), a clear ablation study (including noise-label and single-teacher baselines), the systematic sweep over teacher count k, and the public release of code. The main weakness is that the reported state-of-the-art score is selected using test-set information, and an internal inconsistency in Table 2 undermines the precise SOTA claim.

major comments (3)
  1. [§4.1, §4.3, Figure 2] The headline SOTA of 64.39 is obtained under a model-selection protocol that uses the ICBHI test set. In §4.1, the optimal k for each teacher method is 'chosen based on empirical results in Figure 2,' and Figure 2 plots test-set Scores for k=0,...,30. In §4.3, the curated teacher ensemble is formed by selecting 'the 5 highest score checkpoints,' again based on test-set performance. The authors' statement that this 'does not cause leakage of test data' is incorrect: the teacher count, teacher type, and teacher checkpoints are selected using test labels, which is test-set leakage in the model-selection sense. The reported 64.39 is the maximum of a large grid of correlated configurations and is therefore expected to overstate the performance of any fixed configuration. Please repeat the analysis with a validation split (or nested CV) used to choose k, teacher type, and curated checkpoints, and report the fixed-configuration test score; otherwise the specific SOTA claim is not established.
  2. [§4.2 vs. Table 2] There is an internal inconsistency between the text and Table 2. Section 4.2 states that 'all Table 2 students used BTS++[k=5] as the mean teacher,' yet the BTS row in Table 2 reports 64.39, which Table 3 and §4.1 identify as the random-teacher k=15 result (the mean-teacher k=5 result is 64.38). This matters because the BTS row is the source of the claimed 0.85 improvement over the previous SOTA and is also included in the average gain of 1.16. If the Table 2 BTS row was actually trained with the mean teacher at k=5, the reported numbers should be 64.38 (or the appropriate value); if it was trained with the random teacher at k=15, then the text misdescribes the setup for the other architectures. Please clarify which configuration produced each number and ensure the table, text, and figure captions are mutually consistent.
  3. [§2.3, Table 1, Table 3] The SOTA claim is not supported by any significance testing, and the reported gap is within the reported variability. The distilled BTS-d[k=15] result is 64.39±0.42 over 5 seeds, while the baseline BTS is 63.54±0.80; the difference of 0.85 is smaller than the combined standard deviations. Similarly, the mean-teacher k=5 result 64.38±0.36 overlaps with the baseline. Given that the difference is the central quantitative claim, please provide per-seed results, a paired significance test (e.g., a paired t-test or Wilcoxon signed-rank test over the 5 seeds), or confidence intervals. Without this, 'surpassing the previous best by 0.85' is not statistically substantiated.
minor comments (5)
  1. [Abstract] The abstract spells the dataset name as 'ICHBI'; the correct acronym is ICBHI. Please fix this typo.
  2. [§3.1] The sentence 'By aligning these modalities within a shared latent space, This multimodal approach achieves...' has an unnecessary capital 'T' in 'This'. Please correct the capitalization.
  3. [§4.4, Figure 2] The text says 'we have used the mean teacher with k=5 as the default soft label setting for our experiments,' but §4.1 and Table 1 also report the random-teacher k=15 result as the best single-model score. Please clarify what 'default' means and how the two reported numbers relate to the selection procedure.
  4. [Table 1] The table lists BTS-d[k=5] (mean teacher) and BTS-d[k=15] (random teacher) without noting in the table itself that these k values were selected on the test set; a footnote or caption note would help readers interpret the numbers.
  5. [§3.4] The second-generation ensemble BTS-d++ is described as 'compressing the teacher ensemble into a student ensemble,' but the result is still an ensemble at inference time. The wording could be clarified to avoid confusion with the single-model distillation claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the soft-label distillation result is an empirical evaluation, not a consequence of the method's definitions or of a self-citation chain.

full rationale

The paper's derivation chain is not circular in the sense required by this audit. The student is trained by cross-entropy against soft labels obtained either by averaging teacher logits or by sampling one teacher's logits (Eqs. 1-5), and the reported performance gains are empirical outcomes on the fixed ICBHI test split, not quantities that reduce to the soft-label construction by definition. The BTS model from prior work by the same group is used as a fixed teacher and as a baseline; the distillation equations do not import the paper's own conclusions, and the improvement over BTS is an independent measurement. The only audit-adjacent issue is the selection of teacher count k, teacher type, and curated teacher checkpoints using ICBHI test-set scores (Sections 4.1 and 4.3). That is a model-selection and reproducibility concern, not a logical circularity: the reported 64.39 is an evaluated score, not a prediction forced by the method's definitions. There is also an internal inconsistency between Section 4.2's stated mean-teacher k=5 setting and the BTS row of Table 2, whose values match the random-teacher k=15 result from Table 3; this is a reporting error rather than a circular step. No load-bearing self-citation or uniqueness import is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities or fitted physical parameters, but the teacher ensemble size, teacher type, and curated subset are hyperparameters selected on the test set, which is a form of data-dependent tuning.

free parameters (2)
  • teacher ensemble size k = 5 (mean) or 15 (random)
    Chosen based on test set performance in Figure 2.
  • teacher selection method = mean/random/curated
    Selected based on test scores; curated uses 5 highest-scoring checkpoints.
assumptions (4)
  • domain assumption ICBHI training/test split and annotations are correct and patient-disjoint
    The paper relies on the official split and labels.
  • standard math Softmax cross-entropy with soft labels is a suitable surrogate loss (no temperature scaling needed)
    They use standard CE; removing softmax fails to converge.
  • domain assumption A teacher ensemble trained on the same training data produces soft labels that improve the student's test generalization
    Core mechanism; supported by ablations but no theoretical guarantee.
  • domain assumption The BTS model from the authors' prior work serves as the base architecture and its training recipe is transferable
    The paper adopts the BTS experimental settings from [20].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles." pith.science (2026). https://pith.science/paper/3JS6M2OS

@misc{pith2026250522027,
  author       = {Pith},
  title        = {Pith review of: Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JS6M2OS}},
  note         = {Machine review of arXiv:2505.22027}
}
read the original abstract

Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.

Figures

Figures reproduced from arXiv: 2505.22027 by the authors.

Figure 1
Figure 1. , and the student model learns to mimic the teacher’s responses. The teacher ensemble creates soft labels by comput￾ing the mean of the logits from all predictors, or by sampling the logits of a single predictor from the ensemble. The follow￾ing sections explain the process in detail. 3.1. BTS Model The BTS (Bridging Text and Sound) [20] model is a multimodal framework designed to integrate respiratory sound with te… view at source ↗
Figure 3
Figure 3. Teacher model BTS++ validation loss and ICBHI score compared to the student BTS-d ICBHI score as k in￾creases. at around 65.7, while the test-time compute cost grows linearly as k increases. Although the score of the ensemble keeps in￾creasing throughout the range of k, the score of student model tapers off after the teacher validation loss converges. At k = 5, the distilled model performs similarly to the teacher e… view at source ↗
Figure 2
Figure 2. The diminishing beneficial effect of increasing teacher count k. 4.4. Diminishing Returns of Growing Teacher Ensemble Size In this section, we perform a series of experiments to study the optimal size for the teacher ensemble. We train and evaluate a student model with k set at different multiples of 5 from 0 teachers to 30 teachers, and include k = 1 and k = 3 to better view the effect at the lower end of the scale… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 24 canonical work pages

  1. [1]

    Previous works primarily focused on CNN architectures [1–8], such as ResNet [9], EfficientNet [10], and CNN6 [11]

    Introduction Respiratory sound classification (RSC) has been an active re- search area due to its potential to aid in diagnosing respi- ratory diseases. Previous works primarily focused on CNN architectures [1–8], such as ResNet [9], EfficientNet [10], and CNN6 [11]. More recently, pretrained Audio Spectro- gram Transformer (AST) [12, 13] models trained o...

    work page

  2. [2]

    Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles

    Preliminaries 2.1. Dataset Description The ICBHI respiratory sound dataset [21] is a widely recog- nized benchmark for RSC tasks. It consists of approximately 5.5 hours of respiratory sound recordings, comprising a total of 6,898 breathing cycles. The dataset is officially divided into training (60%) and testing (40%) subsets at the breathing cycle level,...

    work page Pith review arXiv 2025

  3. [3]

    The teacher ensemble creates soft labels by comput- ing the mean of the logits from all predictors, or by sampling the logits of a single predictor from the ensemble

    Method We employ a response-based knowledge distillation approach, where the teacher is an ensemble of predictors as depicted in Figure 1, and the student model learns to mimic the teacher’s responses. The teacher ensemble creates soft labels by comput- ing the mean of the logits from all predictors, or by sampling the logits of a single predictor from th...

    work page

  4. [4]

    Main Results Table 1 presents the main results compared to previous RSC work on the ICBHI dataset

    Experiments 4.1. Main Results Table 1 presents the main results compared to previous RSC work on the ICBHI dataset. For a fair comparison, we sep- arate the ensemble models (labeled with “++”) from the non- ensemble models as they multiply the test-time compute cost. Without additional test-time compute cost, the distilled BTS- d outperforms all previous ...

    work page

  5. [5]

    The second generation ensemble BTS-d++[ k = 5] reaches 65.45 with the same compute cost, highlighting the value of good quality teachers. Increasing k to 15 increases the score to 64.75 and further increasingk to 30 for the full available ensem- ble raises the score to 65.69, but with a massive 30x compute cost over the baseline BTS. Based on these result...

    work page

  6. [6]

    Our approach effectively transferred knowledge from the ensemble of teacher models to lightweight student models, achieving state-of-the-art performance on the ICBHI dataset

    Conclusion In this study, we applied architecture-agnostic knowledge dis- tillation for RSC using soft label training to extract knowledge from teacher ensembles. Our approach effectively transferred knowledge from the ensemble of teacher models to lightweight student models, achieving state-of-the-art performance on the ICBHI dataset. We demonstrated tha...

    work page

  7. [7]

    Acknowledgement This research was supported by Brian Impact Foundation, a non-profit organization dedicated to the advancement of science and technology for all

    work page

  8. [8]

    Adventitious respiratory classification using attentive residual neural networks,

    Z. Yang, S. Liu, M. Song, E. Parada-Cabaleiro, and B. W. Schuller, “Adventitious respiratory classification using attentive residual neural networks,” in Interspeech, 2020

    work page 2020

Show all 38 references
  1. [9]

    Lungrn+ nl: An improved adventitious lung sound classification using non-local block resnet neural net- work with mixup data augmentation

    Y . Ma, X. Xu, and Y . Li, “Lungrn+ nl: An improved adventitious lung sound classification using non-local block resnet neural net- work with mixup data augmentation.” in Interspeech, 2020, pp. 2902–2906

    2020

  2. [10]

    Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting,

    S. Gairola, F. Tom, N. Kwatra, and M. Jain, “Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting,” 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2021

    2021

  3. [11]

    Example-based Explanations with Adversarial Attacks for Res- piratory Sound Analysis,

    Y . Chang, Z. Ren, T. T. Nguyen, W. Nejdl, and B. W. Schuller, “Example-based Explanations with Adversarial Attacks for Res- piratory Sound Analysis,” in Proc. Interspeech 2022 , 2022, pp. 4003–4007

    2022

  4. [12]

    Prototype learning for inter- pretable respiratory sound analysis,

    Z. Ren, T. T. Nguyen, and W. Nejdl, “Prototype learning for inter- pretable respiratory sound analysis,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2022, pp. 9087–9091

    2022

  5. [13]

    A domain transfer based data augmenta- tion method for automated respiratory classification,

    Z. Wang and Z. Wang, “A domain transfer based data augmenta- tion method for automated respiratory classification,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 9017–9021

    2022

  6. [14]

    Lung sound classification using co-tuning and stochastic normalization,

    T. Nguyen and F. Pernkopf, “Lung sound classification using co-tuning and stochastic normalization,” IEEE Transactions on Biomedical Engineering, vol. 69, no. 9, pp. 2872–2882, 2022

    2022

  7. [15]

    Pretraining respiratory sound rep- resentations using metadata and contrastive learning,

    I. Moummad and N. Farrugia, “Pretraining respiratory sound rep- resentations using metadata and contrastive learning,” in 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2023, pp. 1–5

    2023

  8. [16]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  9. [17]

    Efficientnet: Rethinking model scaling for convolutional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning. PMLR, 2019, pp. 6105–6114

    2019

  10. [18]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumb- ley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  11. [19]

    AST: Audio Spectrogram Transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in Proc. Interspeech 2021, 2021, pp. 571–575

    2021

  12. [20]

    Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Clas- sification,

    S. Bae, J.-W. Kim, W.-Y . Cho, H. Baek, S. Son, B. Lee, C. Ha, K. Tae, S. Kim, and S.-Y . Yun, “Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Clas- sification,” in Proc. INTERSPEECH 2023, 2023, pp. 5436–5440

    2023

  13. [21]

    Im- agenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Im- agenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  14. [22]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” inProc. IEEE ICASSP 2017, New Orleans, LA, 2017

    2017

  15. [23]

    Ad- versarial fine-tuning using generated respiratory sound to address class imbalance,

    J.-W. Kim, C. Yoon, M. Toikkanen, S. Bae, and H.-Y . Jung, “Ad- versarial fine-tuning using generated respiratory sound to address class imbalance,” arXiv preprint arXiv:2311.06480, 2023

    2023 arXiv

  16. [24]

    Stethoscope-guided supervised contrastive learning for cross- domain adaptation on respiratory sound classification,

    J.-W. Kim, S. Bae, W.-Y . Cho, B. Lee, and H.-Y . Jung, “Stethoscope-guided supervised contrastive learning for cross- domain adaptation on respiratory sound classification,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ...

    2024

  17. [25]

    Masked modeling duo: Towards a universal audio pre-training framework,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Towards a universal audio pre-training framework,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 2024

    2024

  18. [26]

    Towards open respiratory acoustic foundation models: Pretraining and benchmarking,

    Y . Zhang, T. Xia, J. Han, Y . Wu, G. Rizos, Y . Liu, M. Mosuily, J. Chauhan, and C. Mascolo, “Towards open respiratory acoustic foundation models: Pretraining and benchmarking,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Tr...

    2024

  19. [27]

    Bts: Bridging text and sound modalities for metadata-aided res- piratory sound classification,

    J.-W. Kim, M. Toikkanen, Y . Choi, S.-E. Moon, and H.-Y . Jung, “Bts: Bridging text and sound modalities for metadata-aided res- piratory sound classification,” in Interspeech 2024 , 2024, pp. 1690–1694

    2024

  20. [28]

    A respiratory sound database for the development of auto- mated classification,

    B. Rocha, D. Filos, L. Mendes, I. V ogiatzis, E. Perantoni, E. Kaimakamis, P. Natsiavas, A. Oliveira, C. J´acome, A. Marques et al., “A respiratory sound database for the development of auto- mated classification,” in Precision Medicine Powered by pHealth and Connected Health:...

    2017

  21. [29]

    Towards inference efficient deep ensemble learning,

    Z. Li, K. Ren, Y . Yang, X. Jiang, Y . Yang, and D. Li, “Towards inference efficient deep ensemble learning,” inProceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 7, 2023, pp. 8711–8719

    2023

  22. [30]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  23. [31]

    Knowledge distilla- tion: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distilla- tion: A survey,” International Journal of Computer Vision , vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  24. [32]

    Knowledge distillation and student- teacher learning for visual intelligence: A review and new out- looks,

    L. Wang and K.-J. Yoon, “Knowledge distillation and student- teacher learning for visual intelligence: A review and new out- looks,” IEEE transactions on pattern analysis and machine intel- ligence, vol. 44, no. 6, pp. 3048–3068, 2021

    2021

  25. [33]

    Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,

    H. Zhou, L. Song, J. Chen, Y . Zhou, G. Wang, J. Yuan, and Q. Zhang, “Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,” in International Conference on Learning Representations , 2021. [Online]. Available: https: //openreview.net/forum?id...

    2021

  26. [34]

    From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels,

    Z. Yang, A. Zeng, Z. Li, T. Zhang, C. Yuan, and Y . Li, “From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels,” in Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2023, pp. 17 ...

    2023

  27. [35]

    Distillation scaling laws,

    D. Busbridge, A. Shidani, F. Weers, J. Ramapuram, E. Lit- twin, and R. Webb, “Distillation scaling laws,” arXiv preprint arXiv:2502.08606, 2025

    2025 arXiv

  28. [36]

    Repaugment: Input-agnostic representation-level augmentation for respiratory sound classification,

    J.-W. Kim, M. Toikkanen, S. Bae, M. Kim, and H.-Y . Jung, “Repaugment: Input-agnostic representation-level augmentation for respiratory sound classification,” in 2024 46th Annual Inter- national Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2024, pp. 1–6

    2024

  29. [37]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (I...

    2023

  30. [38]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” Interspeech 2019, Sep 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.