Pith. sign in

REVIEW 4 major objections 4 minor 48 references

Multi-Distillation from Speech and Music Representation Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 23-million-parameter encoder distilled from both a speech and a music model matches the specialists on their home tasks and beats them with 20 labels per class.

desk verdict A careful ablation study of speech+music distillation with a real test-set selection problem; the compact student is promising but the headline numbers are optimistic. read the letter →

arxiv 2506.07237 v2 pith:UJCJXOF6 submitted 2025-06-08 eess.AS

Jui-Chiang Wei , Yi-Cheng Lin , Fabian Ritter-Gutierrez , Hung-yi Lee This is my paper
classification eess.AS
keywords knowledgedistillationmulti-teacherself-supervisedlearningspeechrepresentationmusicfew-shotcross-domainaudio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Compressing two domains into one encoder usually costs accuracy on at least one domain. The paper claims this trade-off can be avoided: a 23-million-parameter student distilled jointly from a speech teacher (HuBERT or WavLM) and a music teacher (MERT) reaches 80.40 average accuracy across seven speech and music tasks, versus 81.16 for a 46-million-parameter ensemble and 79.69 for the full HuBERT teacher. The sharper result is few-shot learning with 20 examples per class, where the WavLM+MERT student averages 56.34 percent accuracy and beats each single-domain distilled teacher on all four tasks tested. If these numbers hold, cross-domain multi-teacher distillation is a practical way to build one small audio encoder for real-world audio that mixes speech and music.

What carries the argument

The load-bearing mechanism is a multi-teacher layerwise distillation objective that aligns the student's hidden states with each teacher's hidden states at layers 4, 8, and 12. Per layer and teacher, the loss is $L_{l,m} = \tfrac{1}{D_T}\|h^S_{l,m}-h^T_{l,m}\|_1 - \log\sigma(\cos(h^S_{l,m},h^T_{l,m}))$, combining L1 distance for magnitude with a cosine-similarity term for direction. Three engineering choices make the alignment work across domains: Data-Domain Separation routes speech data only to the speech teacher and music data only to the music teacher; a hybrid feature translator uses a linear layer for speech features and a convolutional layer for music features before comparison; and the music loss is down-weighted ($w_M=0.3$, speech at 1) because MERT is found much easier to distill than HuBERT or WavLM, so equal weighting would let music dominate.

What would settle it

Run the same downstream probes on test clips made by mixing LibriSpeech utterances with Music4ALL tracks at several speech-to-music ratios; if the 23M multi-distilled model fails to beat the best single-domain distilled teacher on the domain it was not trained for, the central cross-domain claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a single small student can hold both speech and music knowledge if the distillation is engineered around the asymmetry between teachers. The student is initialized from the speech teacher's first layers and trained for 200,000 updates on LibriSpeech and Music4ALL. Speech samples are supervised only by the speech teacher and music samples only by the music teacher; a hybrid translator maps the student's output through a linear layer for speech and a convolutional layer for music; and the music teacher's loss is weighted at 0.3 because it is much easier to distill than the speech teachers. With this recipe the HuBERT+MERT student closes almost all of the gap to the 46M ensemble (80.40 versus 81.16) while using half the parameters, and in 20-shot evaluation the WavLM+MERT student exceeds both single-domain teachers on every task tested, with the HuBERT+MERT student exceeding them on two tasks and staying within half a point on the other two.

Load-bearing premise

The paper assumes that a model that does well on separate speech-only and music-only benchmarks will also do well on real audio that mixes speech and music, but training and evaluation never use mixed audio.

Editorial extensions

If this is right

  • A deployment that would otherwise need two domain-specific encoders can use one 23M student, because the average accuracy gap to a 46M ensemble is about 0.8 points on the seven non-ASR benchmarks.
  • In low-label regimes, the unified model is not just smaller but more accurate: with 20 shots per class, WL+MR reaches 56.34 average accuracy versus 51.04 for the best single-domain teacher.
  • Adding music data to speech distillation improves overall accuracy even without domain separation, and domain-separated losses improve it further, so the two corpora are complementary rather than conflicting.
  • The hybrid translator is the component that recovers speech-centric tasks after music training: it lifts intent classification from 85.95 to 91.96 for the HuBERT+MERT model compared with the purely convolutional translator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A deciding test the paper leaves open is mixed audio: clips with speech over background music. If the student beats both single-domain teachers there, the real-world motivation is confirmed; if not, the benefit may be limited to domain-separated benchmarks.
  • The success of a fixed low music weight ($w_M=0.3$) suggests an adaptive weighting rule could be derived from per-teacher distillation difficulty, e.g., set weights inversely to early-training loss drop, removing the manual knob the paper acknowledges.
  • The same recipe should extend to other audio domains, such as environmental sound or multi-speaker audio, as long as data can be routed by domain and the harder-to-distill teacher receives the higher weight; the paper names this direction but does not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multi-teacher distillation framework that compresses a speech teacher (HuBERT or WavLM) and a music teacher (MERT) into a single 23M-parameter student model. The framework combines data-domain separation, feature translators (linear, convolutional, or hybrid), and per-teacher loss weighting. The student is evaluated on five SUPERB speech tasks and three MARBLE music tasks, plus few-shot variants of four tasks. The authors report that the best student (HB+MR) reaches an average accuracy of 80.40% (excluding ASR), close to a 46M ensemble (81.16%) and slightly above the original HuBERT (79.69%), and they claim in the abstract that the model matches or outperforms domain-specific specialists, especially in few-shot settings.

Significance. If the central claim holds, the paper demonstrates a practical path to a compact unified audio encoder that covers both speech and music, which would be valuable for real-world multimedia applications. The work systematically ablates three design axes (data routing, feature alignment, loss weighting) and evaluates on established external benchmarks (SUPERB and MARBLE), which are strengths. The parameter reduction from a 46M ensemble to a 23M student with only a small average accuracy drop is also a concrete contribution. However, the headline 'matches/outperforms' claim is weakened by hyperparameter selection on the test benchmarks and by the fact that the original teachers still win on most of their own domain tasks.

major comments (4)
  1. [Section V-B2 and V-B3, Tables IV and V] The feature translator and the music loss weight wM are selected by the highest overall average accuracy computed on the downstream test sets, and the headline result in Table II (HB+MR, 80.40%) is produced by that selection. Because the same test benchmarks are used for both model selection and final reporting, the reported advantage over HuBERT (79.69%) is optimistically biased; a held-out validation split or a post-selection correction is needed before the 'outperforms' claim is supportable.
  2. [Abstract and Section V-A, Table II] The claim that the model 'matches the performance of domain-specific models' is contradicted by Table II, where HuBERT outperforms the HB+MR student on all five speech tasks (ASR 6.42 vs. 14.65 WER, KS 96.30 vs. 95.26, IC 98.34 vs. 93.36, ER 64.92 vs. 61.85, SID 81.42 vs. 69.29) and MERT outperforms it on SingID and PID. The student's advantage appears only in the average across tasks; the abstract should be rephrased to 'competitive on average' rather than 'matches'.
  3. [Table V and Section V-A] The numbers are internally inconsistent: the text states that wM=0.3 gives the highest overall average (80.36% for HB+MR, 77.95% for WL+MR), but the table lists 80.19 and 78.30 respectively; Section V-A text reports 80.7% for HB+MR while Table II lists 80.40. Additionally, Table IV appears to be computed with wM=1.0 (its linear-translator row matches the wM=1.0 row in Table V), so the final configuration (wM=0.3 plus hybrid translator) is never shown in a single ablation table. These inconsistencies make the core result difficult to audit.
  4. [Sections I and VII, Table I] The stated motivation is real-world audio that mixes speech and music, but training uses separate LibriSpeech and Music4ALL corpora and evaluation uses only single-domain tasks from SUPERB and MARBLE. The limitations section acknowledges this, but the significance of the work as a 'unified audio encoder' for mixed audio remains untested; an experiment with mixed or concatenated audio content would substantially strengthen the central claim.
minor comments (4)
  1. [Table I and Section V-C] The strings 'V ocalSet' and 'V ocal Identification' contain accidental spaces and should read 'VocalSet' and 'Vocal Identification'.
  2. [Equation (1)] The operator σ is not defined in the text; please define the sigmoid function and clarify the argument of the cosine-similarity term in the equation.
  3. [Reference [13]] Reference [13] is the authors' own prior work on distilling a speech and music encoder; the paper should explicitly discuss how the current framework differs from and improves on that work.
  4. [Availability] No code or trained model is released; providing training and evaluation code would allow independent verification of the reported numbers and of the hyperparameter selection procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the distillation method is evaluated on external benchmarks and the reported losses and ablations do not reduce to the claimed results by construction.

full rationale

The paper's central pipeline is an empirical distillation procedure: a 23M-parameter student is trained with a layerwise L1 plus cosine-similarity loss (Eq. 1) toward HuBERT/WavLM and MERT teachers, and then evaluated on the external SUPERB and MARBLE benchmarks. The loss is explicitly adopted from prior DistilHuBERT work, and while some cited works share authorship (e.g., DistilHuBERT and the authors' own speech-plus-music paper [13]), the citations are not load-bearing derivations: the final claims are supported by measured accuracies on independent downstream tasks, not by the truth of the cited papers. The tuning of the music loss weight and translator selection by highest average accuracy in Tables IV and V is a test-set selection and reproducibility concern, not a circularity, because the reported numbers are empirical results of training and evaluation rather than quantities defined to equal the optimization target. The paper's own limitations section acknowledges manual loss-weight tuning and the lack of mixed-audio evaluation, which are validity threats but not self-referential reductions. No equation in the paper is equivalent to another by construction, and no fitted parameter is renamed as a prediction of the benchmarks. The few-shot results are also direct evaluations on held-out splits of the same external test sets, not consequences of the training objective by definition. Overall, the derivation chain is self-contained with respect to the claimed contributions; the main risks are benchmark-selection bias and limited mixed-domain validation, not circular reasoning.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on data and teacher models supplied by other papers; the only new free choices are the loss weight and translator architecture, both tuned on the evaluation benchmarks. No new theoretical entities are introduced. The key unstated assumption is that separate-domain benchmarks transfer to mixed-domain audio.

free parameters (2)
  • wM (music teacher loss weight) = 0.3
    Chosen by sweeping {0.1, 0.3, 0.5, 1.0} and selecting the value with the highest downstream average accuracy on the same test benchmarks (Table V). This is a hand-tuned hyperparameter, not a theoretical quantity.
  • Feature translator architecture = hybrid (linear for speech, convolutional for music)
    Selected after comparing linear, convolutional, and hybrid translators on downstream test accuracy (Table IV), making the architecture choice a data-driven design choice.
assumptions (4)
  • domain assumption HuBERT/WavLM and MERT are representative teachers whose layerwise representations can be transferred via L1 plus cosine loss.
    Section III assumes the DistilHuBERT loss transfers cross-domain knowledge without further exploration ('we do not explore it further').
  • domain assumption Frozen-feature linear probing with a learnable weighted sum is a valid measure of representation quality.
    Section IV-C applies SUPERB and MARBLE protocols under the standard assumption that downstream linear classifier performance reflects the upstream representation.
  • domain assumption LibriSpeech and Music4ALL are sufficient domain samples, and single-domain benchmarks are representative of real-world mixed audio.
    Section IV-A pretrains only on separate speech and music corpora, while Section I motivates mixed speech plus music scenarios. The paper never evaluates on mixed audio.
  • domain assumption Distillation layers {4, 8, 12} are sufficient.
    Section III selects layers 4, 8, and 12 based on prior work; the paper does not ablate this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Distillation from Speech and Music Representation Models." pith.science (2026). https://pith.science/paper/UJCJXOF6

@misc{pith2026250607237,
  author       = {Pith},
  title        = {Pith review of: Multi-Distillation from Speech and Music Representation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJCJXOF6}},
  note         = {Machine review of arXiv:2506.07237}
}
read the original abstract

Real-world audio often mixes speech and music, yet models typically handle only one domain. This paper introduces a multi-teacher distillation framework that unifies speech and music models into a single one while significantly reducing model size. Our approach leverages the strengths of domain-specific teacher models, such as HuBERT for speech and MERT for music, and explores various strategies to balance both domains. Experiments across diverse tasks demonstrate that our model matches the performance of domain-specific models, showing the effectiveness of cross-domain distillation. Additionally, we conduct few-shot learning experiments, highlighting the need for general models in real-world scenarios where labeled data is limited. Our results show that our model not only performs on par with specialized models but also outperforms them in few-shot scenarios, proving that a cross-domain approach is essential and effective for diverse tasks with limited data.

Figures

Figures reproduced from arXiv: 2506.07237 by the authors.

Figure 1
Figure 1. Two pretrained teacher models: one for speech (e.g., [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. Illustration of multi-distillation structure [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 33 canonical work pages

  1. [13]

    Distilling a speech and music encoder with task arithmetic

    F. Ritter-Gutierrez, Y .-C. Lin, J.-C. Wei, J. H. M. Wong, E. S. Chng, N. F. Chen, and H. yi Lee, “Distilling a speech and music encoder with task arithmetic,” 2025. [Online]. Available: https://arxiv.org/abs/2505.13270

    work page Pith review arXiv 2025

  2. [1]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløeet al., “Self-supervised speech representation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, 2022

    work page 2022

  3. [2]

    Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,

    T.-h. Feng, A. Dong, C.-F. Yeh, S.-w. Yang, T.-Q. Lin, J. Shi, K.-W. Chang, Z. Huang, H. Wu, X. Changet al., “Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,” in2022 IEEE Spoken Language Technology Workshop (SLT), 2023

    work page 2022

  4. [3]

    Marble: Music audio representation benchmark for universal evaluation,

    R. Yuan, Y . Ma, Y . Li, G. Zhang, X. Chen, H. Yin, Y . Liu, J. Huang, Z. Tian, B. Denget al., “Marble: Music audio representation benchmark for universal evaluation,”Advances in Neural Information Processing Systems, 2023

    2023

  5. [4]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, 2021

    work page 2021

  6. [5]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, 2022

    work page 2022

  7. [6]

    MERT: Acoustic music understanding model with large-scale self-supervised training,

    Y . LI, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dannenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Z. Wang, Y . Guo, and J. Fu, “MERT: Acoustic music understanding model with large-scale self-supervised training,” inThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  8. [7]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449– 12 460, 2020

    2020

Show all 48 references
  1. [8]

    TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech,

    A. T. Liu, S.-W. Li, and H.-y. Lee, “TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech,”IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 29, pp. 2351– 2366, 2021

    2021

  2. [9]

    Dynamic-superb phase-2: A collaboratively expand- ing benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-y. Huanget al., “Dynamic-superb phase-2: A collaboratively expand- ing benchmark for measuring the capabilities of spoken language models with 180 tasks,”arXiv preprint arXiv:2411.05361, 2024

    2024 arXiv

  3. [10]

    Jukebox: A Generative Model for Music,

    P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A Generative Model for Music,” 2020. [Online]. Available: https://arxiv.org/abs/2005.00341

    2020 arXiv

  4. [11]

    Contrastive Learn- ing of Musical Representations,

    J. Spijkervet and J. A. Burgoyne, “Contrastive Learn- ing of Musical Representations,” 2021. [Online]. Available: https://arxiv.org/abs/2103.09410

    2021 arXiv

  5. [12]

    Encodecmae: Leveraging neural codecs for universal audio representation learning,

    L. Pepino, P. Riera, and L. Ferrer, “Encodecmae: Leveraging neural codecs for universal audio representation learning,” 2024. [Online]. Available: https://arxiv.org/abs/2309.07391

    2024 arXiv

  6. [14]

    Towards learning universal audio representations,

    L. Wang, P. Luc, Y . Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.- B. Alayrac, S. Dieleman, J. Carreiraet al., “Towards learning universal audio representations,” inICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEE...

    2022

  7. [15]

    Ensemble knowledge distillation of self- supervised speech models,

    K.-P. Huang, T.-H. Feng, Y .-K. Fu, T.-Y . Hsu, P.-C. Yen, W.-C. Tseng, K.-W. Chang, and H.-Y . Lee, “Ensemble knowledge distillation of self- supervised speech models,” inICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    2023

  8. [16]

    Theia: Distilling diverse vision foundation models for robot learning,

    J. Shang, K. Schmeckpeper, B. B. May, M. V . Minniti, T. Kelestemur, D. Watkins, and L. Herlant, “Theia: Distilling diverse vision foundation models for robot learning,” in8th Annual Conference on Robot Learning,

  9. [17]

    Ea-kd: Entropy-based adaptive knowledge distillation,

    C.-P. Su, C.-H. Tseng, B. Pu, L. Zhao, Z. Chen, and S.-J. Lee, “Ea-kd: Entropy-based adaptive knowledge distillation,” 2025

    2025

  10. [18]

    Learning small-size dnn with output-distribution-based criteria,

    J. Li, R. Zhao, J. T. Huang, and Y . Gong, “Learning small-size dnn with output-distribution-based criteria,” inInterspeech, 2014

    2014

  11. [19]

    Distilling the knowledge in a neural network,

    G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”NIPS 2014 Deep Learning Workshop, vol. abs/1503.02531, 2015

    2014 arXiv

  12. [20]

    Distilhubert: Speech represen- tation learning by layer-wise distillation of hidden-unit bert,

    H.-J. Chang, S.-w. Yang, and H.-y. Lee, “Distilhubert: Speech represen- tation learning by layer-wise distillation of hidden-unit bert,” inICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022

    2022

  13. [21]

    Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,

    Y . Lee, K. JANG, J. Goo, Y . Jung, and H.-R. Kim, “Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,” in23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022, 2022

    2022

  14. [22]

    Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,

    R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y . Zhang, T. Ko, and H. Li, “Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,”Proc. Interspeech 2022, 2022

    2022

  15. [23]

    On the social bias of speech self-supervised models,

    Y .-C. Lin, T.-Q. Lin, H.-C. Lin, A. T. Liu, and H. yi Lee, “On the social bias of speech self-supervised models,” inInterspeech 2024, 2024

    2024

  16. [24]

    Robustdistiller: Compressing universal speech repre- sentations for enhanced environment robustness,

    H. R. Guimar ˜aes, A. Pimentel, A. R. Avila, M. Rezagholizadeh, B. Chen, and T. H. Falk, “Robustdistiller: Compressing universal speech repre- sentations for enhanced environment robustness,” inICASSP, 2023

    2023

  17. [25]

    Noise robust distillation of self-supervised speech models via correlation metrics,

    F. Ritter-Gutierrez, K.-P. Huang, D. Ng, J. H. Wong, H.-Y . Lee, E. S. Chng, and N. F. Chen, “Noise robust distillation of self-supervised speech models via correlation metrics,” in2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2024

    2024

  18. [26]

    Improving generalizability of distilled self- supervised speech processing models under distorted settings,

    K.-P. Huang, Y .-K. Fu, T.-Y . Hsu, F. R. Gutierrez, F. Wang, L.-H. Tseng, Y . Zhang, and H. yi Lee, “Improving generalizability of distilled self- supervised speech processing models under distorted settings,” inIEEE- SLT Workshop, 2022

    2022

  19. [27]

    Fast jukebox: Accelerating music generation with knowledge distillation,

    M. Pezzat-Morales, H. Perez-Meana, and T. Nakashika, “Fast jukebox: Accelerating music generation with knowledge distillation,”Applied Sciences, 2023

    2023

  20. [28]

    Knowledge distillation for singing voice detection,

    S. Paul, G. R. M, K. S. Rao, and P. P. Das, “Knowledge distillation for singing voice detection,” inInterspeech 2021, 2021

    2021

  21. [29]

    Symbolic-domain musical instrument classification using knowledge distillation from audio-teacher to symbolic-student,

    S. Sawada, “Symbolic-domain musical instrument classification using knowledge distillation from audio-teacher to symbolic-student,” in2024 32nd European Signal Processing Conference (EUSIPCO), 2024

    2024

  22. [30]

    Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks,

    K. Zuchniak, “Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks,”arXiv preprint arXiv:2302.07215, 2023

    2023 arXiv

  23. [31]

    Multiple teacher distillation for robust and greener models,

    A. Ilichev, N. Sorokin, I. Piontkovskaya, and V . Malykh, “Multiple teacher distillation for robust and greener models,” inProceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), 2021

    2021

  24. [32]

    Multilingual neural machine translation with knowledge distillation,

    X. Tan, Y . Ren, D. He, T. Qin, Z. Zhao, and T.-Y . Liu, “Multilingual neural machine translation with knowledge distillation,” inInternational Conference on Learning Representations, 2018

    2018

  25. [33]

    One teacher is enough? pre- trained language model distillation from multiple teachers,

    C. Wu, F. Wu, and Y . Huang, “One teacher is enough? pre- trained language model distillation from multiple teachers,”ArXiv, vol. abs/2106.01023, 2021

    2021 arXiv

  26. [34]

    Unified and effective ensemble knowledge distillation,

    C. Wu, F. Wu, T. Qi, and Y . Huang, “Unified and effective ensemble knowledge distillation,”ArXiv, vol. abs/2204.00548, 2022

    2022 arXiv

  27. [35]

    Distilling knowledge from ensembles of acoustic models for joint ctc-attention end-to-end speech recognition,

    Y . Gao, T. Parcollet, and N. D. Lane, “Distilling knowledge from ensembles of acoustic models for joint ctc-attention end-to-end speech recognition,”2021 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), pp. 138–145, 2020

    2021

  28. [36]

    Distilling knowledge from ensembles of neural networks for speech recognition,

    Y . Chebotar and A. Waters, “Distilling knowledge from ensembles of neural networks for speech recognition,” inInterspeech, 2016

    2016

  29. [37]

    Knowledge distillation from multiple foundation models for end-to-end speech recognition,

    X. Yang, Q. Li, C. Zhang, and P. C. Woodland, “Knowledge distillation from multiple foundation models for end-to-end speech recognition,” arXiv preprint arXiv:2303.10917, 2023

    2023 arXiv

  30. [38]

    Librispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015

    2015

  31. [39]

    Speech model pre-training for end-to-end spoken language understand- ing,

    L. Lugosch, M. Ravanelli, P. Ignoto, V . S. Tomar, and Y . Bengio, “Speech model pre-training for end-to-end spoken language understand- ing,” inInterspeech 2019, 2019, pp. 814–818

    2019

  32. [40]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,”CoRR, vol. abs/1804.03209, 2018. [Online]. Available: http://arxiv.org/abs/1804.03209

    2018 arXiv

  33. [41]

    V oxceleb: Large- scale speaker verification in the wild,

    A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “V oxceleb: Large- scale speaker verification in the wild,”Computer Speech & Language, vol. 60, p. 101027, 2020

    2020

  34. [42]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  35. [43]

    V ocalset: A singing voice dataset

    J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “V ocalset: A singing voice dataset.” inISMIR, 2018, pp. 468–474

    2018

  36. [44]

    Neural audio synthesis of musical notes with wavenet au- toencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet au- toencoders,” inInternational conference on machine learning. PMLR, 2017, pp. 1068–1077

    2017

  37. [45]

    Music4all: A new music database and its applications,

    I. A. Pegoraro Santana, F. Pinhelli, J. Donini, L. Catharin, R. B. Mangolin, Y . M. e. G. da Costa, V . Delisandra Feltrim, and M. A. Domingues, “Music4all: A new music database and its applications,” in2020 International Conference on Systems, Signals and Image Pro- cessing (...

    2020

  38. [46]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”

  39. [2017]

    Available: https://arxiv.org/abs/1412.6980

    [Online]. Available: https://arxiv.org/abs/1412.6980

    arXiv

  40. [2024]

    Available: https://openreview.net/forum?id=ylZHvlwUcI

    [Online]. Available: https://openreview.net/forum?id=ylZHvlwUcI

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.