Pith. sign in

REVIEW 5 major objections 7 minor 33 references

An Effective Training Framework for Light-Weight Automatic Speech Recognition Models

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims a one-time encoder representation learning phase, followed by a short CTC fine-tuning, produces small Conformer ASR models that beat from-scratch equivalents trained for 150 epochs while using about one-third the…

desk verdict Useful recipe and honest ablations, but the 3x speedup claim does not hold under the paper's own epoch budget and the core loss equation is not reproducible as written. read the letter →

arxiv 2505.16991 v2 pith:QDHNFEA2 submitted 2025-05-22 cs.CV

Abdul Hannan , Alessio Brutti , Shah Nawaz , Mubashir Noman This is my paper · ORCID
classification cs.CV
keywords AutomaticSpeechRecognitionEfficientTrainingFeatureRepresentationLearningLight-weightModelKnowledgeDistillationConformerCTCfine-tuningWordErrorRate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many speech models are too large for low-resource devices, and the usual fixes, pruning, distillation, layer skipping, either degrade accuracy or still require long training of the smaller model. The authors propose a two-step alternative: first train a light-weight encoder to imitate the internal representations of a frozen large reference model (encoder representation learning, EncRL), then attach a CTC decoder and fine-tune for only a short number of epochs. Their central claim is that this recipe yields 6-, 4-, and 2-layer Conformer models that outperform equally sized models trained from scratch for 150 epochs, with about a three-fold reduction in per-model training epochs and a reported WER improvement of up to 12.54% on LibriSpeech and TED-LIUM. If the claim holds, the practical payoff is that one reference model can quickly seed several small ASR models at different resource budgets, without prolonged training of each one.

What carries the argument

The load-bearing component is the EncRL loss, $\mathcal{L}_{\mathrm{EncRL}} = \mathcal{L}_{\mathrm{CLIP}} + \mathcal{L}_{\mathrm{MSE}}$, applied while the large reference model is frozen. $\mathcal{L}_{\mathrm{CLIP}}$ is a symmetric cross-entropy on the last-layer encoder outputs of the reference and the light-weight model that pulls features from the same utterance together and pushes features from different utterances apart; $\mathcal{L}_{\mathrm{MSE}}$ is the mean-squared error between the two models' classifier embeddings. This phase produces a reusable 6-layer encoder, and the subsequent CTC fine-tuning turns the learned representations into transcriptions for each target model depth.

What would settle it

Track the complete compute budget for the whole pipeline, including the reference model's 150-epoch training, the 100-epoch EncRL phase, and the 50-epoch fine-tuning, and compare it with a single from-scratch training run; if the end-to-end cost for one target model is not lower and the WER is not better at equal cost, the claimed three-fold speed-up is an accounting artifact rather than a training saving. A concrete check is to compare the EncRL-inclusive 6-layer model against a from-scratch 6-layer model trained for the same total number of epochs.

Watch

Extended reading notes

Core claim

The authors report that a frozen 12-layer Conformer reference trained for Z=150 epochs can transfer its knowledge to a 6-layer student encoder through an EncRL phase of 2Z/3 epochs. The EncRL loss combines a CLIP-style symmetric cross-entropy term on the last-layer encoder features, which aligns same-utterance features and separates different utterances, with an MSE term that aligns the classifier embeddings of reference and student. After this one-time phase, a CTC decoder is attached and the model is fine-tuned for Z/3 epochs; the same phase also seeds shallower 4- and 2-layer models initialized from subsets of the 6-layer encoder weights. On LibriSpeech test-clean, the 6-layer model reaches 6.27% WER and the 2-layer model 14.68%, compared with 8.61% and 15.57% for the same sizes trained from scratch for 150 epochs; on TED-LIUM v3, the 4-layer model drops from 49.30% to 22.86% WER. These numbers are the paper's evidence that small ASR models benefit more from representation transfer from a large reference than from prolonged training.

Load-bearing premise

The load-bearing assumption is that the 2Z/3-epoch EncRL phase can be treated as a one-time cost that is not counted when reporting per-model training epochs; if it is counted, producing a single small model takes Z epochs in total, the same as training from scratch.

Editorial extensions

If this is right

  • Training one 12-layer reference model once makes it possible to obtain 6-, 4-, and 2-layer ASR models by fine-tuning each for only Z/3 epochs, instead of training every size from scratch for Z epochs.
  • On LibriSpeech test-clean, the 6-layer model reaches 6.27% WER versus 8.61% for the 150-epoch from-scratch baseline, and the 2-layer model reaches 14.68% versus 15.57%.
  • On TED-LIUM v3, the 4-layer model improves from 49.30% to 22.86% WER, the largest single improvement reported in the paper.
  • The representation-learning phase itself is necessary: initializing the 6-layer model with reference weights but skipping EncRL gives 7.41% WER, while the full two-step recipe gives 6.27%, and omitting the fine-tuning phase leaves the model unable to transcribe at all, with roughly 95% WER.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the one-time EncRL cost is counted, training W models with this recipe costs $(W+2)Z/3$ epochs in total versus $WZ$ for from-scratch training, so the realized speed-up is $3W/(W+2)$: 1x for W=1, 1.5x for W=2, and only approaching 3x for many fine-tuned models.
  • An untested extension would be reusing a single EncRL checkpoint to fine-tune models on different target datasets or acoustic conditions; the paper only fine-tunes models on the same dataset used to train the reference, so the checkpoint's cross-domain transfer value is not established.
  • Because the EncRL phase only needs encoder outputs from a frozen model, the same two-step recipe could in principle be driven by any large pre-trained speech encoder, removing the need to train the reference model from scratch; the paper does not explore this alternative.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a two-step training framework for lightweight ASR models. In the first step (EncRL), a small Conformer encoder is trained for 2Z/3 epochs to align its feature representations with those of a larger, frozen reference model using a combination of CLIP-style and MSE losses. In the second step, a CTC decoder is attached and the model is fine-tuned for Z/3 epochs. The authors claim that this yields small models that outperform equivalent models trained from scratch for Z epochs, while achieving a three-fold training speed-up and up to 12.54% WER improvement. Experiments are reported on LibriSpeech and TED-LIUM v3 for encoder depths of 6, 4, and 2 layers.

Significance. If the claims were fully supported, the framework would offer a practical way to obtain multiple lightweight ASR models from a single large model with reduced training cost and competitive accuracy. The paper includes comparisons against from-scratch baselines, early-exit dynamic models, and pruning, and it provides a useful ablation of the loss components. However, the central efficiency claim is not supported by the paper's own epoch accounting, and a core loss definition is mathematically ill-posed. The anomalous TED-LIUM baseline and the absence of variance estimates further weaken the empirical conclusions. The qualitative idea of representation learning followed by brief fine-tuning has potential, but the current quantitative framing overstates the benefits.

major comments (5)
  1. [Section 4.1 and Sections 3.1-3.2] The claimed three-fold training speed-up is not supported by the stated epoch budget. The paper states that for W models the method requires (W x Z)/3 training epochs versus W x Z for from-scratch training, but this omits the 2Z/3-epoch EncRL phase (Section 3.1) and the Z-epoch training of the reference model (Section 3, first paragraph). Counting only the EncRL phase, the total is W * Z/3 + 2Z/3 = (W+2)Z/3, giving a speed-up of 3W/(W+2) relative to W*Z; this equals 1 for W=1 and is only 1.8x for W=3, the largest number of model sizes evaluated in Tables 1 and 2. If the Z-epoch reference training is also counted, the method is slower than the from-scratch baseline for W=1 and W=2. Therefore the abstract's "three-fold training speed-up" is not a property of the method under its own epoch budget, and the central quantitative claim is unsupported as stated.
  2. [Section 3.1, Eq. (3)] The definition of the feature-alignment loss is not mathematically well-defined. The expression L_CLIP = max_{i≠j} min_{i=j} [ sum_{i,j} e^i_ref ⊙ e^j_LW ] is syntactically invalid: the max and min bind conflicting index scopes, the conditions i=j and i≠j cannot both apply to the same sum, and the equation does not correspond to any standard contrastive or symmetric cross-entropy loss. Since this loss is a core component of the proposed method, the method is not reproducible as written. The authors should provide the actual contrastive loss formula (e.g., InfoNCE with temperature) and specify the index normalization.
  3. [Table 2 (TED-LIUM v3)] The from-scratch 4-layer baseline (49.30% WER) is anomalously high compared to the 6-layer baseline (21.82%) and to the reported 4-layer result from [18] (35.4% WER). This suggests the 4-layer baseline was undertrained or otherwise miscalibrated, and it directly inflates the claimed 26.44% absolute improvement for the proposed 4-layer model. The authors should retrain this baseline with the same protocol as the other baselines and report the result; without this, the headline improvement on TED-LIUM is not credible.
  4. [Tables 1-4] All WER numbers are reported as point estimates from a single run, with no standard deviations, confidence intervals, or multiple seeds. Given that several claimed improvements are small (e.g., 0.36% absolute on LibriSpeech test-other for the 4-layer model, and 0.53% in the ablation of Section 4.2.2), the differences may be within run-to-run variability. The authors should report variance over at least three random seeds, or at minimum state the expected training noise level, for the key comparisons.
  5. [Section 4.2.2 and Section 4.1] The experimental protocol for producing the 4- and 2-layer models is ambiguous. The main text does not specify whether these models receive their own EncRL phase or are initialized from the last n layers of the 6-layer EncRL model. This ambiguity matters because the speed-up accounting changes: running separate EncRL for each model size adds 2Z/3 epochs per size. The ablation in Section 4.2.2 suggests the two options differ by 0.53% WER, so the choice is not inconsequential. Please specify the exact protocol used for Tables 1 and 2.
minor comments (7)
  1. [Abstract] The phrase "up to 12.54% word error rate improvement" is not derivable from any table entry; specify whether it is absolute or relative and identify the comparison baseline.
  2. [Section 3.1] The phrase "symmetric cross-entropy L_CLIP loss" is misleading; the CLIP loss is an InfoNCE-style contrastive loss, not cross-entropy. Also, no weighting coefficient between L_CLIP and L_MSE in Eq. (2) is specified; state whether they are used with equal weight.
  3. [Section 3.2, Eq. (5)] The notation L_ctc = f_CTC(t_i - t_pred) is not a mathematical definition; f_CTC is undefined and the subtraction of token sequences is not well-defined. Use the standard CTC loss notation, e.g., L_ctc = -log p(t_i | a_i).
  4. [Tables 1 and 2] The row labels "Conformer*" and "Conformer (ours)" should clarify the number of epochs used for each baseline and method; the current table header "Epochs" is confusing because the reference row reports 150, the ours rows report 50, and the asterisk rows are described only in the caption.
  5. [Section 4.1, Table 3] The column headers in Table 3 appear shifted: the WER values and parameter counts are interleaved without clear column alignment. Please format the table so each method has its own WER and Params columns.
  6. [Section 4.2.3] The comparison between pruning percentages and the "equivalent pruning amount" based on layer counts is not well-defined; pruning 50% of weights is not equivalent to removing 6 of 12 encoder layers. Clarify the equivalence.
  7. [General] There are several typos and awkward phrasings, including "ins spite" (Section 4.1), "despite of" (Abstract), and "Vitality of Finetuning" (Section 4.1), where "Viability" or "Necessity" would be more standard.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: reported WER values are empirical evaluations on public test splits; the only quantitative concern is an epoch-accounting omission, not a definitional or self-citation reduction.

full rationale

The paper's central claim is that a two-step EncRL-plus-finetuning procedure produces small Conformer models with better WER than from-scratch training in fewer epochs. That claim is supported by Tables 1-3, which report WER on held-out LibriSpeech and TED-LIUM test sets for the proposed models, from-scratch baselines, and published/prior-work baselines. The WER numbers are measured, not derived from the training losses; nothing in the EncRL loss (Eqs. 2-4) or fine-tuning CTC loss (Eq. 5) numerically forces the reported test-set WER. The paper does cite prior work by the same authors (Early Exit [18] and LDASR [19]), but only as baselines to compare against, not as justification for the framework, so the self-citations are not load-bearing. No uniqueness theorem is invoked, and no parameter is fitted to test data and then renamed a prediction. The only notable weakness is the Section 4.1 claim that 'to train W models ... our method requires (W x Z)/3 training epochs' and 'achieving 3x faster training': this omits the 2Z/3-epoch EncRL phase and the Z epochs used to train the reference model. That is a cost-accounting error, not circularity, because the central WER results do not reduce to this arithmetic. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. It relies on standard machine-learning assumptions about architecture, loss functions, and feature transfer. The main implicit assumptions are that encoder features are sufficient for knowledge transfer and that the chosen loss combination is correctly implemented despite the malformed equation.

free parameters (3)
  • loss weight ratio = 1:1 (implicit)
    The total EncRL loss is written as LCLIP + LMSE with no weighting coefficients; the balance between the two terms is a free choice that likely affects the ablation results.
  • epoch split ratio = 2/3 for EncRL, 1/3 for fine-tuning
    The 2Z/3 and Z/3 split is chosen by hand and directly determines the claimed speedup; no sensitivity analysis is provided.
  • reference model size = 12 encoder layers, feed-forward 256
    The choice of Conformer-small with 12 layers as the reference model is an architectural assumption, not derived from the problem.
assumptions (3)
  • domain assumption CTC decoding is a valid objective for ASR fine-tuning
    The paper uses CTC loss during fine-tuning without comparing to other sequence losses, assuming it is appropriate for the task.
  • domain assumption The feature spaces of the reference and light-weight models are directly comparable
    The feature-alignment loss (LCLIP and LMSE) assumes that the last-layer encoder features of the 12-layer and smaller models live in spaces that can be meaningfully aligned, which is not demonstrated.
  • domain assumption The reference model's knowledge is fully captured by its encoder features and classifier embeddings
    The method ignores the decoder and relies on encoder representations, implicitly assuming that the information needed for ASR is contained in these layers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Effective Training Framework for Light-Weight Automatic Speech Recognition Models." pith.science (2026). https://pith.science/paper/QDHNFEA2

@misc{pith2026250516991,
  author       = {Pith},
  title        = {Pith review of: An Effective Training Framework for Light-Weight Automatic Speech Recognition Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDHNFEA2}},
  note         = {Machine review of arXiv:2505.16991}
}
read the original abstract

Recent advancement in deep learning encouraged developing large automatic speech recognition (ASR) models that achieve promising results while ignoring computational and memory constraints. However, deploying such models on low resource devices is impractical despite of their favorable performance. Existing approaches (pruning, distillation, layer skip etc.) transform the large models into smaller ones at the cost of significant performance degradation or require prolonged training of smaller models for better performance. To address these issues, we introduce an efficacious two-step representation learning based approach capable of producing several small sized models from a single large model ensuring considerably better performance in limited number of epochs. Comprehensive experimentation on ASR benchmarks reveals the efficacy of our approach, achieving three-fold training speed-up and up to 12.54% word error rate improvement.

Figures

Figures reproduced from arXiv: 2505.16991 by the authors.

Figure 1
Figure 1. Here we illustrate the overall framework of the proposed method. (i) In the feature learning phase, the input utterance is fed to a large ASR model (reference model) to extract the features eref and bref . Afterwards, the same input is passed through the smaller model to obtain features eLW and bLW . To learn the knowledge of reference model, MSE loss is used between the outputs (bref and bLW ) of the classifiers of… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 29 canonical work pages

  1. [18]

    Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition,

    J. W. Yoon, H. Lee, H. Y . Kim, W. I. Cho, and N. S. Kim, “Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1626–1638, 2021

    work page 2021

  2. [1]

    Several efforts have been directed to create user and device-personalized models with adaptable model sizes while minimizing the latency using different tech- niques

    Introduction Accelerated progress in Internet of Things (IoT) [1] and edge- cloud continuum [2] demands significant reduction in the neural architecture sizes for smooth operation with minimal computa- tional and memory burden. Several efforts have been directed to create user and device-personalized models with adaptable model sizes while minimizing the ...

    work page

  3. [2]

    An Effective Training Framework for Light-Weight Automatic Speech Recognition Models

    Background Representation Learning using Mask Auto-Encoders: Mask auto-encoders [21, 22] are self-supervised frameworks that are utilized to learn rich semantic features from large amount of unlabeled data. Mask auto-encoder masks the large portion of its input X ∈ RH×W , and feeds the unmasked patches to the encoder to obtain encoded features Xe. In de- ...

    work page Pith review arXiv 2025

  4. [3]

    Proposed Framework To mitigate the performance limitations and facilitate fast train- ing, we propose a two-step feature learning based framework in Figure 1. The proposed framework requires: (i) One time feature representation learningto mimic the enriched represen- tation of the large Reference model , (ii) Finetuning for lim- ited number of epochs, out...

    work page

  5. [4]

    Experiments and Results Implementation Details: The model is trained on NVIDIA A40 GPU with a batch size of 64 using AdamW optimizer with L2 normalization of 1e−6 and 10000 warmup iterations followed by exponentially decreasing the learning rate till the end of training. For each audio input, mel-spectrograms (us- ing 80 mel-filterbanks) are computed for ...

    work page

  6. [5]

    We demon- strated that a reference model can be employed to train a general light-weight encoder-only model that serves as a starting point for multiple light-weight networks

    Conclusion In this work, we proposed a two-step framework to create ultra light-weight models in a resource efficient manner without sig- nificantly compromising on the final performance. We demon- strated that a reference model can be employed to train a general light-weight encoder-only model that serves as a starting point for multiple light-weight net...

    work page

  7. [6]

    Towards a person- alized clustered federated learning: A speech recognition case study,

    B. Farahani, S. Tabibian, and H. Ebrahimi, “Towards a person- alized clustered federated learning: A speech recognition case study,”IEEE Internet of Things Journal, 2023

    work page 2023

  8. [7]

    W H2D2N 2: Distributed ai-enabled ok-asn ser- vice for web of things,

    K. Liang et al., “W H2D2N 2: Distributed ai-enabled ok-asn ser- vice for web of things,” ACM Transactions on Asian and Low- Resource Language Information Processing , vol. 22, no. 5, pp. 1–16, 2023

    work page 2023

Show all 33 references
  1. [8]

    Learned token pruning for transformers,

    S. Kim et al., “Learned token pruning for transformers,” in Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York, NY , USA: Association for Computing Machinery, 2022, p. 784–794

    2022

  2. [9]

    Model compression by iterative pruning with knowledge distillation and its application to speech enhancement

    Z. Wei, L. Hao, and X. Zhang, “Model compression by iterative pruning with knowledge distillation and its application to speech enhancement.” in INTERSPEECH, 2022, pp. 941–945

    2022

  3. [10]

    Quantization aware training with absolute-cosine regularization for automatic speech recognition,

    H. D. Nguyen, A. Alexandridis, and A. Mouchtaris, “Quantization aware training with absolute-cosine regularization for automatic speech recognition,” in Interspeech 2020, 2020, pp. 3366–3370

    2020

  4. [11]

    Lightweight and efficient end-to-end speech recognition using low-rank trans- former,

    G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu et al., “Lightweight and efficient end-to-end speech recognition using low-rank trans- former,” in ICASSP, 2020, pp. 6144–6148

    2020

  5. [12]

    Bottleneck low-rank transformers for low-resource spoken language understanding,

    P. Wang and H. Van hamme, “Bottleneck low-rank transformers for low-resource spoken language understanding,” 2022

    2022

  6. [14]

    Student- teacher network learning with enhanced features,

    S. Watanabe, T. Hori, J. Le Roux, and J. R. Hershey, “Student- teacher network learning with enhanced features,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2017, pp. 5275–5279

    2017

  7. [15]

    Efficient knowledge distillation from an ensemble of teachers

    T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ram- abhadran, “Efficient knowledge distillation from an ensemble of teachers.” in Interspeech, 2017, pp. 3697–3701

    2017

  8. [16]

    Investigation of sequence-level knowledge distillation methods for ctc acoustic models,

    R. Takashima, L. Sheng, and H. Kawai, “Investigation of sequence-level knowledge distillation methods for ctc acoustic models,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 6156–6160

    2019

  9. [17]

    Knowledge distillation from offline to streaming rnn transducer for end-to-end speech recognition

    G. Kurata and G. Saon, “Knowledge distillation from offline to streaming rnn transducer for end-to-end speech recognition.” in Interspeech, 2020, pp. 2117–2121

    2020

  10. [19]

    Knowledge distillation via module re- placing for automatic speech recognition with recurrent neural network transducer,

    K. Zhao, H. D. Nguyen, A. Jain, N. Susanj, A. Mouchtaris, L. Gupta, and M. Zhao, “Knowledge distillation via module re- placing for automatic speech recognition with recurrent neural network transducer,” in 23rd Interspeech Conference, 2022

    2022

  11. [20]

    Cons-kd: Dropout-robust knowledge distillation for ctc-based automatic speech recognition,

    J. W. Yoon, H. Lee, J. Y . Kang, and N. S. Kim, “Cons-kd: Dropout-robust knowledge distillation for ctc-based automatic speech recognition,” IEEE Access, 2024

    2024

  12. [21]

    Sparsification via compressed sensing for automatic speech recognition,

    K. Zhen, H. D. Nguyen, F.-J. Chang, A. Mouchtaris, and A. Ras- trow, “Sparsification via compressed sensing for automatic speech recognition,” in ICASSP 2021-2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 6009–6013

    2021

  13. [22]

    Dynamic data pruning for automatic speech recognition,

    Q. Xiao, P. Ma, A. Fernandez-Lopez, B. Wu, L. Yin, S. Petridis, M. Pechenizkiy, M. Pantic, D. C. Mocanu, and S. Liu, “Dynamic data pruning for automatic speech recognition,” in Interspeech 2024, 2024, pp. 4488–4492

    2024

  14. [23]

    Training dynamic models using early exits for automatic speech recognition on resource- constrained devices,

    G. A. Wright, U. Cappellazzo, S. Zaiem, D. Raj, L. O. Yang, D. Falavigna, and A. Brutti, “Training dynamic models using early exits for automatic speech recognition on resource- constrained devices,”CoRR, vol. abs/2309.09546, 2023. [Online]. Available: https://doi.org/10.48550...

    2023 doi arXiv

  15. [24]

    LDASR: An experi- mental study on layer drop using conformer-based architecture,

    A. Hannan, A. Brutti, and D. Falavigna, “LDASR: An experi- mental study on layer drop using conformer-based architecture,” in Proc. of EUSIPCO, 2024

    2024

  16. [25]

    Fine-tuning strategies for faster inference using speech self-supervised models: a comparative study,

    S. Zaiem et al., “Fine-tuning strategies for faster inference using speech self-supervised models: a comparative study,” in ICAS- SPW. IEEE, 2023, pp. 1–5

    2023

  17. [26]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2022, pp. 16 000–16 009

    2022

  18. [27]

    Rethinking transformers pre-training for multi- spectral satellite imagery,

    M. Noman, M. Naseer, H. Cholakkal, R. M. Anwer, S. Khan, and F. S. Khan, “Rethinking transformers pre-training for multi- spectral satellite imagery,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , 2024, pp. 27 811–27 819

    2024

  19. [28]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https: //arxiv.org/abs/1503.02531

    2015 arXiv

  20. [29]

    Lessons from build- ing acoustic models with a million hours of speech,

    S. H. Krishnan Parthasarathi and N. Strom, “Lessons from build- ing acoustic models with a million hours of speech,” in ICASSP, 2019, pp. 6670–6674

    2019

  21. [30]

    Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,

    T. Kim, J. Oh, N. Y . Kim, S. Cho, and S.-Y . Yun, “Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI- 21, Z.-H. Zhou, Ed. International...

    2021 doi

  22. [31]

    SentencePiece: A simple and lan- guage independent subword tokenizer and detokenizer for neural text processing,

    T. Kudo and J. Richardson, “SentencePiece: A simple and lan- guage independent subword tokenizer and detokenizer for neural text processing,” in EMNLP, Brussels, Belgium, Nov. 2018, pp. 66–71

    2018

  23. [32]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech 2019, 2019, pp. 2613–2617

    2019

  24. [33]

    Librispeech: an asr corpus based on public do- main audio books,

    Panayotov et al., “Librispeech: an asr corpus based on public do- main audio books,” in ICASSP, 2015, pp. 5206–5210

    2015

  25. [34]

    TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation,

    F. Hernandez, V . Nguyen, S. Ghannay, N. Tomashenko et al. , “TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation,” in Speech and Computer: SPECOM 2018, Leipzig, Germany, 2018, pp. 198–208

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.