Pith. sign in

REVIEW 3 major objections 4 minor 33 references

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For large-scale speech-to-text models, the learning-rate warmup must rise sub-exponentially to avoid gradient explosions, yet once the model converges, the warmup shape has almost no effect on the final word error rate.

desk verdict Useful empirical comparison, but the 'sub-exponential' necessity claim is not supported by the paper's own polynomial result. read the letter →

arxiv 2505.23420 v1 pith:OYGGK2WS submitted 2025-05-29 cs.CL

Marco Gaido , Sara Papi , Luisa Bentivogli , Alessio Brutti , Mauro Cettolo , Roberto Gretter , Marco Matassoni , Mohamed Nabih
show 1 more author
Matteo Negri
This is my paper
classification cs.CL
keywords learningratewarmupspeech-to-textmodelconvergenceConformerexplodinggradientsschedulerlarge-scaletrainingautomaticspeechrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the learning-rate warmup, the opening stretch of training in which the learning rate climbs from zero to its peak, is responsible for the difficulty of getting large speech-to-text models to converge, and whether the warmup's shape changes the quality of the final model. Training an 878M-parameter Conformer model on roughly 150,000 hours of English and Italian speech, the authors compare the standard linear warmup, the two-stage linear warmup used by the OWSM models, and two new schedules, polynomial and exponential. They find that the warmup shape decides convergence: the exponential and two-stage linear schedules train stably, while the linear and polynomial schedules drive the gradient norm to spike past 100 after about 25,000 steps and the model never converges. They also find that the exponential policy's faster early progress does not carry into a better final model; on test word error rates the slower two-stage schedule ends up equal or slightly ahead. The practical conclusion is that a warmup rising sub-exponentially, staying low through the middle of the warmup window, is needed for stable large-scale speech-to-text training, and that once convergence is achieved the warmup contributes little to final quality.

What carries the argument

The load-bearing objects are the four warmup functions compared under identical conditions: the inverse-square-root scheduler's linear warmup $\eta_i=\eta\cdot\min(i/w,\sqrt{w}/\sqrt{i})$; the OWSM piecewise-linear warmup with an intermediate plateau near $\eta/10$; the paper's polynomial warmup $\eta_i=\eta(i/w)^\alpha$; and its exponential warmup $\eta_i=\eta\,(e^{\alpha i/w}-1)/(e^\alpha-1)$, with $\alpha=1.5$ for both new policies. The mechanism doing the explanatory work is the timing of the learning rate relative to an instability window: the diverging linear and polynomial schedules keep the learning rate high, or still rising, in the 15,000–50,000-step middle of the warmup, where the gradient norm explodes past 100–200, while the converging exponential and piecewise-linear schedules stay lower exactly there. The gradient norm during warmup thereby serves as the paper's diagnostic instrument: low norms (below 25) track converging runs, and spikes after step ~25,000 flag runs that will diverge.

What would settle it

Retrain the identical 878M-parameter Conformer on the same data with the polynomial warmup at a lower exponent (for example $\alpha=1.2$ or $\alpha<1$) or with tighter gradient clipping and compare final word error rates with the exponential run: if a converging polynomial schedule appears with equal final quality, then 'exponential or sub-exponential warmup required' is an artifact of the $\alpha=1.5$ operating point, and the real constraint is simply staying below a depth-dependent gradient-instability threshold. A second check is to test whether the polynomial warmup converges at 18 encoder layers but diverges at 24, which would locate the claim in encoder depth rather than warmup shape.

Watch

Extended reading notes

Core claim

The paper's central claim is that large-scale speech-to-text training demands a learning-rate warmup that follows an exponential or sub-exponential rise, and that once this stability requirement is met the warmup has only a minor influence on the final model. The evidence comes from a controlled comparison on a 878M-parameter Conformer with 24 encoder layers: with the standard linear warmup (the inverse-square-root scheduler with $w=50{,}000$ and peak $\eta=2\times10^{-4}$), models deeper than 18 encoder layers diverge, and the proposed polynomial warmup $\eta_i=\eta(i/w)^{1.5}$ diverges in the same way, with gradient-norm spikes beyond 100 around steps 25,000–30,000. Two schedules converge: the OWSM piecewise-linear warmup (a slow ramp to $\eta/10$ at step $w/2$ and then to $\eta$ at step $w$) and the paper's exponential warmup $\eta_i=\eta\,(e^{1.5\,i/w}-1)/(e^{1.5}-1)$, which offers the highest learning rate of all four policies in the first ~15,000 steps but sits below the polynomial's in the 15,000–50,000 window where the explosions occur. The exponential policy shows the fastest initial drop in perplexity, yet by the end of training the piecewise-linear model has closed the gap and posts equal or slightly better word error rates on both languages (for example 18.4 versus 19.1 on English CommonVoice). The authors conclude that the warmup's job is to steer the model through a gradient-instability window, so warmup shape should be chosen for reliable convergence with a fast but safe initial learning rate, not for final performance.

Load-bearing premise

The load-bearing premise is that the hand-set schedule constants ($\alpha=1.5$ for both new policies, $w=50{,}000$, peak $\eta=2\times10^{-4}$, $w'=25{,}000$, $\eta'=\eta/10$), the 24-encoder-layer depth, gradient clipping at 10, and a single run per policy represent large-scale speech-to-text training generally, so that another setting would not flip which warmups converge.

Editorial extensions

If this is right

  • The standard linear warmup of the inverse-square-root scheduler is unsafe for Conformer encoders beyond about 18 layers, so deeper stacks require a sub-exponential or stepped warmup.
  • A warmup that reaches a higher learning rate earlier buys faster initial perplexity drops but not better final word error rates, since the slower piecewise-linear model finishes equal or slightly ahead on every test set.
  • Gradient-norm spikes above roughly 100 in the 25,000–30,000 step range signal an imminent divergence, giving practitioners an early and cheap diagnostic for non-converging large-scale speech-to-text runs.
  • Encoder depth, not decoder depth, drives the convergence problem, because adding decoder layers did not destabilize training in the preliminary experiments.
  • Warmup hyperparameters can be tuned for stability and for speed of the safe early phase, since final model quality is largely insensitive to warmup shape once the model converges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests a single operating point ($\alpha=1.5$, $w=50{,}000$, 24 encoder layers), so a natural extension is a stability frontier: for each encoder depth, find the largest $\alpha$ (or the highest mid-warmup learning rate) that still converges, turning 'sub-exponential required' into a precise design curve.
  • The gradient-norm tripwire suggests an adaptive scheme the paper does not propose: a warmup that pauses or lowers its slope whenever the gradient norm exceeds a threshold could make large speech-to-text training robust without hand-setting $\alpha$.
  • Because the mechanism is gradient instability in deep, subcomponent-heavy encoder blocks, the same convergence ordering should transfer to other modular encoders such as Branchformer, a testable prediction, not a result the paper ran.
  • The result also cautions against reading early learning curves as predictors of final quality: the exponential policy looked best at step 20,000 yet finished slightly behind, so early-curve comparisons alone cannot rank warmup policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates how LR warmup policies affect convergence and final quality in large-scale speech-to-text training. The authors train an 878M-parameter Conformer encoder-decoder on ~150k hours of English and Italian ASR data, comparing inverse-square-root linear warmup, OWSM's piecewise-linear warmup, and two proposed alternatives: polynomial warmup η(i/w)^α and exponential warmup η(e^{αi/w}-1)/(e^α-1), both with α=1.5, warmup length 50k, and peak LR 2e-4. Preliminary depth experiments show divergence with deep Conformer encoders under the standard scheduler. In the main comparison, the exponential and piecewise-linear policies converge while the polynomial policy diverges with gradient-norm spikes. The paper concludes that large-scale S2T trainings demand a sub-exponential LR warmup, that higher warmup LR accelerates early convergence, and that warmup policy has minimal impact on final model quality once convergence is achieved. Code, logs, and intermediate checkpoints are released.

Significance. If the conclusions hold, the paper provides useful practical guidance for warmup design in large ASR/LSM training, with evidence from unusually large real-world runs. The main strength is the scale and realism of the experiments (150k hours, 878M parameters, 170k steps) and the open release of code, logs, and checkpoints, which supports reproducibility and follow-up analysis. The learning curves, gradient-norm plots, and WER table are internally consistent with the qualitative narrative. However, the significance is limited by the fact that the headline claim about 'sub-exponential' warmup is not actually supported by the tested policies, and the secondary claim about final performance rests on a single run per policy without statistical testing. The paper is a useful empirical contribution in a narrower form, but the current generalization claims exceed the evidence.

major comments (3)
  1. [Abstract, §1, §5; §2.3, §4.2] The central claim that 'large-scale S2T trainings demand a sub-exponential LR warmup' is internally inconsistent with the reported experiments. The polynomial policy η·(i/w)^1.5 in §2.3 is a sub-exponential function, yet §4.2 reports that it fails to converge under the same hyperparameters that make the exponential and piecewise-linear policies converge. Thus the experiment does not show that sub-exponentiality is what is required; it shows that this particular polynomial shape is unstable. The gradient-norm analysis in Appendix C further attributes divergence to the LR level in the 25k–50k interval, which is an artifact of the hand-set α=1.5 and η rather than a property of the entire sub-exponential family. The Limitations section explicitly concedes that tuning α could produce a converging polynomial, undercutting the necessity claim. I request a reformulation of the conclusion (e.g., 'the warmup should rise quickly early and stay below a threshold in the later warmup phase') or, better, a small grid of α values for the polynomial policy to map where convergence is lost.
  2. [§4.2, Table 1, Limitations] The secondary claim that warmup policy 'does not significantly affect the final result' is supported only by one training run per policy. In Table 1, the WER differences between piecewise-linear and exponential policies (e.g., 18.4 vs. 19.1 on CV-en and 17.8 vs. 18.3 on VGen-it) are all in the same direction, but without repeated runs, error bars, or significance tests these differences cannot be distinguished from run-to-run noise. The Limitations section acknowledges this explicitly. Since the conclusion is stated without qualification, either at least one condition should be run with multiple seeds for a statistical comparison, or the conclusion should be downgraded to 'in these single runs, the exponential warmup did not improve final WER.'
  3. [§3, §4.1, §A] The generality of the convergence finding rests on a single architecture configuration: 24 encoder layers and 12 decoder layers in a Conformer model, 878M parameters, gradient clipping at 10.0, η=2e-4, and w=50k. The preliminary experiments in §4.1 show that convergence behavior is sensitive to the distribution of encoder vs. decoder depth, and Appendix A notes that auxiliary CTC losses were tested but did not affect the convergence outcome, which is useful evidence. Still, the headline 'large-scale S2T trainings demand' generalizes from this one point in hyperparameter space. A second architecture (e.g., Branchformer) or a second depth setting would materially strengthen the claim; otherwise, the conclusion should be explicitly restricted to the tested Conformer configuration and nearby schedules.
minor comments (4)
  1. [§2.2] There is a typo in the hyperparameter specification: 'the intermediate LR w′ to η/10' should read 'the intermediate LR η′ to η/10', since w′ denotes the intermediate warmup step count.
  2. [§4.1] The sentence 'models with 18 encoder layers and 6 decoder layers diverge, while models with 12 encoder and 12 decoder layers converge' is potentially confusing because both configurations have 24 layers; clarifying that both have the same total depth would make the point about depth distribution clearer.
  3. [§4.2] The parenthetical 'in which the English data is more than 80%' is unclear; it likely refers to the proportion of English training data, but the antecedent is not stated. Please rephrase.
  4. [Figures 2 and 3] The perplexity curves would be easier to interpret if the y-axis units and any log-scale transformation were stated explicitly in the captions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are direct empirical observations from fixed LR schedules, not conclusions that reduce to their own inputs.

full rationale

The paper's derivation chain is empirical rather than formal. The LR schedules are defined by explicit formulas (Sections 2.1-2.4) with hand-set constants (w=50k, eta=2e-4, alpha=1.5), and the convergence/non-convergence results and WER values are measured from training runs, not implied by the schedule definitions. The central claims about sub-exponential warmup and final performance are observations over a grid of tried schedules. The polynomial policy, although itself sub-exponential, diverges, which weakens the paper's generalization but does not make the claim circular; it is a scope or robustness concern. Self-citations to OWSM/Peng et al. (2024) supply the baseline piecewise-linear scheduler and data provenance (FAMA), but no load-bearing uniqueness theorem or ansatz is imported to force the conclusions. The Limitations section honestly concedes that tuning alpha could yield a converging polynomial, which further indicates that the conclusion is an empirical finding under specific hyperparameters rather than a definitional tautology. No fitted parameter is renamed as a prediction, and no known result is repackaged. Accordingly, no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central comparisons rest on hyperparameters chosen by hand, on the OWSM scheduler constants, and on the architecture depth selected in preliminary runs. These are not fitted in a circular way, but they limit the generality of the empirical claims.

free parameters (8)
  • Exponential warmup exponent alpha = 1.5
    Hand-set in Section 2.4; not tuned. The convergence/failure ordering depends on this value.
  • Polynomial warmup exponent alpha = 1.5
    Hand-set in Section 2.3; the polynomial policy diverges at this value, and the authors note a tuned alpha might converge.
  • Warmup length w = 50,000 steps
    Fixed in Section 2.1 for all policies; changes how much of training is affected by warmup shape.
  • Peak learning rate eta = 2e-4
    Fixed in Section 2.1; no ablation, though peak LR interacts with warmup stability.
  • Piecewise intermediate warmup w' = 25,000 steps (w/2)
    Taken from OWSM in Section 2.2; not ablated.
  • Piecewise intermediate learning rate eta' = eta/10
    Taken from OWSM in Section 2.2; not ablated.
  • Model depth = 24 encoder, 12 decoder layers
    Selected in preliminary experiments (Section 4.1) to expose divergence; the central claim about depth-dependent convergence is conditional on this choice.
  • Gradient clipping threshold = 10.0
    Applied to all runs (Appendix A); interacts with whether warmup policies explode.
assumptions (4)
  • domain assumption Conformer and Branchformer encoders are harder to train because their residual-wrapped subcomponents cause exploding gradients at depth.
    Section 4.1 infers this from depth correlations, but no ablation removes residual connections or adds normalization.
  • domain assumption Validation perplexity on CommonVoice dev sets and WER on test sets adequately measure convergence and final model quality.
    Section 4.2 and Table 1 rely on these metrics without external or human evaluation.
  • domain assumption One training epoch (170k steps) is enough to assess final model performance.
    All comparisons use the 170k-step checkpoint; no multi-epoch or convergence-based stopping is used.
  • domain assumption Gradient norm spikes identify the cause of divergence, not just a correlate.
    Appendix C reports spikes for non-converging runs, but the causal link to warmup shape is inferred.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence." pith.science (2026). https://pith.science/paper/OYGGK2WS

@misc{pith2026250523420,
  author       = {Pith},
  title        = {Pith review of: The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYGGK2WS}},
  note         = {Machine review of arXiv:2505.23420}
}
read the original abstract

Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

Figures

Figures reproduced from arXiv: 2505.23420 by the authors.

Figure 1
Figure 1. LR schedulers with inverse square root, piecewise-linear, polynomial, and exponential warmup policies. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Perplexity on the English and Italian valida [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Perplexity on the English and Italian valida [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Perplexity on the training set for the polyno [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Perplexity on the training set for the piecewise [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Gradient norm comparison across the piecewise-linear, polynomial, and exponential warmup policies. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages

  1. [1]

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. https://aclanthology.org/2020.lrec-1.520 Common voice: A massively-multilingual speech corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222, Marsei...

    2020

  2. [2]

    Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. https://doi.org/10.1109/ASRU46091.2019.9003774 A comparative study on end-to-end speech to text translation . In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 792--799

    arXiv 2019

  3. [3]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. 1994. https://doi.org/10.1109/72.279181 Learning long-term dependencies with gradient descent is difficult . IEEE Transactions on Neural Networks, 5(2):157--166

    doi:10.1109/72.279181 1994

  4. [4]

    Seamless Communication et al. 2023. https://arxiv.org/abs/2308.11596 SeamlessM4T: Massively Multilingual & Multimodal Machine Translation . Preprint, arXiv:2308.11596

    arXiv 2023

  5. [5]

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. https://doi.org/10.1109/SLT54892.2023.10023141 Fleurs: Few-shot learning evaluation of universal representations of speech . In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798--805

    arXiv 2023

  6. [6]

    Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, and Matteo Negri. 2024. MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, United Sta...

    work page 2024

  7. [7]

    Gomez, and J \"u rgen Schmidhuber

    Alex Graves, Santiago Fern \'a ndez, Faustino J. Gomez, and J \"u rgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks . In Proceedings of the 23rd international conference on Machine learning (ICML), pages 369--376, Pittsburgh, Pennsylvania

    work page 2006

  8. [8]

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. https://doi.org/10.21437/Interspeech.2020-3015 Conformer: Convolution-augmented transformer for speech recognition . In Interspeech 2020, pages 5036--5040

    doi:10.21437/interspeech.2020-3015 2020

Show all 33 references
  1. [9]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. https://doi.org/10.1109/CVPR.2016.90 Deep residual learning for image recognition . In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778

    2016 doi

  2. [10]

    Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. https://www.aclweb.org/anthology/2020.acl-demos.34 ESP net- ST : All-in-one speech translation toolkit . In Proceedings of the 58th Annual Meeting of the Associat...

    2020

  3. [11]

    Kahn , M

    J. Kahn , M. Rivière , W. Zheng , E. Kharitonov , Q. Xu , P. E. Mazaré , J. Karadayi , V. Liptchinsky , R. Collobert , C. Fuegen , T. Likhomanenko , G. Synnaeve , A. Joulin , A. Mohamed , and E. Dupoux . 2020. Libri-Light: A Benchmark for ASR with Limited or No Supervision . I...

    2020

  4. [12]

    Dayal Singh Kalra and Maissam Barkeshli. 2024. https://openreview.net/forum?id=NVl4SAmz5c Why warmup the learning rate? underlying mechanisms and improvements . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [13]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  6. [14]

    Sara Papi, Marco Gaido, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, and Matteo Negri. 2025. FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian

    2025

  7. [15]

    Sara Papi, Marco Gaido, Andrea Pilzer, and Matteo Negri. 2024. https://doi.org/10.18653/v1/2024.acl-long.200 When good and reproducible results are a giant with feet of clay: The importance of software quality in NLP . In Proceedings of the 62nd Annual Meeting of the Associati...

    2024 doi

  8. [16]

    Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. 2022. https://proceedings.mlr.press/v162/peng22a.html Branchformer: Parallel MLP -attention architectures to capture local and global context for speech recognition and understanding . In Proceedings of the 39th Inte...

    2022

  9. [17]

    Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee weon Jung, and Shinji Watanabe. 2024. https://doi.org/10.21437/Interspeech.2024-1194 Owsm v3.1: Better and faster open whisper-style s...

    2024 doi

  10. [18]

    Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee-Weon Jung, Soumi Maiti, and Shinji Watanabe. 2023. https://doi.org/10.1109/ASRU57964.2023.103...

    2023

  11. [19]

    PleIAs. 2024. P le I A s/ Y ou T ube- C ommons · D atasets at H ugging F ace --- huggingface.co. https://huggingface.co/datasets/PleIAs/YouTube-Commons. [Accessed 10-06-2024]

    2024

  12. [20]

    Martin Popel and Ondřej Bojar. 2018. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110:43--70

    2018

  13. [21]

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. https://doi.org/10.21437/Interspeech.2020-2826 MLS: A Large-Scale Multilingual Dataset for Speech Research . In Proc. Interspeech 2020, pages 2757--2761

    2020 doi

  14. [22]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. https://proceedings.mlr.press/v202/radford23a.html Robust speech recognition via large-scale weak supervision . In Proceedings of the 40th International Conference on Machine Lear...

    2023

  15. [23]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners

    2019

  16. [24]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826

    2016

  17. [25]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Neural Informa...

    2017

  18. [26]

    Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021 a . https://doi.org/10.18653/v1/2021.acl-long.80 V ox P opuli: A large-scale multilingual speech corpus for representation learning, semi-...

    2021 doi

  19. [27]

    Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020. https://doi.org/10.18653/v1/2020.aacl-demo.6 Fairseq S 2 T : Fast speech-to-text modeling with fairseq . In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Com...

    2020 doi

  20. [28]

    Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021 b . https://doi.org/10.21437/Interspeech.2021-2027 CoVoST 2 and Massively Multilingual Speech Translation . In Proc. Interspeech 2021, pages 2247--2251

    2021 doi

  21. [29]

    Matt White, Ibrahim Haddad, Cailean Osborne, Xiao-Yang Liu Yanglet, Ahmed Abdelmonsef, and Sachin Varghese. 2024. https://arxiv.org/abs/2403.13784 The model openness framework: Promoting completeness and openness for reproducibility, transparency, and usability in artificial i...

    2024 arXiv

  22. [30]

    Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, and Shinji Watanabe. 2023. https://doi.org/10.18653/v1/2023.eacl-main.119 CTC alignments improve autoregressive translation . In Proceedings of the 17th Conference of the European Chapter ...

    2023 doi

  23. [31]

    Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Rama...

    2023 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.