Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition

A F M Saif; Brian Kingsbury; George Saon; Lisha Chen; Songtao Lu; Tianyi Chen; Xiaodong Cui

REVIEW 4 major objections 5 minor 46 references

Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that joint bilevel optimization of supervised and unsupervised speech losses, called BL-JUST, consistently outperforms the standard pre-training plus fine-tuning recipe.

desk verdict BL-JUST is a plausible, broadened extension of the authors' earlier bilevel ASR work, but the LibriSpeech headline gain is the best of four penalty schedules selected on the test set, so read the magnitude as optimistic. read the letter →

arxiv 2412.08548 v1 pith:P34HE3PI submitted 2024-12-11 cs.CL

Xiaodong Cui , A F M Saif , Songtao Lu , Lisha Chen , Tianyi Chen , Brian Kingsbury , George Saon This is my paper · ORCID

classification cs.CL

keywords bileveloptimizationjointunsupervisedandsupervisedtrainingautomaticspeechrecognitionpenalty-basedgradientdescentsemi-supervisedlearningpre-trainingfine-tuningself-supervisedrepresentationConformeracousticmodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that automatic speech recognition models are better trained by solving the supervised and unsupervised objectives together in one bilevel optimization than by the standard two-stage recipe of unsupervised pre-training followed by supervised fine-tuning. The proposed method, BL-JUST, treats the supervised loss on labeled speech as an upper-level problem whose solution is required to stay near a local optimum of the unsupervised loss on unlabeled speech, and it solves that pair with penalty-based bilevel gradient descent. Across LibriSpeech, Switchboard, and an industrial payload dataset, BL-JUST reports lower word error rates than PT+FT in every configuration tested, including 4.1% versus 5.1% on LibriSpeech test-clean with 100 labeled and 860 unlabeled hours. The stakes are practical: labeled speech is expensive, unlabeled speech is abundant, and if joint training indeed finds matched local optima of both losses, it offers a more efficient use of that unlabeled data for building task-ready acoustic models.

What carries the argument

The carrying object is the penalty reformulation $F_\gamma(\theta,\phi,\eta) = f(\theta,\phi) + \gamma\,(g(\theta,\eta) - v)$, where $v = \min_{\theta,\eta} g(\theta,\eta)$ is the optimal value of the unsupervised loss and the parenthesis is the value-function gap. The algorithm maintains a shared backbone $\theta$, a supervised head $\phi$, and an unsupervised head $\eta$; each epoch it first runs self-supervised exploration to approach a local optimum of the unsupervised loss, then applies the penalty-based bilevel gradient descent update with the penalty coefficient $\gamma$ increasing linearly from 0 to a preset maximum, then ends with a small-learning-rate supervised fine-tuning. Lemma 1 is the theoretical link: for $\gamma$ large enough, every local or global solution of the penalty problem is an $\epsilon$-approximate solution of the bilevel problem, so the procedure is meant to converge to a matched local optimum of both losses.

What would settle it

A concrete check: train the same Conformer models with BL-JUST and PT+FT on LibriSpeech with 100 labeled and 860 unlabeled hours, measure the final gradient norm of the CPC loss, and also run BL-JUST with a constant penalty factor after the unsupervised gap has narrowed. If the final CPC gradient norm is not substantially smaller under BL-JUST than under PT+FT, or if the constant-penalty version matches the scheduled-penalty version in WER, the matched-optimum story would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that the disconnection in the conventional PT+FT pipeline carries a real cost: pre-training ignores the labeled task and fine-tuning ignores the unlabeled data, so the final model is not required to be good for both objectives and can land in a supervised local optimum with a high unsupervised loss. BL-JUST avoids this by solving $\min_{\theta,\phi} f(\theta,\phi)$ subject to $\theta \in \arg\min_{\theta',\eta} g(\theta',\eta)$, where $f$ is the supervised loss (CTC or RNN-Transducer) and $g$ is the unsupervised loss (CPC or BEST-RQ). The paper reports that this joint formulation outperforms PT+FT in every experiment and also beats constant-penalty joint training (JUST), pseudo-labeling, and alternating optimization. It takes the simultaneously faster drop of both loss curves and the smaller final unsupervised-gradient norm under BL-JUST as evidence that the training reaches matched local optima of the two losses, yielding representations that remain useful for unlabeled speech while being adapted to the ASR task.

Load-bearing premise

The theoretical guarantee that the penalty method exactly solves the bilevel problem requires the unsupervised loss to satisfy a strong-gradient condition (the Polyak-Łojasiewicz inequality, which says the gradient does not vanish while the loss is still far from its minimum) and requires the supervised loss and the penalty gradient to be smooth; the paper never verifies these conditions for CPC, BEST-RQ, CTC, or RNNT losses on Conformer networks, so if they fail the matched-optimum interpretation rests on measured gradient norms rather than on the convergence proof.

Editorial extensions

If this is right

If the paper is right, the widely used PT+FT recipe leaves accuracy on the table: coupling supervised and unsupervised losses with a growing penalty reduces WER at the same data and model sizes.
The reported gains are not tied to one self-supervised loss; they appear with CPC plus CTC on LibriSpeech and with BEST-RQ plus RNNT or CTC on Switchboard and Payload, which points to the bilevel coupling itself as the active ingredient.
The ablation results imply that the self-supervised exploration step is the most important component of the implementation, and that the final fine-tuning step still adds a further improvement; removing both raises test-clean WER from 4.1% to 5.9%.
Scaling within the method is a direct continuation of the paper's results: a 15-block Conformer trained with BL-JUST reaches 8.1% average WER on Switchboard, better than the 8.2% of the 74M-parameter version.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's speech experiments, the same value-function-gap penalty is modality-agnostic, so the bilevel coupling is a natural candidate for any field with a large unlabeled pool and a small labeled pool, such as computer vision or language modeling; this is an editorial inference, not a claim of the paper.
The paper tests only a linear penalty schedule, but its theory only requires $\gamma$ to be large enough; an adaptive schedule that grows $\gamma$ in response to the measured unsupervised gap could, in principle, reach the same optimum with less tuning, though the paper does not test this.
If the matched-optimum interpretation is correct, BL-JUST models should transfer better to out-of-domain speech than PT+FT models at equal primary-test WER, because the unsupervised constraint keeps the shared representations from drifting too far from the unlabeled distribution; this is a testable prediction the paper does not report.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The quick take: the paper's claim that a bilevel formulation (supervised loss as leader, unsupervised loss as follower) beats disconnected pretrain+finetune is supported by a wide set of experiments, but the LibriSpeech number quoted in the abstract is the best of four penalty schedules chosen directly on test WER. The spread across schedules is as large as the claimed advantage, so the 4.1% vs 5.1% margin is not robust as reported.

What's actually new: the method generalizes JUST and extends the authors' ICASSP 2024 paper with genuinely new evidence—Switchboard with RNNT, Payload with 90k hours, BEST-RQ, larger conformers, and direct comparisons to pseudo-labeling and alternating optimization. The ablations (removing self-supervised exploration, removing final fine-tuning) are clean and show both pieces matter. The loss curves in Fig. 3 provide a nice supporting detail. The empirical pattern across three datasets and two loss families is consistent, which is more than most short ASR papers offer.

Where it's soft. The stress-test concern lands. Table V sweeps four penalty schedules and reports the best one for LibriSpeech 100h/860h; the other three give 4.7–6.7 on test-clean, putting two of them at or below PT+FT's 5.1. No held-out split, no multiple seeds, no error bars. On top of that, all methods use a learning-rate scheduler that monitors test loss, so test-set selection is baked into the whole pipeline. The theory in Section III cites [22] for the penalty equivalence; the PL and Lipschitz assumptions are stated but never verified for CPC, BEST-RQ, CTC, or RNNT on conformers. That is a real limitation, though the empirical claim does not depend on the theory being verified—only on the algorithm working. The self-citation to [22] and [25] is not a problem here: the cited work is a separate theory paper, and the comparisons are against external baselines.

Bottom line: this deserves a serious referee. The method is coherent, the experimental breadth is useful, and the qualitative conclusion—joint bilevel training can use unlabeled data more effectively than disconnected PT+FT—is credible. What needs fixing before the numbers are trusted: a proper held-out validation split for hyperparameters, at least three seeds or a significance statement, and a toned-down "consistently outperforms" claim. I'd send it to review, and I'd want the revised version to address uncertainty quantification before acceptance.

My recommendation: take it, but require the validation work.

Referee Report

4 major / 5 minor

Summary. The paper proposes BL-JUST, a bilevel training framework for ASR in which supervised training on labeled data is the upper-level problem and unsupervised training on unlabeled data is the lower-level problem, with the two coupled through a penalty-based bilevel gradient descent method borrowed from Shen and Chen [22]. The authors evaluate BL-JUST on LibriSpeech, Switchboard, and an internal payload dataset, using CPC and BEST-RQ as unsupervised losses and CTC and RNNT as supervised losses, and report that BL-JUST consistently outperforms PT+FT as well as JUST, pseudo-labeling, and alternating optimization. The headline results are 4.1% vs. 5.1% WER on LibriSpeech test-clean under a 100h labeled / 860h unlabeled split, and 8.2% vs. 9.6% on the Switchboard average. The paper also includes an ablation study and an analysis of loss and gradient norms to support the 'matched local optima' interpretation.

Significance. If the empirical claim is robust, the result is practically significant: it suggests that, for semi-supervised ASR, jointly optimizing supervised and unsupervised objectives within a bilevel formulation can beat the standard disconnected pretraining-then-finetuning recipe. The breadth of the evaluation—three datasets, two unsupervised losses, two supervised losses, and multiple architectures—is a genuine strength, as is the inclusion of ablations and comparisons with several semi-supervised baselines. However, the current evidence is weakened by test-set-based hyperparameter selection, the absence of variance or significance information, and theoretical assumptions that are not verified for the actual losses and models. The central idea is plausible and worth publishing if the empirical protocol is tightened, but the present manuscript does not yet support the strength of the stated claims.

major comments (4)

[§V-C, Table V and Appendix A] The penalty schedule for LibriSpeech is selected on the test sets. Table V reports four increasing-rate schedules with test-clean WERs of 6.7, 4.1, 4.7, and 5.2, so the spread across configurations (2.6 percentage points) is larger than the claimed BL-JUST advantage over PT+FT on test-clean (1.0 percentage point). Furthermore, Appendix A states that the learning-rate scheduler 'monitor[s] the test loss every 10 epochs' for the baseline, PT+FT fine-tuning, and BL-JUST training. This is test-set contamination for the entire LibriSpeech comparison, not just for the penalty value. The manuscript needs a held-out validation split for schedule selection and learning-rate scheduling, plus multiple seeds with error bars or significance tests, before the 'consistently outperforms' claim is supported by the LibriSpeech experiments.
[§III, Lemma 1 and Theorem 1] The theoretical equivalence and convergence guarantee rely on Assumptions (a)-(c): L-Lipschitz continuity of f, the Polyak-Łojasiewicz inequality for g, and Lipschitz continuity of ∇Fγ. These assumptions are never verified for CPC, BEST-RQ, CTC, or RNNT losses on Conformer networks. Since the lower-level problems in this paper are trained with SGD rather than solved exactly, and since the PL condition is a strong global condition, the penalty-based reformulation is not guaranteed to approximate the bilevel problem in this setting. The paper should either verify the assumptions empirically (for example, by estimating the PL constant along the training trajectory) or explicitly recast the theory as a heuristic motivation and place the burden on the empirical comparisons.
[§IV, Algorithm 1] The final supervised fine-tuning step (the N3 loop) is outside the penalty-based bilevel formulation analyzed in Section III. After this step, the model is no longer guaranteed to satisfy the lower-level constraint, so Lemma 1 and Theorem 1 do not apply to the final model whose WER is reported. The ablation study shows that removing fine-tuning degrades WER, but it also means the reported system is a hybrid of the penalty method and a final PT+FT-like stage. Please clarify the role of this step relative to the theoretical claims, or report the performance before the final fine-tuning as the BL-JUST solution and the fine-tuned version as a post-processing variant.
[§V, Tables I-III] No variance, confidence intervals, or significance tests are reported for any table. All comparisons appear to be single runs, and the claimed gains on LibriSpeech (1.0-1.9 absolute WER) and Switchboard (1.4 absolute on average) are moderate in size. Given the test-set selection issue in Table V, the lack of uncertainty quantification makes it impossible to determine whether the headline gains are robust. At minimum, the authors should report multiple random seeds for the key comparisons and, ideally, a small hyperparameter sensitivity analysis on a validation set.

minor comments (5)

[§V-D] The text says 'BL-JUST also reaches a better CTC loss than PF+FT'; 'PF+FT' should be 'PT+FT'.
[Appendices B and C] 'The masked frames are replaced with Gaussian noise with with 0 mean and 0.1 variance' contains a duplicated 'with' in both appendices.
[Table V] The row label 'simi' appears to be a placeholder and should be replaced with the intended method description.
[Algorithm 1] The penalty update γk = (k - 1)γm/K reaches (K-1)/K γm at k=K rather than γm; either the formula or the description 'increases to a pre-defined maximum value' should be corrected. Also, the values of N1, N2, and N3 are not specified in the appendices, although they are part of the algorithm's input.
[References] Reference [2] and [27] contain the typo 'Maganize' for 'Magazine'.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity: the WER claims are empirical and the PBGD theory is independently cited; the main caveats are minor self-citation and test-set-based hyperparameter selection, which affect evaluation strength rather than derivation equivalence.

full rationale

The paper's derivation chain is not circular at the construction level. The upper-level supervised loss f and lower-level unsupervised loss g are independently defined from labeled and unlabeled data (Eqs. 1-4), and the penalty reformulation F_gamma = f + gamma(g - v) (Eq. 9) is a standard reformulation of the constraint theta in argmin g; Lemma 1 and Theorem 1 are quoted from [22] with explicit assumptions (Lipschitz continuity, Polyak-Lojasiewicz) that are not shown to hold for CPC, BEST-RQ, CTC, or RNNT losses, but failing to verify assumptions is a theoretical-support gap, not a circular reduction. The empirical headline results (4.1% vs 5.1% on LibriSpeech test-clean; 8.2% vs 9.6% on Switchboard average) are measured against independent PT+FT and semi-supervised baselines on public or externally defined datasets, so they do not reduce to the inputs by definition. The self-citations to [22] and [25] involve overlapping authors, but [22] is a separate theory paper whose assumptions do not include the ASR performance claim, and [25] is only described as the authors' prior work; neither forbids alternatives nor forces the conclusion. The penalty schedule for gamma is selected on the test sets (Table V reports the best of four schedules, with 0.002/0.20 giving 4.1/11.3, while the other schedules give 4.7-6.7 on test-clean), and Appendix A's learning-rate scheduler monitors test loss; this is a real evaluation-validity concern and weakens the strength of the 'consistently outperforms' claim, but it is test-set selection bias, not an equation-level equivalence or a fitted parameter renamed as a prediction. Overall, no circular step meets the evidentiary bar, and the score of 2 reflects only the minor, non-load-bearing self-citation and the evaluation caveats, not an actual circularity finding.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or formal entities; it reframes known losses in a bilevel optimization.

free parameters (3)

Penalty schedule (increase rate, final gamma value) = LibriSpeech: rate 0.002, max 0.2; Switchboard: rate 0.007, max 0.2; Payload: max 0.1
The schedule is a hand-tuned hyperparameter. In Table V the best rate and max value are selected using test WER, so this is a fitted value that directly affects the reported gains.
Inner-loop iteration counts N1, N2, N3 = not reported
Algorithm 1 uses N1 unsupervised exploration steps and N2 joint steps per epoch, and N3 final fine-tuning steps, but the appendices do not give values. These control how closely the lower-level optimum is approximated.
Learning rates alpha, beta, tau and fine-tuning length = e.g., alpha=5e-3, beta=5e-4, final FT lr=5e-5 for 20 epochs on LibriSpeech
Per-dataset training hyperparameters chosen by hand; standard but not model-derived.

assumptions (3)

ad hoc to paper Assumptions (a)-(c) of Section III: f is L-Lipschitz in phi; g obeys the Polyak-Lojasiewicz inequality; gradient of F_gamma is L_gamma-Lipschitz.
Invoked before Lemma 1 to justify the penalty reformulation; not checked for the actual losses and networks.
domain assumption Large overparameterized networks have many local minima of similar quality, so a point approximately optimal for both losses exists.
Stated in the introduction with reference [15] to motivate the existence of matched local optima.
domain assumption The shared-backbone architecture with separate heads (Fig. 2) is sufficient for ASR and allows independent optimization of theta, eta, phi.
The method's design and update rules assume this decomposition; it is conventional but not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition." pith.science (2026). https://pith.science/paper/P34HE3PI

@misc{pith2026241208548,
  author       = {Pith},
  title        = {Pith review of: Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P34HE3PI}},
  note         = {Machine review of arXiv:2412.08548}
}

read the original abstract

In this paper, we propose a bilevel joint unsupervised and supervised training (BL-JUST) framework for automatic speech recognition. Compared to the conventional pre-training and fine-tuning strategy which is a disconnected two-stage process, BL-JUST tries to optimize an acoustic model such that it simultaneously minimizes both the unsupervised and supervised loss functions. Because BL-JUST seeks matched local optima of both loss functions, acoustic representations learned by the acoustic model strike a good balance between being generic and task-specific. We solve the BL-JUST problem using penalty-based bilevel gradient descent and evaluate the trained deep neural network acoustic models on various datasets with a variety of architectures and loss functions. We show that BL-JUST can outperform the widely-used pre-training and fine-tuning strategy and some other popular semi-supervised techniques.

Figures

Figures reproduced from arXiv: 2412.08548 by the authors.

**Figure 1.** An illustration of the two-stage pre-training followed by finetuning (PT+FT) in the upper panel and bilevel joint unsupervised and supervised training (BL-JUST) in the lower panel. In this paper, we propose bilevel joint unsupervised and supervised training (BL-JUST) to overcome this limitation of the two-stage PT+FT. The motivation of BL-JUST is to train a model that simultaneously minimizes both the unsupervised … view at source ↗

**Figure 2.** The network architecture for bilevel joint unsupervised [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The unsupervised CPC loss (upper panel) and super [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 33 canonical work pages

[22]

On penalty-based bilevel gradient descent method,

H. Shen and T. Chen, “On penalty-based bilevel gradient descent method,” arXiv preprint arXiv:2302.05185 , 2023

arXiv 2023
[25]

Joint unsupervised and supervised training for automatic speech recognition via bilevel optimization,

A. Saif, X. Cui, H. Shen, S. Lu, B. Kingsbury, and T. Chen, “Joint unsupervised and supervised training for automatic speech recognition via bilevel optimization,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10 931–10 935

work page 2024
[1]

Deep learning,

Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” Nature, pp. 436– 444, May 2015

work page 2015
[2]

Deep neural networks for acoustic modeling in speech recognition,

G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,” IEEE Signal Processing Maganize, pp. 82–97, November 2012

work page 2012
[3]

Toward human parity in conversational speech recognition,

W. Xiong, J. Droppo, X. Huang, F. Seide, M. L. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Toward human parity in conversational speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2410–2423, 2017

work page 2017
[4]

English conversational telephone speech recognition by humans and machines,

G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in Interspeech, 2017, pp. 132–136

work page 2017
[5]

Lessons from building acoustic models with a million hours of speech,

S. H. K. Parthasarathi and N. Str ¨om, “Lessons from building acoustic models with a million hours of speech,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 6670– 6674

work page 2019
[6]

Re- alizing petabyte scale acoustic modeling,

S. H. K. Parthasarathi, N. Sivakrishnan, P. Ladkat, and N. Strom, “Re- alizing petabyte scale acoustic modeling,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 9, no. 2, pp. 422–432, 2019

work page 2019

Show all 46 references

[7]

New types of deep neural network learning for speech recognition and related applications: An overview,

L. Deng, G. Hinton, and B. Kingsbury, “New types of deep neural network learning for speech recognition and related applications: An overview,” in IEEE international conference on acoustics, speech and signal processing, 2013, pp. 8599–8603

2013
[8]

vq-wav2vec: Self- supervised learning of discrete speech representations,

A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self- supervised learning of discrete speech representations,” arXiv preprint arXiv:1910.05453, 2019

1910 arXiv
[9]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449– 12 460, 2020

2020
[10]

Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

2021
[11]

Self-supervised learning with random-projection quantizer for speech recognition,

C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in International Conference on Machine Learning (ICML), 2022, pp. 3915– 3924

2022
[12]

To transfer or not to transfer,

M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich, “To transfer or not to transfer,” in NIPS workshop on transfer learning, no. 3, 2005

2005
[13]

A survey on transfer learning,

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans- actions on knowledge and data engineering , vol. 22, no. 10, pp. 1345– 1359, 2009

2009
[14]

Characterizing and avoiding negative transfer,

Z. Wang, Z. Dai, B. P ´oczos, and J. Carbonell, “Characterizing and avoiding negative transfer,” in Proceedings of Conference on Computer Vision and Pattern Recognition , 2019, pp. 11 293–11 302

2019
[15]

The loss surfaces of multilayer networks,

A. Choromanska, M. Henaff, M. Mathieu, G. Ben Arous, and Y . LeCun, “The loss surfaces of multilayer networks,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), 2015, pp. 192–204

2015
[16]

Investigating bi- level optimization for learning and vision from a unified perspective: A survey and beyond,

R. Liu, J. Gao, J. Zhang, D. Meng, and Z. Lin, “Investigating bi- level optimization for learning and vision from a unified perspective: A survey and beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10 045–10 067, 2021

2021
[17]

Bilevel methods for image reconstruc- tion,

C. Crockett and J. A. Fessler, “Bilevel methods for image reconstruc- tion,” Foundations and Trends® in Signal Processing , vol. 15, no. 2-3, pp. 121–289, 2022

2022
[18]

Learning with limited samples: Meta-learning and applications to com- munication systems,

L. Chen, S. T. Jose, I. Nikoloska, S. Park, T. Chen, and O. Simeone, “Learning with limited samples: Meta-learning and applications to com- munication systems,” Foundations and Trends® in Signal Processing , vol. 17, no. 2, pp. 79–208, 2023

2023
[19]

Meta-DAG: Meta causal discovery via bilevel optimization,

S. Lu and T. Gao, “Meta-DAG: Meta causal discovery via bilevel optimization,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023

2023
[20]

Bilevel programming for hyperparameter optimization and meta-learning,

L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil, “Bilevel programming for hyperparameter optimization and meta-learning,” in International Conference on Machine Learning , 2018, pp. 1568–1577

2018
[21]

Model-agnostic meta-learning for fast adaptation of deep networks,

C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International Conference on Machine Learning, 2017, pp. 1126–1135

2017
[23]

Joint unsupervised and supervised training for multilingual ASR,

J. Bai, B. Li, Y . Zhang, A. Bapna, N. Siddhartha, K. C. Sim, and T. N. Sainath, “Joint unsupervised and supervised training for multilingual ASR,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022, pp. 6402–6406

2022
[24]

Iterative pseudo-labeling for speech recognition,

Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, “Iterative pseudo-labeling for speech recognition,” in Interspeech, 2020

2020
[26]

Towards principled unsupervised learning,

I. Sutskever, R. Jozefowicz, K. Gregor, D. Rezende, T. Lillicrap, and O. Vinyals, “Towards principled unsupervised learning,” arXiv preprint arXiv:1511.06440, 2015

2015 arXiv
[27]

An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning,

Y . Zhang, P. Khanduri, I. Tsaknakis, Y . Yao, M. Hong, and S. Liu, “An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning,” IEEE Signal Processing Maganize, pp. 38–59, April 2024

2024
[28]

First-order penalty methods for bilevel optimization,

Z. Lu and S. Mei, “First-order penalty methods for bilevel optimization,” SIAM Journal on Optimization , vol. 34, no. 2, pp. 1937–1969, 2024

1937
[29]

On penalty methods for nonconvex bilevel optimization and first-order stochastic approxima- tion,

J. Kwon, D. Kwon, S. Wright, and R. D. Nowak, “On penalty methods for nonconvex bilevel optimization and first-order stochastic approxima- tion,” in International Conference on Learning Representations (ICLR) , 2024

2024
[30]

Optimization methods for large- scale machine learning,

L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large- scale machine learning,” SIAM review, vol. 60, no. 2, pp. 223–311, 2018

2018
[31]

Google USM: scaling automatic speech recognition beyond 100 languages,

Y . Z. et. al., “Google USM: scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037 , 2023

2023 arXiv
[32]

Librispeech: an asr corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015

2015
[33]

Conformer: Convolution- augmented transformer for speech recognition,

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution- augmented transformer for speech recognition,” in Interspeech, 2020, pp. 5036–5040

2020
[34]

Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,

T. Kudo and J. Richardson, “Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226 , 2018

2018 arXiv
[35]

SpecAugment: A simple data augmentation method for automatic speech recognition,

D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Interspeech, 2019, pp. 2613–2617

2019
[36]

Representation learning with contrastive predictive coding,

A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

2018 arXiv
[37]

Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning , 2006, pp. 369–376

2006
[38]

Sequence transduction with recurrent neural networks,

A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711 , 2012

2012 arXiv
[39]

Speech recognition with deep recurrent neural networks,

A. Graves and A.-r. Mohamed and G. Hinton, “Speech recognition with deep recurrent neural networks,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2013, pp. 6645– 6649

2013
[40]

Advancing RNN transducer technology for speech recognition,

G. Saon, Z. Tueske, D. Bolanos, and B. Kingsbury, “Advancing RNN transducer technology for speech recognition,” in International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP) , 2021

2021
[41]

Improving RNN transducer modeling for end-to-end speech recognition,

J. Li, R. Zhao, H. Hu, and Y . Gong, “Improving RNN transducer modeling for end-to-end speech recognition,” in Automatic Speech Recognition and Understanding Workshop (ASRU) , 2019

2019
[42]

Sequence noise injected training for end-to-end speech recognition,

G. Saon, Z. Tuske, K. Audhkhasi, and B. Kingsbury, “Sequence noise injected training for end-to-end speech recognition,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 6261–6265

2019
[43]

Regularization of neural networks using DropConnect,

L. Wan, M. Zeiler, S. Zhang, Y . LeCun, and R. Fergus, “Regularization of neural networks using DropConnect,” in Proceedings of the 35th 10 International Conference on Machine Learning (ICML), 2013, pp. 1058– 1066

2013
[44]

Alignment-length synchronous decoding for RNN transducer,

G. Saon, Z. Tuske, and K. Audhkhasi, “Alignment-length synchronous decoding for RNN transducer,” in International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , 2020, pp. 7804–7808

2020
[45]

ADAM: a method for stochastic optimiza- tion,

D. P. Kingma and J. L. Ba, “ADAM: a method for stochastic optimiza- tion,” in International Conference on Learning Representations (ICLR) , 2015

2015
[46]

Super-convergence: very fast training of neural networks using large learning rates,

L. N. Smith and N. Topin, “Super-convergence: very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications , 2019. APPENDIX A TRAINING RECIPES FOR LIBRI SPEECH EXPERIMENTS Baseline I...

2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [22]

On penalty-based bilevel gradient descent method,

H. Shen and T. Chen, “On penalty-based bilevel gradient descent method,” arXiv preprint arXiv:2302.05185 , 2023

arXiv 2023

[2] [25]

Joint unsupervised and supervised training for automatic speech recognition via bilevel optimization,

A. Saif, X. Cui, H. Shen, S. Lu, B. Kingsbury, and T. Chen, “Joint unsupervised and supervised training for automatic speech recognition via bilevel optimization,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10 931–10 935

work page 2024

[3] [1]

Deep learning,

Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” Nature, pp. 436– 444, May 2015

work page 2015

[4] [2]

Deep neural networks for acoustic modeling in speech recognition,

G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,” IEEE Signal Processing Maganize, pp. 82–97, November 2012

work page 2012

[5] [3]

Toward human parity in conversational speech recognition,

W. Xiong, J. Droppo, X. Huang, F. Seide, M. L. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Toward human parity in conversational speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2410–2423, 2017

work page 2017

[6] [4]

English conversational telephone speech recognition by humans and machines,

G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in Interspeech, 2017, pp. 132–136

work page 2017

[7] [5]

Lessons from building acoustic models with a million hours of speech,

S. H. K. Parthasarathi and N. Str ¨om, “Lessons from building acoustic models with a million hours of speech,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 6670– 6674

work page 2019

[8] [6]

Re- alizing petabyte scale acoustic modeling,

S. H. K. Parthasarathi, N. Sivakrishnan, P. Ladkat, and N. Strom, “Re- alizing petabyte scale acoustic modeling,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 9, no. 2, pp. 422–432, 2019

work page 2019