Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints

A F M Saif; Brian Kingsbury; Tianyi Chen; Xiaodong Cui

REVIEW 2 major objections 4 minor 32 references

Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Heterogeneous self-supervised acoustic pre-training should not just minimize the averaged loss over all mixed data; this paper claims that adding per-source local constraints—requiring each data source to reach its own local optimum after…

desk verdict PTLOC is FOMAML/Reptile-style meta-learning applied to self-supervised speech pretraining; the WER gains are plausible but confounded by extra training epochs and warm-starting, so the central causal claim is not yet isolated. read the letter →

arxiv 2508.19990 v2 pith:FRADWVQT submitted 2025-08-27 cs.LG cs.CL

Xiaodong Cui , A F M Saif , Brian Kingsbury , Tianyi Chen This is my paper · ORCID

classification cs.LGcs.CL

keywords self-supervisedlearningspeechrecognitionbileveloptimizationlocalconstraintsheterogeneousdatapre-trainingmodel-agnosticmetaworderrorrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-supervised speech pre-training usually mixes data from many domains or languages and minimizes one averaged loss, which can leave individual sources poorly represented. This paper proposes PTLOC, a pre-training scheme that adds local constraints: after $K$ steps of gradient descent initialized from the shared model, each data source should be near its own local loss optimum. The authors formulate this as a bilevel optimization problem, solve it with a first-order approximation, and show experimentally in multi-domain and multilingual ASR that the resulting pre-trained model fine-tunes better. Relative word-error-rate gains over the plain averaged-loss baseline are 15–40% in multi-domain tasks and 2.9–9.6% in multilingual tasks after three rounds of alternating CSSL and PTLOC initialization. The practical claim is that preserving per-source optima during pre-training gives a better initialization for downstream supervised fine-tuning.

What carries the argument

The load-bearing object is the bilevel objective in Eq. (2): an upper-level averaged global loss over all $M$ data sources and $M$ lower-level problems, each defined by a $K$-step gradient descent from the shared initial parameters $ heta$. The first-order approximation in Eq. (10)—setting each Hessian to zero—reduces the upper-level gradient to the average of the local gradients evaluated after the $K$ local updates, making the method computationally practical. The paper justifies the approximation by observing that in sufficiently trained deep models most Hessian eigenvalues are near zero, and the special case $K=1$ makes PTLOC closely resemble MAML, though here the same training data are used in both levels and no labels are involved.

What would settle it

Run a compute-matched control in which plain CSSL is trained for the same total number of pre-training epochs as the iterative PTLOC schedule (multi-domain: 80/60/80/60/80/60 epochs; multilingual: 100/80/100/80/100/80), with each round initialized from the previous round's model exactly as PTLOC is, and then fine-tune on the same downstream tasks. If the control reproduces the 15–40% and 2.9–9.6% relative WER gains, the local constraints are not the cause.

Watch

Extended reading notes

Core claim

The central discovery is that constraining a shared speech model to reach a local optimum for each heterogeneous data source after $K$-step gradient descent—rather than only minimizing the average loss over all mixed data—improves the model's adaptivity to downstream supervised fine-tuning. The paper frames this as bilevel optimization: the upper level minimizes the averaged global loss, and the lower level requires each source's $K$-step gradient descent, started from the shared parameters, to land near that source's local optimum. Using a first-order approximation that drops the Hessian terms, the gradient update becomes the average of the local-source gradients after the $K$ local steps, which the authors connect to model-agnostic meta-learning. In both multi-domain (BEST-RQ-based) and multilingual (CPC-based) experiments, iterating between CSSL and PTLOC with mutual initialization gives consistent WER reductions on downstream ASR tasks relative to the initial CSSL baseline.

Load-bearing premise

The load-bearing premise is that the WER improvements are caused by the local constraints rather than by PTLOC receiving extra pre-training epochs and warm-start initialization: the paper always initializes PTLOC from a CSSL-trained model and trains it for an additional 60–80 epochs (Sections 4 and 5.1, Tables 2–3), and no compute-matched CSSL baseline is reported.

Editorial extensions

If this is right

If PTLOC is correct, pre-trained speech models trained on mixed domains or languages should be pre-trained with per-source local constraints instead of only the averaged loss, yielding consistently lower word error rates after downstream fine-tuning (Tables 3 and 5).
A single local update ($K=1$) is enough to capture most of the benefit in the multi-domain experiments, so the method does not require long inner loops, though it does add extra outer training rounds.
Because CSSL itself can be improved by initializing from a PTLOC model, alternating the two objectives is a usable recipe for squeezing more adaptivity out of the same unlabeled data.
The bilevel view specializes to first-order MAML at $K=1$, so any future improvement in efficient first-order bilevel solvers should transfer directly to speech self-supervised pre-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would apply PTLOC to heterogeneous non-speech self-supervised learning, such as multimodal or multi-dataset image and audio pre-training, where the same averaged-loss failure mode is expected to appear.
An implication the paper leaves implicit is that the method's advantage should depend on the sharpness of the loss landscape at initialization; the zero-Hessian approximation is more accurate in flatter regions, so gains should concentrate in later iterative rounds, which the reported tables do show.
The iterative mutual initialization between CSSL and PTLOC can be viewed as alternating minimization between two objectives; if so, comparing PTLOC against a compute-matched single CSSL run of equal total epochs would separate the effect of the local constraints from the effect of more training.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a short conference-style paper that applies first-order MAML (equivalently Reptile) updates to self-supervised acoustic pretraining on heterogeneous data sources. The algorithmic novelty is thin, and the authors say so themselves by discussing the MAML connection. The real question is empirical: does adding per-source local constraints improve downstream ASR fine-tuning? The answer they give is believable but not yet clean.

What the paper does well: the formulation is clear and honest. The bilevel derivation is standard and correctly simplified with the first-order Hessian approximation, and they justify the approximation with an external observation about Hessian spectra in deep networks. They also disclose that PTLOC is always initialized from a CSSL-trained model. The experiments cover two realistic setups (multi-domain BEST-RQ English and multilingual CPC on CoVoST), and the multilingual gains, while small, are consistent across four languages. The iterative mutual-initialization schedule is an interesting practical trick.

The soft spot is load-bearing. PTLOC is warmed up with a full CSSL run and then trained for 60-80 additional epochs. The CSSL baseline gets only 80 or 100 epochs total. The iterative protocol compounds this: the final PTLOC.3 row has received three CSSL runs plus three PTLOC runs, so roughly 380-420 epochs versus 80-100 for the baseline. The 15-40% multi-domain improvements could come from longer training, warm-start effects, the iterative round-robin itself, or the local-constraint update. Without a compute-matched CSSL continuation (CSSL trained for the same total epochs) or a CSSL-from-PTLOC control, the causal attribution to PTLOC is not established. The math is not the problem; the experimental isolation is. There are also no error bars, no seeds, and no code or data release, so the reader cannot tell how robust the single-run numbers are.

A minor point: K=3 does not beat K=1, so the multi-step part of the algorithm that distinguishes it from plain FOMAML/Reptile shows no benefit in the reported results. That is worth flagging but not a fatal flaw.

Who this is for: researchers working on multi-domain or multilingual self-supervised ASR pretraining, and anyone interested in how meta-learning updates transfer to SSL. It deserves a serious referee because the question is real and the confound is fixable with additional experiments. If the authors can show that a longer-trained CSSL baseline does not match PTLOC, the paper becomes much stronger. I would currently not cite it as evidence for local constraints improving adaptivity, but I would cite it as an application of FOMAML to SSL pretraining.

Recommendation: send to peer review with a request for compute-matched baselines, multiple seeds, and ideally code or data. The central claim needs that control before the results can be trusted.

Referee Report

2 major / 4 minor

Summary. This paper proposes PTLOC, a self-supervised pre-training method that adds local constraints for each data source, formulated as a bilevel optimization problem where the upper level minimizes the averaged global loss and the lower level solves each data source's loss via K-step gradient descent. The authors derive a first-order approximation to the gradient, present pseudocode, and situate the method relative to model-agnostic meta-learning. Experiments on multi-domain (BEST-RQ) and multilingual (CPC) datasets report WER improvements over a conventional self-supervised baseline, with claimed relative improvements of 15%–40% and 2.9%–9.6% after iterative mutual initialization of CSSL and PTLOC.

Significance. The paper addresses a practically relevant problem: standard mixing of heterogeneous data in self-supervised acoustic pre-training may not preserve source-specific feature representations. The proposed formulation is a principled extension of bilevel optimization to this setting, and the connection to first-order MAML is clearly disclosed. The derivation is transparent and the algorithm is simple to implement. If the empirical claims are supported, the method could offer a practical improvement for multi-domain and multilingual pre-training. However, the experimental evidence is currently weakened by an unequal-compute comparison that conflates the effect of the local-constraint objective with additional training and warm-start initialization. The paper also provides clear pseudocode and a reproducible experimental description for its settings, which is a strength.

major comments (2)

[§5.1, §5.2, Tables 2, 3, 5] The comparisons in Tables 2, 3, and 5 do not match compute between PTLOC and the baseline. In Section 5.1, the CSSL baseline is trained for 80 epochs, while each PTLOC run is initialized from a CSSL model and trained for an additional 60 epochs; in the iterative protocol, PTLOC.3 therefore receives 3×80 + 3×60 = 420 pre-training epochs, versus 80 for the CSSL.1 baseline row. In Section 5.2, CSSL trains for 100 epochs and PTLOC adds 80, so PTLOC.3 receives 540 epochs versus 100 for CSSL.1. As a result, the reported 15–40% and 2.9–9.6% relative WER improvements cannot be attributed to the local-constraint update alone; they could reflect longer training, the warm-start schedule, or the iterative cycling. A compute-matched control—such as a CSSL model trained for the same total number of epochs with a comparable learning-rate schedule, or a CSSL continuation of the same duration—is required to support the paper's central claim that PTLOC improves adaptivity.
[§5.1, §5.2, Tables 3 and 5] Even with matched total compute, the iterative mutual-initialization protocol itself may benefit both CSSL and PTLOC through repeated warm-starts and re-initializations. The paper does not include a control in which CSSL is iteratively initialized from CSSL (e.g., CSSL.1 → CSSL.2 → CSSL.3) with the same number of epochs per round. Without such a control, the statement in Section 5.1 that 'PTLOC always outperforms its initial CSSL model' may be explained simply by PTLOC being the later model in the sequence and therefore receiving additional training and warm-start improvements. A CSSL-only iterative chain is needed to isolate the contribution of the local-constraint objective itself.

minor comments (4)

[§3, Eq. (4)] The definition of g_i(\phi_i) in Eq. (4) uses \ell_i(\phi_i(\theta); x, D_i) on the right-hand side, but the left-hand side is a function of \phi_i only; this is inconsistent with the subsequent gradient computation in Eq. (5), which treats g_i as a function of \phi_i. It should presumably read \ell_i(\phi_i; x, D_i).
[§5.1, Table 3 caption] The caption of Table 3 refers to 'PTEC' in 'CSSL and PTEC alternately initialize each other'; this appears to be a typo for 'PTLOC'.
[§5.1] The sentence 'Among these eight domains, data from the first five domains is used for pre-training, while that from the last five is used for downstream FT and test' is ambiguous because the last five domains overlap with the first five (GB English and Librispeech appear in both). The following sentence clarifies the overlap, but the wording should be adjusted to avoid confusion.
[§5.1, §5.2] The paper reports single WER numbers per model without standard errors or significance tests; adding multiple runs or variance estimates would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PTLOC's bilevel objective and first-order solver are derived from stated definitions, and the reported WER improvements are measured rather than constructed.

full rationale

The paper's claimed derivation chain is self-contained. PTLOC is defined by the bilevel objective in Eq. (2), whose upper level is the averaged global self-supervised loss and whose lower level is per-source optimization via K-step gradient descent. Section 3 performs standard gradient unrolling, and Eq. (10) is obtained by explicitly setting the Hessian to zero in Eq. (11), an approximation justified by an external empirical observation [14]. The K=1 case in Eq. (12) is explicitly acknowledged to resemble MAML, so no hidden importation of the result occurs. No parameter is fitted to the downstream WER numbers and then reported as a prediction: the learning rates and epoch counts are fixed in advance, and the WERs are measured after supervised fine-tuning. The only self-citations, refs. [7] and [16], are background references for bilevel optimization and the MAML bilevel interpretation; they are not load-bearing for the central claim. The paper also discloses in Section 4 that PTLOC is always initialized from a CSSL-trained model, which introduces an empirical confound: PTLOC receives additional pre-training epochs beyond the CSSL baseline, so the causal attribution of the WER gains to the local constraints is not fully controlled. That is an experimental-design limitation, not a circularity in the derivation. No step reduces an output to an input by construction, and no uniqueness theorem or self-citation chain is invoked to force the result.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim (empirical WER improvement) rests mainly on the experimental design rather than exotic axioms. The optimization derivation uses a zero-Hessian approximation and finite-batch K-step 'inner' solves, and the alternating scheme has no convergence proof. There are no invented entities or fitted physical constants; the only free hyperparameter is K.

free parameters (1)

K (number of local gradient steps) = 1 (K=3 also evaluated, with similar or worse WER)
Hand-chosen hyperparameter; the paper selects K=1 for remaining experiments based on computational cost and comparable performance, but the choice is not fitted to data.

assumptions (4)

domain assumption Second-order Hessian terms are zero (Eq. 11)
The first-order approximation in Section 3.2 sets all Hessians to zero, justified in Section 4 by the empirical observation from [14] that many Hessian eigenvalues vanish near deep-model optima. This is not a theorem and changes the algorithm if violated.
domain assumption K-step gradient descent on a single batch approximates the lower-level argmin over the full source (Eq. 5, Algorithm 1)
The lower problem in Eq. (2) is defined on the entire data source D_i, but it is solved by K updates on one randomly sampled batch; the finite-batch approximation is never analyzed.
domain assumption Alternating lower and upper optimization converges to a useful solution (Algorithm 1)
The algorithm alternates solving all lower problems then one upper step; no convergence guarantee for this nonconvex bilevel heuristic is provided.
domain assumption Reaching a local optimum after K steps for each source preserves per-source feature characteristics (Section 2)
The motivation for the local constraints assumes that local adaptability implies good per-source representations; this is a modeling belief, not a derived statement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints." pith.science (2026). https://pith.science/paper/FRADWVQT

@misc{pith2026250819990,
  author       = {Pith},
  title        = {Pith review of: Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRADWVQT}},
  note         = {Machine review of arXiv:2508.19990}
}

abstract

Self-supervised pre-training using unlabeled data is widely used in automatic speech recognition. In this paper, we propose a new self-supervised pre-training approach to dealing with heterogeneous data. Instead of mixing all the data and minimizing the averaged global loss in the conventional way, we impose additional local constraints to ensure that the model optimizes each source of heterogeneous data to its local optimum after $K$-step gradient descent initialized from the model. We formulate this as a bilevel optimization problem, and use the first-order approximation method to solve the problem. We discuss its connection to model-agnostic meta learning. Experiments are carried out on self-supervised pre-training using multi-domain and multilingual datasets, demonstrating that the proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages

[1]

In speech related applications, self-supervised pre-training has been actively investigated and broadly used [1–4]

INTRODUCTION Since labeled data is expensive to obtain while unlabeled data is readily available, a common practice in machine learning is a two- stage approach where a large amount of unlabeled data is first used for self-supervised pre-training and the pre-trained foundation model is then fine-tuned using labeled data in downstream tasks. In speech rela...

work page
[2]

PROBLEM FORMULA TION Suppose the unlabeled data D is collected from M sources: D={D 1,···,D M}. Conventional self-supervised learning (CSSL) trains a model that minimizes the following empirical risk min θ 1 M MX i=1 X x∈Di ℓi(θ;x,D i)(1) wherex∈D i is a sample from sourceDi;θ is the model param- eters;ℓi(θ;x,D i) is the loss defined on data source Di. In...

work page Pith review arXiv 2025
[3]

OPTIMIZA TION To solve the bilevel optimization problem in Eq. (2), we first define the following functions to simplify the derivation f(θ)≜ 1 M MX i=1 X x∈Di ℓi(ϕ∗ i (θ);x,D i)(3) gi(ϕi)≜ X x∈Di ℓi(ϕi(θ);x,D i), i∈[M](4) wheref(·) andgi(·) denote the upper level and lower level problems in Eq. 2 respectively. 3.1. Solving the lower level problem There ar...

work page
[4]

We then expand eachϕi k with its gradient descent update in Eq

=I which is due toϕi 0 =θ since the gradient descent starts from the shared parameterθ. We then expand eachϕi k with its gradient descent update in Eq. (5) ∇θf(θ)≈ 1 M MX i=1 ∇ϕigi(ϕi K ) KY k=1 ∇ϕi k−1 [ϕi k−1−α∇ϕigi(ϕi k−1)] = 1 M MX i=1 ∇ϕigi(ϕi K ) KY k=1 [I−α∇ 2 ϕigi(ϕi k−1)] ≈ 1 M MX i=1 ∇ϕigi(ϕi K )(10) where in the last step we apply the first-ord...

work page
[5]

3.1 and 3.2, the pseudo-code imple- mentation of PTLOC is given in Algorithm 1

IMPLEMENTA TION Based on the derivation in Secs. 3.1 and 3.2, the pseudo-code imple- mentation of PTLOC is given in Algorithm 1. The PTLOC training is carried out inT iterations, where the problems of the lower and upper levels are alternately optimized. Specifically, in each iteration, theM lower-level problems on each data source are first (approximatel...

work page
[6]

One is a self-supervised English acoustic model pre-training using speech data from multiple domains based on BEST-RQ [4]

EXPERIMENTS We evaluate the proposed PTLOC on two sets of experiments. One is a self-supervised English acoustic model pre-training using speech data from multiple domains based on BEST-RQ [4]. The other is a self-supervised multilingual acoustic model pre-training based on contrastive predictive coding (CPC) [21]. 5.1. Multi-Domain Pre-Training In this e...

work page
[7]

Local constraints are imposed to ensure that the models optimize each data source to its local optimum after K- step gradient descent initialized from the model

SUMMARY In this paper, we propose PTLOC to deal with data heterogeneity in self-supervised pre-training. Local constraints are imposed to ensure that the models optimize each data source to its local optimum after K- step gradient descent initialized from the model. We use the first-order approximation to solve the resulting bilevel optimization problem. ...

work page
[8]

vq- wav2vec: Self-supervised learning of discrete speech represen- tations,

Alexei Baevski, Steffen Schneider, and Michael Auli, “vq- wav2vec: Self-supervised learning of discrete speech represen- tations,”arXiv preprint arXiv:1910.05453, 2019

arXiv 1910

Show all 32 references

[9]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,”Advances in neural infor- mation processing systems, vol. 33, pp. 12449–12460, 2020

2020
[10]

Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vo...

2021
[11]

Self-supervised learning with random-projection quantizer for speech recog- nition,

C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recog- nition,” inInternational Conference on Machine Learning (ICML), 2022, pp. 3915–3924

2022
[12]

Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond,

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin, “Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond,”IEEE Trans- actions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10045–10067, 2021

2021
[13]

Bilevel methods for image reconstruction,

Caroline Crockett and Jeffrey A Fessler, “Bilevel methods for image reconstruction,”Foundations and Trends® in Signal Processing, vol. 15, no. 2-3, pp. 121–289, 2022

2022
[14]

Learning with lim- ited samples: Meta-learning and applications to communication systems,

Lisha Chen, Sharu Theresa Jose, Ivana Nikoloska, Sangwoo Park, Tianyi Chen, and Osvaldo Simeone, “Learning with lim- ited samples: Meta-learning and applications to communication systems,”Foundations and Trends® in Signal Processing, vol. 17, no. 2, pp. 79–208, 2023

2023
[15]

Meta-DAG: Meta causal discovery via bilevel optimization,

Songtao Lu and Tian Gao, “Meta-DAG: Meta causal discovery via bilevel optimization,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

2023
[16]

Bilevel programming for hy- perparameter optimization and meta-learning,

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil, “Bilevel programming for hy- perparameter optimization and meta-learning,” inInternational Conference on Machine Learning, 2018, pp. 1568–1577

2018
[17]

Model- agnostic meta-learning for fast adaptation of deep networks,

Chelsea Finn, Pieter Abbeel, and Sergey Levine, “Model- agnostic meta-learning for fast adaptation of deep networks,” inInternational Conference on Machine Learning, 2017, pp. 1126–1135

2017
[18]

Truncated back-propagation for bilevel optimization,

A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots, “Truncated back-propagation for bilevel optimization,” inInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2019

2019
[19]

A generic first- order algorithmic framework for bi-level programming beyond lower-level singleton,

R. Liu, P. Mu, X. Yuan, S. Zeng, and J. Zhang, “A generic first- order algorithmic framework for bi-level programming beyond lower-level singleton,” inInternational Conference on Machine Learning (ICML), 2020

2020
[20]

Memory-efficient gradient unrolling for large-scale bi-level optimization,

Q. Shen, Y . Wang, Z. Yang, X. Li, H. Wang, Y . Zhang, J. Scar- lett, Z. Zhu, and K. Kawaguchi, “Memory-efficient gradient unrolling for large-scale bi-level optimization,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

2024
[21]

Eigenvalues of the Hessian in deep learning: singularity and beyond,

L. Sagun, L. Bottou, and Y . LeCun, “Eigenvalues of the Hessian in deep learning: singularity and beyond,”arXiv:1611.07476, 2016

2016 arXiv
[22]

Sign-MAML: efficient model- agnostic meta-learning by SignSGD,

C. Fan, P. Ram, and S. Liu, “Sign-MAML: efficient model- agnostic meta-learning by SignSGD,” in5th NeurIPS Workshop on Meta-Learning, 2021

2021
[23]

Sharp- MAML: sharpness-aware model-agnostic meta learning,

M. Abbas, Q. Xiao, L. Chen, P.-Y . Chen, and T. Chen, “Sharp- MAML: sharpness-aware model-agnostic meta learning,” in Proceedings of the International Conference on Machine Learn- ing (ICML), 2022

2022
[24]

Meta learning for end- to-end low-resource speech recognition,

J.-Y . Hsu, Y .-J. Chen, and H. y. Lee, “Meta learning for end- to-end low-resource speech recognition,” inInternational Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7844–7848

2020
[25]

Mul- tilingual meta-transfer learning for low-resource speech recog- nition,

R. Zhou, T. Koshikawa, A. Ito, T. Nose, and C.-P. Chen, “Mul- tilingual meta-transfer learning for low-resource speech recog- nition,”IEEE Access, pp. 158493–158504, 2024

2024
[26]

Meta-learning for Indian languages: performance analysis and improvements with linguistic similarity measures,

C. S. Anoop and A. G. Ramakrishnan, “Meta-learning for Indian languages: performance analysis and improvements with linguistic similarity measures,”IEEE Access, vol. 11, pp. 82050– 820645, 2023

2023
[27]

Model-agnostic meta-learning for fast text-dependent speaker embedding adaptation,

W. Lin and M.-W. Mak, “Model-agnostic meta-learning for fast text-dependent speaker embedding adaptation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1866–1876, 2023

2023
[28]

Representa- tion learning with contrastive predictive coding,

Aaron van den Oord, Yazhe Li, and Oriol Vinyals, “Representa- tion learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[29]

Conformer: Convolution-augmented transformer for speech recognition,

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech, 2020, pp. 5036–5040

2020
[30]

Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProceedings of the 35th International Conference on Machine Learning (ICML), 2006, pp. 369–376

2006
[31]

Covost 2 and mas- sively multilingual speech-to-text translation,

Changhan Wang, Anne Wu, and Juan Pino, “Covost 2 and mas- sively multilingual speech-to-text translation,”arXiv preprint arXiv:2007.10310, 2020

2007 arXiv
[32]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,

Taku Kudo and John Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”arXiv preprint arXiv:1808.06226, 2018

2018 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

In speech related applications, self-supervised pre-training has been actively investigated and broadly used [1–4]

INTRODUCTION Since labeled data is expensive to obtain while unlabeled data is readily available, a common practice in machine learning is a two- stage approach where a large amount of unlabeled data is first used for self-supervised pre-training and the pre-trained foundation model is then fine-tuned using labeled data in downstream tasks. In speech rela...

work page

[2] [2]

PROBLEM FORMULA TION Suppose the unlabeled data D is collected from M sources: D={D 1,···,D M}. Conventional self-supervised learning (CSSL) trains a model that minimizes the following empirical risk min θ 1 M MX i=1 X x∈Di ℓi(θ;x,D i)(1) wherex∈D i is a sample from sourceDi;θ is the model param- eters;ℓi(θ;x,D i) is the loss defined on data source Di. In...

work page Pith review arXiv 2025

[3] [3]

OPTIMIZA TION To solve the bilevel optimization problem in Eq. (2), we first define the following functions to simplify the derivation f(θ)≜ 1 M MX i=1 X x∈Di ℓi(ϕ∗ i (θ);x,D i)(3) gi(ϕi)≜ X x∈Di ℓi(ϕi(θ);x,D i), i∈[M](4) wheref(·) andgi(·) denote the upper level and lower level problems in Eq. 2 respectively. 3.1. Solving the lower level problem There ar...

work page

[4] [4]

We then expand eachϕi k with its gradient descent update in Eq

=I which is due toϕi 0 =θ since the gradient descent starts from the shared parameterθ. We then expand eachϕi k with its gradient descent update in Eq. (5) ∇θf(θ)≈ 1 M MX i=1 ∇ϕigi(ϕi K ) KY k=1 ∇ϕi k−1 [ϕi k−1−α∇ϕigi(ϕi k−1)] = 1 M MX i=1 ∇ϕigi(ϕi K ) KY k=1 [I−α∇ 2 ϕigi(ϕi k−1)] ≈ 1 M MX i=1 ∇ϕigi(ϕi K )(10) where in the last step we apply the first-ord...

work page

[5] [5]

3.1 and 3.2, the pseudo-code imple- mentation of PTLOC is given in Algorithm 1

IMPLEMENTA TION Based on the derivation in Secs. 3.1 and 3.2, the pseudo-code imple- mentation of PTLOC is given in Algorithm 1. The PTLOC training is carried out inT iterations, where the problems of the lower and upper levels are alternately optimized. Specifically, in each iteration, theM lower-level problems on each data source are first (approximatel...

work page

[6] [6]

One is a self-supervised English acoustic model pre-training using speech data from multiple domains based on BEST-RQ [4]

EXPERIMENTS We evaluate the proposed PTLOC on two sets of experiments. One is a self-supervised English acoustic model pre-training using speech data from multiple domains based on BEST-RQ [4]. The other is a self-supervised multilingual acoustic model pre-training based on contrastive predictive coding (CPC) [21]. 5.1. Multi-Domain Pre-Training In this e...

work page

[7] [7]

Local constraints are imposed to ensure that the models optimize each data source to its local optimum after K- step gradient descent initialized from the model

SUMMARY In this paper, we propose PTLOC to deal with data heterogeneity in self-supervised pre-training. Local constraints are imposed to ensure that the models optimize each data source to its local optimum after K- step gradient descent initialized from the model. We use the first-order approximation to solve the resulting bilevel optimization problem. ...

work page

[8] [8]

vq- wav2vec: Self-supervised learning of discrete speech represen- tations,

Alexei Baevski, Steffen Schneider, and Michael Auli, “vq- wav2vec: Self-supervised learning of discrete speech represen- tations,”arXiv preprint arXiv:1910.05453, 2019

arXiv 1910