Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On a single-speaker rtMRI corpus, audio-only phoneme recognition (PER 0.21) beats the multimodal model (0.26), so the paper concludes that adding articulatory video does not help.

desk verdict Solid corpus paper with a promising interpretability direction, but the headline audio-vs-multimodal comparison is not supported as reported because the models were selected on the test set. read the letter →

arxiv 2505.24059 v1 pith:SLKFH2D6 submitted 2025-05-29 cs.LG

Sean Foley , Hong Nguyen , Jihwan Lee , Sudarsana Reddy Kadiri , Dani Byrd , Louis Goldstein , Shrikanth Narayanan This is my paper
classification cs.LG
keywords multimodalphonemerecognitionreal-timeMRIvocaltractarticulationattentionweightssingle-speakercorpusConformerWavLMphoneticclasses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether watching the moving vocal tract on real-time MRI video helps a phoneme recognizer do its job, once audio is already available. Trained on about 38 minutes of single-speaker audio-video speech, the audio-only model achieves a phoneme error rate of 0.21, the multimodal model 0.26, and video-only 0.49, so the paper concludes that adding articulatory video does not improve phoneme recognition. The accuracy story is not the whole story: attention-weight comparisons show the multimodal model shifts its focus toward articulatory constriction intervals, where tongue gestures begin before their acoustic consequences. That timing evidence bears on the longstanding question of whether speech targets are acoustic, articulatory, or both.

What carries the argument

The central object is the Conformer's self-attention weights at the last layer, extracted, averaged over heads, and z-scored, then compared between the audio-only and multimodal models at phoneme-level time intervals. This is the instrument the paper uses to see articulatory versus acoustic timing: a shift of attention toward constriction formation is read as evidence about when phonologically relevant information is available in each modality. The rest of the machinery is the fusion design (features concatenated along the temporal dimension, so the multimodal input has dimension $2 \times 768$) and the use of pretrained feature extractors, WavLM layer 9 for audio and a ViT CLS token fine-tuned on MRI video, with CTC as the training loss.

What would settle it

Train a second-stream control model on the same data with the video features replaced by a second acoustic feature stream (for example, a different WavLM layer or a time-shifted copy of the same features), keeping everything else identical. If that control also shows the same attention shifts and a similar PER increase to 0.26, then the shift is a fusion or capacity artifact rather than evidence about articulatory timing; if it does not, the causal reading of the attention weights is supported.

Watch

Extended reading notes

Core claim

Using a long-form, single-speaker real-time MRI corpus, the authors train unimodal and multimodal Conformer models for phoneme recognition with CTC loss, with audio features taken from WavLM and video features from a ViT. On the held-out test set, the audio model reaches PER 0.21 and the multimodal model 0.26, while video alone reaches 0.49; the paper states plainly that adding the video features does not improve performance. The models perform similarly across manner classes but diverge by place of articulation, with the multimodal model worst on velars and best on coronals. The paper's positive finding is interpretive: the latent spaces of audio and multimodal models encode phonetic similarity in much the same way, but attention weights differ markedly for liquids and vowels, and for a phrase like "a roll" the multimodal model attends to moments when tongue constrictions form, before the acoustics show them, whereas audio attention tracks the formant structure.

Load-bearing premise

The attention-timing conclusion assumes that the observed differences in attention weights between the audio-only and multimodal models are caused by the articulatory content of the video, rather than by the two models differing in input dimension, fusion strategy, and independent training runs.

Editorial extensions

If this is right

  • Adding articulatory video to audio does not help, and slightly hurts, phoneme error rate on this single-speaker corpus, so for pure phoneme recognition the acoustic stream appears sufficient.
  • The value of the articulatory stream is interpretative rather than accuracy-driving: it reveals that phoneme-relevant articulatory information is present before its acoustic onset, and that models can learn to exploit it.
  • Manner classes such as liquids and vowels, which are recognized well by both models, are exactly where the two models' attention differs most, suggesting the video stream is not simply adding redundant information.
  • Place-of-articulation performance diverges across modalities, implying that some articulatory distinctions have stronger visual signatures (velars) and others stronger acoustic signatures (coronals).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the fusion is simple temporal concatenation and the models are trained separately, a fair test would vary fusion mechanism (e.g., cross-attention or feature gating); the paper's 'no gain' result should not yet be read as a general statement about all multimodal fusion strategies.
  • Editorial extension: the attention shift toward constriction formation suggests articulatory information might be more valuable for tasks other than phoneme identity, such as detecting coarticulation, prosodic boundaries, or articulatory targets themselves; these are testable with the same corpus.
  • Editorial extension: a control with a second audio stream would determine whether the PER gap and attention differences stem from doubling the input dimensionality rather than from the modality content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper trains unimodal audio, unimodal video, and multimodal (audio+video) Conformer/CTC models for phoneme recognition on a new single-speaker rtMRI corpus. It reports that the audio-only model achieves a PER of 0.21, the video-only model 0.49, and the multimodal model 0.26, concluding that adding articulatory video does not improve over audio. It further interprets per-class PERs, latent-space t-SNE projections, and attention-weight differences, arguing that the multimodal model shifts attention to articulatory constriction intervals.

Significance. The study addresses a worthwhile question: whether articulatory rtMRI video provides information beyond acoustics in phoneme recognition, using a relatively rare long-form single-speaker corpus with manually corrected alignments. The interpretability analyses (latent-space comparisons, attention-weight contrasts) are a useful addition to the multimodal speech literature. The paper ships no code or data release, but the empirical protocol is mostly transparent. If the main comparison is re-run with a valid model-selection procedure, the findings could be a valuable contribution. At present, however, the headline claim is not supported by the reported experiment.

major comments (3)
  1. [4.1 / 3.3] The central PER comparison is invalidated by the model-selection protocol. Section 4.1 states that the reported results are from 'the best performing model on the test set, based on the CTC loss,' and Section 3.3 describes a grid search over learning rate, batch size, and weight decay. Because the same test set is used both for selection and for reporting, the PERs in Table 1 are minima over hyperparameter settings for that particular test partition, not unbiased estimates of generalization. Audio and multimodal models differ in input dimensionality, fusion, and optimization trajectories, so their degrees of test-set overfitting can differ; the observed 0.21 vs 0.26 gap may be a selection artifact. The authors should either (a) use a proper validation split for model selection and report test-set results for the selected configuration, or (b) report the full distribution of PERs across grid points, and adjust all claims in the Discussion (e.g., 'adding the video features does not improve performance') accordingly.
  2. [4.1 / Figure 2] The paper provides no repeated runs or confidence intervals for the overall PERs in Table 1, and the per-class bootstrap intervals are conditional on a model already selected on the test set. The claim in Section 4.1 that the audio model's performance on affricates is 'not significantly better than the multimodal model' is not backed by any significance test or distribution. Without a measure of variability (e.g., across training seeds, or a proper bootstrap over test chunks for a model fixed before seeing the test set), the 5-point PER difference between audio and multimodal models cannot be distinguished from noise, and the wording 'significantly' should be justified or removed.
  3. [4.3 / Figure 5] The attention-timing conclusion is based on a single phrase ('a roll') and lacks a control condition. The audio-only and multimodal models differ in input dimension (768 vs 2*768), fusion strategy, and training trajectories, so the observed differences in attention weights need not be caused by the articulatory modality. To support the claim that 'the multimodal model attention is highly localized to intervals containing crucial constriction information,' the authors should (a) quantify attention differences over many utterances rather than one example, and (b) include a control in which the second input stream is a non-articulatory or duplicated audio stream; otherwise the interpretation is confounded by architecture and optimization.
minor comments (5)
  1. [2.1] The text says audio and video features are 'concatenated along the temporal dimension' but defines the multimodal input as M in R^{t x 2D}, which is feature-dimension concatenation. Please correct the wording or the shape.
  2. [4.2] The t-SNE projections are stochastic, so statements about centroids being 'in nearly the same position' should be supported by multiple runs or by a quantitative distance measure in a stable embedding.
  3. [Table 1] The K- rows are from a different corpus and a different task setup; please label them explicitly as not directly comparable to the current results.
  4. [2.2 / Figure 4] The bootstrap description is incomplete: specify the resampling unit (utterances, chunks, or phoneme tokens) and whether the intervals in Figure 4 are for the absolute differences or for something else.
  5. [3.2] There is a typo in 'V oice Activity Detection' and an inconsistent acronym spelling 'V AD'; also, the symbol substitutions for phonemes in Figure 5 should be replaced with standard IPA or ARPABET notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the experimental comparison is self-contained and externally grounded.

full rationale

Reviewing the paper for circular reasoning, I find no step in which a result is derived from a quantity defined in terms of that result, nor any prediction that is statistically forced by a fitted parameter renamed as an outcome, nor a load-bearing uniqueness claim imported from the authors' prior work. The central comparison (audio PER 0.21 vs multimodal PER 0.26, Table 1) is an empirical evaluation of independently trained models on a new single-speaker corpus. Pretrained feature extractors (WavLM layer 9, ViT CLS token) are external to the task and are not tuned to the phoneme-recognition objective. Self-citations appear for the USC 75-Speaker dataset [12] and for the attention-weight extraction protocol [16], but these are methodological inputs or comparison baselines, not conclusions that force the paper's headline result. The attention-timing interpretation in Section 4.3 is an interpretation of model weights, not a derivation; the claim that the multimodal model attends to constriction intervals is supported by visual comparison of attention to MRI frames, not by formal equivalence. A noteworthy statistical concern is the model-selection procedure in Section 4.1, where results are reported from the best model on the test set based on CTC loss; this undermines the strength of the performance comparison but is a correctness/validity issue, not a circularity, because the reported numbers are not defined as the thing they are used to conclude. Therefore the circularity score is 0.

Assumptions & free parameters 8 free parameters · 7 assumptions · 0 invented entities

The central claims rest on the quality of the pretrained feature extractors, the accuracy of phoneme alignments, and the interpretive assumption that attention differences isolate modality effects. These are domain assumptions rather than derivational axioms; no mathematically free constants are fit to the target result, though grid-selected hyperparameters affect the reported PERs.

free parameters (8)
  • learning rate = 1e-3
    Selected by grid search over [1e-3, 5e-4, 1e-4]; affects model convergence and final PER comparison.
  • batch size = 8
    Grid-searched over [8, 16, 32]; chosen 8.
  • weight decay = not reported
    Grid searched over [1e-3, 5e-4, 1e-4], but the final selected value is not stated in the text.
  • number of Conformer layers = 3
    Chosen by hand; affects model capacity and the representations analyzed.
  • attention heads = 4
    Chosen by hand; attention weights are averaged across these heads.
  • kernel size = 31
    Chosen by hand for the Conformer convolution module.
  • dropout = 0.3
    Chosen by hand; affects regularization and training dynamics.
  • LSTM latent size = 128
    Chosen by hand; the latent size feeds the final linear layer.
assumptions (7)
  • domain assumption WavLM layer 9 features encode rich phonetic information.
    Section 3.2 cites [29] for this; the audio model's performance depends on it.
  • domain assumption ViT CLS token fine-tuned on USC 75-Speaker MRI video transfers to the new 0.55T single-speaker corpus.
    Section 3.2; no fine-tuning on the target corpus is reported.
  • domain assumption Phoneme alignments from MFA with manual correction are accurate enough for per-phoneme PER and temporal slicing.
    Section 3.1; errors in alignments would propagate to all class-level results.
  • domain assumption Attention weights averaged over heads and keys measure temporal focus.
    Section 2.2; the analysis relies on this operationalization.
  • standard math t-SNE with perplexity 30 preserves meaningful cluster structure.
    Section 4.2; qualitative interpretation assumes projection validity.
  • standard math CTC loss is standard for phoneme recognition.
    Section 2.1; standard approach.
  • domain assumption Differences in attention between audio and multimodal models isolate the effect of adding video.
    Sections 2.1 and 4.3; this is the weakest interpretive premise, confounded by architecture differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition." pith.science (2026). https://pith.science/paper/SLKFH2D6

@misc{pith2026250524059,
  author       = {Pith},
  title        = {Pith review of: Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLKFH2D6}},
  note         = {Machine review of arXiv:2505.24059}
}
read the original abstract

Although many previous studies have carried out multimodal learning with real-time MRI data that captures the audio-visual kinematics of the vocal tract during speech, these studies have been limited by their reliance on multi-speaker corpora. This prevents such models from learning a detailed relationship between acoustics and articulation due to considerable cross-speaker variability. In this study, we develop unimodal audio and video models as well as multimodal models for phoneme recognition using a long-form single-speaker MRI corpus, with the goal of disentangling and interpreting the contributions of each modality. Audio and multimodal models show similar performance on different phonetic manner classes but diverge on places of articulation. Interpretation of the models' latent space shows similar encoding of the phonetic space across audio and multimodal models, while the models' attention weights highlight differences in acoustic and articulatory timing for certain phonemes.

Figures

Figures reproduced from arXiv: 2505.24059 by the authors.

Figure 1
Figure 1. Model architecture for the current study. modules, a self-attention layer, and a convolution module (see [21] for further details), accessed via Torchaudio. The output of the Conformer is decoded by a single LSTM layer, with a final linear layer used for prediction. In the unimodal cases, the input to the Conformer consists of either acoustic features A ∈ R t×D or video features V ∈ R t×D, where t is the temporal di… view at source ↗
Figure 2
Figure 2. PER results by phonetic class for models in the cur￾rent study, grouped by manner (above) and place of articulation (below). and /k/; though in general there do not appear to be uniform changes in the latent space for consonants when adding video features. The vowel feature spaces (not shown) show similar clusters across the two models, with most of the peripheral vow￾els showing well-formed clusters; though the vow… view at source ↗
Figure 3
Figure 3. It can be seen clearly that both models learn distinct representations for the liquids, nasals and sibilants /s z S/. Where the representations are less distinct is in the oral labial and coro￾nal stops and labio-dental fricatives, which show a great deal of overlap. Also, in the audio feature space, the phonemes /t/ and /k/ have centroids in nearly the same position, despite these phonemes having different places o… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Log-mel spectrogram (top) and audio and multimodal attention weights (middle) for the phrase “a roll”, with MRI frames for the phonemes /@/ and /ô/ (bottom). Lighter attentions have higher values. affricates and stops. This is unsurprising as nasals and liquids both ha…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages

  1. [1]

    In speech production, individ- ual variability in both articulation and its consequent acoustics is robust [4, 5]

    Introduction Speech is inherently multimodal, with the integration of sen- sory and motor information being essential to both speech pro- duction and perception [1, 2, 3]. In speech production, individ- ual variability in both articulation and its consequent acoustics is robust [4, 5]. A crucial aspect of this variability concerns the realization of targe...

    work page

  2. [2]

    Method 2.1. Model The primary module for the model architecture is a Conformer, following the design in [21], which consists of two feed-forward arXiv:2505.24059v1 [cs.LG] 29 May 2025 ViT Conformer Fusion LSTM FC WavLM Layer 9 CLS Token Figure 1: Model architecture for the current study. modules, a self-attention layer, and a convolution module (see

    work page Pith review arXiv 2025

  3. [3]

    Dataset We use a new single speaker rtMRI corpus with simultaneously recorded audio and video

    Experiments 3.1. Dataset We use a new single speaker rtMRI corpus with simultaneously recorded audio and video. The corpus contains speech data from one male native speaker of American English producing the 460 sentences used in the USC TIMIT corpus, passages used in the USC 75-Speaker Dataset [12] and spontaneous speech prompted on topics such as food, h...

    work page

  4. [4]

    Phoneme error rate (PER) The overall PER results on the held-out test set are presented in Table 1

    Results 4.1. Phoneme error rate (PER) The overall PER results on the held-out test set are presented in Table 1. The results reported here are from the best perform- ing model on the test set, based on the CTC loss. The baseline Wav2Vec2Phoneme model achieved a zero-shot PER of 0.36. While no previous study has performed phoneme recognition on the corpus ...

    work page

  5. [5]

    Discussion In this study, we trained unimodal audio and video models and a combined multimodal model on phoneme recognition using a long-form single-speaker rtMRI corpus. The PER results showed that our audio model significantly outperforms the base- line model and those from similar previous work, which attests to the robustness of the WavLM representati...

    work page

  6. [6]

    Acknowledgments This work was supported by NIH grant T32 DC009975

    work page

  7. [7]

    The architecture of speech production and the role of the phoneme in speech processing,

    G. Hickok, “The architecture of speech production and the role of the phoneme in speech processing,” Language, Cognition and Neuroscience, vol. 29, no. 1, pp. 2–20, 2014

    work page 2014

  8. [8]

    M. D. Faytak, Articulatory uniformity through articulatory reuse: insights from an ultrasound study of S¯uzh¯ou Chinese. University of California, Berkeley, 2018

    work page 2018

Show all 43 references
  1. [9]

    Sensorimotor integration in speech processing: computational basis and neural organization,

    G. Hickok, J. Houde, and F. Rong, “Sensorimotor integration in speech processing: computational basis and neural organization,” Neuron, vol. 69, no. 3, pp. 407–422, 2011

    2011

  2. [10]

    Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech,

    J. H. Venezia, P. Fillmore, W. Matchin, A. L. Isenberg, G. Hickok, and J. Fridriksson, “Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech,” NeuroImage, vol. 126, pp. 196–207, 2016

    2016

  3. [11]

    The role of temporal modulation in sensorimotor interaction,

    L. Goldstein, “The role of temporal modulation in sensorimotor interaction,” Frontiers in Psychology, vol. 10, p. 2608, 2019

    2019

  4. [12]

    Variability of articulator positions and formants across nine english vowels,

    D. H. Whalen, W.-R. Chen, M. K. Tiede, and H. Nam, “Variability of articulator positions and formants across nine english vowels,” Journal of phonetics, vol. 68, pp. 1–14, 2018

    2018

  5. [13]

    S. K. Harper, Individual differences in phonetic variability and phonological representation. University of Southern California, 2021

    2021

  6. [14]

    Articulatory phonology: An overview,

    C. P. Browman and L. Goldstein, “Articulatory phonology: An overview,”Phonetica, vol. 49, no. 3-4, pp. 155–180, 1992

    1992

  7. [15]

    Multimodal representations for syn- chronized speech and real-time mri video processing,

    ¨O. D. K¨ose and M. Sarac ¸lar, “Multimodal representations for syn- chronized speech and real-time mri video processing,”IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 1912–1924, 2021

    1912

  8. [16]

    Towards speech classification from acous- tic and vocal tract data in real-time mri,

    Y . Yue, M. Proctor, L. Zhou, R. Gupta, T. Piyadasa, A. Gully, K. Ballard, and C. Jin, “Towards speech classification from acous- tic and vocal tract data in real-time mri,” inProceedings of INTER- SPEECH 2024, 2024

    2024

  9. [17]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” arXiv preprint arXiv:2201.02184, 2022

    2022 arXiv

  10. [18]

    Robust audiovisual speech recognition models with mixture-of- experts,

    Y . Wu, Y . Peng, Y . Lu, X. Chang, R. Song, and S. Watanabe, “Robust audiovisual speech recognition models with mixture-of- experts,” arXiv preprint arXiv:2409.12370, 2024

    2024 arXiv

  11. [19]

    Speech production real-time mri at 0.55 t,

    Y . Lim, P. Kumar, and K. S. Nayak, “Speech production real-time mri at 0.55 t,” Magnetic Resonance in Medicine , vol. 91, no. 1, pp. 337–343, 2024

    2024

  12. [20]

    A multispeaker dataset of raw and reconstructed speech production real-time mri video and 3d volumetric images,

    Y . Lim, A. Toutios, Y . Bliesener, Y . Tian, S. G. Lingala, C. Vaz, T. Sorensen, M. Oh, S. Harper, W. Chen et al., “A multispeaker dataset of raw and reconstructed speech production real-time mri video and 3d volumetric images,” Scientific data, vol. 8, no. 1, p. 187, 2021

    2021

  13. [21]

    The output of the Conformer is decoded by a single LSTM layer, with a final linear layer used for prediction

    for further details), accessed via Torchaudio. The output of the Conformer is decoded by a single LSTM layer, with a final linear layer used for prediction. In the unimodal cases, the input to the Conformer consists of either acoustic features A ∈ Rt×D or video featuresV ∈ Rt×...

  14. [22]

    Towards automatic speech identification from vocal tract shape dynamics in real-time mri,

    P. Saha, P. Srungarapu, and S. Fels, “Towards automatic speech identification from vocal tract shape dynamics in real-time mri,” arXiv preprint arXiv:1807.11089, 2018

    2018 arXiv

  15. [23]

    Cnn-based phoneme classifier from vocal tract mri learns embed- ding consistent with articulatory topology

    K. Van Leeuwen, P. Bos, S. Trebeschi, M. J. van Alphen, L. V oskuilen, L. E. Smeele, F. van der Heijden, R. van Sonet al., “Cnn-based phoneme classifier from vocal tract mri learns embed- ding consistent with articulatory topology.” in Interspeech, 2019, pp. 909–913

    2019

  16. [24]

    Eval- uation of a novel 8-channel rx coil for speech production mri at 0.55 t,

    F. Mu ˜noz, Y . Lim, S. X. Cui, H. Stark, and K. S. Nayak, “Eval- uation of a novel 8-channel rx coil for speech production mri at 0.55 t,” Magnetic Resonance Materials in Physics, Biology and Medicine, vol. 36, no. 3, pp. 419–426, 2023

    2023

  17. [25]

    Direct articulatory observa- tion reveals phoneme recognition performance characteristics of a self-supervised speech model,

    X. Shi, T. Feng, K. Huang, S. R. Kadiri, J. Lee, Y . Lu, Y . Zhang, L. Goldstein, and S. Narayanan, “Direct articulatory observa- tion reveals phoneme recognition performance characteristics of a self-supervised speech model,” JASA Express Letters , vol. 4, no. 11, 2024

    2024

  18. [26]

    Usc-timit: A database of multimodal speech production data,

    S. Narayanan, A. Toutios, V . Ramanarayanan, A. Lammert, J. Kim, S. Lee, K. Nayak, Y .-C. Kim, Y . Zhu, L. Goldsteinet al., “Usc-timit: A database of multimodal speech production data,” USC, Tech. Rep., 2013.[Online] http://sail. usc. edu/span/usc- timit . . . , Tech. Rep., 2013

    2013

  19. [27]

    Database of volumetric and real-time vocal tract mri for speech science

    T. Sorensen, Z. I. Skordilis, A. Toutios, Y .-C. Kim, Y . Zhu, J. Kim, A. C. Lammert, V . Ramanarayanan, L. Goldstein, D. Byrd et al., “Database of volumetric and real-time vocal tract mri for speech science.” in Interspeech, 2017, pp. 645–649

    2017

  20. [28]

    Characterization of inter-speaker articulatory variability: A two- level multi-speaker modelling approach based on mri data,

    A. Serrurier, P. Badin, L. Lamalle, and C. Neuschaefer-Rube, “Characterization of inter-speaker articulatory variability: A two- level multi-speaker modelling approach based on mri data,” The Journal of the Acoustical Society of America, vol. 145, no. 4, pp. 2149–2170, 2019

    2019

  21. [29]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al. , “Conformer: Convolution- augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  22. [30]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd International Conference on Machine Learning , ser. ICML ’06. New York, NY , USA: As...

    2006

  23. [31]

    Simple and effective zero-shot cross-lingual phoneme recognition,

    Q. Xu, A. Baevski, and M. Auli, “Simple and effective zero-shot cross-lingual phoneme recognition,” arXiv preprint arXiv:2109.11680, 2021

    2021 arXiv

  24. [32]

    Ro- bust and efficient medical imaging with self-supervision,

    S. Azizi, L. Culp, J. Freyberg, B. Mustafa, S. Baur, S. Kornblith, T. Chen, P. MacWilliams, S. S. Mahdavi, E. Wulczynet al., “Ro- bust and efficient medical imaging with self-supervision,” arXiv preprint arXiv:2205.09723, 2022

    2022 arXiv

  25. [33]

    State-of-the-art speech production mri protocol for new 0.55 tesla scanners,

    P. Kumar, Y . Tian, Y . Lim, S. X. Cui, C. Hagedorn, D. Byrd, U. K. Sinha, S. Narayanan, and K. S. Nayak, “State-of-the-art speech production mri protocol for new 0.55 tesla scanners,” in Inter- speech 2024, 2024, pp. 2590–2594

    2024

  26. [34]

    Announcing the electro- magnetic articulography (day 1) subset of the mngu0 articulatory corpus,

    K. Richmond, P. Hoole, and S. King, “Announcing the electro- magnetic articulography (day 1) subset of the mngu0 articulatory corpus,” inTwelfth Annual Conference of the International Speech Communication Association, 2011

    2011

  27. [35]

    Real time speech enhancement in the waveform domain,

    A. Defossez, G. Synnaeve, and Y . Adi, “Real time speech enhancement in the waveform domain,” arXiv preprint arXiv:2006.12847, 2020

    2006 arXiv

  28. [36]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  29. [37]

    Ev- idence of vocal tract articulation in self-supervised learning of speech,

    C. J. Cho, P. Wu, A. Mohamed, and G. K. Anumanchipalli, “Ev- idence of vocal tract articulation in self-supervised learning of speech,” arXiv preprint arXiv:2210.11723, 2022

    2022 arXiv

  30. [38]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    D. Alexey, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv: 2010.11929 , 2020

    2010 arXiv

  31. [39]

    A sim- ple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A sim- ple framework for contrastive learning of visual representations,” arXiv preprint arXiv:2002.05709, 2020

    2002 arXiv

  32. [41]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008

    2008

  33. [42]

    Toward articulatory- acoustic models for liquid approximants based on mri and epg data. part ii. the rhotics,

    A. Alwan, S. Narayanan, and K. Haker, “Toward articulatory- acoustic models for liquid approximants based on mri and epg data. part ii. the rhotics,” The Journal of the Acoustical Society of America, vol. 101, no. 2, pp. 1078–1089, 1997

    1997

  34. [43]

    Articulatory characterization of english liquid- final rimes,

    M. Proctor, R. Walker, C. Smith, T. Szalay, L. Goldstein, and S. Narayanan, “Articulatory characterization of english liquid- final rimes,” Journal of Phonetics, vol. 77, p. 100921, 2019

    2019

  35. [256]

    The number of Conformer layers was set to 3

    The number of attention heads was set at 4, a kernel size of 31, and dropout set at 0.3. The number of Conformer layers was set to 3. For the LSTM layer, the latent size was 128, which served as the input size for the final linear layer. For all mod- els, the Adam optimizer wa...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.