Pith. sign in

REVIEW 3 major objections 4 minor 63 references

Deep Speech Synthesis from Multimodal Articulatory Representations

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that pre-training on multiple articulatory modalities, then fine-tuning on a single scarce one, makes MRI- and EMG-to-speech synthesis substantially more intelligible.

desk verdict Useful multimodal pre-training recipe for articulatory synthesis; the in-architecture gains are real, but the headline comparison and the estimated-EMA leakage need tightening before the central claim is clean. read the letter →

arxiv 2412.13387 v1 pith:3SW6HUVD submitted 2024-12-17 eess.AS cs.SD

Peter Wu , Bohan Yu , Kevin Scheck , Alan W Black , Aditi S. Krishnapriyan , Irene Y. Chen , Tanja Schultz , Shinji Watanabe
show 1 more author
Gopala K. Anumanchipalli
This is my paper · ORCID
classification eess.AScs.SD
keywords articulatorysynthesismultimodalpre-trainingMRI-to-speechEMG-to-speechelectromagneticarticulographytransferlearningspeechintelligibilitydeepfeatureloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Articulatory-to-speech synthesis -- turning measurements of the vocal tract or articulator muscles into audible speech -- usually has far less training data than ordinary speech synthesis. The paper argues that this scarcity can be offset by pre-training a shared encoder on several articulatory modalities at once and then fine-tuning on whichever single modality is actually available. The headline result is a jump in intelligibility on the hard, low-resource MRI-to-speech task: test-set word error rate falls from 69.5% to 33.4% compared with the prior model, and similar gains appear for EMG-to-speech. If these results hold, the recipe gives practitioners a practical transfer-learning path for assistive and silent-speech technologies where recorded articulatory data is scarce. A notable part of the design is that the auxiliary EMA channel is estimated from the audio itself, so multimodal pre-training can run without paired multimodal recordings.

What carries the argument

The load-bearing object is the multimodal encoder, built from unimodal per-modality encoders, a fusion layer that averages the non-zero modality encodings, and a shared transformer-convolution encoder. Because absent modalities are zeroed and the fusion only averages what is present, the same pretrained encoder can be fine-tuned with a single modality without architectural change. An optional deep feature loss pulls the unimodal encoders toward agreement on the same datapoint, and a GAN vocoder pretrained on ordinary speech turns encoder outputs into waveforms; the decoder is deliberately trained without articulatory labels so it can consume any speech corpus.

What would settle it

Run the best MRI-to-speech recipe with the audio-estimated EMA features time-shuffled (or replaced by EMA features from a different utterance) and check whether the word error rate stays near 33.4% or rises toward the unimodal baseline; if it stays low, the gain is not about articulatory content. A paired dataset with concurrently recorded MRI and true EMA would settle it directly: if the benefit disappears when real EMA replaces audio-estimated EMA, the estimated features were leaking target information.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodal pre-training is an effective fix for the scarcity of articulatory data. The authors train a shared encoder jointly on several articulatory modalities -- electromagnetic articulography (EMA), MRI, and surface electromyography (EMG) -- then fine-tune the encoder on whichever single modality is actually available for the target speaker. On a single-speaker MRI task with under ten minutes of training speech, the best variant reaches a test-set word error rate of 33.4%, against 69.5% for the prior model, and the EMG-to-speech system also beats its unimodal baselines on intelligibility and on objective and subjective quality metrics. A distinctive part of the recipe is that the auxiliary EMA channel is not recorded for the MRI and EMG speakers at all: it is linearly estimated from the same audio, using a regression trained on a separate EMA corpus, which lets the authors run tri-modal pre-training without paired multimodal data.

Load-bearing premise

The load-bearing premise is that EMA features linearly estimated from the ground-truth audio are a faithful proxy for real articulator motion and do not leak the target speech; if they carry acoustic information about the utterance being synthesized, the pre-training gains could be an artifact rather than evidence about multimodal alignment.

Editorial extensions

If this is right

  • MRI-to-speech synthesizers trained on less than ten minutes of single-speaker data can reach a test-set word error rate of 33.4%, more than halving the 69.5% error of the prior model.
  • Adding more articulatory modalities during pre-training improves results across ASR word error rate, mel-cepstral distortion, SpeechBERTScore, and mean opinion score, with the largest jump from unimodal to bi-modal training.
  • EMG-to-speech, a task with no strong published baseline, also beats its unimodal counterpart on intelligibility and quality metrics under the same pre-training recipe.
  • Including audio-estimated EMA features and the deep feature loss generally improves synthesis quality, suggesting that modality alignment matters during pre-training.
  • The decoder can be trained on any large speech corpus, so the articulatory-data bottleneck is confined to encoder pre-training and fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same recipe should transfer to other low-resource articulatory inputs, such as ultrasound or silent EMG, whenever a linear map from that modality to EMA can be estimated from existing data.
  • Editorial inference: If the audio-estimated EMA channel is genuinely informative, the practical payoff is that silent-speech and assistive systems can be bootstrapped from large unlabeled speech corpora plus the smallest articulatory recordings.
  • Editorial inference: A direct control experiment -- replacing estimated EMA with time-shuffled or other-utterance EMA during pre-training -- would test whether the gains come from articulatory content or from the extra model capacity the multi-channel input provides.
  • Editorial inference: The reported linear correlations between modalities (EMA to MRI, EMG to EMA) suggest that the choice of which auxiliary modality to synthesize can be guided by how linearly predictable one articulatory representation is from another.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a multimodal pre-training framework for articulatory-to-speech synthesis. A shared encoder with per-modality convolutional encoders and a modality-invariant pooling layer is pre-trained to predict HuBERT-Soft features from EMA, MRI, and EMG inputs, then fine-tuned on a single low-resource articulatory modality; the decoder is a HiFi-CAR vocoder trained on VCTK. For the MRI and EMG tasks, the authors use a WavLM-based linear regressor to estimate EMA features from ground-truth waveforms, since no paired MRI-EMA or EMG-EMA data exist. The paper reports large WER reductions over a prior MRI-to-speech model and consistent improvements of multimodal over unimodal pre-training on multiple objective and subjective metrics.

Significance. The within-architecture unimodal-versus-multimodal ablation is a genuine experimental strength, and the missing-modality zeroing mechanism is a clean way to combine unimodal articulatory corpora. If the pre-training benefit is real, the recipe is practically useful for low-resource articulatory synthesis, transferring 7.9 hours of EMA data to 11-minute MRI and 3.9-hour EMG tasks. The paper also evaluates with three objective metrics plus MOS and is transparent about dataset splits. The unresolved issue is whether the audio-estimated EMA channel provides genuine articulatory alignment or leaks target-speech information; until that is controlled, the central claim is not fully established.

major comments (3)
  1. [Sections 4.1.2/4.1.3/4.2, Eq. (5), Tables 3/4] The auxiliary EMA channel used for MRI and EMG is not measured articulatory data; it is produced by a linear WavLM-to-EMA regressor trained on 2000 HPRC frames and applied to the ground-truth waveforms from which the HuBERT-Soft target y is also extracted. The L1 pre-training objective L1(x,y)=|e(x)-y| can therefore be minimized by copying audio-derived information through the estimated-EMA branch, rather than by learning a genuine articulatory-to-acoustic mapping. The within-architecture unimodal rows are the only conditions without this audio-derived channel, so the reported gains (MRI WER 50.9 to 33.4; EMG WER 24.4 to 19.6) conflate multimodal pre-training with target leakage and with the added EMA pre-training corpus. Section 4.2 says the effect of estimated EMA was studied, but Tables 3/4 do not report those variants. Please add a controlled ablation, for example zeroing the EMA branch for MRI/EMG datapoints while still pre-training on real HPRC EMA, or replacing the estimated EMA with a non-articulatory WavLM feature, and report it in the main results.
  2. [Section 4.4 and Table 2] The correlations in Table 2 do not validate the pseudo-EMA in the domains where it is used. The MRI and EMG datasets have no paired ground-truth EMA; the EMA targets for the MRI-to-EMA (0.577) and EMG-to-EMA (0.510) rows are the audio-estimated features described in Sections 4.1.2 and 4.1.3. A correlation between an articulatory feature and a regressor-derived target that itself comes from the waveform cannot support the statement that EMG and MRI share information with EMA. Please either obtain true paired targets or clearly state that these entries are consistency checks against pseudo-labels, and adjust the interpretation accordingly.
  3. [Abstract, Section 4.2, Table 3] The headline comparison of 33.4% WER versus 69.5% for the previous model [4] is not a controlled test of multimodal pre-training, because the baseline differs in input and output sampling rates, encoder architecture, audio enhancement, and decoder configuration. The within-architecture unimodal Encoder-Decoder row (50.9% WER) is the correct baseline for the pre-training claim and should be foregrounded. In addition, the abstract's '36% word error rate' should be stated as 36.1 absolute percentage points or as a relative WER reduction, to avoid ambiguity.
minor comments (4)
  1. [Abstract] The phrase '36% word error rate' is ambiguous; Table 3 corresponds to 36.1 absolute WER percentage points (69.5% to 33.4%), which is a 51.9% relative reduction.
  2. [Section 4.2] The statement that the estimated-EMA inclusion was studied is not backed by a row or column in Tables 3 and 4; either add these variants or state explicitly that all multimodal rows use estimated EMA.
  3. [Section 4.5] The sentence 'Adding more modalities generally improves performance' is not supported by Table 3, where tri-modal with deep feature loss has WER 41.3 versus 33.4 for bi-modal with deep feature loss; please qualify the generalization and discuss the non-monotonicity.
  4. [Tables 3 and 4] The EMG baseline labeled 'V ocoder [4]' is not literally the model from [4], since Section 4.2 specifies a HiFi-CAR with different upsample scales; rename it to something like 'HiFi-CAR EMG baseline' to avoid implying a published EMG comparison.

Circularity Check

2 steps flagged · score 5.0 of 10

Auxiliary EMA used for MRI/EMG pre-training is fitted from the same waveform that supplies the HuBERT-Soft target, so the claimed multimodal benefit is confounded by an audio-derived shortcut.

  1. fitted input called prediction [Section 3.4 (L1 pre-training objective) and Section 4.1.2 (estimated EMA features)]
    ""We use WavLM [43] and the trained regression model to extract EMA from waveforms in our MRI dataset" ... "We optimize with the L1 loss function, given by L1(x, y) = |e(x) − y|, where e(x) and y are the predicted and ground-truth outputs, respectively" ... "whether including the EMA features estimated from waveforms in Section 4.1 improves model performance.""

    The EMA used as an auxiliary modality for MRI and EMG is not measured articulography: it is the output of a linear WavLM-to-EMA regressor applied to the same 16 kHz waveform from which the HuBERT-Soft target y is extracted (Section 4.1.1). Under L1(x, y) = |e(x) - y|, the encoder can minimize the pre-training loss by copying audio-derived WavLM information through the EMA branch, without learning an articulatory-to-acoustic mapping from MRI or EMG. The reported bi-modal gains over unimodal pre-training (e.g., Table 3: 50.865% to 39.239% WER) are therefore not attributable specifically to multimodal articulatory alignment; they are confounded by a fitted, audio-derived modality. The paper reports no ablation that removes only the estimated-EMA channel while retaining real EMA pre-training.

  2. other [Section 4.4 and Table 2]
    ""The EMG-to-EMA and MRI-to-EMA correlation results are also reasonably high, suggesting that EMG and MRI share information with EMA and can benefit from joint training." Section 4.1.2 defines the EMA for these datasets as: "We use WavLM [43] and the trained regression model to extract EMA from waveforms in our MRI dataset.""

    The regression targets used to compute the MRI-to-EMA (0.577) and EMG-to-EMA (0.510) correlations in Table 2 are the same WavLM-estimated EMA features defined in Section 4.1.2, not independently measured EMA aligned with MRI/EMG. Since both the input features (MRI, EMG) and the EMA targets are deterministic functions of the same underlying waveform, the high correlations can simply reflect shared audio-derived information. The paper uses these correlations as evidence that "EMG and MRI share information with EMA and can benefit from joint training," but the validation is circular: the ground-truth EMA for these modalities is itself a fitted function of the target audio.

full rationale

This is an empirical systems paper rather than a mathematical derivation, so most of the pipeline is not circular: the encoder/decoder architecture, the HiFi-CAR vocoder, the HuBERT-Soft targets, and the comparison against the prior MRI model [4] are all externally grounded. However, the central attribution of the reported improvements to multimodal articulatory pre-training is partially circular. For both MRI-to-speech and EMG-to-speech, the auxiliary EMA modality is not measured; it is generated by a WavLM-to-EMA regression trained on HPRC and then extracted from the same waveforms that provide the HuBERT-Soft training target. Consequently, the L1 pre-training objective can be satisfied by an audio-derived shortcut through the EMA branch, and the bi-modal-versus-unimodal comparisons do not isolate the effect of multimodal articulatory alignment. The Section 4.4 correlation evidence is similarly circular because the EMA targets for MRI and EMG are themselves the estimated features. The measured WER improvements are real outputs of the system, so the paper is not wholly reducible to its inputs, but the interpretive claim that multimodal articulatory pre-training drives the gain is confounded by construction. Score 5 reflects this partial circularity: the fitted, target-derived EMA is central to the method's stated mechanism, yet the final test-time models and measured intelligibility results retain independent empirical content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper relies on a handful of domain assumptions (suitability of HuBERT-Soft, validity of audio-estimated EMA, the pooling design, and the MRI audio enhancement). None of these are derived in the paper; they are inherited from prior work or chosen ad hoc. The audio-estimated EMA is the most consequential, as it is the only way multimodal pre-training is realized for MRI and EMG.

free parameters (2)
  • Loss weighting between L1 and deep feature loss L2 = not reported
    The paper says it also experiments with adding the deep feature loss (Section 3.4) but does not specify the weighting relative to L1, which is needed for replication and affects the results.
  • Training duration (steps/epochs) for encoder pre-training and fine-tuning = not reported
    No training schedule is given, only architecture and optimizer settings (Section 4.2). This is a significant unstated detail for reproduction.
assumptions (4)
  • domain assumption HuBERT-Soft features are a suitable intermediate target for articulatory-to-acoustic synthesis
    The whole pipeline maps articulatory inputs to HuBERT-Soft (Section 3.3, 4.1.1), relying on [40] for its suitability. If this representation were insufficient, the decoder could not produce intelligible speech.
  • domain assumption EMA can be linearly estimated from WavLM features with the regression trained on 2000 frames
    Section 4.1.2 adopts the approach of [42] to synthesize an EMA stream for MRI and EMG datasets. The pre-training benefit for those modalities depends on this estimate being a valid, non-leaking proxy.
  • ad hoc to paper The modality-invariant pooling operation (sum of unimodal encodings, normalized by number of present modalities) retains sufficient information for synthesis
    Equation 4 is a design choice with no analysis of information loss or optimality; the paper only argues invariance, not sufficiency.
  • domain assumption Deep speech enhancement of noisy MRI audio yields a clean-enough target for training and evaluation
    Section 4.1.2 inherits the enhancement from [4]. If enhancement introduces artifacts, the reported WER/MOS would overstate real-world performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Speech Synthesis from Multimodal Articulatory Representations." pith.science (2026). https://pith.science/paper/3SW6HUVD

@misc{pith2026241213387,
  author       = {Pith},
  title        = {Pith review of: Deep Speech Synthesis from Multimodal Articulatory Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3SW6HUVD}},
  note         = {Machine review of arXiv:2412.13387}
}
read the original abstract

The amount of articulatory data available for training deep learning models is much less compared to acoustic speech data. In order to improve articulatory-to-acoustic synthesis performance in these low-resource settings, we propose a multimodal pre-training framework. On single-speaker speech synthesis tasks from real-time magnetic resonance imaging and surface electromyography inputs, the intelligibility of synthesized outputs improves noticeably. For example, compared to prior work, utilizing our proposed transfer learning methods improves the MRI-to-speech performance by 36% word error rate. In addition to these intelligibility results, our multimodal pre-trained models consistently outperform unimodal baselines on three objective and subjective synthesis quality metrics.

Figures

Figures reproduced from arXiv: 2412.13387 by the authors.

Figure 1
Figure 1. Multimodal encoder architecture. prior loss functions requiring more than one modality as input [33, 27]. Moreover, our approach supports settings where only one input modality is available per datapoint, as is often the case [15, 29, 8], elaborated in Section 3.4. 3.3. Decoder Given the output of the encoder as input, our decoder outputs an acoustic waveform. Our decoder architecture is HiFi-CAR, an auto-regressive… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 62 canonical work pages

  1. [4]

    Parallel V ocalized Speech

    Results 4.1. Datasets We study jointly training an articulatory synthesizer with elec- tromagnetic articulography (EMA), real-time magnetic reso- nance imaging (MRI), and surface electromyography (EMG) datasets. We summarize characteristics of these three modali- ties in Table 1 and provide detailed descriptions of each dataset below. 4.1.1. EMA Dataset E...

    work page 2000

  2. [1]

    Since these mod- els are biologically grounded, they can be applied to decod- ing speech from biosignals for health technology applications [6, 7, 8, 9, 10, 11, 12, 13]

    Introduction Articulatory synthesis incorporates information about the vocal tract into speech synthesizers to improve interpretability, gen- eralizability, and efficiency [1, 2, 3, 4, 5]. Since these mod- els are biologically grounded, they can be applied to decod- ing speech from biosignals for health technology applications [6, 7, 8, 9, 10, 11, 12, 13]...

    work page

  3. [2]

    Deep Speech Synthesis from Multimodal Articulatory Representations

    Related Work 2.1. Deep Articulatory Synthesis Deep articulatory synthesis involves synthesizing acoustics from articulatory features using deep learning [2, 6, 7, 8, 3, 4, 9, 5]. Current approaches can generally be described as ei- ther direct or involving an intermediate feature. Direct synthe- sis maps articulatory inputs to acoustics with a single end-...

    work page Pith review arXiv 2024

  4. [3]

    Like [8], the encoder and decoder are trained separately, since the former requires articulatory la- bels whereas the latter does not

    Methods We propose an encoder-decoder-based framework for multi- modal articulatory synthesis. Like [8], the encoder and decoder are trained separately, since the former requires articulatory la- bels whereas the latter does not. Our encoder utilizes a mul- timodal fusion layer to jointly encode multiple modalities, and our decoder is an articulatory voco...

    work page

  5. [5]

    When training the encoder, like [3], we randomly crop a 0.6 to 2.0 second window from each sample in the size-64 batch, with the window length fixed within the batch

    Our unimodal encoders have a kernel size of 5, stride 1, and padding 2 so that their input sequence lengths match the output lengths. When training the encoder, like [3], we randomly crop a 0.6 to 2.0 second window from each sample in the size-64 batch, with the window length fixed within the batch. Our decoder, described in Section 3.3, contains 4 upsam-...

    work page

  6. [6]

    Dnn-based ultrasound-to-speech conversion for a silent speech interface,

    T. G. Csap ´o et al., “Dnn-based ultrasound-to-speech conversion for a silent speech interface,” Interspeech, 2017

    work page 2017

  7. [7]

    Sottovoce: An ultrasound imaging-based silent speech interaction using deep neural networks,

    N. Kimura et al., “Sottovoce: An ultrasound imaging-based silent speech interaction using deep neural networks,” in CHI, 2019

    work page 2019

  8. [8]

    Our MRI-to-speech synthesizer outperforms the test-set ASR WER of the previous model [4] by 36% WER, with our EMG- to-speech model similarly outperforming the baseline

    Conclusion We devise a multimodal pre-training framework for improv- ing the performance of deep MRI- and EMG-to-speech models. Our MRI-to-speech synthesizer outperforms the test-set ASR WER of the previous model [4] by 36% WER, with our EMG- to-speech model similarly outperforming the baseline. On all of our objective and subjective synthesis quality met...

    work page

Show all 63 references
  1. [9]

    Multi-speaker speech synthesis from electromyographic signals by soft speech unit prediction,

    K. Scheck and T. Schultz, “Multi-speaker speech synthesis from electromyographic signals by soft speech unit prediction,” in ICASSP, 2023

    2023

  2. [10]

    Articulatory synthesis from underlying dy- namics,

    C. P. Browman et al., “Articulatory synthesis from underlying dy- namics,” JASA, 1984

    1984

  3. [11]

    Data driven articulatory syn- thesis with deep neural networks,

    S. Aryal and R. Gutierrez-Osuna, “Data driven articulatory syn- thesis with deep neural networks,”Computer Speech & Language, 2016

    2016

  4. [12]

    Deep speech synthesis from articulatory representa- tions,

    P. Wu et al., “Deep speech synthesis from articulatory representa- tions,” Interspeech, 2022

    2022

  5. [13]

    Deep speech synthesis from mri-based articulatory repre- sentations,

    ——, “Deep speech synthesis from mri-based articulatory repre- sentations,” Interspeech, 2023

    2023

  6. [14]

    Self-Supervised Solution to the Control Prob- lem of Articulatory Synthesis,

    P. K. Krug et al., “Self-Supervised Solution to the Control Prob- lem of Articulatory Synthesis,” in Interspeech, 2023

    2023

  7. [15]

    Quantifying kinematic aspects of reduction in a contrasting rate production task,

    M. K. Tiede et al., “Quantifying kinematic aspects of reduction in a contrasting rate production task,” JASA, 2017

    2017

  8. [16]

    Libritts: A corpus derived from librispeech for text-to-speech,

    H. Zen et al. , “Libritts: A corpus derived from librispeech for text-to-speech,” Interspeech, 2019

    2019

  9. [17]

    An improved model for voicing silent speech,

    D. Gaddy and D. Klein, “An improved model for voicing silent speech,” arXiv, 2021

    2021

  10. [18]

    Multibench: Multiscale benchmarks for multi- modal representation learning,

    P. P. Liang et al., “Multibench: Multiscale benchmarks for multi- modal representation learning,” NeurIPS, 2021

    2021

  11. [19]

    A high-performance neuroprosthesis for speech decoding and avatar control,

    S. L. Metzger et al. , “A high-performance neuroprosthesis for speech decoding and avatar control,” Nature, 2023

    2023

  12. [20]

    Unconstrained dysfluency modeling for dysfluent speech transcription and detection,

    J. Lian et al., “Unconstrained dysfluency modeling for dysfluent speech transcription and detection,” in ASRU, 2023

    2023

  13. [21]

    Real-time control of an articulatory-based speech synthesizer for brain computer interfaces,

    F. Bocquelet et al. , “Real-time control of an articulatory-based speech synthesizer for brain computer interfaces,” PLoS compu- tational biology, 2016

    2016

  14. [22]

    Brain-computer interface: applications to speech decoding and synthesis to augment communication,

    S. Luo et al., “Brain-computer interface: applications to speech decoding and synthesis to augment communication,”Neurothera- peutics, 2023

    2023

  15. [23]

    Announcing the electromagnetic articulog- raphy (day 1) subset of the mngu0 articulatory corpus,

    K. Richmond et al., “Announcing the electromagnetic articulog- raphy (day 1) subset of the mngu0 articulatory corpus,” in Inter- speech, 2011

    2011

  16. [24]

    Talnet: V oice reconstruction from tongue and lip articulation with transfer learning from text-to-speech synthe- sis,

    J.-X. Zhang et al., “Talnet: V oice reconstruction from tongue and lip articulation with transfer learning from text-to-speech synthe- sis,” in AAAI, 2021

    2021

  17. [25]

    The multimodal information based speech pro- cessing (misp) 2022 challenge: Audio-visual diarization and recognition,

    Z. Wang et al., “The multimodal information based speech pro- cessing (misp) 2022 challenge: Audio-visual diarization and recognition,” in ICASSP, 2023

    2022

  18. [26]

    Cross-Speaker Training and Adaptation for Electromyography-to-Speech Conversion,

    K. Scheck et al. , “Cross-Speaker Training and Adaptation for Electromyography-to-Speech Conversion,” inEMBC, 2024

    2024

  19. [27]

    Ema2s: An end-to-end multimodal articulatory-to-speech system,

    Y .-W. Chen et al. , “Ema2s: An end-to-end multimodal articulatory-to-speech system,” in ISCAS, 2021

    2021

  20. [28]

    Stream-ets: Low-latency end-to-end speech synthesis from electromyography signals,

    K. Scheck et al. , “Stream-ets: Low-latency end-to-end speech synthesis from electromyography signals,” in Speech Communi- cation, 2023

    2023

  21. [29]

    Large-scale multi-modal pre-trained models: A comprehensive survey,

    X. Wang et al., “Large-scale multi-modal pre-trained models: A comprehensive survey,”Machine Intelligence Research, 2023

    2023

  22. [30]

    A real-time speech driven talking avatar based on deep neural network,

    K. Zhao et al., “A real-time speech driven talking avatar based on deep neural network,” in APSIPA, 2013

    2013

  23. [31]

    Audio-visual speech recognition using deep bottleneck features and high-performance lipreading,

    S. Tamura et al. , “Audio-visual speech recognition using deep bottleneck features and high-performance lipreading,” inAPSIPA, 2015

    2015

  24. [32]

    Speech gesture generation from the trimodal con- text of text, audio, and speaker identity,

    Y . Yoonet al., “Speech gesture generation from the trimodal con- text of text, audio, and speaker identity,”TOG, 2020

    2020

  25. [33]

    Speech denoising with deep feature losses,

    F. G. Germain et al., “Speech denoising with deep feature losses,” arXiv preprint arXiv:1806.10522, 2018

    2018 arXiv

  26. [34]

    Hifi-gan: High-fidelity denoising and dereverbera- tion based on speech deep features in adversarial networks,

    J. Su et al., “Hifi-gan: High-fidelity denoising and dereverbera- tion based on speech deep features in adversarial networks,” in Interspeech, 2017

    2017

  27. [35]

    Data-driven multimodal synthe- sis,

    R. Carlson and B. Granstr ¨om, “Data-driven multimodal synthe- sis,” Speech communication, 2005

    2005

  28. [36]

    Electromagnetic articulography: Use of alternating magnetic fields for tracking movements of multiple points inside and outside the vocal tract,

    P. W. Sch ¨onle et al. , “Electromagnetic articulography: Use of alternating magnetic fields for tracking movements of multiple points inside and outside the vocal tract,” Brain and Language , 1987

    1987

  29. [37]

    Estimating articulatory movements in speech production with transformer networks,

    S. Udupa et al. , “Estimating articulatory movements in speech production with transformer networks,” arXiv, 2021

    2021

  30. [38]

    A multispeaker dataset of raw and reconstructed speech production real-time mri video and 3d volumetric images,

    Y . Lim et al., “A multispeaker dataset of raw and reconstructed speech production real-time mri video and 3d volumetric images,” Scientific data, 2021

    2021

  31. [39]

    Deep residual learning for image recognition,

    K. He et al., “Deep residual learning for image recognition,” in CVPR, 2016

    2016

  32. [40]

    0.702 0.224 0.423 - Table 3: MRI-to-speech, training proposed model with tri-modal (EMA, MRI, EMG), bi-modal (EMA, MRI), and unimodal (MRI) data. Model CER (%) ↓ WER (%) ↓ MCD↓ SpeechBERTScore↑ MOS↑ Tri-modal Encoder-Decoder with Deep Feature Loss 26.427 41.326 8.3359 0.7305 3...

    1961

  33. [41]

    Attention is all you need,

    A. Vaswani et al., “Attention is all you need,”NeurIPS, 2017

    2017

  34. [42]

    We train regression models for the fol- lowing speech features: EMA, MRI, EMG, and HuBERT-Soft [40]

    and Section 4.1.1, and test on 200 frames randomly chosen from the remaining data. We train regression models for the fol- lowing speech features: EMA, MRI, EMG, and HuBERT-Soft [40]. For experiments with EMG, we choose frames from the EMG dataset in Section 4.1, and likewise ...

  35. [43]

    Cognitron: A self-organizing multilayered neural network,

    K. Fukushima, “Cognitron: A self-organizing multilayered neural network,” Biological cybernetics, 1975

    1975

  36. [44]

    Chunked autoregressive gan for conditional waveform synthesis,

    M. Morrison et al., “Chunked autoregressive gan for conditional waveform synthesis,” ICLR, 2021

    2021

  37. [45]

    Silent speech interfaces,

    B. Denby et al., “Silent speech interfaces,” Speech Communica- tion, 2010

    2010

  38. [46]

    Speaker-independent acoustic-to-articulatory speech inversion,

    P. Wu et al., “Speaker-independent acoustic-to-articulatory speech inversion,” inICASSP, 2023

    2023

  39. [47]

    Transposed convolutions have strides of 8, 5, 4, and 2, in that order, in order to upsample 50 Hz inputs to 16000 Hz waveforms

    followed by a transposed convolution like [34]. Transposed convolutions have strides of 8, 5, 4, and 2, in that order, in order to upsample 50 Hz inputs to 16000 Hz waveforms. Like [34], each upsampling block is followed by 3 residual blocks with kernel sizes of 3, 7, and 11. ...

    2000

  40. [48]

    The secret source: In- corporating source features to improve acoustic-to-articulatory speech inversion,

    Y . M. Siriwardena and C. Espy-Wilson, “The secret source: In- corporating source features to improve acoustic-to-articulatory speech inversion,” inICASSP, 2023

    2023

  41. [49]

    Crepe: A convolutional representation for pitch estimation,

    J. W. Kim et al., “Crepe: A convolutional representation for pitch estimation,” in ICASSP, 2018

    2018

  42. [50]

    A comparison of discrete and soft speech units for improved voice conversion,

    B. van Niekerk et al., “A comparison of discrete and soft speech units for improved voice conversion,” inICASSP, 2022

    2022

  43. [51]

    Speech Synthesis from Articulatory Movements Recorded by Real-time MRI,

    Y . Otani et al., “Speech Synthesis from Articulatory Movements Recorded by Real-time MRI,” in Interspeech, 2023

    2023

  44. [52]

    Evidence of vocal tract articulation in self- supervised learning of speech,

    C. J. Cho et al. , “Evidence of vocal tract articulation in self- supervised learning of speech,” in ICASSP, 2023

    2023

  45. [53]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” JSTSP, 2022

    2022

  46. [54]

    Component gestures in the production of oral and nasal labial stops,

    K. S. Harris et al., “Component gestures in the production of oral and nasal labial stops,” JASA, 1962

    1962

  47. [55]

    Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,

    C. Veaux et al., “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,” CSTR, 2017

    2017

  48. [56]

    Rectifier nonlinearities improve neural network acoustic models,

    A. L. Maas et al., “Rectifier nonlinearities improve neural network acoustic models,” in ICML, 2013

    2013

  49. [57]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” ICLR, 2015

    2015

  50. [58]

    Espnet2-tts: Extending the edge of tts re- search,

    T. Hayashi et al. , “Espnet2-tts: Extending the edge of tts re- search,” arXiv, 2021

    2021

  51. [59]

    An adaptive algorithm for mel-cepstral analysis of speech

    T. Fukada et al., “An adaptive algorithm for mel-cepstral analysis of speech.” in ICASSP, 1992

    1992

  52. [60]

    SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation met- rics,

    T. Saeki et al., “SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation met- rics,” arXiv, 2024

    2024

  53. [61]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al., “Robust speech recognition via large-scale weak supervision,” in ICML, 2023

    2023

  54. [62]

    A comparative study of different emg features for acoustics-to-emg mapping

    M. Sharma et al., “A comparative study of different emg features for acoustics-to-emg mapping.” in Interspeech, 2021

    2021

  55. [63]

    STE-GAN: Speech-to- Electromyography Signal Conversion using Generative Ad- versarial Networks,

    K. Scheck and T. Schultz, “STE-GAN: Speech-to- Electromyography Signal Conversion using Generative Ad- versarial Networks,” in Interspeech, 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.