Conformer-based Ultrasound-to-Speech Conversion

G\'abor Gosztolya; Ibrahim Ibrahimov; Zaink\'o Csaba

REVIEW 3 major objections 6 minor 31 references

Conformer-based Ultrasound-to-Speech Conversion

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conformer with bi-LSTM improves perceived naturalness in ultrasound-to-speech conversion, while the simpler Conformer Base matches baseline at a lower training cost.

desk verdict A clean, honest empirical comparison of Conformer variants for ultrasound-to-speech, but the only positive result—better perceptual naturalness—rests on one uncorrected borderline MUSHRA p-value. read the letter →

arxiv 2506.03831 v1 pith:KXKXDQSL submitted 2025-06-04 cs.SD cs.MMeess.AS

Ibrahim Ibrahimov , Zainkó Csaba , Gábor Gosztolya This is my paper

classification cs.SDcs.MMeess.AS

keywords silentspeechinterfacesultrasoundtongueimagingconformerultrasound-to-speechconversionmelspectrogrammappingHiFi-GANvocoderMUSHRAlisteningtestspeaker-specificmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether convolution-augmented transformer architectures can replace the usual CNN in mapping ultrasound tongue images to mel spectrograms for silent speech interfaces. It trains speaker-specific models for four speakers and compares two Conformer variants against a 2D-CNN baseline. Objective metrics show no reliable gain, but a MUSHRA listening test rated the bi-LSTM variant's output more natural than the baseline, while the lighter Conformer Base matched baseline quality and trained about three times faster. If accepted, this gives the field a new acoustic-feature mapping option that may be more efficient and perceptually competitive, even when standard objective scores do not improve.

What carries the argument

The conformer block, a convolution-augmented transformer that combines self-attention for global context with convolution for local feature extraction, is the core mapping module from 64 by 128 ultrasound frame sequences to 80-dimensional mel spectrograms. The bi-LSTM variant adds two bidirectional LSTM layers after layer normalization to capture both past and future sequence context. A HiFi-GAN vocoder then synthesizes the audio waveform. The efficiency claim is carried by parameter counts and relative training times: Conformer Base has 2.66 million parameters and trains in about 30 percent of the baseline time, while the bi-LSTM variant has 5.35 million parameters and trains in about 80 percent of the baseline time.

What would settle it

Run a preregistered listening test with native listeners, all test utterances per speaker, and multiple-comparison correction; if the bi-LSTM variant no longer beats the baseline significantly, the central positive claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a Conformer with bi-LSTM gives better perceived naturalness than a standard 2D-CNN baseline in ultrasound-to-speech conversion, while Conformer Base matches baseline perceptual quality with about one-third of the training time. This positive conclusion rests on the MUSHRA listening test, because the objective measures (MSE and mel cepstral distortion) showed no statistically significant improvement for either proposed model, and Conformer Base was significantly worse on MCD for some speakers. The authors conclude that both Conformer-based systems are viable alternatives for ultrasound-to-speech conversion, with a clear preference for the bi-LSTM variant.

Load-bearing premise

The claim of better perceptual quality rests on a single MUSHRA test with 27 non-native listeners rating just five utterances per speaker, and the overall p-value of 0.037 is borderline and not corrected for multiple comparisons.

Editorial extensions

If this is right

The bi-LSTM Conformer could serve as a drop-in replacement for the CNN mapping stage when perceptual naturalness is the goal in ultrasound-to-speech systems.
Conformer Base offers a faster-training alternative with similar perceived quality, which is useful for tuning speaker-specific models.
Because both proposed models train faster than the CNN baseline, they improve the feasibility of real-time silent speech interfaces.
The mismatch between subjective preference and objective metrics suggests that MSE and mel cepstral distortion alone may be insufficient for evaluating ultrasound-to-speech conversion quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The perceptual gain may come from better capture of spectral detail such as formant structure, which would mean that a stronger objective metric targeting formant accuracy could reconcile the subjective and objective results.
A larger listening test with native listeners and more utterances per speaker is still needed; if the reported p-value does not survive multiple-comparison correction, the positive conclusion would weaken considerably.
The architecture likely transfers to other articulatory-to-speech mappings, such as lip video or electromagnetic articulography, where both global sequence context and local feature extraction matter.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a straight empirical comparison, well reported, with artifacts released. The main positive finding—Conformer with bi-LSTM sounds more natural than a 2D-CNN baseline—rests on a single MUSHRA contrast (p=0.037) with no multiple-comparison correction, so treat it as provisional.

What's new: first application of Conformer to ultrasound-to-mel mapping, in two flavors (Base and bi-LSTM). On four speakers from UltraSuite-TaL80, both variants train faster than the baseline; Base is about 30% of baseline training time, bi-LSTM about 80%. MSE and MCD show no consistent gain; some speakers are significantly worse. The authors report these negatives honestly and don't massage them. Code and synthesized samples are on Zenodo, which makes the comparison reproducible.

Soft spots, in order: (1) The central claim of better naturalness comes from one MUSHRA with 27 non-native listeners and 20 utterances. The p=0.037 is borderline and, as far as the paper describes, uncorrected for the multiple pairs/speaker subsets examined. The per-speaker confidence intervals in Fig. 3 overlap. If the effect doesn't replicate under a mixed-effects model with listener and utterance random effects, the conclusion collapses to 'not worse, but faster to train.' (2) The claim about 'potential for real-time applications' is based on training time, not inference latency; that's a non-sequitur. (3) The input-sequence representation in Section 3.3 is hard to follow—treating each beam line as a time step gives a sequence length of 64 times the time dimension, and then 'feeding a 64x128 partition' is underspecified. (4) The visual spectrogram argument in Section 5 is speculative; the authors acknowledge it.

The citation pattern looks appropriate: they cite the prior Conformer+bi-LSTM work for accelerometer-based synthesis and the existing UTS baselines. No sign of self-citation padding beyond what's relevant.

Bottom line: this is a modest, careful contribution for the silent speech interface subfield. It doesn't open a new direction, but it's a legitimate architecture comparison with honest reporting. The main statistical claim needs a robustness check. I'd send it to peer review—it's the kind of paper that belongs in the record after a round of revisions—and I'd tell the authors to reanalyze the MUSHRA data and temper the real-time language.

Not a must-read for outsiders; useful for people working on UTS.

Referee Report

3 major / 6 minor

Summary. The paper compares two Conformer-based acoustic feature generators (Conformer Base and Conformer with bi-LSTM) against a standard 2D-CNN baseline for ultrasound-to-speech (UTS) conversion. Speaker-specific models map ultrasound tongue image frames to 80-dimensional mel spectrograms, which are then synthesized to audio with a HiFi-GAN vocoder. Experiments use four speakers from the UltraSuite-TaL80 corpus. Objective evaluation via MSE and mel-cepstral distortion shows no consistent improvement and, for several speakers, significant degradation. A MUSHRA listening test (27 listeners, 20 utterances) yields a statistically significant preference for the Conformer with bi-LSTM over the baseline (p=0.037), while Conformer Base matches the baseline; both proposed models train faster than the baseline. The paper concludes that both models are viable alternatives, with a clear preference for the bi-LSTM variant.

Significance. If the perceptual advantage is real, the paper provides useful evidence that Conformer-based architectures can be competitive with or better than CNN baselines in UTS, while being more parameter-efficient and faster to train. The authors are commendably transparent: they report that objective metrics show no improvement, they release code and synthesized samples, and they do not overclaim beyond the subjective result. The main significance hinges on a single MUSHRA p-value, so the statistical robustness of that result is load-bearing for the paper's central claim.

major comments (3)

[Section 4.4] The central claim that Conformer with bi-LSTM provides better perceptual quality rests entirely on a single MUSHRA comparison with p=0.037, and the paper does not state which statistical test was used, what the unit of analysis was (listener means, utterance means, or pooled ratings), or whether any multiple-comparison correction was applied. Three pairwise comparisons are reported (Base vs baseline, bi-LSTM vs baseline, bi-LSTM vs Base), so an uncorrected p=0.037 is borderline. The per-speaker 95% confidence intervals in Figure 3 overlap for every speaker, so the significant average result is not visible at the speaker level. The authors should report the exact test, provide a mixed-effects model with listener and utterance random effects, or apply a proper correction; otherwise the 'clear preference' is not fully supported.
[Section 3.3] The input sequence representation is ambiguous and appears internally inconsistent. The text states that each frame's 64 beam lines are treated as separate time steps, yielding a sequence length of 64 × (time dimension), and then says the network is fed with a '64×128 partition of the sequence'. If each ultrasound frame corresponds to 64 time steps, a 128-frame partition would span 8192 time steps, not a 64×128 grid. The relationship between input partitions, the flattening of the final linear layer output, and the one-frame-per-time-step mel output must be specified precisely. As written, the description is not sufficiently clear to reproduce the method or to determine whether the model is causal with respect to the output frame sequence.
[Sections 4.2 and 4.3] The objective results do not corroborate the subjective preference and in several cases significantly contradict it. Conformer Base has significantly higher MSE for speaker 03mn (p=0.026, Table 1) and significantly higher MCD for speakers 01fi, 03mn, and 04me (p≤0.001, Table 2); Conformer with bi-LSTM has significantly higher MCD for speaker 04me (p=0.002). The paper acknowledges the discrepancy but offers only a speculative visual-spectrogram explanation in Section 5. To make the 'viable alternatives' conclusion convincing despite the objective degradation, the authors should either report an analysis of why MSE/MCD fail to predict perceived quality (e.g., per-utterance correlation between objective and subjective scores) or temper the conclusion to 'no worse than baseline in perception, with computational benefits'.

minor comments (6)

[Section 2] The first sentence contains a word-order error: 'We conducted experiments our using' should read 'We conducted our experiments using'.
[Section 4.4] The phrase 'with apvalue of 0.037' is a typo; it should be 'with a p-value of 0.037'. Also, 'Conformer with biLSTM' appears in Section 4.4 while the rest of the paper uses 'bi-LSTM'; please standardize.
[Section 3.5] The phrase 'we chose the first variation' is unclear; it should specify that the VCTK V1 pre-trained HiFi-GAN checkpoint was used, rather than 'first variation'.
[Tables 1 and 2] The p-values are presumably from two-sided Mann-Whitney U tests, but this is not stated in the table captions. Please specify the test direction and whether any correction for testing four speakers was applied.
[Figure 3] The error bars are described only as '95% confidence intervals'; please state the bootstrap or analytical procedure used to compute them and whether they are across listeners or across utterances.
[Section 3.3] The phrase 'the exact same sandwich-style structure' is informal; use 'the same sandwich-style structure' and cite the original conformer description more precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a direct empirical comparison against a held-out test set and a fixed baseline, with no fitted parameter or self-citation chain defining the reported result.

full rationale

The paper makes no derivation claim that reduces to its own inputs. The reported outcomes are (1) objective MSE and MCD comparisons between two proposed conformer variants and a fixed 2D-CNN baseline on a held-out test set, and (2) a MUSHRA listening test comparing naturalness ratings of the generated utterances. The proposed models are trained on a train/development split, and all metrics are computed on utterances reserved before training; there is no parameter fitted to the test set and then renamed as a prediction. The baseline is taken from an open-access implementation cited as [13], but the comparison is a direct head-to-head measurement, not a derivation from the baseline's properties. The only self-citations (e.g., [9], [13], [23]) concern prior baseline architectures and the observation of speaker variability; these are contextual and not load-bearing for the central conclusion. The MUSHRA result (p=0.037) is a single, borderline significance test without multiple-comparison correction, which is a statistical robustness concern, not a circularity concern. The conclusion that 'Conformer with bi-LSTM' offers better perceptual quality is an empirical claim whose evidence strength may be questioned, but it is not circular by construction. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper makes no first-principles derivation and introduces no new entities. It relies on domain assumptions about the ultrasound-to-speech mapping, the fixed vocoder, the representativeness of the test material, and the generalizability of the listening test. The only free parameters listed are hand-chosen conformer hyperparameters, which are not fitted to the target data.

free parameters (1)

Conformer hyperparameters (encoder dim 256, 32 attention heads, conv kernel size 31, feedforward expansion factor 3…
Taken from prior conformer work or chosen without systematic tuning; not fitted to the test data, but they affect the comparison and are hand-selected design choices.

assumptions (4)

domain assumption Ultrasound tongue image frames contain sufficient articulatory information to reconstruct the corresponding acoustic speech signal.
Foundational premise of ultrasound-to-speech conversion; cited prior work supports this link, but the paper does not independently establish it.
domain assumption The pre-trained HiFi-GAN vocoder trained on VCTK generalizes to the mel spectrograms generated by all compared acoustic models for all four speakers.
The vocoder is fixed and not retrained; its behavior with different mel inputs could mask or amplify model differences, and this is not controlled.
domain assumption The 10 held-out test sentences per speaker, and the 5 randomly selected per speaker for listening tests, are representative of the speech material in ultrasound-to-speech conversion.
The test set is a contiguous block of shared read sentences, and only a subset is used for subjective evaluation; representativeness is assumed.
domain assumption MUSHRA naturalness ratings from 27 non-native English listeners generalize to the target user population of silent speech interfaces.
The positive conclusion depends on this subjective measurement being meaningful and transferable; no cross-validation or repeated listening test is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conformer-based Ultrasound-to-Speech Conversion." pith.science (2026). https://pith.science/paper/KXKXDQSL

@misc{pith2026250603831,
  author       = {Pith},
  title        = {Pith review of: Conformer-based Ultrasound-to-Speech Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXKXDQSL}},
  note         = {Machine review of arXiv:2506.03831}
}

read the original abstract

Deep neural networks have shown promising potential for ultrasound-to-speech conversion task towards Silent Speech Interfaces. In this work, we applied two Conformer-based DNN architectures (Base and one with bi-LSTM) for this task. Speaker-specific models were trained on the data of four speakers from the Ultrasuite-Tal80 dataset, while the generated mel spectrograms were synthesized to audio waveform using a HiFi-GAN vocoder. Compared to a standard 2D-CNN baseline, objective measurements (MSE and mel cepstral distortion) showed no statistically significant improvement for either model. However, a MUSHRA listening test revealed that Conformer with bi-LSTM provided better perceptual quality, while Conformer Base matched the performance of the baseline along with a 3x faster training time due to its simpler architecture. These findings suggest that Conformer-based models, especially the Conformer with bi-LSTM, offer a promising alternative to CNNs for ultrasound-to-speech conversion.

Figures

Figures reproduced from arXiv: 2506.03831 by the authors.

**Figure 1.** Schematic diagram of the ultrasound-to-speech system pipeline, showing the Conformer-based ultrasound-to-mel mapping and the HiFi-GAN vocoder for speech synthesis. Training set with development set flows through the solid arrow, while test set flows through the dashed arrow after the training phase was done (output shape of each layer is mentioned in parentheses). the synthesized speech samples using the HiFi-GAN ne… view at source ↗

**Figure 2.** a) Conformer Base; b)Conformer with bi-LSTM (output shape of each layer is mentioned in parentheses). 3.2. Baseline As a baseline, we trained a standard 2D-CNN architecture for each speaker to map UTIF to an 80-dimensional mel spectrogram using the open-access implementation2 provided by Csapo et al. [13]. The speaker-specific model consisted of ´ 4.09 million trainable parameters. 3.3. Conformer Base We implemented… view at source ↗

**Figure 3.** Results of the MUSHRA listening test with respect to naturalness, speaker by speaker (first two rows) and on average (bottom-right corner). The error bars show the 95% confidence intervals. 4.4. Subjective evaluation To assess the perceived naturalness of the synthesized speech, we also conducted a MUSHRA listening test [25]. Listeners rated the naturalness of each synthesized utterance relative to the corresponding… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

[1]

Conformer-based Ultrasound-to-Speech Conversion

Introduction Speech as a communication method is a vital part of human life, involving several body parts (tongue, lips, etc.) during production. However, for various reasons (speech disorders, environment, etc.), producing audible speech is sometimes not possible or desired. To offer an alternative communication method in these scenarios, Silent Speech I...

work page Pith review arXiv 2025
[2]

This dataset includes recording of the tongue movements using an ultrasound system called ”Micro” by Articulate Instruments Ltd

Dataset We conducted experiments our using four participants: two females (01fi, 02fe) and two males (03mn, 04me), selected from the UltraSuite-TaL80 corpus [21] 1. This dataset includes recording of the tongue movements using an ultrasound system called ”Micro” by Articulate Instruments Ltd. at a frame rate of 81.5 fps, simultaneously with audio recordin...

work page
[3]

Methodology 3.1. Experimental setup Both baseline and proposed models were trained for a maximum of 20 epochs with a batch size of 128, employing an early stopper (patience level of 3) based on development set MSE metric. All experiments were conducted on a server equipped with an Intel Core i7-4770 CPU (3.40 GHz, 8 cores, 16 threads) and an NVIDIA TITAN ...

work page
[4]

Results 4.1. Preliminary consideration Given the speaker-specific nature of the models and the variability in UTIF quality across subjects [23], a speaker-wise analysis of UTS systems is necessary. With four speakers, we examined objective measurements individually to identify speaker-wise trends. However, subjective evaluations were conducted both per sp...

work page
[5]

Conclusions and discussion In the ultrasound-to-speech area there is a need for more effective acoustic feature generation techniques. In this work we presented two models based on the Conformer architecture for mapping ultrasound tongue images to 80-dimensional mel spectrograms, and used the VCTK V1 variant of the HiFi-GAN vocoder for the speech synthesi...

work page
[6]

This work is dedicated to the memory of Dr

Acknowledgements This study was supported by the NRDI Office of the Hungarian Ministry of Innovation and Technology (grant TKP2021-NV A-09), and within the framework of the Artificial Intelligence National Laboratory Program (RRF-2.3.1-21-2022-00004) and the European Union’s HORIZON Research and Innovation Programme under grant agreement No 101120657, pro...

work page 2022
[7]

Towards optimizing electrode configurations for silent speech recognition based on high-density surface electromyography,

M. Zhu, H. Zhang, X. Wang, X. Wang, Z. Yang, C. Wang, O. W. Samuel, S. Chen, and G. Li, “Towards optimizing electrode configurations for silent speech recognition based on high-density surface electromyography,”Journal of Neural Engineering, vol. 18, no. 1, p. 016005, 2021

work page 2021
[8]

Silent speech interfaces: Non-invasive neuromuscular signal processing for assistive communication,

S. Khan, K. U. Ali, A. R. A. Qadri, and M. Rizvi, “Silent speech interfaces: Non-invasive neuromuscular signal processing for assistive communication,” inProceedings of ICAIC, 2025, pp. 1–6

work page 2025

Show all 31 references

[9]

Ultrasound-and MRI-based speech synthesis applying neural networks,

R. Trencs ´enyi and L. Czap, “Ultrasound-and MRI-based speech synthesis applying neural networks,” inProceedings of ICCC. IEEE, 2024, pp. 1–6

2024
[10]

Articulatory synthesis based on real-time magnetic resonance imaging data,

A. Toutios, T. Sorensen, K. Somandepalli, R. Alexander, and S. S. Narayanan, “Articulatory synthesis based on real-time magnetic resonance imaging data,” inProceedings of Interspeech, 2016, pp. 1492–1496

2016
[11]

Silent versus modal multi-speaker speech recognition from ultrasound and video,

M. S. Ribeiro, A. Eshky, K. Richmond, and S. Renals, “Silent versus modal multi-speaker speech recognition from ultrasound and video,” inProceedings of Interspeech, 2021, pp. 641–645

2021
[12]

Melder: The design and evaluation of a real-time silent speech recognizer for mobile devices,

L. Pandey and A. S. Arif, “Melder: The design and evaluation of a real-time silent speech recognizer for mobile devices,” inProceedings of the CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–23

2024
[13]

Statistical conversion of silent articulation into audible speech using full-covariance HMM,

T. Hueber and G. Bailly, “Statistical conversion of silent articulation into audible speech using full-covariance HMM,” Computer, Speech & Language, vol. 36, pp. 274–293, 2016

2016
[14]

Multi-modal co-learning for silent speech recognition based on ultrasound tongue images,

M. Guo, J. Wei, R. Zhang, Y . Zhao, and Q. Fang, “Multi-modal co-learning for silent speech recognition based on ultrasound tongue images,”Speech Communication, vol. 165, p. 103140, 2024

2024
[15]

DNN-based ultrasound-to-speech conversion for a silent speech interface,

T. G. Csap ´o, T. Gr ´osz, G. Gosztolya, L. T ´oth, and A. Mark ´o, “DNN-based ultrasound-to-speech conversion for a silent speech interface,” inProceedings of Interspeech, 2017, pp. 3672–3676

2017
[16]

Ultrasound-based silent speech interface using convolutional and recurrent neural networks,

E. Moliner and T. Csap ´o, “Ultrasound-based silent speech interface using convolutional and recurrent neural networks,”Acta Acustica united with Acustica, vol. 105, 07 2019

2019
[17]

Ultra2speech – A deep learning framework for formant frequency estimation and tracking from ultrasound tongue images,

P. Saha, Y . Liu, B. Gick, and S. Fels, “Ultra2speech – A deep learning framework for formant frequency estimation and tracking from ultrasound tongue images,” inProceedings of MICCAI, Lima, Peru, Oct 2020, pp. 473–482

2020
[18]

SottoV oce: An ultrasound imaging-based silent speech interaction using deep neural networks,

N. Kimura, M. Kono, and J. Rekimoto, “SottoV oce: An ultrasound imaging-based silent speech interaction using deep neural networks,” inProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–11

2019
[19]

Ultrasound-based articulatory-to-acoustic mapping with WaveGlow speech synthesis,

T. G. Csap ´o, C. Zaink ´o, L. T ´oth, G. Gosztolya, and A. Mark ´o, “Ultrasound-based articulatory-to-acoustic mapping with WaveGlow speech synthesis,” inProceedings of Interspeech, 2020, pp. 2727–2731

2020
[20]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inProceedings of NIPS, vol. 30. Curran Associates, Inc., 2017

2017
[21]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” inProceedings of NIPS, 2020

2020
[22]

Novel transformer networks for improved sequence labeling in genomics,

J. Clauwaert and W. Waegeman, “Novel transformer networks for improved sequence labeling in genomics,”IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 19, no. 1, pp. 97–106, 2022

2022
[23]

Streaming automatic speech recognition with the transformer model,

N. Moritz, T. Hori, and J. Le, “Streaming automatic speech recognition with the transformer model,” inProceedings of ICASSP. IEEE, 2020, pp. 6074–6078

2020
[24]

Decoding silent speech from high-density surface electromyographic data using transformer,

R. Song, X. Zhang, X. Chen, X. Chen, X. Chen, S. Yang, and E. Yin, “Decoding silent speech from high-density surface electromyographic data using transformer,”Biomedical Signal Processing and Control, vol. 80, p. 104298, 2023

2023
[25]

Conformer: Convolution-augmented transformer for speech recognition,

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Proceedings of Interspeech, 2020, pp. 5036–5040

2020
[26]

Speech synthesis from three-axis accelerometer signals using conformer-based deep neural network,

J. Kwon, J. Hwang, J. E. Sung, and C.-H. Im, “Speech synthesis from three-axis accelerometer signals using conformer-based deep neural network,”Computers in Biology and Medicine, vol. 182, p. 109090, 2024

2024
[27]

TaL: A synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos,

M. S. Ribeiro, J. Sanger, J.-X. X. Zhang, A. Eshky, A. Wrench, K. Richmond, and S. Renals, “TaL: A synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos,” in Proceedings of SLT, Shenzhen, China, 2021, pp. 1109–1116

2021
[28]

HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” Proceedings of NIPS, vol. 33, pp. 17 022–17 033, 2020

2020
[29]

Adaptation of tongue ultrasound-based silent speech interfaces using Spatial Transformer Networks,

L. T ´oth, A. Honarmandi Shandiz, G. Gosztolya, and T. G. Csap´o, “Adaptation of tongue ultrasound-based silent speech interfaces using Spatial Transformer Networks,” inProceedings of Interspeech, 2023, pp. 1169–1173

2023
[30]

On a test of whether one of two random variables is stochastically larger than the other,

H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”Annals of Mathematical Statistics, vol. 18, no. 1, pp. 50–60, 1947

1947
[31]

ITU-R Recommendation BS.1534: Method for the Subjective Assessment of Intermediate Audio Quality,

International Telecommunication Union, “ITU-R Recommendation BS.1534: Method for the Subjective Assessment of Intermediate Audio Quality,” Tech. Rep. BS.1534, 2001

2001

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Conformer-based Ultrasound-to-Speech Conversion

Introduction Speech as a communication method is a vital part of human life, involving several body parts (tongue, lips, etc.) during production. However, for various reasons (speech disorders, environment, etc.), producing audible speech is sometimes not possible or desired. To offer an alternative communication method in these scenarios, Silent Speech I...

work page Pith review arXiv 2025

[2] [2]

This dataset includes recording of the tongue movements using an ultrasound system called ”Micro” by Articulate Instruments Ltd

Dataset We conducted experiments our using four participants: two females (01fi, 02fe) and two males (03mn, 04me), selected from the UltraSuite-TaL80 corpus [21] 1. This dataset includes recording of the tongue movements using an ultrasound system called ”Micro” by Articulate Instruments Ltd. at a frame rate of 81.5 fps, simultaneously with audio recordin...

work page

[3] [3]

Methodology 3.1. Experimental setup Both baseline and proposed models were trained for a maximum of 20 epochs with a batch size of 128, employing an early stopper (patience level of 3) based on development set MSE metric. All experiments were conducted on a server equipped with an Intel Core i7-4770 CPU (3.40 GHz, 8 cores, 16 threads) and an NVIDIA TITAN ...

work page

[4] [4]

Results 4.1. Preliminary consideration Given the speaker-specific nature of the models and the variability in UTIF quality across subjects [23], a speaker-wise analysis of UTS systems is necessary. With four speakers, we examined objective measurements individually to identify speaker-wise trends. However, subjective evaluations were conducted both per sp...

work page

[5] [5]

Conclusions and discussion In the ultrasound-to-speech area there is a need for more effective acoustic feature generation techniques. In this work we presented two models based on the Conformer architecture for mapping ultrasound tongue images to 80-dimensional mel spectrograms, and used the VCTK V1 variant of the HiFi-GAN vocoder for the speech synthesi...

work page

[6] [6]

This work is dedicated to the memory of Dr

Acknowledgements This study was supported by the NRDI Office of the Hungarian Ministry of Innovation and Technology (grant TKP2021-NV A-09), and within the framework of the Artificial Intelligence National Laboratory Program (RRF-2.3.1-21-2022-00004) and the European Union’s HORIZON Research and Innovation Programme under grant agreement No 101120657, pro...

work page 2022

[7] [7]

Towards optimizing electrode configurations for silent speech recognition based on high-density surface electromyography,

M. Zhu, H. Zhang, X. Wang, X. Wang, Z. Yang, C. Wang, O. W. Samuel, S. Chen, and G. Li, “Towards optimizing electrode configurations for silent speech recognition based on high-density surface electromyography,”Journal of Neural Engineering, vol. 18, no. 1, p. 016005, 2021

work page 2021

[8] [8]

Silent speech interfaces: Non-invasive neuromuscular signal processing for assistive communication,

S. Khan, K. U. Ali, A. R. A. Qadri, and M. Rizvi, “Silent speech interfaces: Non-invasive neuromuscular signal processing for assistive communication,” inProceedings of ICAIC, 2025, pp. 1–6

work page 2025