{"id":"4e8afea2-f3a5-4ed5-a963-584f9dcb3a22","arxiv_id":"2412.18836","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A system that uses AV-HuBERT to recognize words from rtMRI vocal tract videos and VITS to speak them in a new voice reaches 15.18% WER on USC-TIMIT.","lead":"Researchers built a pipeline that turns real-time MRI scans of a person's vocal tract into the words they are saying, then feeds those words into a text-to-speech engine to produce audible speech in any chosen voice. The system reports far lower error rates than previous MRI-to-speech methods, a step toward silent speech interfaces.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthesized-speech WER is mostly a text-recognition + TTS score: without an oracle-text or same-ASR direct baseline, the claimed improvement over [1] is not established.","rationale":"I read the paper in good faith: the method is transparently a cascade from rtMRI to text and durations, then to speech via VITS. The core text-recognition results (Table I) are plausible and well-structured. However, the strongest claim in the abstract and conclusion is the 15.18% WER on synthesized speech and the 575.89% improvement over [1]. That claim rests on Table IV, whose conditions are not adequately controlled. The reader's concern about the unnamed ASR is real, but I think the deeper issue is that the synthesized-audio WER conflates three components: MRI-to-text recognition, MRI-to-duration alignment, and a separately trained clean-voice TTS. The comparison to [1] is not on the same task because [1] performs direct acoustic prediction from MRI without a text or language-model stage. A simple oracle-text ablation would disambiguate how much of the reported intelligibility comes from the TTS and how much from the MRI-based components. Without it, the paper's headline claim overstates what has been demonstrated, even though the underlying text-recognition system may be a solid contribution. This does not require changing the reader's CONDITIONAL verdict; it strengthens the conditions under which the paper should be accepted: the authors must name the ASR and report the oracle-text baseline and a same-ASR comparison to prior work. I do not see evidence of internal inconsistency, and the masked-articulator analysis is a genuine positive. The concern is about the support for the central speech-synthesis claim, not about the correctness of the text-prediction component.","tokens_in":8280,"tokens_out":6292,"duration_ms":61945,"concrete_test":"Fix the evaluation by transcribing, with one named ASR (e.g., Whisper large-v2) and the same decoding settings: (1) MRI2Speech's synthesized LJSpeech audio; (2) oracle ground-truth text passed through the same source-speaker duration predictor and LJSpeech VITS decoder; (3) original noisy USC-TIMIT audio; and (4) audio synthesized by Otani et al.'s system if available. Compare WER/CER. If condition (2) ≈ condition (1) ≈ 15%, the headline WER is dominated by text prediction and TTS, so the claimed improvement over direct acoustic synthesis is not demonstrated; if condition (2) is much lower and condition (1) tracks predicted-text WER, the same conclusion holds in weaker form. The ASR name and any language model must be identical across all conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing result is the 15.18% WER on synthesized speech (Table IV), but the synthesis path in Section II-C does not decode acoustics from articulation: it feeds the text predicted by AV-HuBERT and MRI-derived durations into a VITS model trained on clean LJSpeech. The acoustic content is therefore generated by a standard TTS from discrete phonemes; the only MRI-dependent information is which phoneme sequence is recognized and its duration. An ASR WER on that audio therefore measures a cascade (MRI→text→TTS) and is bounded from below by text-prediction errors; it does not measure articulatory-to-acoustic conversion. The comparison to Otani et al. [1] (102.6% WER) is not apples-to-apples, because that baseline predicts acoustic features directly and has no text/LM/TTS. No oracle-text baseline (e.g., ground-truth text + the same duration predictor and LJSpeech decoder) is reported, and the ASR used for Table IV is unnamed. Either issue can dominate the 575.89% relative-improvement claim: if the oracle-text condition already gives ~15% WER, the improvement is attributable to the TTS; if the chosen ASR is particularly favorable to clean synthetic voices, the number is not comparable to [1]'s ASR. This is a claims-support problem in Section IV-B, not an internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MRI2Speech, a two-stage approach for speech synthesis from real-time MRI (rtMRI) videos. In the first stage, a fine-tuned AV-HuBERT model predicts text from silent rtMRI frames using CTC loss. In the second stage, a VITS-based stochastic duration predictor is trained on noisy audio-text pairs to estimate speaker-specific phoneme durations. At inference, the predicted text and durations are fed to a speech decoder: for the speaker's own voice, a multi-speaker VITS is used, and for a novel clean voice, a single-speaker VITS trained on LJSpeech is used with durations from the source speaker. Experiments on USC-TIMIT and ASD1 report text-prediction CER/WER of 10.95%/14.38% and 11.14%/22.80% respectively, and a synthesized-speech WER of 15.18% on USC-TIMIT with the LJSpeech voice, which the authors claim is a 575.89% improvement over the previous SOTA (WER 102.6%). The paper also includes a masking analysis showing that internal articulators contribute more than lips to text prediction.","tokens_in":8537,"tokens_out":7146,"duration_ms":61982,"significance":"The text-recognition results are the strongest contribution of this paper: fine-tuning AV-HuBERT on rtMRI video yields single-digit-to-teen CERs that substantially improve over prior visual speech recognition from rtMRI, and the masking analysis provides an interesting finding about the model's reliance on non-lip articulators. The method is cleanly described and the demo page is a useful resource. However, the paper's central synthesis claim is not supported by the reported evaluation. The synthesized-speech WER is a cascade score of (a) MRI-to-text recognition, (b) duration prediction, and (c) a standard text-to-speech decoder, and the comparison to the direct acoustic-feature prediction of Otani et al. [1] is not apples-to-apples. If reframed as a silent speech interface with a TTS back-end, the contribution is meaningful; as presented, the claim of 'articulatory-to-speech synthesis' with a 575.89% improvement over SOTA overreaches the evidence.","major_comments":[{"comment":"The WER reported in Table IV is computed on audio generated by a VITS decoder from predicted text and predicted durations; it is not a direct measure of articulatory-to-acoustic conversion. The ASR model used to compute the CER/WER is never specified. Without an oracle-text baseline (ground-truth text plus the same duration predictor and the same VITS decoder), the 15.18% WER cannot be attributed to the MRI-to-speech pipeline, and the comparison to Otani et al. [1] is not controlled because [1] uses a different synthesis paradigm and likely a different ASR. This leaves the central claim of a 575.89% improvement over SOTA unsupported.","section":"IV-B, Table IV"},{"comment":"The synthesis path feeds the phoneme sequence predicted by AV-HuBERT and the predicted durations into a VITS model trained on clean LJSpeech; the acoustic waveform is generated by a standard text-to-speech decoder. Consequently, the only MRI-dependent information entering the acoustics is the discrete phoneme identity and duration. This is a cascade of MRI-to-text followed by text-to-speech, not an articulatory-to-acoustic mapping. The paper should either reframe its contribution as a silent speech interface with a TTS back-end or provide an evaluation that isolates the articulatory contribution to the acoustic output.","section":"II-C"},{"comment":"The comparison to Pandey and Arif [19] is confounded by mismatched test protocols. The authors report their own results averaged over leave-one-speaker-out folds, while the numbers for [19] are taken from the original paper and are based on only two randomly selected unseen speakers. The test splits are not the same, and no re-evaluation of [19] on the authors' splits is reported. This undermines the quantitative claim of a 65.58% CER / 59.47% WER reduction over [19].","section":"IV-A, Table III"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any CER/WER result. Given the high variability across speakers (e.g., Table III CER ranges from 7.04% to 14.66% on ASD1), the improvement claims need variance estimates and, where appropriate, per-speaker paired significance tests (e.g., Wilcoxon signed-rank test across speakers).","section":"General (Tables I, III, IV)"}],"minor_comments":[{"comment":"The text lists 'Electrophysiology [5], Electrophysiology [5]' with a duplicated reference; the second occurrence should be a different modality or removed.","section":"Introduction"},{"comment":"The abstract states 'Our method achieves a 15.18% Word Error Rate (WER) on the USC-TIMIT MRI corpus' without specifying that this is the synthesized-speech WER (Table IV) while the text-prediction WER is 14.38% (Table I). Please clarify which result is being cited.","section":"Abstract and IV-B"},{"comment":"The phrase 'vocal fold articulators' is used to refer to structures such as lips, tongue, velum, and glottis; these are vocal tract articulators, not the vocal folds. Please correct the terminology throughout the section and in the Table II caption.","section":"IV-A"},{"comment":"The reported '575.89% improvement' uses the non-standard formula (old−new)/new; the standard relative reduction is (old−new)/old ≈ 85.2%. Please report both or clarify the definition.","section":"Conclusion"},{"comment":"Equation (1) is not precisely stated: the left-hand side is p(k|e_t), but the right-hand side sums over alignments π of the full sequence e^T. Please rewrite in standard CTC notation, e.g., p(k|x) = sum over π in β^{-1}(k) of product_t p(π_t | e_t).","section":"II-A, Eq. (1)"},{"comment":"The word 'vellum' should be 'velum'.","section":"IV-A"},{"comment":"The masking experiment in Table II lacks procedural details, such as how the lip region is cropped or masked and how the mask is defined. Please specify the preprocessing steps to make the experiment reproducible.","section":"II-C, Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The text-recognition part of the paper is a solid and useful contribution, and the authors are to be commended for releasing a demo page. However, the synthesis evaluation is the central selling point of the paper and is currently not rigorous: the ASR is unnamed, no oracle-text baseline is provided, and the comparison to [1] is not controlled. These issues are addressable with additional experiments or a careful reframing, so I recommend major revision rather than rejection. Please also ask the authors to verify that the numbers for [19] are computed on matched test splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s the short version. The text-recognition route is the real contribution, and the numbers in Table I are good: 10.95% CER / 14.38% WER on USC-TIMIT, with credible cross-speaker results in Table III. The masking experiment is also a nice touch. If the paper were only about predicting text from rtMRI, I’d have little to argue with.\n\nThe soft spot is Section IV-B. The synthesized-speech evaluation does not do what the abstract claims. The system is a cascade: AV-HuBERT predicts text, the duration predictor estimates phoneme timing, and a VITS decoder trained on clean LJSpeech does the actual acoustic generation. So the 15.18% WER on synthesized audio is substantially a score for the text recognizer plus a TTS front-end, not a measure of articulatory-to-acoustic conversion. The comparison with Otani et al. is not apples-to-apples, because that baseline predicts acoustic features directly with no language model and no TTS. What is missing is an oracle-text condition: feed ground-truth text through the same duration predictor and LJSpeech decoder, and report WER on that. Without it, you cannot tell whether the improvement comes from the text recognizer or from the TTS. Also, the ASR used to transcribe the synthesized speech is never named. That matters because ASR choice can dominate the reported WER, and it makes the headline number unfalsifiable as reported. The “575.89% improvement” in the conclusion is a relative-difference figure that will mislead; the meaningful statement is an 85% WER reduction, and only if the comparison holds.\n\nNone of this is fatal to the paper’s main insight. The text-prediction results stand on their own, and the approach is a genuine departure from prior acoustic-prediction work. The flaws are all in the claims-support around the synthesis evaluation, and they are fixable: release details of the ASR, add an oracle-text baseline, and either rerun [1] under the same conditions or temper the comparison. I’d send this to reviewers. It is a solid empirical contribution to the silent speech interface community, and the evaluation issues are the kind a careful referee can pin down.","headline":"The text-recognition result from rtMRI is a real advance; the 15.18% WER headline on synthesized speech is oversold, but the paper deserves a serious referee.","tokens_in":9118,"tokens_out":2332,"would_cite":true,"duration_ms":21361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speech can be synthesized from real-time MRI video of the vocal tract by first decoding the MRI into text and then aligning that text in a clean voice, reaching a 15.18% WER on USC-TIMIT.","keywords":["real-time MRI","speech synthesis","articulatory-to-acoustic conversion","silent speech interface","self-supervised audio-visual model","text prediction from video","duration prediction","voice conversion"],"falsifier":"Rerun the synthesized-speech evaluation on the released samples with a publicly named automatic speech recognizer and with human listeners on the same USC-TIMIT test split; if either measurement moves far from the reported 15.18% WER, or fails to beat the prior system [1] by a wide margin, the central claim of intelligibility improvement would be overturned.","tokens_in":8064,"feed_emoji":"🗣️","tokens_out":8478,"duration_ms":70469,"temperature":0.7,"pith_summary":"MRI2Speech claims that intelligible speech can be synthesized from real-time MRI video of the vocal tract by first predicting text rather than by directly mapping MRI frames to noisy acoustic features. The argument is that ground-truth audio recorded inside an MRI scanner contaminates every mel-spectrogram with scanner noise, so prior two-stage systems produce speech with poor intelligibility (WER up to 102.6%). The paper's system instead fine-tunes a multimodal self-supervised model (AV-HuBERT) to recognize text from silent rtMRI frames, uses a separate duration predictor trained on audio-text pairs to align phonemes to the video, and then feeds the text and durations into a normalizing-flow-based speech decoder. On USC-TIMIT, the predicted text reaches 10.95% CER/14.38% WER, and speech synthesized in a clean LJSpeech voice reaches 9.27% CER/15.18% WER, a large drop from the prior state of the art. If correct, this makes real-time MRI a usable signal for silent-speech interfaces and for studying how internal articulators encode speech.","feed_headline":"MRI scans of the vocal tract become speech at 15% WER","feed_subtitle":"By predicting text and timing from silent MRI, it synthesizes clean speech in a new voice at far lower error rates.","key_machinery":"The load-bearing mechanism is a two-stage decomposition: a fine-tuned AV-HuBERT transformer (a self-supervised audio-visual speech model with its audio encoder removed, fine-tuned with CTC loss to emit text from silent rtMRI video), a VITS-style stochastic duration predictor (a conditional variational autoencoder with normalizing flow and monotonic alignment search, trained on noisy audio-text pairs, whose frozen weights predict per-phoneme durations for a given speaker), and a HiFi-GAN-based decoder trained on clean LJSpeech audio, into which the source speaker's durations are injected. The crucial move is that the duration predictor decouples what is said (text from MRI) from when it is said (duration from audio-text alignment) and who says it (the clean-voice decoder).","core_discovery":"The central discovery is that the correlation between rtMRI articulatory video and intelligible speech is best exploited through a text intermediate: a pre-trained audio-visual self-supervised model can be adapted to read the vocal tract from MRI sequences and output a CTC transcription, while a stochastic duration predictor learned from noisy audio and text supplies per-speaker timing. Because the acoustic decoder for the final voice is no longer trained on noisy MRI audio (the clean-voice decoder only borrows source-speaker durations), the scanner noise is bypassed rather than suppressed, and the synthesized speech is both more intelligible and voice-flexible. The paper reports this as 15.18% WER on USC-TIMIT for synthesized speech in a novel clean voice, versus 102.6% WER for the prior method, as well as credible zero-shot text prediction on unseen speakers of both USC-TIMIT and ASD1.","pith_inferences":["A direct extension the paper does not test is porting the same text-prediction stage to non-MRI articulatory sensors such as ultrasound tongue imaging or electromagnetic articulography; because the intermediate is text, only the video encoder would need re-training.","The masking study suggests a finer-grained articulatory atlas is possible: masking individual regions (velum, tongue dorsum, glottis) rather than the whole lip could reveal which articulators carry which phonemic cues.","Since the final voice is supplied by a clean text-to-speech decoder, end-to-end error is approximately text-recognition error plus alignment error, so the most direct path to further gains is improving the text recognizer or adding a language model, not replacing the decoder.","A fixed, named automatic speech recognizer and a small human-listening study would make the 15.18% WER comparable across future systems; the paper currently leaves the recognizer unnamed, so cross-paper comparisons are provisional."],"forward_implications":["Real-time MRI video alone, without clean target audio, is sufficient to drive intelligible speech synthesis: 10.95% CER/14.38% WER on predicted text and 9.27% CER/15.18% WER on synthesized speech in a clean voice on USC-TIMIT.","The duration predictor makes synthesis voice-flexible: alignment learned from a noisy MRI audio source can be applied to a decoder trained on clean LJSpeech audio, producing a novel voice without re-recording the content.","The method generalizes to unseen speakers, with average 13.56% CER/17.06% WER on USC-TIMIT and 11.75% CER/21.07% WER on ASD1, across different languages and MRI hardware.","Masking the lip region lowers WER by 25.33% relative to lip-only input, indicating the model reads internal articulators (velum, tongue, glottis) rather than relying on lip movements alone.","The large gap from the prior state of the art (102.6% WER) implies that the earlier strategy of predicting mel-spectrograms from noisy MRI audio was the main bottleneck, not the MRI signal itself."],"supporting_citations":[{"why":"Supplies the pre-trained AV-HuBERT model that is fine-tuned for text prediction from silent rtMRI video.","marker":"[23]"},{"why":"Defines the CTC loss used to train text prediction from frame-level MRI video.","marker":"[25]"},{"why":"Defines the VITS conditional variational autoencoder whose duration predictor supplies speaker-specific phoneme alignment.","marker":"[28]"},{"why":"Supplies the clean LJSpeech voice used for the novel-voice decoder and the 9.27% CER/15.18% WER evaluation.","marker":"[29]"},{"why":"Provides the USC-TIMIT rtMRI corpus, the primary dataset for the reported text and synthesized-speech error rates.","marker":"[20]"},{"why":"Provides the ASD1 French rtMRI dataset used to test cross-database and unseen-speaker generalization.","marker":"[21]"},{"why":"Provides the prior state-of-the-art baseline with WER 102.6% that the paper's synthesized-speech results are compared against.","marker":"[1]"},{"why":"Provides prior text-recognition baselines on unseen speakers that the paper's zero-shot results improve upon.","marker":"[19]"}],"fun_headline_variants":["MRI speech synthesis bypasses scanner noise via text","From silent MRI to clean speech: AV-HuBERT predicts text","15% WER: MRI vocal tract to speech in new voices","Text intermediate turns MRI articulatory video into speech","Zero-shot speech from rtMRI: text first, then duration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the word error rates obtained with the paper's (unspecified) automatic speech recognizer faithfully measure the intelligibility of the synthesized speech; if that recognizer is biased toward the clean LJSpeech voice or the TIMIT vocabulary, the reported 15.18% WER may not hold for other voices or listeners.","fun_headline_variants_meta":{"raw":{"variants":["MRI speech synthesis bypasses scanner noise via text","From silent MRI to clean speech: AV-HuBERT predicts text","15% WER: MRI vocal tract to speech in new voices","Text intermediate turns MRI articulatory video into speech","Zero-shot speech from rtMRI: text first, then duration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1297,"prompt_tokens":906,"completion_tokens":391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":522,"tokens_out":391,"duration_ms":3959,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:25:30.803309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the synthesized-speech evaluation on the released samples with a publicly named automatic speech recognizer and with human listeners on the same USC-TIMIT test split; if either measurement moves far from the reported 15.18% WER, or fails to beat the prior system [1] by a wide margin, the central claim of intelligibility improvement would be overturned.","supporting_citations":[{"cited_title":"Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,","cited_arxiv_id":null,"evidence_quote":"Defines the VITS conditional variational autoencoder whose duration predictor supplies speaker-specific phoneme alignment."},{"cited_title":"Real- time magnetic resonance imaging and electromagnetic articulography database for speech production research (tc),","cited_arxiv_id":null,"evidence_quote":"Provides the USC-TIMIT rtMRI corpus, the primary dataset for the reported text and synthesized-speech error rates."},{"cited_title":"Multimodal dataset of real-time 2d and static 3d mri of healthy french speakers,","cited_arxiv_id":null,"evidence_quote":"Provides the ASD1 French rtMRI dataset used to test cross-database and unseen-speaker generalization."},{"cited_title":"Speech Synthesis from Articulatory Movements Recorded by Real-time MRI,","cited_arxiv_id":null,"evidence_quote":"Provides the prior state-of-the-art baseline with WER 102.6% that the paper's synthesized-speech results are compared against."},{"cited_title":"Silent speech and emotion recognition from vocal tract shape dynamics in real-time mri,","cited_arxiv_id":null,"evidence_quote":"Provides prior text-recognition baselines on unseen speakers that the paper's zero-shot results improve upon."}],"review_version":1}