{"id":"89cefaf9-e253-41d5-9a35-a2632957f6d5","arxiv_id":"2501.14790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EEG signals can be decoded into visemes (lip-shape units) and reconstructed into talking-face animations, with best performance for overt speech and much weaker for imagined speech.","lead":"Researchers decoded lip movements from brain waves recorded while people spoke, mouthed, or imagined speech, using a diffusion model to map EEG signals to visemes, the building blocks of mouth shapes. The work points toward brain-controlled talking faces for communication aids, though the signals may partly reflect face muscle activity rather than pure brain activity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"30–499 Hz band likely contains facial EMG; reported overt/mimed viseme decoding may reflect muscle artifacts rather than neural speech signals.","rationale":"I independently reviewed the paper and identified the same load-bearing concern as the reader: the 30–499 Hz bandpass filter allows EMG contamination to masquerade as neural signal. The paper's strongest quantitative results (overt and mimed speech) coincide with the conditions where articulatory muscle activity is present, while the no-articulation condition (imagined speech) shows a dramatic drop in performance. No artifact rejection or control analysis is provided, so the claim that the model decodes 'visual speech intentions from speech-related non-invasive brain signals' is not yet supported. This is the single most critical assumption because if it fails, the neuroprosthesis framing loses its foundation for the better-performing conditions. Secondary weaknesses, such as the absence of quantitative metrics for the talking-face reconstruction and the dimensional inconsistency in the loss equations, also undermine the presentation but would not by themselves invalidate the core viseme classification claim. The reader's CONDITIONAL verdict is appropriate: the paper should be accepted only after the authors demonstrate neural origin via artifact control and provide reconstruction metrics. No verdict change is needed.","tokens_in":8514,"tokens_out":3660,"duration_ms":41477,"concrete_test":"Re-run the Table I experiment with the same model and data under three conditions: (i) replace the 30–499 Hz bandpass with a 1–30 Hz low-pass, (ii) apply ICA-based muscle artifact rejection to the 30–499 Hz data before training, and (iii) add a control condition where participants move their lips silently without phonation. If overt/mimed viseme error rates rise to roughly the imagined-speech level (≈78%) under (i) or (ii), or if the control condition yields high-confidence viseme classifications, the reported decoding is EMG-mediated rather than neural.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that visemes are decoded from speech-related brain signals (abstract; Section I). The load-bearing assumption is stated in Section II-B2: 'We applied a 5th-order Butterworth bandpass filter in the range of 30–499 Hz containing speech-related information in brain signals.' No ICA or other muscle-artifact rejection is reported. The 30–499 Hz band on the scalp is dominated by facial and ocular EMG, not by neural high-gamma, which is weak and non-invasive EEG. The observed performance gradient — overt VER 34.07%, mimed 48.33%, imagined 77.96% — is exactly what would be expected if articulatory muscle activity drives the strongest results: overt speech involves full articulation, mimed speech involves lip movements without phonation, and imagined speech has no overt articulator movement. If the overt and mimed results are EMG-mediated, the neuroprosthesis claim collapses for the two conditions with the best numbers; the only condition without overt muscle confound (imagined) drops to roughly 22% accuracy, just above chance (6.7%). The paper provides no control condition, no low-frequency comparison, and no analysis of the discriminative spatial/temporal features. This is an unresolved artifact-control problem, not an accusation of misconduct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Park et al. propose a diffusion-based viseme decoding framework for non-invasive EEG, with 15 viseme classes collapsed from 39 phonemes. EEG was recorded from three participants during overt, mimed, and imagined speech, and from ten repeated sentences. The model combines a DDPM, a conditional autoencoder, and a KAN classifier; it is evaluated on isolated trials (Table I) and then fine-tuned on sentence-level segments to reconstruct talking-face sequences. The paper reports lower viseme error rates than three baseline networks in overt and mimed speech, and moderate gains in imagined speech, and argues this supports dynamic neural communication and speech neuroprosthesis.","tokens_in":1290,"tokens_out":1492,"duration_ms":75541,"significance":"If the results are artifact-free, the contribution is useful: the 15-class viseme vocabulary follows an established phonetic-to-visual mapping, the overt and mimed decoding metrics improve over EEGNet, DeepConvNet, and ShallowConvNet on held-out trials, and the sentence-level fine-tuning recipe with forced alignment is practical. The central claim, however, depends on the high-frequency EEG band actually carrying neural speech information; the manuscript provides no artifact-rejection or control analysis to rule out facial EMG, and the sentence-level reconstruction claim is not quantified. With those gaps filled, the work would be a meaningful step; as written, the evidence is not yet sufficient for the neuroprosthesis claim.","major_comments":[{"comment":"The 30-499 Hz band-pass filter is applied without any stated muscle-artifact rejection (e.g., ICA, SSP, or regression). In scalp EEG, this band is dominated by facial and ocular EMG, especially during overt and mimed articulation; the observed ordering of viseme error rates (overt 34.07%, mimed 48.33%, imagined 77.96%) is exactly the ordering of articulator involvement. Because the imagined condition, the only one without overt vocal-tract movement, has F1 = 21.04% and AUC = 66.67%, the current numbers cannot distinguish neural decoding from EMG decoding. Please add control analyses: artifact rejection followed by re-training, decoding in a low-frequency band, spatial topography of discriminative features, and a no-articulation control condition. These analyses are necessary to support the claim of decoding speech-related brain signals and the neuroprosthesis conclusion.","section":"II-B2 / Table I"},{"comment":"Equation (2) defines L_CAE as ||L_DDPM(theta) - D_psi(E_phi(x0), xhat_theta(xt,t))||. Since L_DDPM in Eq. (1) is a scalar norm, subtracting it from the decoder output, a vector or tensor, is dimensionally inconsistent; as written, the objective is undefined. Please state the intended loss (likely ||x0 - D_psi(...)|| or a similar reconstruction error) and confirm that the implementation matches the corrected equation. Without a well-defined loss, the training procedure is not reproducible.","section":"II-A2, Eq. (2)"},{"comment":"The claim of continuous sentence-level reconstruction is not supported by any quantitative metric. The text states that the ten predefined sentences were inferred 'in their complete form,' but no viseme sequence accuracy, edit distance, frame-level agreement, or comparison with a chance baseline is reported, and no held-out sentence evaluation is described. Since the model is fine-tuned on phoneme-aligned segments of the same repeated sentences, this is also vulnerable to sentence-specific memorization. Please report per-sentence decoding metrics on held-out sentences and clarify whether any sentence was excluded from fine-tuning.","section":"III-C / Fig. 5"}],"minor_comments":[{"comment":"The phrase 'with mostly focusing on decoding short intentions' is ungrammatical; consider revising to 'with most work focusing on decoding short intentions'.","section":"Abstract"},{"comment":"The dataset description should clarify whether the 8,100 isolated trials and 7,629 sentence-level trials are totals across the three conditions and three participants, and per-subject trial counts should be reported.","section":"II-B1"},{"comment":"The table reports only means and standard deviations across subjects; add subject-wise results and a statistical comparison (e.g., permutation test or paired test) against each baseline and against chance level.","section":"Table I"},{"comment":"The sentence 'C_rho, is composed of Kolmogorov-Arnold Networks' has a stray comma; the intended statement is that C_rho is composed of KAN layers.","section":"II-A3"},{"comment":"Please state explicitly whether any EOG/EMG artifact rejection (ICA, SSP, or regression of EOG channels) was performed; the current text only describes band-pass and notch filtering.","section":"II-B2"},{"comment":"For Figures 3 and 4, specify which model layer and which subject pool generated the t-SNE projection and confusion matrices, and clarify whether the matrices are aggregated over all participants.","section":"III-B"},{"comment":"No code or data availability statement is provided; releasing the preprocessing and training code would materially improve reproducibility, especially given the unusual loss definition in Eq. (2).","section":"General"}],"recommendation":"major_revision","confidential_remarks":"I do not question the authors' good faith; the EMG concern is a standard artifact-control issue. If the requested control analyses cannot be produced with the existing data, the overt and mimed speech claims should be withdrawn and only the imagined-speech results retained. As it stands, I would not accept the manuscript without major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real attempt to decode visemes from EEG and drive a talking face, but the signal-processing choice (30–499 Hz band) makes the headline numbers hard to trust as neural. The overt and mimed results are exactly in the order you'd expect if the classifier is reading lip-muscle EMG through the scalp. What is new: no one has done viseme-level decoding from non-invasive EEG with diffusion and then reconstructed a talking face, as far as the cited literature goes. That's a sensible extension of the authors' own Diff-E work. The t-SNE showing phoneme encodings clustered into viseme groups is a nice qualitative sanity check, and the classification consistently beats EEGNet/DeepConvNet/ShallowConvNet on all three conditions, with reasonable subject variance reported. The soft spots are substantial. First, the 30–499 Hz bandpass filter with no ICA or artifact rejection is a known recipe for facial EMG contamination. Overt speech has full articulation, mimed speech has lip movement, imagined speech does not—and the error rates go 34%, 48%, 78% in exactly that order. The imagined-speech accuracy is barely above chance (22% vs 6.7%), so the practical neuroprosthesis claim rests on the two conditions most likely to be contaminated. The authors don't provide any control analysis—no low-frequency comparison, no spatial/temporal feature analysis, no EMG electrode check. Second, the sentence-level reconstruction claim is qualitative. There are no quantitative metrics for the talking-face output, no user study, no consistency score. Figure 5 is a cartoon. The claim that predefined sentences were 'successfully inferred in their complete form' needs numbers. Third, the loss equations have a unit problem. Eq. (1) is a scalar norm, Eq. (2) subtracts that scalar from D_psi, which is a tensor. Either the notation is sloppy or there's a real bug in the code. Fourth, the sentence data: each of ten sentences was repeated five times, and the random 8:2 split presumably operates on trials, so the same sentence can appear in both training and test. That could inflate the fine-tuning numbers if the model memorizes sentence-level patterns. Net: this is a promising proof-of-concept with an unresolved artifact problem. It deserves referee time so the authors can address the EMG confound and provide reconstruction metrics, but I wouldn't build on the current numbers.","headline":"Viseme decoding from EEG with diffusion and talking-face reconstruction is a new combo, but the 30–499 Hz band makes the strong overt/mimed results look EMG-driven until proven otherwise.","tokens_in":9289,"tokens_out":2006,"would_cite":false,"duration_ms":19470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that viseme-level decoding from non-invasive EEG can reconstruct coherent lip movements and talking-face animations, bridging brain signals to dynamic visual communication.","keywords":["brain-computer interface","EEG decoding","viseme","speech neuroprosthesis","diffusion model","talking face reconstruction","imagined speech","neural communication"],"falsifier":"Run the same decoding pipeline on EEG where lip motion is completely suppressed, such as in fully paralyzed users or with instructed articulation prevented, and compare performance against models trained after EMG-artifact removal; if the 30-499 Hz band loses its discriminative content in those conditions, the neuroprosthesis claim based on overt and mimed speech would not survive.","tokens_in":8273,"feed_emoji":"🧠","tokens_out":5002,"duration_ms":47604,"temperature":0.7,"pith_summary":"The paper tries to show that a person's intended lip movements during speech can be read from non-invasive scalp EEG and turned into a moving talking face. The authors map 39 English phonemes onto 15 viseme classes, train a diffusion-based model to classify each short EEG segment as one of those visemes, and then chain the predictions together to reconstruct continuous sentences. They report viseme error rates of 34% for overt speech, 48% for mimed speech, and 78% for imagined speech, with the overt and mimed results beating three standard EEG classifiers. The stated goal is to move speech brain-computer interfaces from decoding isolated words to generating dynamic, face-to-face visual communication, with applications for people who cannot speak.","feed_headline":"Reading lip shapes from EEG enables a brain-driven talking face","feed_subtitle":"A diffusion model maps 39 phonemes to 15 visemes and reconstructs continuous sentences from non-invasive scalp signals.","key_machinery":"The load-bearing mechanism is a denoising diffusion probabilistic model (DDPM) trained to reconstruct the original EEG segment rather than predict the added noise, paired with a conditional autoencoder whose decoder is skip-connected to the diffusion denoiser and whose encoder output is passed to an adaptive classifier. Channel attention weights EEG channels by relevance. The 39 phonemes are consolidated into 15 viseme classes based on shared lip shapes, and a forced-alignment tool segments continuous recorded speech into phoneme-level EEG windows for fine-tuning, allowing single-trial viseme classifiers to be adapted to continuous sentence decoding.","core_discovery":"In the authors' own framing, the discovery is that viseme-level decoding from speech-related EEG is feasible, and that a denoising-diffusion autoencoder can extract the lip-shape information hidden in noisy non-invasive signals. Using 15 viseme classes defined by facial-animation groupings, the model learns phoneme-to-viseme representations that cluster in latent space by lip shape, and then, after fine-tuning on phoneme-segmented sentence EEG, decodes continuous viseme sequences that are reconstructed into synchronized talking-face animations. The authors interpret this as bridging brain signals and dynamic visual interfaces, going beyond word- or sentence-level text decoding toward unconstrained reconstruction from small speech fragments.","pith_inferences":["If the 30-499 Hz band turned out to contain muscle artifact, the strong overt and mimed results would be explained by lip-movement EMG, so the imagined-speech results are the cleaner test of neural origin.","The approach could be extended to other languages by replacing the phoneme-to-viseme mapping table, since the viseme classes are articulation-based rather than language-specific.","Combining viseme decoding with an audio vocoder or text-to-speech could produce multimodal output from the same latent viseme sequence.","A direct neural-versus-artifact control, such as decoding from EEG recorded with lip motion absent or after EMG regression, would settle whether the high-frequency signal carries phonetic content."],"forward_implications":["Viseme-level decoding gives a granular output unit finer than words, so speech BCIs could reconstruct sentences not in a fixed vocabulary.","Overt and mimed speech decode at 34% and 48% viseme error rate, enough to drive realistic talking-face animations for face-to-face neural communication.","Even imagined speech, at 78% viseme error rate, shows classification well above chance and may support silent communication.","The same diffusion framework can be fine-tuned from isolated trials to continuous sentence data, pointing a path from offline classification to online dynamic output generation."],"supporting_citations":[{"why":"Provides the diffusion-based backbone architecture and training details for decoding speech-related EEG.","marker":"[24]"},{"why":"Supplies the denoising diffusion probabilistic model formalism used as the core generative mechanism.","marker":"[25]"},{"why":"Motivates the conditional autoencoder with skip connections to the diffusion model to correct information loss.","marker":"[26]"},{"why":"Segments recorded audio into phoneme-level timing to create sentence-level EEG training windows.","marker":"[28]"},{"why":"Defines the viseme-to-phoneme mapping and facial-animation parameter framework used to consolidate the 15 viseme classes.","marker":"[29]"},{"why":"Supports the claim that EEG contains speech-related information in the 30-499 Hz band.","marker":"[7]"},{"why":"Serves as one of the standard EEG decoding baselines that the proposed method outperforms.","marker":"[34]"},{"why":"Provides two additional established convolutional baselines for comparison against the proposed framework.","marker":"[35]"}],"fun_headline_variants":["Diffusion model decodes visemes from EEG to animate talking faces","EEG signals encode lip shapes for talking face reconstruction","Brain-driven talking face emerges from EEG viseme decoding","Non-invasive EEG yields coherent lip movements for speech neuroprosthesis","Viseme-based neural decode creates dynamic talking faces from EEG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing belief is that the 30-499 Hz EEG band holds genuine speech-related neural information, so that the strong overt and mimed accuracies come from brain activity rather than from facial-muscle electrical contamination.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion model decodes visemes from EEG to animate talking faces","EEG signals encode lip shapes for talking face reconstruction","Brain-driven talking face emerges from EEG viseme decoding","Non-invasive EEG yields coherent lip movements for speech neuroprosthesis","Viseme-based neural decode creates dynamic talking faces from EEG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2727,"prompt_tokens":852,"completion_tokens":1875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":1802}},"tokens_in":468,"tokens_out":1875,"duration_ms":16476,"temperature":1.0,"reasoning_tokens":1802,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:22:24.362160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same decoding pipeline on EEG where lip motion is completely suppressed, such as in fully paralyzed users or with instructed articulation prevented, and compare performance against models trained after EMG-artifact removal; if the 30-499 Hz band loses its discriminative content in those conditions, the neuroprosthesis claim based on overt and mimed speech would not survive.","supporting_citations":[{"cited_title":"Diff-E: Diffusion-based learning for decoding imagined speech EEG,","cited_arxiv_id":null,"evidence_quote":"Provides the diffusion-based backbone architecture and training details for decoding speech-related EEG."},{"cited_title":"Denoising diffusion probabilistic models,","cited_arxiv_id":null,"evidence_quote":"Supplies the denoising diffusion probabilistic model formalism used as the core generative mechanism."},{"cited_title":"Unsupervised representation learning from pre-trained diffusion probabilistic models,","cited_arxiv_id":null,"evidence_quote":"Motivates the conditional autoencoder with skip connections to the diffusion model to correct information loss."},{"cited_title":"Montreal forced aligner: Trainable text-speech alignment using kaldi,","cited_arxiv_id":null,"evidence_quote":"Segments recorded audio into phoneme-level timing to create sentence-level EEG training windows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the viseme-to-phoneme mapping and facial-animation parameter framework used to consolidate the 15 viseme classes."},{"cited_title":"Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that EEG contains speech-related information in the 30-499 Hz band."},{"cited_title":"EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces,","cited_arxiv_id":null,"evidence_quote":"Serves as one of the standard EEG decoding baselines that the proposed method outperforms."},{"cited_title":"Deep learning with convolutional neural networks for EEG decoding and visualization,","cited_arxiv_id":null,"evidence_quote":"Provides two additional established convolutional baselines for comparison against the proposed framework."}],"review_version":1}