{"id":"907bc7bd-757f-469d-be03-96bfa3b741c8","arxiv_id":"2411.09211","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A diffusion-based model classifies short EEG/EMG segments into 15 viseme classes, and an LSTM trained on the test sentences themselves reconstructs the spoken sentences.","lead":"This paper decodes lip-movement-related viseme classes from EEG and EMG signals during spoken sentences, then reconstructs the sentences using a language model. The work is a candidate assistive communication method for people who cannot speak, but the sentence reconstruction step is trained on the exact test sentences, so the headline result is weaker than it appears.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported sentence reconstruction is not evidence of neural decoding because the LSTM reconstructor is trained on the ground-truth viseme sequences of the exact 50 test sentences, so it can memorize the output set.","rationale":"I agree with the reader's weak-assumption analysis. The central claim is that spoken sentences can be reconstructed from short EEG/EMG segments; the only evidence for sentence-level reconstruction is Section III-B, which depends entirely on an LSTM explicitly trained on the ground-truth viseme sequences of the 50 test sentences. This makes the evaluation circular: the reconstructor has already seen the full set of possible sentence outputs and their exact viseme alignments, so it can map even partially correct viseme predictions to the correct sentence by retrieval. The claim would only be supported if the LSTM were trained on a disjoint sentence set or evaluated on unseen sentences. I also note the table/reported-numbers mismatch in Section III-A (the EEG+EMG numbers quoted are identical to EEG-only numbers and do not match Table I), which further weakens confidence in the quantitative reporting, but the reconstruction protocol is the decisive issue. The paper does contain useful components — a diffusion-based viseme decoder, phoneme-to-viseme condensation, and a real recorded dataset — but the headline result as evaluated does not establish the claimed capability. I therefore see no reason to change the reader's REJECT verdict.","tokens_in":6586,"tokens_out":3466,"duration_ms":35020,"concrete_test":"Leave-one-sentence-out evaluation of the reconstruction stage: for each of the 50 test sentences, train the LSTM on viseme sequences from the other 49 test sentences plus the 424 training sentences (with their ground-truth visemes), then decode the held-out sentence from the predicted viseme sequence and record whether the output matches. If sentence-level accuracy is near zero or substantially below the reported 'all,' the original result is an artifact of training on the test sentences. A useful control is to feed shuffled predicted viseme sequences through the same LSTM; if it still maps them to correct sentences, memorization is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the evaluation protocol for the headline result. Section II-D states the LSTM reconstructor 'was trained with ground truth viseme sequences to ensure that the predicted viseme sequences could accurately guide the reconstruction of the target sentences,' and Fig. 2 states it 'was trained using the original viseme sequences for 50 predefined sentences.' Since the same 50 sentences are the test set, the reconstruction stage is trained on the ground-truth outputs of the exact test items. With only 50 possible sentences, the LSTM can learn a closed-set mapping from any noisy viseme sequence to the nearest training sequence, so 'all were accurately inferred within the pre-defined sentence set' (Section III-B) does not demonstrate that the neural signal contains sufficient information for sentence reconstruction. The underlying viseme classifier achieves at best 31.8% top-1 accuracy over 15 classes (Table I), so the apparent sentence-level success is attributable to the memorizing reconstructor rather than to neural decoding. This is an internal-validity flaw, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a framework for decoding visemes from non-invasive EEG/EMG signals during overt speech and reconstructing full spoken sentences using a vision-BCI pipeline. The viseme classifier uses a diffusion-based model with a KAN classifier, and the reported classification performance is modest, with top-1 accuracy around 27–34% over 15 classes. The paper's central claim is that the predicted viseme sequences for 50 test sentences were all accurately reconstructed into the original sentences using an LSTM model.","tokens_in":6801,"tokens_out":4824,"duration_ms":49214,"significance":"If the sentence-level reconstruction result were valid, the work would be a meaningful step toward practical non-invasive BCI communication and dynamic visual speech output. However, the central claim is not supported by the current evidence because the reconstruction LSTM is trained on the ground-truth viseme sequences of the exact 50 test sentences, making the reported perfect reconstruction a memorization artifact rather than a demonstration of neural decoding. The paper also contains an internal inconsistency between the results text and Table I. The viseme classification accuracy itself is low, and the paper provides no baseline or statistical validation for the sentence-level claim, so the main contribution is not established.","major_comments":[{"comment":"The LSTM reconstruction model is trained with the ground-truth viseme sequences of the 50 sentences that are also used for testing; Fig. 2 explicitly states that the model was trained using the original viseme sequences for 50 predefined sentences. With only 50 possible output sentences, the model can learn a closed-set mapping from any noisy viseme sequence to the nearest training sequence, so the statement that all were accurately inferred within the pre-defined sentence set does not demonstrate that the neural signal contains sufficient information for sentence reconstruction. The authors must retrain the LSTM on a disjoint set of sentences, such as the 424 training sentences, and report held-out reconstruction accuracy.","section":"Section II-D and Fig. 2, Section III-B"},{"comment":"The text reporting the EEG-only results gives exactly the same numeric values as the EEG+EMG results; for example, the 64 ms EEG-only values are reported as 27.33, 54.09, 22.96, and 75.03, which are the EEG-only values in Table I, while the EEG+EMG row of Table I lists 31.81, 56.44, 27.73, and 78.38. This inconsistency makes it impossible to verify the claim that incorporating EMG improved decoding performance, and the authors should correct the numeric values and clarify which condition each result refers to.","section":"Section III-A"}],"minor_comments":[{"comment":"The phrase 'These are beginning to explore the representation of visual speech intentions from neural signals' is grammatically awkward and should be reworded for clarity.","section":"Introduction"},{"comment":"The LSTM architecture details, including hidden size, number of layers, training epochs, and the loss function, are not provided; these details are needed for reproducibility.","section":"Section II-D"},{"comment":"The caption states that the LSTM model was trained using the original viseme sequences for 50 predefined sentences; this should be flagged as a methodological concern rather than presented as an accepted design choice.","section":"Fig. 2"},{"comment":"The text appears to contain duplicated numerical paragraphs, making the comparison between EEG+EMG and EEG-only conditions difficult to follow.","section":"Section III-A"},{"comment":"The table does not report standard deviations or significance tests for the classification metrics, so it is unclear whether the differences between conditions are reliable.","section":"Table I"}],"recommendation":"reject","confidential_remarks":"The paper is within the journal's scope, but the evaluation protocol for the headline sentence-reconstruction result is circular. The authors should be encouraged to rerun the reconstruction stage with a proper train/test split (e.g., training the LSTM on the 424 training sentences only) and to correct the numerical inconsistencies in Section III-A; if held-out accuracy is strong, the work could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the viseme-decoding part of this paper is real but modest; the headline sentence-reconstruction result is not evidence of neural decoding, because the LSTM reconstructor was trained on the ground-truth viseme sequences of the exact 50 test sentences. Treat the 'all sentences reconstructed' claim as a closed-set memorization artifact.\n\nWhat's actually new: the authors apply their earlier Diff-E diffusion model to viseme classification from EEG/EMG, with segment lengths of 64-256 ms. The classification results are above chance (31.8% top-1 over 15 classes, vs 6.7% chance), and there is a small, consistent advantage when EMG is added. The dataset construction—474 MOCHA-TIMIT sentences with phoneme alignment and mapping to 15 viseme classes—is reasonable and reproducible in principle, though no code or data are released.\n\nWhere it falls apart: the sentence reconstruction (Section III-B) is the headline result, and it is circular. Section II-D and Fig. 2 say the LSTM was trained with ground-truth viseme sequences for the 50 predefined sentences; those 50 sentences are exactly the test set. So the reconstructor has memorized the input-output mapping for every possible target sentence. With only 50 sentences, even a poor viseme classifier can be mapped to the nearest memorized sequence. The result 'all were accurately inferred' therefore does not show that the neural signal contains enough information for sentence reconstruction. This is a load-bearing flaw, not a quibble.\n\nThere are two more issues. Section III-A repeats the same numeric results for EEG+EMG and EEG-only in text, despite Table I showing different numbers; that looks like a copy-paste error. And the abstract promises 'dynamic visual outputs' and reconstructed lip movements, but the paper only outputs viseme labels and reconstructed sentences; no visual generation is shown.\n\nCredit where due: the paper is clearly written and the viseme classification setup is honest in its numbers. The diffusion-based decoding approach is a legitimate application of existing work. But the central claim is unsupported.\n\nBottom line: this is not a paper I would send to a serious referee in its current form. The reconstruction experiment should either be removed or re-run with a held-out sentence set, and the reporting errors fixed. If the authors resubmit with those changes, the viseme classification part could make a short workshop-level contribution. Right now, it reads as a closed-set evaluation dressed up as communication.","headline":"The viseme classification is real but modest; the headline sentence-reconstruction result is a circular closed-set artifact and the paper's central claim does not hold.","tokens_in":7282,"tokens_out":6063,"would_cite":false,"duration_ms":63352,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that short EEG/EMG segments can be decoded into lip-movement units (visemes) and reassembled into complete spoken sentences, at least within a predefined sentence set.","keywords":["brain-computer interface","EEG","EMG","viseme decoding","diffusion model","speech reconstruction","neural communication","overt speech"],"falsifier":"Evaluate the full pipeline on spoken sentences whose viseme sequences appear in no LSTM training data: decode EEG/EMG segments, run the LSTM, and compare the output text with the recorded audio. If open-set reconstruction accuracy falls toward chance, the reported sentence-level success is an artifact of the fixed 50-sentence test set.","tokens_in":6432,"feed_emoji":"🧠","tokens_out":9113,"duration_ms":97151,"temperature":0.7,"pith_summary":"The paper aims to show that a whole spoken sentence can be recovered from non-invasive neural signals by decoding lip-movement units rather than full phonemes. The authors segment EEG and EMG recordings of overt speech from a single subject into short windows, map each window to one of 15 viseme classes with a diffusion-based decoder, and then use an LSTM to turn the predicted viseme sequence back into the original sentence. They report that on their 50 test sentences every reconstruction was accurate within that predefined sentence set. The value of the claim, if it holds, is that BCI output could move from fragmented text to continuous visual speech—lip movements that could drive avatars or talking faces. The paper itself notes that the LSTM was trained on the ground-truth viseme sequences of exactly those 50 sentences, so the reconstruction stage is tied to the fixed test set.","feed_headline":"Brain signals decoded into complete spoken sentences","feed_subtitle":"A diffusion model reads short neural windows as lip-movement units, then an LSTM maps them to a fixed 50-sentence set.","key_machinery":"Visemes are the visual counterpart of phonemes: the 15 condensed lip-shape categories used as the decoding target, which compress the larger phoneme inventory and make classification tractable. The decoding pipeline is a time-conditional U-Net built on denoising diffusion probabilistic models (DDPM), with a conditional autoencoder (encoder $E_\\phi$, decoder $D_\\psi$) whose latent features are skip-connected into the diffusion layers to compensate for information loss; a channel-attention layer weights EEG/EMG channels before classification, and the final classifier is fine-tuned with Kolmogorov-Arnold Networks. The predicted viseme labels are arranged as a one-dimensional sequence and passed to a long short-term memory (LSTM) network, trained on ground-truth viseme sequences of the 50 predefined sentences, which produces the reconstructed sentence.","core_discovery":"On its own terms, the paper's central discovery is that speech-related EEG/EMG can be decoded into a small set of viseme classes at short time scales (64–256 ms), and that the resulting label sequences can be reassembled into complete sentences. With EEG and EMG combined, top-1 viseme accuracy reaches 33.77% and AUC reaches 81.53%; with EEG alone the numbers are lower but still well above chance, and 64 ms segments perform comparably to longer ones. From these decoded sequences, the authors reconstruct all 50 predefined test sentences correctly. The authors interpret this as evidence that a non-invasive, dynamic neural communication channel is feasible—one that captures rapidly changing lip movements during natural speech attempts and presents them as visual output rather than static text.","pith_inferences":["A natural extension the authors leave untested is open-vocabulary reconstruction: train the LSTM on viseme sequences from the 424-sentence training set and evaluate on sentences the model has never seen; this would separate genuine decoding from memorization of the 50 test sentences.","Because the EMG channels sit on facial muscles, part of the EEG+EMG advantage may come from muscle activity leaking into nearby EEG electrodes; removing EMG-contaminated EEG epochs would isolate the purely neural signal.","If the decoded viseme stream is accurate enough, off-the-shelf lip-sync networks could turn the predicted sequences directly into talking-face video, bypassing text entirely—a direction the paper mentions as future work."],"forward_implications":["Short 64 ms segments decode visemes at nearly the same accuracy as 256 ms segments, so sentence-level decoding need not wait for the full utterance to finish.","Combining EMG with EEG improves viseme decoding, indicating that muscle activity around the mouth carries articulatory information that can supplement noisy brain signals.","EEG-only decoding remains above chance, suggesting the same viseme approach could extend to imagined speech where no overt muscle signal exists.","A coarse 15-viseme vocabulary is enough to reconstruct predefined sentences, implying that finer phoneme-level decoding may not be necessary for whole-sentence visual communication.","The decoded viseme sequences are a natural input for computer-vision lip-sync and talking-face systems, linking BCI output to realistic dynamic avatars."],"supporting_citations":[{"why":"It defines the phoneme-to-viseme mapping that condenses speech sounds into the 15 viseme classes used as decoding targets.","marker":"[22]"},{"why":"It provides the forced aligner used to segment recorded audio into phoneme intervals for labelling each EEG/EMG segment.","marker":"[23]"},{"why":"It supplies the EEGLAB preprocessing pipeline used for filtering and preparing the neural signals.","marker":"[26]"},{"why":"It supplies the diffusion-based EEG decoding backbone (Diff-E) that the viseme decoder adapts.","marker":"[27]"},{"why":"It defines denoising diffusion probabilistic models, the generative component at the core of the time-conditional U-Net.","marker":"[28]"},{"why":"It provides the conditional autoencoder whose latent features are skip-connected into the diffusion layers to compensate for information loss.","marker":"[29]"},{"why":"It supplies the Kolmogorov-Arnold Networks used to fine-tune the final viseme classifier.","marker":"[30]"},{"why":"It supports the 30–499 Hz bandpass choice as the range associated with speech and articulatory information.","marker":"[7]"}],"fun_headline_variants":["Dynamic neural comm: EEG/EMG visemes rebuild 50 sentences","Brain signals to lip movements to full sentences in 64ms","Non-invasive EEG/EMG decodes visemes into complete speech","Neural signals reconstruct sentences via fast viseme mapping","Capturing lip movements from brain signals to speak fully"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reconstruction result rests on the assumption that training the LSTM on the ground-truth viseme sequences of the exact 50 test sentences, from a single subject, is a valid way to measure sentence reconstruction rather than simply letting the model memorize the target sentences.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic neural comm: EEG/EMG visemes rebuild 50 sentences","Brain signals to lip movements to full sentences in 64ms","Non-invasive EEG/EMG decodes visemes into complete speech","Neural signals reconstruct sentences via fast viseme mapping","Capturing lip movements from brain signals to speak fully"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1728,"prompt_tokens":846,"completion_tokens":882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":797}},"tokens_in":462,"tokens_out":882,"duration_ms":9915,"temperature":1.0,"reasoning_tokens":797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:53:37.354438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the full pipeline on spoken sentences whose viseme sequences appear in no LSTM training data: decode EEG/EMG segments, run the LSTM, and compare the output text with the recorded audio. If open-set reconstruction accuracy falls toward chance, the reported sentence-level success is an artifact of the fixed 50-sentence test set.","supporting_citations":[{"cited_title":"Phoneme–to–viseme mapping for visual speech recognition,","cited_arxiv_id":null,"evidence_quote":"It defines the phoneme-to-viseme mapping that condenses speech sounds into the 15 viseme classes used as decoding targets."},{"cited_title":"Montreal forced aligner: Trainable text–speech alignment using kaldi,","cited_arxiv_id":null,"evidence_quote":"It provides the forced aligner used to segment recorded audio into phoneme intervals for labelling each EEG/EMG segment."},{"cited_title":"EEGLAB: An open source toolbox for analysis of single–trial EEG dynamics including independent component analysis,","cited_arxiv_id":null,"evidence_quote":"It supplies the EEGLAB preprocessing pipeline used for filtering and preparing the neural signals."},{"cited_title":"Denoising diffusion probabilistic models,","cited_arxiv_id":null,"evidence_quote":"It defines denoising diffusion probabilistic models, the generative component at the core of the time-conditional U-Net."}],"review_version":1}