{"id":"4826010c-6360-409f-b5ad-a0722ca93908","arxiv_id":"2502.00547","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Milmer reports 96.72% four-class accuracy on DEAP by fusing facial frames selected via multiple instance learning with EEG tokens in a transformer, but the evaluation protocol is not fully described.","lead":"A new framework called Milmer combines facial expression images and EEG signals for emotion recognition, using multiple instance learning to select key frames and a transformer to fuse the two modalities. On the DEAP dataset, the authors report 96.72% accuracy for four-class emotion classification, but the experimental protocol has several gaps that may inflate this number.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.72% DEAP-4 result is not reproducible as reported: Section 4 never specifies a trial-disjoint split, and Table 4 appears to select the 64-token setting from the same test set used for the headline metric.","rationale":"The reader's weakest-assumption analysis points at test-set hyperparameter selection, and I agree. My stress-test adds that the paper does not even specify the level at which data are split (subject, trial, or window), which is more fundamental than the multiple-comparison issue in Table 4. The paper also states that subject-dependent results are high while subject-independent shows a gap, indicating the headline may come from the easier protocol; prior comparisons in Table 1 are drawn from papers with different protocols. The ablation tables do not report variability such as standard deviation or confidence intervals, and the conclusion begins with an unrelated IoT credibility paragraph, which suggests rushed assembly. None of this is an attack on the authors; it means the evidence in the manuscript does not support 96.72% as a robust state-of-the-art claim. The architecture (Swin + MIL + cross-attention) is plausible, and the code link is a positive, but the missing protocol and apparent test-set tuning make the central claim unverified. I therefore keep the reader's rejection unchanged.","tokens_in":18678,"tokens_out":5090,"duration_ms":54452,"concrete_test":"Inspect the released repository's data loader and training script. Verify (1) train/test splits are trial-disjoint rather than window-random, (2) the 64-token choice in Table 4 was selected on a validation split not used for final metrics, and (3) which protocol underlies Table 1. Then rerun the full pipeline under 5-fold trial-disjoint cross-validation with token size selected inside each training fold; if the mean test accuracy is below 96.72% by more than 2 percentage points or the exact result is not reproducible, the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Milmer reaches 96.72% accuracy and 96.71% F1 on DEAP-4, outperforming all prior methods. For this to be trustworthy, the evaluation must be unbiased. Section 4.1 describes 3-second windowing and pairing each window with 10 facial frames, but it never states how windows are split into train/validation/test, whether the split is by subject, by trial, or random, or whether Table 1 reports the subject-dependent or subject-independent protocol. The text says subject-dependent results are much better than subject-independent but gives no numeric protocol for either. Table 4 then varies the visual-token output size from 196 down to 16 and reports 96.72% for CA 64; no held-out validation set is mentioned, so the headline number appears to be the best of several configurations evaluated on test data. If windows from the same 60-second trial appear in both training and test partitions, temporal correlation between adjacent windows can inflate accuracy. Participant 11 is also excluded without a robustness check. The reported result therefore lacks a demonstrated unbiased evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Milmer, a multimodal emotion recognition framework that fuses EEG signals and facial video frames on the DEAP dataset. The method uses a fine-tuned Swin Transformer for visual features, attention-based multiple instance learning with top-K frame selection, a cross-attention module that compresses the visual tokens, and a transformer fusion module with modality-type and position embeddings. The paper reports accuracy and F1 results for DEAP-2, DEAP-3, and DEAP-4, with the headline claim of 96.72% accuracy and 96.71% F1 on the four-class DEAP-4 task, together with ablation studies for the proposed modules.","tokens_in":18945,"tokens_out":4255,"duration_ms":44333,"significance":"If the reported accuracies were obtained under an unbiased evaluation protocol, the contribution would be significant for multimodal affective computing: the DEAP-4 result is more than 13 points above the strongest cited comparator, and the MIL plus cross-attention design addresses a real gap in using temporal facial information. The authors also promise to release code, which would aid reproducibility. However, the paper does not specify the train/validation/test split, and Table 4 indicates that the visual token size was selected as the best among several configurations without a described held-out validation set. Until the evaluation protocol is clarified and the selection procedure is shown to be unbiased, the numerical claims cannot be credited.","major_comments":[{"comment":"The train/validation/test split is never specified. The text states that data are segmented into 3-second windows and that each EEG window is paired with 10 facial frames, but it does not say whether the split is by subject, by trial, or random, nor which split is used for the headline numbers in Table 1. Because adjacent windows from the same 60-second trial are temporally correlated, a random window-level split can substantially inflate classification accuracy. This omission is load-bearing because the abstract's 96.72% claim and all subsequent comparisons are uninterpretable without knowing the protocol.","section":"Section 4.1"},{"comment":"The visual token output size N=64 is selected as the best among seven tested values (196, 147, 128, 96, 64, 32, 16), and the paper does not mention a separate validation set for this selection. If the test set was used to choose N, the reported 96.72% is optimistically biased. The authors should select hyperparameters on a held-out validation split and report test-set accuracy only for the chosen configuration, along with the validation-based selection criterion.","section":"Table 4"},{"comment":"Participant 11 is excluded because three facial expression videos are missing, and no robustness analysis is provided. Only 22 of the 32 DEAP participants have facial data, so excluding Participant 11 leaves 21 subjects; the paper does not assess how this exclusion affects the results, for example by rerunning with the remaining facial videos of Participant 11 or by reporting per-participant accuracy. Since the dataset is small, this exclusion could materially change the reported numbers.","section":"Section 4.1"},{"comment":"The ablation results are reported without standard deviations, repeated runs, or a statement of the underlying split, so it is impossible to judge whether differences such as 96.72% versus 95.99% (Table 4) or 94.46% versus 96.72% (Table 6) are significant. Given that the model has multiple stochastic components, single-run accuracies without error bars are insufficient support for the claim that the full configuration is the best.","section":"Tables 5 and 6"}],"minor_comments":[{"comment":"The conclusion begins with an unrelated paragraph about assessing the credibility of Internet of Things devices, which appears to be a copy-paste error. This paragraph should be removed and replaced with a conclusion that accurately summarizes the emotion recognition work presented in the paper.","section":"Section 5"},{"comment":"The sentence \"Our code are available at ...\" should be corrected to \"Our code is available at ...\".","section":"Abstract and Section 6"},{"comment":"The labels \"HAL V\" and \"LAL V\" contain an awkward space and are inconsistent with the compact notation \"HAHV\" and \"LAHV\" used in the same equation; the formatting should be made uniform.","section":"Equation (5)"},{"comment":"The sentence beginning \"Each modality contributes unique strengths...\" is repeated verbatim in the introduction; one copy should be deleted.","section":"Section 1"},{"comment":"Reference [13] cites ViLT, a vision-and-language pretraining model, but the surrounding text discusses multimodal emotion recognition fusion; a more directly relevant citation is needed.","section":"References"},{"comment":"There is a typo in the first row: \"Gupta wt al.\" should be \"Gupta et al.\".","section":"Table 1"},{"comment":"The phrase \"experiments conduct under subject-dependent\" should be \"experiments conducted under subject-dependent\".","section":"Section 4.2"}],"recommendation":"reject","confidential_remarks":"The evaluation protocol is the central problem: the paper never states the split, and Table 4 appears to select the visual token size on the test set. These are load-bearing issues that invalidate the headline accuracy as reported. The unrelated IoT paragraph in the conclusion also raises concerns about the care with which the manuscript was assembled. I would be willing to reconsider a substantially revised version that describes a proper subject-disjoint split, uses a validation set for hyperparameter selection, reports error bars, and addresses the Participant 11 exclusion, but the current submission does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable engineering contribution, but the headline number is not verifiable as reported. The evaluation protocol is underspecified, and Table 4 is consistent with selecting the visual token count on the test set.\n\nThe parts that work: the MIL variant selects top-K instances instead of pooling, which is a sensible fit for this problem; the cross-attention compression to balance visual and EEG tokens is well motivated; and the ablations show each component moves accuracy by a few points. The framework is a legitimate extension, not a conceptual breakthrough, but it is more careful than the typical concatenation baseline.\n\nThe soft spots are in the evaluation. Section 4.1 never says whether the split is by subject, trial, or random window. That matters a lot: 3-second windows from the same 60-second trial are highly correlated, and a random split can inflate accuracy. The text mentions subject-dependent and subject-independent experiments, but the main tables report one number without saying which protocol it came from. Table 4 varies the visual token size across seven values and reports accuracy for each, with the best (96.72%) at 64; no validation set is mentioned, so the headline number appears to be the best test-set configuration. That is selection bias. Excluding Participant 11 without any robustness check is also a problem, because only 22 of 32 participants have facial data and the exclusion shrinks an already small sample. None of these are fatal to the method, but they are fatal to the credibility of the headline result.\n\nOne more thing: the conclusion contains a paragraph about IoT device credibility that has nothing to do with emotion recognition. Probably a copy-paste error. Minor, but it signals a lack of final proofreading.\n\nThe take-away: the framework is plausible and the ablations are informative, but as written the paper does not show that 96.72% is an unbiased estimate. It deserves a serious referee because the problem is important and the method is reasonable, but the revision needs to be major: specify the splitting protocol, report the subject-independent numbers, pick hyperparameters on a validation set or with nested CV, and provide confidence intervals. I would not cite the headline result until that is done.","headline":"Plausible engineering combination, but the 96.72% DEAP-4 result is not verifiable as reported because the evaluation protocol is underspecified and the token-size selection in Table 4 appears to be done on the test set.","tokens_in":19467,"tokens_out":2216,"would_cite":false,"duration_ms":22115,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EEG plus facial video pushes four-class emotion accuracy to 96.72%.","keywords":["multimodal emotion recognition","multiple instance learning","EEG signal processing","facial expression recognition","transformer fusion","cross-attention mechanism","DEAP dataset","valence-arousal classification"],"falsifier":"Run the DEAP-4 experiment with a separate validation split used to pick the 64-token visual size, include all participants who have facial videos, and report accuracy under a strict subject-independent protocol; if the number drops materially below 96.72% or below the 83.20% of the cited prior method under the same protocol, the headline claim is protocol-bound rather than a property of the fusion design.","tokens_in":63,"feed_emoji":"🧠","tokens_out":6376,"duration_ms":115705,"temperature":0.7,"pith_summary":"The paper argues that multimodal emotion recognition fails when the two modalities are fused too crudely or when facial expressions are compressed into one frame, and that both defects can be fixed with a transformer-based pipeline. It introduces Milmer, which pairs filtered EEG segments with a bag of facial frames, selects the most informative frames by attention-based multiple instance learning, compresses their visual tokens with cross-attention, and lets a transformer fuse the two modalities. On the DEAP dataset the framework reports 96.72% accuracy and 96.71% F1 in the four-class valence-arousal task, above the best prior result it cites (83.20% accuracy). The authors also report the strongest accuracy in the two- and three-class tasks and show in ablations that each module contributes.","feed_headline":"EEG plus facial video pushes four-class emotion accuracy to 96.72%","feed_subtitle":"A bag-of-frames transformer fuses EEG with facial video, beating the previous four-class accuracy by 13 points.","key_machinery":"The machinery is a three-module pipeline: an EEG preprocessing stage (1-50 Hz bandpass filter plus Fast ICA artifact removal), a facial feature extraction and balancing stage, and a transformer fusion stage. The distinctive object is the attention-based multiple instance learning step: a bag of facial frames is weighted by AMIL attention scores, and the top-K frames are kept instead of pooling them to one vector, preserving temporal information for fusion. A cross-attention layer then compresses the many visual tokens to N learnable queries, balancing token counts between modalities before a transformer encoder with modal-type and position embeddings pools a CLS token for classification. All three modules are shown in ablation to contribute, with the MIL plus cross-attention combination giving the largest single jump.","core_discovery":"Milmer's core claim is that emotion classification improves when facial expression videos are treated as bags of frames rather than single images, and when the resulting visual tokens are balanced against EEG tokens before fusion. For each 3-second segment, ten evenly spaced facial frames are encoded by a fine-tuned Swin Transformer, attention weights pick the top-K frames, and cross-attention compresses the M visual tokens into N learnable queries sized to match the EEG token count. A transformer fusion module with modal-type, positional, and CLS embeddings then produces the four-class valence-arousal prediction. The paper reports 96.72% accuracy and 96.71% F1 on DEAP-4, exceeding the best compared method by 13.52 accuracy points, and similar advantages on DEAP-3 and DEAP-2.","pith_inferences":["A strict subject-independent rerun is the natural check: the paper's own confusion-matrix analysis shows a gap, so the headline 96.72% should be read as subject-dependent performance.","The balancing idea is modular: the same cross-attention token-compression could apply to any pair of modalities with unequal token counts, such as audio-text fusion.","Testing on other synchronized video-physiology emotion datasets would show whether the gains are specific to the DEAP setup or structural to the bag-of-frames approach."],"forward_implications":["Four-class DEAP emotion recognition can be treated as a nearly saturated benchmark under the paper's subject-dependent evaluation, with accuracy over 96%.","Treating a video segment as a bag of frames, rather than picking one representative image, raises recognition accuracy even when the same backbone is used.","Fusion quality depends on balancing the token counts of heterogeneous modalities: shrinking visual tokens toward the EEG token count improves results up to a point, then hurts.","Transformer-based fusion outperforms concatenation and DeepCCA-style correlation fusion for EEG-plus-face emotion recognition by more than seven accuracy points.","A drop in accuracy under subject-independent evaluation shows the framework gains from intra-subject pattern learning, not only from general emotion features."],"supporting_citations":[{"why":"Supplies the DEAP dataset, the benchmark on which all comparisons and ablations are run.","marker":"[1]"},{"why":"Provides the Swin Transformer backbone that extracts visual tokens from facial frames.","marker":"[14]"},{"why":"Introduces attention-based multiple instance learning, which the top-K selection step adapts.","marker":"[30]"},{"why":"Defines the transformer and attention primitives used in cross-attention and fusion.","marker":"[33]"},{"why":"Recent multimodal contrastive-learning baseline whose 83.20% accuracy and 84.10% F1 on DEAP-4 Milmer claims to beat.","marker":"[42]"},{"why":"Supplies the Fast ICA implementation used for EEG artifact removal.","marker":"[35]"}],"fun_headline_variants":["MIL + EEG + facial video: 96.72% four-class emotion accuracy","Bag-of-frames fusion of EEG and faces: 96.72% accuracy","Transformer MIL fusion reaches 96.72% accuracy","Multimodal MIL with cross-attention: 96.72% on DEAP-4","EEG and facial video via MIL: 96.72% accuracy, 13.5 gain"],"cache_read_input_tokens":21632,"weakest_assumption_plain":"The reported 96.72% depends on the evaluation being unbiased when hyperparameters such as the visual token size are chosen, and on excluding Participant 11 not materially changing the result.","fun_headline_variants_meta":{"raw":{"variants":["MIL + EEG + facial video: 96.72% four-class emotion accuracy","Bag-of-frames fusion of EEG and faces: 96.72% accuracy","Transformer MIL fusion reaches 96.72% accuracy","Multimodal MIL with cross-attention: 96.72% on DEAP-4","EEG and facial video via MIL: 96.72% accuracy, 13.5 gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3945,"prompt_tokens":953,"completion_tokens":2992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2884}},"tokens_in":569,"tokens_out":2992,"duration_ms":20180,"temperature":1.0,"reasoning_tokens":2884,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:33:59.112569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DEAP-4 experiment with a separate validation split used to pick the 64-token visual size, include all participants who have facial videos, and report accuracy under a strict subject-independent protocol; if the number drops materially below 96.72% or below the 83.20% of the cited prior method under the same protocol, the headline claim is protocol-bound rather than a property of the fusion design.","supporting_citations":[{"cited_title":"Deap: A database for emotion anal- ysis; using physiological signals,","cited_arxiv_id":null,"evidence_quote":"Supplies the DEAP dataset, the benchmark on which all comparisons and ablations are run."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted win- dows,","cited_arxiv_id":null,"evidence_quote":"Provides the Swin Transformer backbone that extracts visual tokens from facial frames."},{"cited_title":"Attention-based deep multi- ple instance learning,","cited_arxiv_id":null,"evidence_quote":"Introduces attention-based multiple instance learning, which the top-K selection step adapts."},{"cited_title":"Emotion recognition using eeg signals and audiovisual features with contrastive learning,","cited_arxiv_id":null,"evidence_quote":"Recent multimodal contrastive-learning baseline whose 83.20% accuracy and 84.10% F1 on DEAP-4 Milmer claims to beat."},{"cited_title":"Meg and eeg data analysis with mne-python,","cited_arxiv_id":null,"evidence_quote":"Supplies the Fast ICA implementation used for EEG artifact removal."}],"review_version":1}