{"id":"1626d5aa-d7b8-4898-a8fc-0e5ba8a3b2e5","arxiv_id":"2412.09317","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Combining probabilities from fine-tuned wav2vec2 and ViViT models gives modest emotion-recognition results, but the evaluation design does not support a reliable conclusion.","lead":"The paper fine-tunes a speech model on CREMA-D and a video model on RAVDESS and combines their emotion probabilities with five simple fusion rules. It reports that averaging and rule-based fusion give acceptable results, but the evaluation uses a small, possibly leaking test set and selects fusion methods after seeing the same test results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RAVDESS file-level split leaks actor identity and sentence content into the video test set, so the reported fusion gains do not establish the central claim.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the RAVDESS train/test split is not clean. Among several methodological problems, this is the one that directly undermines the causal claim that video plus audio fusion improves or achieves acceptable emotion recognition. The 24-actor dataset with repeated statements makes file-level sampling especially dangerous because identity and sentence content overlap between training and test. The paper's own numbers are not reported in prose, but the figures are consistent with inflated video accuracy from memorization. A clean actor-level split is a minimal, decisive experiment: if the improvement disappears, the central claim is unsubstantiated. Secondary issues, including test-set-driven selection of fusion rules and the absence of numeric tables and error bars, would remain even if the leakage were fixed, but the leakage is the primary reason the current evidence cannot support the conclusion. Therefore the reader's REJECT verdict should stand.","tokens_in":8572,"tokens_out":5106,"duration_ms":57587,"concrete_test":"Re-run the video model and fusion evaluation with an actor-exclusive split: hold out all files from a disjoint set of RAVDESS actors for testing, train the video model on the remaining actors' video-only files, and evaluate on the held-out actors using the same 105-file protocol or a larger balanced sample. If video accuracy and fusion accuracy drop materially, for example from roughly 88% toward chance or the majority-class baseline, the leak is confirmed and the central claim fails. Also report exact numeric accuracy and F1 for each fusion rule on this clean split, with the audio model trained only on CREMA-D as in the paper, so the comparison is not hidden in figures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is in Section 2.2. The authors sample 105 full-AV RAVDESS files for testing and remove only the video-only counterparts of those exact files from the video training set. RAVDESS contains only 24 actors, each reciting the same two statements across all emotions, intensities, and repetitions. After this removal, the same actors and the same sentences remain in training, often with the same emotion at a different repetition or intensity. The ViViT video model can therefore exploit actor appearance, lighting, and sentence-specific mouth movements to reach roughly 88% on the 105-file test set without learning emotion-general visual cues. Section 3.3 then selects fusion rules after viewing results on this same leaked test sample, writing 'After disparity in the results... another test framework is created.' This test-set peeking further inflates the comparison of averaging, weighted averaging, confidence threshold, dynamic weighting, and rule-based logic. The Discussion's limitation paragraph acknowledges cultural and environmental bias but does not mention this identity/sentence leakage. Consequently, the strongest claim, that simple decision-level fusion of independently fine-tuned unimodal transformers yields acceptable emotion recognition, is not supported by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal emotion recognition pipeline that combines a fine-tuned wav2vec2-large audio model and a fine-tuned ViViT video model, and it compares five decision-level fusion strategies (simple averaging, weighted averaging, confidence-level thresholding, dynamic confidence-based weighting, and rule-based logic) on a test sample of 105 RAVDESS video files. The authors report that fusion improves over the unimodal baselines, that averaging and rule-based logic are the best fusion methods, and that the combined video+audio approach yields \"acceptable results\" on emotion recognition. The central claim is that simple decision-level fusion of two independently fine-tuned unimodal transformers can achieve usable emotion recognition without training a fusion network.","tokens_in":8802,"tokens_out":4648,"duration_ms":45222,"significance":"If the claims were established, the paper would provide a lightweight alternative to learned multimodal fusion, which is a meaningful practical contribution. The authors provide links to Kaggle notebooks for training and evaluation, which is helpful for reproducibility. However, the experimental design has two load-bearing weaknesses: the RAVDESS test set is likely leaked in terms of actor identity and sentence content, and the fusion methods are selected after observing results on the same test set. These issues call the central conclusion into question. The paper also contains no statistical analysis, and the audio and video models are evaluated under different domain-shift conditions. The significance of the claimed finding is therefore contingent on a reworked evaluation that the current manuscript does not provide.","major_comments":[{"comment":"The construction of the 105-file test set introduces actor and sentence leakage that can inflate video accuracy. RAVDESS contains only 24 actors, each reciting the same two statements across all emotions, intensities, and repetitions. Removing only the video-only counterparts of the 105 test files leaves in training many videos from the same actors and the same two statements, frequently with the same emotion at a different intensity or repetition. The ViViT video model can therefore exploit actor identity, lighting, and sentence-specific mouth movements to reach the reported ~88% accuracy without learning emotion-general visual cues. Because all fusion results are computed on this same test set, the central claim that combining video and audio inputs improves emotion recognition is not established. A leave-actors-out split, or at least a split that ensures no actor and no statement appear in both training and test, is required to support the claim.","section":"Section 3.3"},{"comment":"The four additional fusion strategies (weighted average, confidence threshold, dynamic weighting, rule-based logic) were created after inspecting the averaging method's results on the same 105-file test set. The abstract and Section 3.3 state, \"After disparity in the results... another test framework is created,\" and Section 3.3 explicitly says both V1 and V2 were run on the same 105-file RAVDESS sample. This is a post hoc selection of fusion rules on the test data, so the comparison among the five methods is not a fair out-of-sample evaluation. The conclusion that averaging and rule-based logic are the best methods may simply reflect overfitting to these 105 files. The paper should pre-register the fusion rules or use a separate validation set for method selection and a different held-out test set for final evaluation.","section":"Section 3.3"},{"comment":"The audio and video unimodal models are evaluated under different domain-shift conditions, which confounds the comparison that motivates the fusion design. The audio model is fine-tuned on CREMA-D and tested on RAVDESS audio, whereas the video model is fine-tuned on RAVDESS and tested on RAVDESS video. The lower audio accuracy (around 59-72%) relative to video accuracy (~88%) may reflect a train/test domain shift rather than an inherent difference in modality difficulty. Since the weighted average, confidence threshold, and dynamic weighting methods are all motivated by the video model's higher accuracy, the validity of this motivation depends on an apples-to-apples comparison. The paper should either fine-tune the audio model on RAVDESS audio as well or explicitly analyze and control for the domain shift.","section":"Sections 3.1, 3.2"},{"comment":"All conclusions about fusion performance are based on a single test set of 105 files, with no confidence intervals, error bars, or statistical tests. The differences among the five fusion methods are small (a few percentage points), and the ranking of averaging and rule-based logic as the best methods could easily be within sampling noise. The paper should report per-class accuracy, confusion matrices, and bootstrapped confidence intervals or a significance test (e.g., McNemar's test) to support the claimed ranking. Without this, the qualitative conclusions in the Discussion are not strongly supported.","section":"Section 3.3, Discussion"}],"minor_comments":[{"comment":"There are typographical errors, including \"avarage\" for \"average\" and \"the models that been used\" for \"the models that were used.\" The abstract also says five methods are used but lists only four; the averaging method is presumably the fifth, but this should be explicit.","section":"Abstract"},{"comment":"The dataset name is inconsistently written as \"RA VDESS\" in several places; it should be \"RAVDESS\" throughout.","section":"Section 2.2"},{"comment":"There are typos in the text: \"prepossessed\" should be \"preprocessed,\" and \"tdqm\" should be \"tqdm.\" Also, the library name \"A V\" should be \"PyAV\" or \"av\".","section":"Section 2.3"},{"comment":"The figures show accuracy bars but the numeric values are not reported in the captions or axes; the reader cannot determine the exact accuracy differences between methods. Please add a table with exact values or label the bars.","section":"Figures 2-6"},{"comment":"The paper uses \"sentiment analysis\" and \"emotion recognition\" interchangeably. The task is emotion recognition (six discrete emotion classes), not sentiment analysis (typically positive/negative). Clarifying this terminology would avoid confusion with the sentiment-analysis literature cited in Table 1.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"This is a short conference manuscript whose central claim depends on an experimental evaluation with a leaked test split and post hoc fusion-rule selection. These are not local presentation issues; they require redoing the training/testing protocol and the fusion comparison. The manuscript's scope is a empirical demonstration, and without a valid evaluation the paper does not meet the standard for publication. I would also note that the choice of different datasets for the two modalities adds a further confound that the authors do not address."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a transparent student-level application paper, not a research contribution. It fine-tunes wav2vec2-large on CREMA-D and ViViT on RAVDESS, then combines the output probabilities with five decision-level fusion rules. The prose is clear and the authors link their Kaggle notebooks, which is good practice. But the evaluation is broken in a way that undermines the central claim that multimodal fusion helps.\n\nThe main problem is the RAVDESS split in Section 2.2. The authors randomly sample 105 full-AV files for testing and remove only the video-only counterparts of those exact files from the training set. RAVDESS has 24 actors, each saying the same two sentences. So the same actors and sentences appear in both training and test, often with the same emotion at a different repetition or intensity. The video model can memorize actor identity, lighting, and sentence-specific mouth movements. Its ~88% test accuracy is therefore not evidence of emotion generalization. The audio model, in contrast, was trained on CREMA-D and tested on RAVDESS audio, which is a real cross-dataset generalization test. That distinction is never discussed, and it makes the two model accuracies non-comparable.\n\nThe second issue is test-set peeking. Section 3.3 says the fusion rules were designed \"after disparity in the results\" on the same 105 files. Choosing among averaging, weighted averaging, confidence thresholds, dynamic weighting, and rule-based logic after seeing their test-set performance makes the comparison circular. There are also no numeric results in the text, only figures, and no error bars or significance tests.\n\nOn the positive side, the paper is honest about its limitations regarding cultural bias and controlled environments, but it does not mention the leakage or the peeking. The citation pattern is fine. There is no mathematical contribution, and the experimental design cannot support the conclusion. The \"acceptable results\" claim is vague.\n\nWho is this for? It could serve as a cautionary example in a methods class about split leakage and test-set-driven method selection. As a research paper, it does not deserve a serious referee. I would desk reject, or route it to a student workshop where the feedback is pedagogical. The authors can fix this with actor-level splitting, a separate held-out set for fusion selection, numeric tables with error bars, and a properly evaluated unimodal baseline.","headline":"Honest and transparent, but the leaky RAVDESS split and test-set-peeking make the fusion claim unsupported.","tokens_in":9311,"tokens_out":3742,"would_cite":false,"duration_ms":36473,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining audio and video models improves emotion recognition.","keywords":["sentiment analysis","emotion recognition","multimodal fusion","audio classification","video classification","wav2vec2","ViViT","decision-level fusion"],"falsifier":"Train the video and fusion models on a RAVDESS split where no actor appears in both training and test (for example, hold out six actors entirely), then rerun the five fusion rules; if the combined accuracy falls to or below the better unimodal model's accuracy, the claimed fusion benefit is refuted.","tokens_in":8366,"feed_emoji":"🎭","tokens_out":5703,"duration_ms":49649,"temperature":0.7,"pith_summary":"The paper sets out to show that emotion recognition from videos can be built by fusing the outputs of two separately fine-tuned unimodal transformer models, one for audio and one for video, rather than training a joint multimodal network. Using the CREMA-D audio dataset and the RAVDESS video dataset, the authors fine-tune wav2vec2-large and ViViT, then combine their per-emotion probability outputs with five simple decision rules. On a 105-video test sample, the combined systems match or exceed the better unimodal model, with averaging and a rule-based logic performing best when the two models have similar accuracy. The paper concludes that the combined approach gives acceptable results on the emotion recognition tasks, while flagging that consistency needs further testing.","feed_headline":"Combining audio and video models improves emotion recognition","feed_subtitle":"Averaging the probability outputs of two fine-tuned transformers handles six emotions without training a fusion network.","key_machinery":"The mechanism is decision-level fusion of per-emotion probability vectors. Each unimodal model emits a probability distribution over the six emotions (anger, disgust, fear, happiness, neutral, and sadness), and the fusion rules operate purely on these vectors and on each model's accuracy or prediction confidence: the mean probability, a weighted mean scaled by model accuracy, a threshold that trusts the video model when its top probability exceeds 0.7, a dynamic weighting based on inverse total confidence, and a rule that returns the agreed emotion when both models are confident or otherwise the higher-confidence prediction. No fusion parameters are learned.","core_discovery":"The central claim is that decision-level fusion of two independently fine-tuned unimodal transformers yields acceptable emotion recognition accuracy. The audio model (wav2vec2-large) and the video model (ViViT) each output a probability distribution over six emotions; the five fusion rules — simple averaging, accuracy-weighted averaging, a video-confidence threshold, dynamic confidence-based weighting, and rule-based logic — convert these two distributions into a single prediction. In the first model version, where both modalities had similar accuracy, averaging and rule-based logic produced the best results; in the second version, where the video model was much stronger, the accuracy-weighted average kept the combined score at or above the video model's level. The paper presents this as evidence that combining video and audio inputs is a usable approach for emotion recognition.","pith_inferences":["The RAVDESS test set is drawn from the same 24 actors and two sentences used in training, so the reported accuracy, especially the video model's roughly 88% in the second version, may partly reflect identity and sentence memorization rather than emotion generalization; a person-disjoint split would test this.","A direct experiment would be to hold out all videos from a subset of actors and rerun the five fusion rules; if the fusion advantage disappears, the central claim would be weakened.","Applying the same framework to out-of-distribution videos — different lighting, backgrounds, languages, or cultures — would reveal whether the acceptable results persist beyond the controlled RAVDESS setting.","The authors' own discussion of the EU AI Act suggests that any deployment, such as the proposed therapy robot, would need regulatory review before use."],"forward_implications":["If averaging two unimodal probability vectors improves accuracy, multimodal sentiment analysis can be improved without training a fusion network.","The rule-based logic, which trusts agreement or the higher-confidence model, performs as well as averaging, so confidence signals carry useful information for combining models.","When one modality is much more accurate, the accuracy-weighted average prevents the weaker model from pulling the combined result down.","The six-emotion, single-culture, controlled-recording setup means the approach still needs to be tested on diverse, in-the-wild data before it can be generalized."],"supporting_citations":[{"why":"Supplies the CREMA-D audio dataset used to fine-tune the audio model.","marker":"[7]"},{"why":"Supplies the RAVDESS video dataset used to fine-tune the video model and to draw the 105-file test set.","marker":"[8]"},{"why":"Provides the wav2vec2-large architecture fine-tuned for the audio emotion classifier.","marker":"[9]"},{"why":"Provides the ViViT video vision transformer architecture fine-tuned for the video emotion classifier.","marker":"[10]"}],"fun_headline_variants":["Fusing audio and video models sharpens sentiment analysis","Audio-visual fusion boosts emotion recognition accuracy","Two transformers better than one for sentiment from video and audio","Merging audio and video outputs improves emotion detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the RAVDESS train/test split is clean, so scores reflect emotion recognition instead of memory of the same 24 actors and two sentences showing up in both training and test.","fun_headline_variants_meta":{"raw":{"variants":["Fusing audio and video models sharpens sentiment analysis","Audio-visual fusion boosts emotion recognition accuracy","Two transformers better than one for sentiment from video and audio","Merging audio and video outputs improves emotion detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2004,"prompt_tokens":882,"completion_tokens":1122,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1060}},"tokens_in":498,"tokens_out":1122,"duration_ms":9013,"temperature":1.0,"reasoning_tokens":1060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:22.745580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the video and fusion models on a RAVDESS split where no actor appears in both training and test (for example, hold out six actors entirely), then rerun the five fusion rules; if the combined accuracy falls to or below the better unimodal model's accuracy, the claimed fusion benefit is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CREMA-D audio dataset used to fine-tune the audio model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the wav2vec2-large architecture fine-tuned for the audio emotion classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ViViT video vision transformer architecture fine-tuned for the video emotion classifier."}],"review_version":1}