{"id":"ba910ca9-96fd-4719-aac2-f19cee8fe67e","arxiv_id":"2505.04642","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A late-fusion dense network using TF-IDF, audio statistics, XGBoost stacking, and oversampling is claimed to reach 92.55% accuracy on six-class IEMOCAP, exceeding published transformer baselines.","lead":"The paper reports 92.5% accuracy on the IEMOCAP emotion recognition benchmark using a simple fusion of dense neural networks and handcrafted features, without transformers. It argues that careful feature engineering can beat much larger multimodal models, but the evaluation protocol raises serious leakage concerns.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92.55% accuracy claim is load-bearing, but the evaluation protocol allows train/test leakage through speaker overlap and ambiguous oversampling timing, so the comparison to prior work is not yet valid.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the load-bearing point. My independent reading confirms that the paper's split description is internally inconsistent (80/10/10 versus 80/20) and never mentions speaker or session separation on IEMOCAP. The oversampling timing ambiguity is even more damaging than the split ambiguity alone: with stratified sampling with replacement, if oversampling precedes the split, the same generated row can appear on both sides of the train/test boundary, turning part of the reported accuracy into memorization. Since the paper provides no code, data, or error bars, there is no way to rule this out from the manuscript. The comparison against TFN, MAG-BERT, MMIM, and UniMSE is only meaningful if the test protocol matches the baselines; the paper does not establish that. I do not see a comparably severe alternative concern: the architectural simplicity itself is not a flaw, and the absence of pretrained transformers is not inherently invalid. The empirical claim is the center of the paper, and its measurement is not shown to be sound. Therefore the verdict should remain REJECT, with the caveat that a corrected evaluation could in principle change the conclusion.","tokens_in":9336,"tokens_out":2637,"duration_ms":35580,"concrete_test":"Obtain or reimplement the exact pipeline and rerun under a speaker-independent protocol: (1) create the train/test split by session or speaker so no speaker appears in both sets; (2) apply the class-balanced oversampling only after the split, so synthetic duplicates cannot cross into the test set; (3) measure test accuracy and weighted F1 on this new held-out set. If the accuracy falls materially below 92.55% or approaches the 84-86% range of the published baselines, the original number is an artifact of the split/oversampling order. A secondary check: inspect the original pipeline's split code to see whether any oversampled duplicate rows appear in both train and test.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that the proposed late-fusion model reaches 92.55% test accuracy and 92.34% weighted F1 on six-class IEMOCAP, outperforming TFN, Multimodal Transformer, MAG-BERT, MMIM, and UniMSE. Every downstream conclusion depends on this number being measured under a protocol comparable to the published baselines. The paper's protocol does not establish that. Section 3.6 says the split is \"stratified sampling (80/10/10)\" while Section 4.1 says the dataset was \"split into an 80:20 train-test configuration\" with an additional 10% of training data for validation; no speaker or session separation is mentioned anywhere. IEMOCAP's standard evaluation is speaker-independent (leave-one-session-out or speaker-disjoint splits) because utterances from the same speaker are highly correlated in emotional expression, prosody, and facial behavior; a random utterance-level split lets the model exploit speaker identity rather than generalizable multimodal sentiment. Additionally, the oversampling in Sections 3.3 and 4.3 is described before the split, but the paper never states whether oversampling is applied before or after the train/test split. If oversampling with replacement is applied before the split, exact duplicate rows can appear in both training and test sets, directly inflating accuracy. The comparison table then mixes this potentially leaky number with externally reported results whose protocols are not specified. The accuracy drop under a proper speaker-independent split is unknown, so the headline claim is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight late-fusion model for six-class emotion classification on IEMOCAP, using TF-IDF text features, handcrafted audio features, MoCap-derived video features, per-class oversampling, XGBoost-based feature enrichment, and dense encoders followed by concatenation and a softmax layer. The central claim is that this simple architecture achieves 92.55% test accuracy and 92.34% weighted F1, outperforming TFN, Multimodal Transformer, MAG-BERT, MMIM, and UniMSE on IEMOCAP. The paper argues that strong feature engineering and modular design can outperform attention-heavy architectures at lower computational cost.","tokens_in":9632,"tokens_out":3610,"duration_ms":40804,"significance":"If the reported accuracy were obtained under a protocol comparable to the published baselines, the result would be noteworthy: a parameter-light, non-pretrained fusion model beating transformer-based multimodal systems by roughly six points on a standard benchmark would challenge the prevailing emphasis on architectural complexity. The manuscript is clearly written and transparent about its design choices, and it gives useful details on regularization and class balancing. However, no code or data is provided, and the central accuracy claim rests on an evaluation protocol that appears to allow speaker leakage and possibly duplicate oversampled rows across train and test. The reported comparison to prior work is therefore not yet credible, and the significance of the claimed improvement cannot be assessed from the manuscript as written.","major_comments":[{"comment":"The data-split description is internally contradictory. Section 3.6 states that a stratified sampling (80/10/10) split was used, while Section 4.1 states that the dataset was split into an 80:20 train-test configuration with an additional 10% of the training data set aside for validation. These protocols yield different test-set sizes and different training/validation proportions, so the reported 92.55% test accuracy cannot be reproduced or interpreted without clarification. Please specify the exact final train/validation/test sizes and the random seed used for stratification.","section":"3.6 and 4.1"},{"comment":"The evaluation does not use a speaker-independent split. IEMOCAP's standard evaluation protocols are speaker-independent (e.g., leave-one-session-out or speaker-disjoint splits), because utterances from the same speaker are highly correlated in prosody, facial behavior, and lexical choices; a random utterance-level split allows the model to exploit speaker identity rather than learn generalizable multimodal sentiment. Since the paper reports only a stratified random split without any speaker or session separation, the measured accuracy is likely inflated relative to published baselines that respect speaker independence. Please rerun the experiments under a speaker-disjoint or leave-one-session-out protocol and report those results.","section":"3.6 and 4.1"},{"comment":"The timing of the oversampling relative to the split is ambiguous and load-bearing. Section 3.3 says oversampling was performed per class using stratified sampling with replacement and that the final dataset exhibited a balanced distribution, while Section 4.3 refers to this balancing process without clarifying whether it is applied before or after the train/test split. If oversampling with replacement is applied before the split, exact duplicate rows can appear in both training and test sets, directly inflating test accuracy. The manuscript must state explicitly that oversampling is applied only to the training partition, and the experiments should be rerun if that was not the case.","section":"3.3 and 4.3"},{"comment":"The comparison with prior work is not valid because the baseline numbers are taken from papers that use different evaluation protocols, label sets, and possibly different IEMOCAP partitions. Moreover, the references in Table 1 are incorrect: entries labeled TFN, Multimodal Transformer, MAG-BERT, MMIM, and UniMSE cite reference numbers [12]–[16], which in the reference list correspond to MOSI, Wu and Xu, CoreNLP, spaCy, and the Audio Spectrogram Transformer, not to the stated models. Please provide a table in which all methods, including the proposed one, are evaluated on the same speaker-independent split and with the same label mapping, and correct the citations.","section":"Table 1 and Section 5.5"}],"minor_comments":[{"comment":"The reported accuracy is inconsistent across the paper: the abstract says 92%, Section 1.4 says 92.5%, and Section 5.1 says 92.55%. Please unify these numbers.","section":"Abstract and Section 1.4"},{"comment":"Section 4.3 refers to 'target counts defined in Section 3.2.4,' but the target counts actually appear in Section 3.3. Please fix the cross-reference.","section":"4.3"},{"comment":"The emotion_mapped dictionary in Section 3.3 is not explained: the source label order and the meaning of the target class indices should be stated explicitly so that readers can map the six classes to the original IEMOCAP annotations.","section":"3.3"},{"comment":"The feature engineering pipeline is not fully reproducible: the number of TF-IDF features after LASSO and RFE, the XGBoost hyperparameters, and the leaf-embedding dimensions are not reported. Please give these values or provide code.","section":"3.2"},{"comment":"Several citations are mismatched: the BERT sentence in Section 2.1 cites [10], which is listed as the Tensor Fusion Network paper; [12] is listed as MOSI but is cited for TFN; and [23] is used for multiple distinct claims while also duplicating [5] and [13]. Please renumber and correct the bibliography.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central claim is a benchmark-comparison claim, and the evaluation protocol as described cannot support it. The split contradiction and the absence of speaker separation are not cosmetic issues: they directly affect whether the 92.55% number is comparable to the cited baselines. Even if the authors added a speaker-independent evaluation, the conclusions would likely change, and the current manuscript would need substantial new experiments. I see no path to acceptance without a rework of the core experimental protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the evaluation pitfalls, not for the result. The headline claim—92.55% test accuracy on six-class IEMOCAP with a dense late-fusion model—is not supported by the protocol as written.\n\nThe architecture is a straightforward assembly of known pieces: TF-IDF text with LASSO/RFE, handcrafted audio features boosted with XGBoost leaf embeddings, MoCap video passed through XGBoost softmax stacking, then concatenation and a few dense layers. Nothing here is new by itself, and the paper only claims novelty in the final number. To the authors' credit, they ran single-modality and early-fusion baselines, and the efficiency story is plausible. The writing is clear and the pipeline description is detailed enough to reproduce in principle. But no code or data are released.\n\nThe soft spots are structural. Section 3.6 says an 80/10/10 stratified split; Section 4.1 says an 80:20 train-test split with a further 10% validation. Those cannot both be true. More seriously, the split is utterance-level with no speaker or session separation. IEMOCAP's standard protocol is speaker-independent because same-speaker utterances are highly correlated; a random utterance split lets the model exploit speaker identity. The paper also never says whether the oversampling in Section 3.3 happens before or after the split. If before, exact duplicate rows can land in both train and test. Either issue alone would inflate the number; together they make 92.55% unreliable. The comparison against TFN, Multimodal Transformer, MAG-BERT, MMIM, and UniMSE uses published numbers from unknown protocols, so the six-point gap is not meaningful.\n\nThe reference list is also in disarray—citation [10] is called BERT but points to the Tensor Fusion Network paper, and the comparison table cites references that don't match the bibliography. That alone would need a rewrite.\n\nThe central message—simple, well-engineered fusion can compete with attention-heavy models—may be right, but this paper doesn't demonstrate it. The accuracy drop under a proper speaker-independent split is unknown, and there are no error bars or repeated runs.\n\nI wouldn't send this to peer review in its current form. The load-bearing number isn't credible, and the fix is a new experimental campaign, not a minor revision. If the authors rerun under a proper protocol, release code and data, and fix the citations, the question is worth a second look. For now, it's a useful teaching example of why evaluation protocol matters.","headline":"The 92.55% IEMOCAP result is the whole paper, and the evaluation protocol as written does not come close to supporting it.","tokens_in":10181,"tokens_out":4300,"would_cite":false,"duration_ms":48992,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A late-fusion dense model claims 92.55% accuracy on six-class IEMOCAP, beating attention-heavy baselines by about six points.","keywords":["multimodal sentiment analysis","late fusion","dense encoders","feature engineering","TF-IDF","IEMOCAP","emotion classification","oversampling"],"falsifier":"Run the identical encoder under a leave-one-session-out speaker-independent protocol and check for exact feature-vector overlap between the training and test sets after oversampling. If accuracy falls near the 86.5% baseline rather than 92.55%, or duplicates are found, the reported margin is an artifact of the split.","tokens_in":9111,"feed_emoji":"🎭","tokens_out":9198,"duration_ms":97635,"temperature":0.7,"pith_summary":"The paper's claim is that a deliberately simple architecture—three independent dense encoders, one for text, audio, and video, joined by plain concatenation and a single dense fusion layer—reaches 92.55% test accuracy and 92.34% weighted F1 on six-class IEMOCAP. The authors attribute the result to careful feature engineering per modality and oversampling of minority classes rather than to attention, tensor products, or pretrained transformers. A sympathetic reader would take this as evidence that 'better features, simpler fusion' can beat complex multimodal models by a wide margin on standard benchmarks and at a fraction of the compute. The paper reports a roughly six-point margin over the strongest listed baseline, and positions the architecture as suitable for resource-constrained deployment.","feed_headline":"Dense late fusion reports 92.55% on IEMOCAP","feed_subtitle":"Independent per-modality encoders plus concatenation claim a six-point gain over attention-heavy baselines.","key_machinery":"The load-bearing mechanism is late fusion of independently encoded modality representations. Text is summarized as TF-IDF with regularization-based feature selection; audio is summarized with prosodic statistics enriched by gradient-boosted leaf embeddings; video is MoCap features concatenated with gradient-boosted classifier scores. Each modality encoder outputs a 128-dimensional vector, and concatenation followed by a 256-unit dense layer with dropout learns cross-modal interactions. Oversampling with replacement toward fixed class targets is the complementary component, and the paper credits it with balanced recall on fear and surprise.","core_discovery":"On IEMOCAP, after remapping the eight original emotion labels into six classes, the model encodes TF-IDF-selected text vectors, statistical audio features, and MoCap-derived video features through separate 128-unit dense encoders with dropout. The encoded streams are concatenated into $Z_{\\text{fused}} = [z_{\\text{audio}}; z_{\\text{video}}; z_{\\text{text}}]$, passed through a 256-unit ReLU layer with dropout, and classified by softmax. The reported test accuracy is 92.55%, weighted F1 is 92.34%, macro F1 is 93.05%, and macro ROC-AUC is 0.9881; these are the numbers that the paper argues show late fusion with engineered features can outperform or match attention-heavy and tensor-based alternatives.","pith_inferences":["Because the paper uses a stratified random utterance split rather than a speaker- or session-independent split, the honest interpretation is that 92.55% is an upper bound for a random-utterance setting; a session-split evaluation would likely lower the number and narrow the claimed margin.","A direct diagnostic for split leakage would be to compare the exact feature vectors of training and test rows after oversampling; if duplicates appear, the score is inflated by construction.","The gradient-boosted feature enrichment at the audio and video inputs is effectively a learned nonlinear preprocessing step; applying the same trick to text, or testing the architecture without it, would isolate how much of the gain is fusion versus feature engineering.","The same model on other multimodal sentiment datasets with speaker-independent splits would show whether the feature-engineering advantage transfers beyond IEMOCAP; the paper lists such datasets as future work but does not report numbers."],"forward_implications":["If the reported numbers reproduce, a simple dense late-fusion baseline should become the default comparison for IEMOCAP experiments, not an upper bound that only attention-heavy models approach.","The recipe—per-modality feature engineering, class balancing, concatenation fusion—can be applied to other small multimodal emotion datasets with less GPU time than transformer training.","The six-point gap over the listed state-of-the-art transformer baseline implies that much of the reported gain in complex fusion models may come from evaluation protocol differences rather than architectural power.","The model's low parameter count and single-GPU training profile make real-time emotion-aware interfaces, such as dialogue agents or mental-health monitoring, more feasible.","The oversampling design suggests that data-balancing choices are as consequential for minority emotion classes as the fusion strategy itself."],"supporting_citations":[{"why":"Supplies the aligned text, audio, and MoCap features and the categorical emotion labels used for training and evaluation.","marker":"[8]"},{"why":"The tensor-based fusion baseline whose reported accuracy and F1 the paper's Table 1 compares against.","marker":"[12]"},{"why":"The cross-modal attention baseline whose reported accuracy and F1 set the comparison point the paper claims to exceed.","marker":"[13]"},{"why":"A text-pretrained transformer baseline in the comparison table.","marker":"[14]"},{"why":"A mutual-information modeling baseline in the comparison table.","marker":"[15]"},{"why":"The state-of-the-art baseline the paper claims to beat by about six points.","marker":"[16]"}],"fun_headline_variants":["Simple late fusion: 92.55% on IEMOCAP","Dense encoders plus concat beat attention","92% sentiment from streamlined fusion","No complex attention needed: 92.55% accuracy","Late fusion with engineered features wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that a stratified random split of IEMOCAP utterances, with oversampling applied cleanly before training, measures real generalization; the paper does not separate speakers or sessions and does not state whether oversampling happens before or after the split.","fun_headline_variants_meta":{"raw":{"variants":["Simple late fusion: 92.55% on IEMOCAP","Dense encoders plus concat beat attention","92% sentiment from streamlined fusion","No complex attention needed: 92.55% accuracy","Late fusion with engineered features wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1346,"prompt_tokens":864,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":410}},"tokens_in":480,"tokens_out":482,"duration_ms":5844,"temperature":1.0,"reasoning_tokens":410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:55:19.596324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical encoder under a leave-one-session-out speaker-independent protocol and check for exact feature-vector overlap between the training and test sets after oversampling. If accuracy falls near the 86.5% baseline rather than 92.55%, or duplicates are found, the reported margin is an artifact of the split.","supporting_citations":[{"cited_title":"Audio Spectrogram Transformer: Transformer architecture for audio data representation,","cited_arxiv_id":null,"evidence_quote":"The state-of-the-art baseline the paper claims to beat by about six points."},{"cited_title":"The Stanford CoreNLP natural language processing toolkit,","cited_arxiv_id":null,"evidence_quote":"A text-pretrained transformer baseline in the comparison table."},{"cited_title":"spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing,","cited_arxiv_id":null,"evidence_quote":"A mutual-information modeling baseline in the comparison table."},{"cited_title":"IEMOCAP: Interactive emotional dyadic motion capture database,","cited_arxiv_id":null,"evidence_quote":"Supplies the aligned text, audio, and MoCap features and the categorical emotion labels used for training and evaluation."}],"review_version":1}