{"id":"a5825234-ec73-4a99-a94e-64196517e867","arxiv_id":"1909.01763","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-modal two-time-scale framework with progressive training is claimed to beat prior work on LIRIS-ACCEDE, but the results rely on test-set parameter selection and do not improve arousal MSE.","lead":"This paper presents a multi-modal neural system that predicts the emotional valence and arousal of movie clips from audio, faces, scenes, and body actions. The authors report large gains on a standard benchmark, but many of those gains come from tuning on the test set, so the headline claim is not reliable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'large-margin' claim is undercut by the paper's own Table 5: arousal MSE (0.137) is worse than GLA (0.133), and the valence improvements rely on hyperparameters and modality sets selected on the test set without validation.","rationale":"I read the paper as an engineering contribution whose central claim is the large-margin superiority on LIRIS-ACCEDE. The pipeline is reasonable, but the evidence for the headline is insecure in two related ways. The reader's weakest assumption focuses on test-set selection of hyperparameters and modalities; that is indeed a serious concern because Tables 3 and 4 show explicit search over clip count and beta, and Table 1 is used to choose per-task modality sets, all apparently on the test set. I additionally note a more direct, internal problem: Table 5 shows that the proposed method's arousal MSE (0.137) is not better than GLA's (0.133), so even if test-set selection were not an issue, the abstract's unqualified 'better performance with a large margin' is not supported by the official metric across both tasks. The MediaEval 2015 classification numbers are a legitimate secondary signal but do not rescue the regression claim because the task, setup, and metrics differ. My recommendation remains REJECT, consistent with the reader's verdict, because the central claim is not trustworthy as reported. The concrete test I propose would settle the selection concern and would also reveal whether the reported valence margin survives honest evaluation.","tokens_in":11735,"tokens_out":3866,"duration_ms":37995,"concrete_test":"Re-run the pipeline on LIRIS-ACCEDE with a three-way split: train on a random subset of the 54 development movies, validate on the remaining development movies, select beta, clip count, and modality sets on the validation set only, then evaluate exactly once on the official 12-movie test set. Report MSE and PCC for both valence and arousal, with bootstrap 95% confidence intervals over movies. If valence MSE remains near 0.071 and the test-selected models do not degrade relative to Table 5, the concern is resolved; if the numbers move toward GLA/THUHCSI or arousal MSE remains worse than GLA, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the framework beats state-of-the-art on LIRIS-ACCEDE by a large margin. For this to hold, the reported comparisons must be both fairly obtained and actually superior. Neither condition is met. First, under the official metric (the paper states MSE is the official metric), Table 5 gives arousal MSE 0.137 for the proposed method, which is worse than GLA's 0.133; the large margin exists only for valence MSE and for PCC, not for arousal MSE. Second, the valence number is obtained by choosing the clip count (Table 3: 4 clips), the arousal decay beta (Table 4: 0.98/0.99), and per-task modality sets (Table 1) based on test-set performance. No development/validation split or repeated evaluation is reported, so the reported 0.071 valence MSE may reflect selection on the test set rather than a real advantage. The MediaEval 2015 classification results are a different task and do not validate the regression headline. Without code, error bars, or a validation-based selection protocol, the 'large margin' claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a multi-modal framework for continuous valence/arousal prediction on the LIRIS-ACCEDE dataset. Features are extracted with pre-trained audio (VGGish), facial expression, scene, and action (OpenPose) modules; per-modality BiLSTMs provide intra-clip temporal fusion, while inter-clip context is modeled with a TSN-like BiLSTM for valence and an exponential moving average for arousal. The model is trained with a residual-based progressive strategy, and the authors report comparisons to prior MediaEval 2018 systems as well as MediaEval 2015 classification results. The central claim is that the proposed framework achieves better performance than the state of the art by a large margin.","tokens_in":11985,"tokens_out":8124,"duration_ms":75883,"significance":"If the claims were supported, the paper would be a useful contribution to affective video analysis: it combines modality-specific deep features, an interpretable sequential fusion procedure, and separate temporal treatments for valence and arousal. The manuscript also deserves credit for reporting per-modality ablations and for comparing with several published systems on the same dataset. However, the evaluation as reported does not establish the headline claim: hyperparameters and per-task modality sets are selected using test-set performance, and the arousal MSE in the paper's own comparison table is worse than the best competitor under the official metric. The reported large-margin advantage is therefore not credible, and the framework's significance relative to prior art remains unclear.","major_comments":[{"comment":"The final valence and arousal results are obtained by selecting hyperparameters on the test set: the number of context clips is chosen as 4 and the EMA decay beta as 0.98 (or 0.99) because these values give the best numbers in Tables 3 and 4. No development split is used, so the reported valence MSE of 0.071 and arousal PCC of 0.419 are optimistically biased by test-set selection. In addition, Table 4 is internally ambiguous: beta=0.96/0.97 gives the best MSE (0.136), beta=0.99 gives the best PCC (0.427), and the reported arousal MSE of 0.137 corresponds to beta=0.98 without a stated selection criterion. The authors should select these hyperparameters on a validation split and then report test results once.","section":"Experiments and Results, Performance for Long Temporal Fusion (Tables 3 and 4)"},{"comment":"The per-task modality sets are chosen after evaluating modality-specific models on the test set: Table 1 ranks audio, scene, expression, and action on the test set, and the text then excludes action for valence and expression for arousal. Because the same test set is used for the final comparison in Table 5, this modality selection is circular. Modality selection must be performed on a development subset, or with nested cross-validation, and only the final evaluation should touch the test set.","section":"Modality-specific Performance (Table 1) and Multi-modal Features"},{"comment":"The abstract's claim of 'better performance ... with a large margin' is contradicted by the paper's own table under the official metric: the proposed method's arousal MSE (0.137) is worse than GLA's (0.133). The large margin exists only for valence MSE/PCC. The claim should be restricted to valence, or the arousal model needs to be modified and re-evaluated; as written, the conclusion overstates the results.","section":"Comparison with The State-of-the-art (Table 5)"},{"comment":"The experimental section reports single-run numbers on a 12-movie test set with no error bars, per-movie breakdown, or significance testing. Differences as small as 0.001-0.01 in MSE are quoted as decisive advantages, and the PCC differences are not assessed for statistical reliability. The authors should report results across runs, movies, or bootstrap intervals before claiming superiority.","section":"Experiments and Results"}],"minor_comments":[{"comment":"The abstract contains typos such as 'temporal contex' and 'audiences emotional states'; these should be corrected.","section":"Abstract"},{"comment":"The paper states that MSE is the official metric, but the hyperparameter choices in Tables 3 and 4 appear to optimize different quantities for valence and arousal; the selection criterion should be stated explicitly, since selecting beta by PCC while reporting MSE is inconsistent.","section":"Dataset and Metrics"},{"comment":"Table 6 reports classification accuracies on the MediaEval 2015 task, which uses different evaluation protocols and annotations; these results cannot be used as evidence for the valence/arousal regression framework, and this comparison should be clearly separated from the main regression evaluation.","section":"Results, Comparison with MediaEval 2015"},{"comment":"The T-SNE plots are described as demonstrating discriminability, but no quantitative measure is given; consider replacing or supplementing them with a quantitative separability metric.","section":"Figures 3 and 4"},{"comment":"The EMA is defined on clip-level predictions, but the per-second evaluation repeats each clip prediction 10 times; the interaction between the EMA and this repetition should be clarified.","section":"Equation (2)"}],"recommendation":"reject","confidential_remarks":"The evaluation protocol issue is serious: the reported results are selected on the test set, and the arousal result is not superior under the official metric. A revision would require re-running the experiments with a development split and substantially narrowing the central claim, which is beyond a normal minor revision. The paper is clearly written and the per-modality ablation is useful, but the current evidence does not support the headline contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a mixed bag. The framework is a sensible combination of known parts — VGGish, expression and scene CNNs, OpenPose, bi-LSTMs, TSN-like context, EMA smoothing — with a few twists: per-task modality selection, a two-time-scale temporal design, and a progressive training scheme that treats each new modality as fitting a residual relative to the previously fused features. The paper does real work: it ablates its fusion choice against feature-level and decision-level fusion, it reports a secondary MediaEval 2015 classification result, and the improvements on valence MSE and on both PCCs are large, not just incremental. Credit is due for the clarity of the system description and for the comparative ablations.\n\nBut the central claim is not supported. The authors try several clip counts for valence and several decay weights for arousal and then report the best numbers. They also select the per-task modality sets by looking at test-set performance. There is no validation split, no repeated evaluation, no error bars. The reported valence MSE of 0.071 and arousal PCC of 0.419 are therefore partly selected on the same data used for evaluation, so the 'large margin' statement is optimistic at best. The problem is not hypothetical: the paper's own Table 5 shows the arousal MSE of 0.137 is worse than GLA's 0.133, so the improvement is not uniform. The authors do not flag this, and the abstract's blanket 'large margin' claim is misleading.\n\nThe 'residual-based' label is also generous. During training each new modality's LSTM is asked to fit the residual from the previous sum, but at inference the features are simply summed. That is a training curriculum, not a residual architecture. It may still help, but it is not what the name suggests.\n\nThe MediaEval 2015 classification results are a useful sanity check, but they are a different task on a different split and do not validate the regression headline. Without code, a validation protocol, or error bars, the reported state-of-the-art numbers are not trustworthy enough to build on.\n\nMy take: this is a solid systems paper with a flawed evaluation. It deserves a serious referee, because the problems are fixable and the task is practically relevant, but the referee should ask for a proper validation-based selection protocol and honest reporting of all tried configurations. If those numbers survive, it becomes a decent contribution to the affective video analysis subfield; if not, it is a cautionary example of test-set tuning.","headline":"A reasonable engineering pipeline whose headline 'large margin' claim is undercut by test-set selection and by the paper's own arousal MSE, which is worse than a cited baseline.","tokens_in":12488,"tokens_out":2407,"would_cite":false,"duration_ms":21297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a multi-modal video emotion framework with per-task modality selection, two-time-scale temporal fusion, and residual-based progressive training beats prior methods by a large margin on LIRIS-ACCEDE.","keywords":["affective video content analysis","valence and arousal prediction","multi-modal fusion","temporal context modeling","LSTM","progressive training","LIRIS-ACCEDE"],"falsifier":"Rerun the same framework on LIRIS-ACCEDE while fixing beta, clip count, and modality sets using only a validation split before any test labels are seen; if valence MSE rises above 0.071 or arousal PCC falls below 0.419, part of the reported margin over the state of the art is an artifact of test-set selection.","tokens_in":11555,"feed_emoji":"🎬","tokens_out":6227,"duration_ms":55054,"temperature":0.7,"pith_summary":"This paper is trying to establish that predicting a video's continuous emotional effect—its valence and arousal values—gets substantially better when three design decisions are made together. First, each emotion dimension gets its own modality set, chosen by measuring how informative audio, scene, facial-expression, and action features are for that dimension. Second, multi-modal features are fused with a residual-based progressive training rule: modalities are added one at a time, and each new modality is trained only to supply what the already-fused features are missing. Third, temporal context is handled at two time scales, with short in-clip LSTMs per modality and a longer clip-level context that is modeled differently for valence and arousal. On the LIRIS-ACCEDE benchmark the complete pipeline reports valence MSE of 0.071 and arousal PCC of 0.419, better than the state-of-the-art systems it compares with, which is the concrete payoff the paper is arguing for.","feed_headline":"Video emotion predictor posts best valence and arousal scores","feed_subtitle":"Per-task modality choice, two-time-scale temporal fusion, and residual training lift LIRIS-ACCEDE results.","key_machinery":"The load-bearing mechanism is the residual-based progressive training strategy. Modalities are first trained individually, ranked by validation performance, and then added one by one: at step i the already-fused features $f_{i-1}$ are held fixed, and the new modality's LSTMs are trained to fit the residual mapping $H_i(x) = f_i - f_{i-1}$, so an uninformative modality is pushed to learn a near-zero mapping while an informative one fills in missing feature structure. Around this sits the two-time-scale temporal structure: bidirectional LSTMs inside each clip produce modality-specific clip embeddings, which are summed for the clip-level multi-modal feature, and then a second LSTM layer over clips models long context for valence while an exponential moving average post-processing models the slow arousal trend. This machinery is what the paper credits with avoiding the overfitting of joint feature-level training and the information loss of decision-level fusion.","core_discovery":"The central claim is that the right decomposition of video affect prediction makes previously reported numbers look weak: per-task modality selection (audio, scene, and expression for valence; audio, scene, and action for arousal), two-stage temporal fusion, and residual-based progressive training combine into a framework whose valence and arousal predictions on LIRIS-ACCEDE beat the listed prior methods by a large margin. The paper also claims that audio is the most informative single modality for both dimensions, that expressions matter for valence but not arousal, that actions matter for arousal but not valence, and that the temporal-context choice is dimension-dependent: a TSN-like bidirectional LSTM over four clips helps valence, while an exponential moving average with decay weight 0.98 helps arousal by smoothing its slow-changing intensity. The authors present these as integrated findings, not as separate tricks, since each design choice is motivated by an observed property of emotion dynamics.","pith_inferences":["If the reported margin holds under a proper held-out validation split, the same recipe—per-task modality selection plus residual fusion—could transfer to other continuous affect datasets, but the specific rankings (audio first, expression only for valence) may shift with genre or culture.","The exponential moving average result is an implicit argument that arousal can be modeled with a low-order autoregressive process; a natural testable extension is replacing the fixed beta with a learned gating mechanism or comparing against a Kalman filter.","The residual progressive training can be read as a general way to combine modalities of unequal quality, so it may be worth trying on other multi-modal prediction problems, not just emotion, whenever feature-level fusion tends to overfit.","The paper's bad-case analysis suggests that comedy clips with fighting or quarrels are systematically mispredicted as negative valence; a testable extension would add genre-aware features or some form of situational/narrative context to correct this."],"forward_implications":["On the LIRIS-ACCEDE test set, the reported framework improves valence MSE and arousal PCC over every state-of-the-art system listed in the comparison table, including the strongest prior GLA baseline.","Audio features alone carry the most emotional signal, so any successor system that drops or weakens the audio stream is likely to lose performance on this task.","Because the progressive training makes per-clip modality selection dynamic, the fused model can lean on whatever modality is discriminative for a given clip, which is a mechanism the paper argues that concat-based or voting-based fusion lacks.","The valence long-context result of four clips suggests there is an optimal context window for valence prediction, with shorter windows losing context and longer windows adding noise from abrupt emotional shifts.","The arousal result shows that a parameter-free exponential moving average can beat more complex recurrent temporal modeling for arousal, supporting the paper's premise that arousal changes slowly and its trend is strongly auto-regressive."],"supporting_citations":[{"why":"Supplies VGGish, the pre-trained audio feature extractor that the experiments show carries the most emotional information for both tasks.","marker":"Hershey et al. 2017"},{"why":"Supplies the OpenPose backbone for action/pose features, used for the arousal task.","marker":"Cao et al. 2018"},{"why":"Supplies the Xception network fine-tuned on RAF for facial-expression features used for valence.","marker":"Chollet 2017"},{"why":"Provides the LSTM cells used for both intra-clip per-modality fusion and inter-clip valence context.","marker":"Hochreiter and Schmidhuber 1997"},{"why":"Provides the temporal segment network structure that the valence long-context fusion is modeled on.","marker":"Wang et al. 2016"},{"why":"Defines the MediaEval 2015 affective-impact task whose classification results the intra-clip model is compared with.","marker":"Sjoberg et al. 2015"},{"why":"One of the strongest prior LIRIS-ACCEDE baselines (GLA) that the proposed framework claims to beat by a large margin.","marker":"Sun, Liu, and Prasad 2018"}],"fun_headline_variants":["Per-task modalities and two-level temporal fusion beat video affect baselines","Audio leads, expressions for valence, actions for arousal in video affect model","Two-time-scale fusion yields large margin on LIRIS-ACCEDE affect prediction","Residual progressive training merges modalities for better video emotion scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that choosing the arousal decay weight, the valence clip count, and each task's modality set by looking at test-set outcomes does not inflate the reported scores, since no validation-set experiment is shown to confirm those choices generalize.","fun_headline_variants_meta":{"raw":{"variants":["Per-task modalities and two-level temporal fusion beat video affect baselines","Audio leads, expressions for valence, actions for arousal in video affect model","Two-time-scale fusion yields large margin on LIRIS-ACCEDE affect prediction","Residual progressive training merges modalities for better video emotion scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001004,"raw_usage":{"total_tokens":4239,"prompt_tokens":932,"completion_tokens":3307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":3228}},"tokens_in":548,"tokens_out":3307,"duration_ms":23306,"temperature":1.0,"reasoning_tokens":3228,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:54:39.115344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same framework on LIRIS-ACCEDE while fixing beta, clip count, and modality sets using only a validation split before any test labels are seen; if valence MSE rises above 0.071 or arousal PCC falls below 0.419, part of the reported margin over the state of the art is an artifact of test-set selection.","supporting_citations":[{"cited_title":"P.; Gemmeke, J","cited_arxiv_id":null,"evidence_quote":"Supplies VGGish, the pre-trained audio feature extractor that the experiments show carries the most emotional information for both tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Xception network fine-tuned on RAF for facial-expression features used for valence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LSTM cells used for both intra-clip per-modality fusion and inter-clip valence context."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the temporal segment network structure that the valence long-context fusion is modeled on."},{"cited_title":"J.; Liu, T.; and Prasad, G","cited_arxiv_id":null,"evidence_quote":"One of the strongest prior LIRIS-ACCEDE baselines (GLA) that the proposed framework claims to beat by a large margin."}],"review_version":1}