{"id":"d95d985f-5d70-4fd4-a82a-88d7954a6a88","arxiv_id":"2507.02080","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TAGF adds a BiLSTM-based gate that reweights recursive cross-attention outputs for valence-arousal prediction, with results slightly below several existing methods on Aff-Wild2.","lead":"This paper presents an emotion-recognition model that blends audio and facial-video cues using a learned gate across repeated attention passes. It reports mid-tier scores on a standard benchmark, but does not test its headline claim of handling misaligned audio and video.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BiLSTM gate in Eqs. 6–8 runs over recursive-refinement steps, not video frames, so it cannot implement the claimed frame-level robustness to noise and misalignment.","rationale":"The paper's central claim is that temporal awareness embedded in the recursive fusion process captures sequential emotional evolution and yields robustness to cross-modal misalignment. For that claim to hold, the gating mechanism in Eqs. 6–8 must be able to modulate frame-level information as a function of video time. The equations do not provide this: the BiLSTM is applied along the recursive-attention axis, and Eq. 7 produces a single scalar per recursive step, so the same weight is applied to all frames in Eq. 8. Even if the BiLSTM learns useful context about which refinement iteration is most informative, it cannot suppress noisy, occluded, or temporally misaligned frames selectively, which is exactly the capability promised in §3.3.2 and the abstract. This is not a disagreement with external consensus; it is an internal mismatch between the stated contribution and the described mechanism. The experiments provide no alternative support: there is no misalignment experiment, no gating ablation, no error bars, and the reported scores on Aff-Wild2 are below the recursive-attention baselines the method extends. Given that the central contribution is unsupported by both the architecture and the evidence, the reader's REJECT verdict remains appropriate. The paper could be salvaged by reformulating the gate to operate per frame or by explicitly testing misalignment robustness, but as written the rejection stands.","tokens_in":11157,"tokens_out":5562,"duration_ms":68038,"concrete_test":"Remove the BiLSTM gate by replacing Eqs. 6–8 with a uniform average over the T recursive outputs and re-run the Aff-Wild2 validation experiment with three seeds and the same preprocessing. If the CCC does not drop, or if it drops by the same amount under ±3/±6/±9-frame audio shifts relative to the visual stream, the BiLSTM gate adds no measurable temporal or robustness value and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In TAGF, H(t) from Eqs. 3–4 are updated representations of the full frame sequence, so each H(t) is L×d (or, if read literally from §3.3.1, a whole-clip vector that would make frame-level prediction impossible). The temporal encoder f_temp in Eq. 6 therefore processes T recursive-refinement steps, not the L video frames; T is small in recursive attention methods, typically 2–3. Eq. 7 produces one scalar α_t per step, and Eq. 8 applies that same weight to every frame. Consequently the gate has no frame index and cannot suppress an occluded face or off-screen speech at frame l while keeping frame l′: the architecture-level mechanism required for the robustness claim is absent. The actual temporal modeling in the pipeline comes from the TCNs on the backbone features, not from the BiLSTM gate. The experiments also do not test the claim: there is no misalignment or noise ablation, no gating ablation, no error bars, and in Tables 1–2 TAGF's valence is below JCA/GRJCA/HGRJCA and its average CCC (0.552 validation, 0.540 test) is below GRJCA (0.556/0.581) and HGRJCA (0.562/0.591). The cited GRJCA/HGRJCA reference [29] is a psychotherapy article, though this is a citation error rather than the core architectural problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAGF, a Time-aware Gated Fusion framework for audio-visual valence-arousal estimation. The claimed contribution is a BiLSTM-based gate that reweights the outputs of recursive joint cross-attention steps (Eqs. 6–8) so that the fusion becomes time-aware, captures the sequential evolution of emotion, and is robust to cross-modal misalignment and noise. The model is evaluated on the Aff-Wild2 dataset with official validation and test splits, reporting mean CCC values of 0.552 (validation) and 0.540 (test). The central claim is that embedding temporal awareness into recursive fusion improves robustness to noisy or misaligned audio-visual streams without frame-level supervision.","tokens_in":11405,"tokens_out":6569,"duration_ms":73769,"significance":"If the claimed mechanism worked as described, it would be a lightweight, useful extension to recursive joint cross-attention models for dimensional emotion recognition. The paper provides a clean formulation of the gating equations, uses an established external benchmark (Aff-Wild2), and reports results on the official test server, which are all strengths. However, the core architectural claim is not supported by the equations as written: the gating weights are computed over recursive refinement steps, not over video frames, so the mechanism cannot implement the claimed frame-level suppression of misaligned or noisy inputs. In addition, the experimental section contains no misalignment or noise ablation, no gating ablation, and no error bars, and the reported average CCC is below the GRJCA/HGRJCA baselines to which the paper most directly relates. The paper is therefore better characterized as an unsupported incremental modification than as a validated robustness mechanism.","major_comments":[{"comment":"The temporal gate runs over recursive-attention steps, not over video frames, so it cannot deliver the claimed frame-level robustness. Equation (6) defines each H(t) as a vector in R^d and G as an element of R^{T×d}; Eq. (7) produces one scalar α_t per recursive step; and Eq. (8) applies that same scalar to every frame. There is no frame index l in α_t. The BiLSTM therefore models the T recursive-refinement steps, not the L video frames of Eq. (1). This contradicts the statement in Section 3.2 that the module produces \"attention weights for each frame and step\" and the claim in Section 3.3.2 that the model can suppress \"outputs that may be noisy or temporally misaligned, such as those corresponding to occluded faces or off-screen speech frames.\" The actual frame-level temporal modeling in the pipeline is performed by the TCNs described in Section 4.2.1, not by the proposed gate. This mismatch directly affects the central claim in the Abstract and the Conclusion.","section":"Section 3.3.1, Eqs. (6)–(8)"},{"comment":"There is no experiment that tests cross-modal misalignment or noise, despite the Abstract and Conclusion claiming \"strong robustness\" to these conditions. The evaluation reports only clean CCC values on the official splits. Moreover, there is no ablation that removes or replaces the gating module; because Tables 1–2 compare against methods with different backbones, training schedules, and auxiliary components, any observed difference cannot be attributed to the BiLSTM gate. The average CCC of TAGF (0.552 validation, 0.540 test) is below GRJCA (0.556/0.581) and HGRJCA (0.562/0.591), so even the \"competitive performance\" claim requires more careful qualification, and the robustness claim has no direct empirical support.","section":"Section 4.4, Tables 1–2"},{"comment":"The paper reports no error bars, no multiple-seed statistics, and no sensitivity analysis over the number of recursive steps T or the BiLSTM hidden size. The statement in Section 4.4.1 that \"consistent results observed across cross-validation folds beyond the official split\" is not accompanied by any table, figure, or numerical summary. Given that the reported differences between TAGF and the relevant baselines are small (e.g., 0.552 versus 0.556 on the validation set), single-checkpoint CCC values cannot support the claim of stable or robust improvement.","section":"Section 4.2.2 and Section 4.4.1"}],"minor_comments":[{"comment":"The GRJCA and HGRJCA baselines are cited as reference [29], but [29] is Mennin et al., a psychotherapy article in Behavior Therapy, not an audio-visual emotion recognition paper. The correct references appear to be [34] and [37].","section":"Tables 1–2"},{"comment":"The claim in Section 4.4.1 that the model is trained \"without relying on multiple pretrained backbones or external resources\" is contradicted by Section 4.2.1, which uses a ResNet-50 pretrained on MS-Celeb-1M and a VGGish model pretrained on AudioSet.","section":"Section 4.2.1"},{"comment":"Key implementation details are missing: the number of recursive cross-attention steps T, the BiLSTM hidden size, the TCN layer configuration, and whether the gate is applied to visual and audio features jointly or separately. No code or repository is provided.","section":"Section 4.2.2"},{"comment":"There are several typos and formatting issues: \"seperated\" in Section 4.2.1, \"an Multi-Layer Perceptron\" in the Figure 2 caption, and the column header \"CCC A vg\" in Tables 1 and 2, which should be \"CCC Avg.\"","section":"General presentation"},{"comment":"The test set is evaluated through the official challenge server, so describing it as \"fold-0\" is misleading; the test split is not a cross-validation fold.","section":"Table 2 caption"}],"recommendation":"reject","confidential_remarks":"The manuscript is an incremental modification of recursive joint cross-attention, but the mechanism as described cannot perform the claimed frame-level gating, and the experiments do not include the misalignment/noise evaluations that would be required to support the central robustness claim. The incorrect reference for the main baselines in Tables 1–2 is also a verification concern. If the authors were to move the gating to the frame axis, add ablations and misalignment experiments, and substantially narrow the claims, a new submission might be viable, but I do not see a publishable revision within the current manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TAGF is a small, real extension of recursive joint cross-attention, but the paper's central claims about temporal modeling and misalignment robustness don't survive contact with the architecture or the numbers. I'd reject as is, though the idea might be salvageable.\n\nWhat's new: the BiLSTM gate over the T recursive cross-attention outputs (Eqs. 6-8) is a legitimate modification of the static combination used in GRJCA/HGRJCA. The authors report both validation and official test CCC on Aff-Wild2, trained without external data, and the writing is mostly clear. That's the extent of the good news.\n\nThe main problem is the gap between claim and mechanism. The gate runs over the recursion-step index, not over video frames. Each H(t) is a full-sequence representation, so Eq. 7 produces a single scalar per step, applied equally to every frame. That cannot suppress an occluded or misaligned frame at a specific time, which is exactly what the abstract and Section 3.3.2 promise. The paper even says the weights are 'for each frame and step' in Section 3.2, but Eq. 8 says otherwise. No misalignment experiment, no ablation of the gate, and no error bars are reported.\n\nThe numbers are also against the claims. On the validation set, TAGF's average CCC is 0.552, below GRJCA's 0.556 and HGRJCA's 0.562. On the test set, 0.540 sits well below both baselines (0.581 and 0.591). Calling that 'competitive' is a stretch; it is below the exact methods it extends. The reference list also mis-cites [29] as the GRJCA source, when that reference is a psychotherapy article.\n\nIf the authors reframed the contribution as 'soft weighting of recursive refinement steps' and added a real misalignment/noise experiment, this could become a modest workshop-level paper. As submitted, the load-bearing robustness claim is unsupported and the empirical evidence is marginal. I would not accept it for a serious venue, but I would send it to a referee if the venue wants to give the authors a chance to fix the framing and fill in the missing experiments. For now, reject.","headline":"Real but modest extension; central robustness claim unsupported by architecture or results.","tokens_in":11992,"tokens_out":3137,"would_cite":false,"duration_ms":35679,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A BiLSTM gate over recursive attention steps makes audio-visual fusion time-aware and robust to misalignment.","keywords":["multimodal emotion recognition","valence-arousal estimation","temporal gating","recursive cross-attention","BiLSTM","audio-visual fusion","Aff-Wild2","robustness to misalignment"],"falsifier":"Shuffle the order of the recursive outputs before the BiLSTM and retrain; if the CCC stays the same, the gate is not using temporal order. A second check: replace the recursion-axis BiLSTM with a gate applied along the actual frame axis; if that version outperforms TAGF on offset audio-visual inputs, the claimed cross-modal robustness is not explained by the recursion-axis gating.","tokens_in":10877,"feed_emoji":"🎭","tokens_out":12493,"duration_ms":117558,"temperature":0.7,"pith_summary":"This paper claims that the reason multimodal valence-arousal estimation fails under noisy or misaligned audio-visual streams is that existing recursive-attention fusion gates every time step the same way. It proposes TAGF, which treats the $T$ outputs of recursive joint cross-attention as a time-ordered sequence, encodes them with a bidirectional LSTM, and uses the resulting gating vectors to weight a sum of those outputs. On Aff-Wild2, TAGF reports a valence CCC of 0.427 and an arousal CCC of 0.676 on the validation set, and 0.512 and 0.568 on the test set, and the paper argues that the time-aware weights let the model suppress noisy or misaligned frames without frame-level supervision. If correct, the claim matters because it offers a simple, label-free way to make multimodal fusion robust in real-world emotion recognition.","feed_headline":"Time-aware gating lifts arousal CCC to 0.676 on Aff-Wild2","feed_subtitle":"A BiLSTM gate reweights recursive attention outputs so models ignore noisy, misaligned frames without extra labels.","key_machinery":"The central object is a BiLSTM-based temporal gating mechanism placed on top of recursive joint cross-attention. The recursion produces one representation $H^{(t)}$ per joint attention step; the encoder $f_{\\mathrm{temp}}$ (a bidirectional LSTM) reads the ordered sequence $\\{H^{(1)}, \\dots, H^{(T)}\\}$ and emits gating vectors $G = \\{g_1, \\dots, g_T\\}$, and a learned parameter vector $w$ turns each $g_t$ into a softmax scalar $\\alpha_t$ that weights $H^{(t)}$ in the final sum. This machinery converts a static pool-over-recursion-steps into temporal attention pooling that is end-to-end trainable and requires no frame-level labels.","core_discovery":"The central claim is that temporal awareness belongs inside the fusion step, not only in feature extraction: the outputs $\\{H^{(1)}, \\dots, H^{(T)}\\}$ of recursive joint cross-attention form a sequence whose ordering encodes how the modalities refine each other, and a bidirectional LSTM (Eq. 6) can read that sequence to assign each step a context-dependent gate. Aggregation is then $F = \\sum_{t=1}^{T} \\alpha_t H^{(t)}$ with $\\alpha_t = \\operatorname{softmax}(w^\\top g_t)$ (Eqs. 7-8), so the model can upweight informative recursive steps and downweight noisy or temporally misaligned ones. TAGF reports valence CCC 0.427 and arousal CCC 0.676 on the Aff-Wild2 validation split, and 0.512 and 0.568 on the official test split, and attributes its robustness to this time-aware gating.","pith_inferences":["The authors leave implicit that the recursion axis is not the video-time axis: if only a few recursive steps are used, a BiLSTM over whole-sequence outputs cannot literally perform per-frame suppression, so the reported robustness likely also depends on the frame-level temporal context learned by the backbone and temporal convolutional layers.","Nothing in the gating equations is emotion-specific, so the same construction could be tested on other continuous-output multimodal regression tasks with offset or partially dropped streams, such as pain or engagement estimation.","A natural testable extension would run the same gate along the actual frame axis instead of the recursion axis; the paper does not compare these two placements, and that comparison would separate temporal modeling of emotions from temporal modeling of the refinement process."],"forward_implications":["A fusion model can be made time-aware by gating its recursive attention outputs, so the approach transfers to any recursive cross-attention architecture without changing the underlying attention layers.","The time-aware gate suppresses noisy or misaligned steps such as occluded faces or off-screen speech without masking heuristics or frame-level supervision.","Training only on Aff-Wild2 with a single modality pair, TAGF reaches an arousal CCC of 0.676 on validation, second among the compared methods, which suggests the gating helps most where temporal dynamics dominate.","Because the gate weights are computed per video from the recursive outputs, the fusion strategy adapts to each sequence rather than applying one fixed gating pattern across all time steps.","On the official test set, the model's average CCC of 0.540 stays competitive with larger pipelines, supporting the paper's claim that the mechanism generalizes beyond the validation split."],"supporting_citations":[{"why":"Supplies the Aff-Wild2 dataset, its subject-independent splits, and the valence-arousal annotations used in all reported comparisons.","marker":"[17]"},{"why":"Introduces the recursive joint cross-modal attention whose per-step outputs $H^{(t)}$ are the sequence TAGF feeds into the BiLSTM gate.","marker":"[34]"},{"why":"Proposes the recursive joint attention framework and the static gated fusion that TAGF extends by making the weights time-dependent.","marker":"[37]"},{"why":"Provides the joint cross-attention model used as a comparison baseline and the representational lineage for TAGF's attention layers.","marker":"[35]"},{"why":"Defines the VGGish audio preprocessing pipeline that produces the log-mel spectrogram features for the acoustic stream.","marker":"[41]"}],"fun_headline_variants":["TAGF: Temporal gate reweights recursive attention for VA estimation","Time-aware gating boosts arousal CCC to 0.676 on Aff-Wild2","BiLSTM gate in fusion step improves multimodal emotion recognition","TAGF: temporal gating handles misaligned audio-visual for VA","TAGF learns step-wise gates to combat noisy misaligned frames"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the order of the recursive attention steps carries real video-time meaning, so a bidirectional LSTM over those few steps can model moment-to-moment emotional changes; if the order is just an implementation detail, the gate cannot actually suppress individual noisy or shifted frames.","fun_headline_variants_meta":{"raw":{"variants":["TAGF: Temporal gate reweights recursive attention for VA estimation","Time-aware gating boosts arousal CCC to 0.676 on Aff-Wild2","BiLSTM gate in fusion step improves multimodal emotion recognition","TAGF: temporal gating handles misaligned audio-visual for VA","TAGF learns step-wise gates to combat noisy misaligned frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2773,"prompt_tokens":900,"completion_tokens":1873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1776}},"tokens_in":516,"tokens_out":1873,"duration_ms":13515,"temperature":1.0,"reasoning_tokens":1776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:38:42.658518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the order of the recursive outputs before the BiLSTM and retrain; if the CCC stays the same, the gate is not using temporal order. A second check: replace the recursion-axis BiLSTM with a gate applied along the actual frame axis; if that version outperforms TAGF on offset audio-visual inputs, the claimed cross-modal robustness is not explained by the recursion-axis gating.","supporting_citations":[{"cited_title":"Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architec- tures, and beyond","cited_arxiv_id":null,"evidence_quote":"Supplies the Aff-Wild2 dataset, its subject-independent splits, and the valence-arousal annotations used in all reported comparisons."},{"cited_title":"Recursive joint cross- modal attention for multimodal fusion in dimensional emo- tion recognition","cited_arxiv_id":null,"evidence_quote":"Introduces the recursive joint cross-modal attention whose per-step outputs $H^{(t)}$ are the sequence TAGF feeds into the BiLSTM gate."},{"cited_title":"Re- cursive joint attention for audio-visual fusion in regression based emotion recognition","cited_arxiv_id":null,"evidence_quote":"Proposes the recursive joint attention framework and the static gated fusion that TAGF extends by making the weights time-dependent."},{"cited_title":"A joint cross-attention model for audio-visual fusion in dimensional emotion recognition","cited_arxiv_id":null,"evidence_quote":"Provides the joint cross-attention model used as a comparison baseline and the representational lineage for TAGF's attention layers."}],"review_version":1}