{"id":"f8072e29-021d-4f02-8909-87301f5859ea","arxiv_id":"2412.13543","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"QUAG, a query-centric audio-visual cognition network, reports state-of-the-art moment retrieval and segmentation on HIREST and competitive video summarization on TVSum.","lead":"This paper proposes QUAG, a network that combines contrastive audio-visual alignment, cross-attention, and query-based gating to improve moment retrieval, moment segmentation, and step captioning on the HIREST benchmark. It reports gains over the Joint baseline on most metrics and shows competitive transfer to query-based video summarization on TVSum.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies how Whisper transcripts are mapped onto the Nv-frame audio tensor; if the implementation broadcasts a single pooled embedding, MSP's local audio-visual cross-attention (Eq. 3) carries no temporal information and the architecture's central claimed synergy is unsupported.","rationale":"Reader's CONDITIONAL verdict is sound. I examined the central claim that QUAG's query-centric audio-visual representation, built from MSP and QC2, is what makes it state-of-the-art on HIREST. The weakest link is not the equations or the reported tables; it is the unstated construction of the audio tensor. The paper's own text says Ra has the same length as Rv, and the only described audio encoder is MiniLM, a sentence-level model, so the natural reading is that a global transcription embedding is copied across frames. If that is true, the cross-attention in Eq. (3) has a constant key/value sequence and cannot perform the advertised local audio-visual interaction; the entire 'synergy' story reduces to a static bias that could be replicated by an additive query-independent vector. The InfoNCE term in Eq. (2) is computed after mean-pooling, so it does not test or enforce temporal alignment either. This is a genuinely load-bearing modeling assumption because the paper uses it to explain the gains over Joint. The released code can resolve whether the concern lands; the proposed ablation of broadcasting a global mean versus the released Ra, plus code inspection, would settle it. I am not raising statistical-significance issues as primary because, while real, they affect the strength of the empirical claim rather than the meaning of the architecture, and the reader already noted them.","tokens_in":16880,"tokens_out":8706,"duration_ms":87241,"concrete_test":"Open the released QUAG repository and trace how the Whisper/MiniLM output is shaped into Ra with Nv rows; then run the HIREST pipeline with Ra replaced by the per-video mean of that representation broadcast to all Nv rows, keeping all other settings identical. If the broadcast version reproduces the reported R@0.5/R@0.7 and segmentation scores within about 0.5 points, and the repository does not already implement per-timestamp alignment, the temporal audio-visual interaction claimed for MSP is not doing the work; if scores drop materially or the code reveals timestamp-based alignment, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At 'Multi-modal Input Embedding', the authors state that the audio representation Ra has length Nv, but they do not describe how Whisper transcription, a variable-length token stream with its own timestamps, is projected onto the frame timeline. The implementation details say only that Whisper extracts 'speech transcription' and MiniLM maps 'the transcription' into 'a representation'; MiniLM is a sentence encoder, so if the whole transcript is encoded once, the output is a single vector that must be broadcast to Nv rows. Under that construction, Eq. (3)'s MHCA(Rv, Ra, Ra) presents identical key/value vectors at every temporal position, so the 'local fine-grained interaction' degenerates to a global audio bias, and the InfoNCE loss of Eq. (2), computed on mean-pooled vectors, cannot recover temporal audio structure. Because the paper attributes QUAG's gains over Joint specifically to audio-visual synergy and to temporal-channel filtration, this missing alignment is load-bearing. The released code may resolve the ambiguity, but the manuscript as written leaves the central audio-visual mechanism underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QUAG, a query-centric audio-visual cognition network for three HIREST subtasks: moment retrieval, moment segmentation, and step-captioning. The architecture introduces a Modality-Synergistic Perception (MSP) module that applies global contrastive alignment (InfoNCE) and local cross-attention between visual and audio features, followed by a Query-Centric Cognition (QC2) module that performs temporal-channel filtering of the audio-visual representation using the text query. The resulting query-centric representation is fed into a multi-modal encoder and task heads. Experiments on the HIREST dataset report improvements over the Joint baseline and task-specific methods on moment retrieval and segmentation, competitive step-captioning performance, and additional experiments on TVSum for query-based video summarization.","tokens_in":17043,"tokens_out":5164,"duration_ms":48889,"significance":"If the reported results are statistically reliable, QUAG is a reasonable step toward unified multi-task video understanding, and the shallow-to-deep framing is intuitively appealing. The paper ships a public code repository, which strengthens reproducibility. However, the empirical support is weakened by the lack of error bars or significance tests, the small margin over strong baselines on moment retrieval, and a critical underspecification of how audio is temporally aligned with video. The proposed audio-visual synergy mechanism is the main novelty, so the missing alignment detail is a substantial obstacle to validating the central claim.","major_comments":[{"comment":"The paper states that the audio representation Ra has length Nv equal to the video representation, but it never specifies how Whisper transcription, a variable-length token stream, is converted into Nv frame-aligned vectors. If the entire transcription is encoded by MiniLM into a single sentence vector and broadcast across all Nv positions, then in Eq. (3), MHCA(Rv, Ra, Ra) provides identical key and value vectors at every temporal position, so the claimed 'local fine-grained interaction' degenerates to a global audio bias. The InfoNCE loss in Eq. (2) is computed on mean-pooled vectors and cannot recover temporal audio structure. Because the paper attributes QUAG's gains over Joint to audio-visual synergy and to temporal-channel filtration, this missing alignment is load-bearing. Please describe the exact alignment procedure (e.g., Whisper word timestamps, sliding windows, or a learned temporal projection), or, if the implementation broadcasts a pooled embedding, justify why MSP still provides temporal audio information and provide an ablation with a properly aligned audio representation.","section":"Multi-modal Input Embedding / Modality-Synergistic Perception, Eq. (3)"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the HIREST results. The test set contains 1,391 video-query pairs, and the margins over the strongest task-specific baselines are narrow; for example, in Table 1 the gap to UVCOM is +0.42 R@0.5 and +0.33 R@0.7. The claim that QUAG achieves state-of-the-art results is therefore not robustly supported by the reported numbers. Please report the mean and standard deviation over at least three training seeds, and, if possible, a paired bootstrap test or significance test against the strongest baseline.","section":"Tables 1-3, Performance Comparison on HIREST"},{"comment":"The supplementary reports step-captioning metrics for λ values 0, 0.0001, ..., 0.0006 and selects λ=0.0003, but it does not state whether these evaluations were performed on the validation split or the test split. If Table 6 reports test-set numbers for each λ, then the final test results are optimistically biased because the hyperparameter was selected directly on the test set. Please clarify the selection split; if λ was selected on the test set, re-run the evaluation with λ chosen on the validation split and update the main results accordingly.","section":"Supplementary, Study on the Trade-off Parameter λ, Eq. (15)"}],"minor_comments":[{"comment":"In the definition of Pstep, the variable bt is used in the formula, but the following sentence says 'where Wt ∈ RD×1 and be ∈ R1'; this should be bt, not be.","section":"Methodology, Eq. (10)"},{"comment":"The paper claims 'SOTA results on HIREST' without qualification, but the experiments only address three of the four HIREST tasks; video retrieval is described in the task definition but never evaluated in the paper. Please qualify the claim to the three evaluated subtasks.","section":"Abstract and Introduction"},{"comment":"The supplementary says 'The ablation study on Ate and Ach is shown in Table 4 of the main paper', but Table 4 reports the MSP/QC2 module ablation, not the Ate/Ach ablation; the Ate/Ach results actually appear in Table 7 of the supplementary. The cross-reference should be corrected.","section":"Supplementary, Ablation Study for Query-Centric Cognition (QC2)"},{"comment":"The overview figure is dense and the small font sizes make the MSP and QC2 data flows difficult to follow; consider enlarging the figure or presenting the two modules in separate panels.","section":"Figure 2"},{"comment":"The phrase 'the audio information helps perceptive the main objects' is ungrammatical; also, the query representation is described as Rt ∈ RD after projection, but the query consists of Nt tokens and the token-to-vector reduction is not described. Please clarify how the token-level query is pooled or projected.","section":"Multi-modal Input Embedding"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the audio alignment specification. If the released code turns out to broadcast a single sentence embedding across the temporal dimension, the central audio-visual synergy mechanism is unsupported and the paper would need substantial revision. I recommend that the editor require the authors to document the exact audio-to-frame alignment in the manuscript and, if necessary, provide an ablation with a genuinely time-resolved audio representation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a real, if incremental, empirical contribution to the HIREST line. The new thing is the QUAG combination — contrastive global audio-visual alignment, local cross-attention, query-driven temporal-channel gating — and it reports consistent gains over Joint on moment retrieval and segmentation. The ablation supports that both modules matter. The release of code is a plus.\n\nThe main soft spot is one the paper itself leaves open: it never says how Whisper transcriptions become an Nv-length audio tensor. If MiniLM encodes the whole transcript once and the vector is broadcast across frames, then Eq. (3)'s MHCA(Rv, Ra, Ra) has identical keys and values at every temporal position, and the 'local fine-grained interaction' is really a global audio bias. The InfoNCE loss on mean-pooled vectors cannot recover temporal structure either. The code may resolve this, but the manuscript as written underdetermines the central audio-visual mechanism.\n\nOther weaknesses are real but not fatal. No error bars or significance tests, and the margins over task-specific baselines are thin: 72.54 vs 72.12 at R@0.5, 38.86 vs 38.53 at R@0.7. The contrastive weight lambda is selected on the validation set across a range, which is tuning on the benchmark. The 'SOTA' claim is too strong for step-captioning, where SwinBERT beats QUAG on ROUGE-L and SPICE.\n\nWhat the paper does well: it is clearly written, the shallow-to-deep framing is implemented straightforwardly, the ablations are informative, and the TVSum generalization experiment is a sensible check. If the audio construction is what I suspect (a pooled broadcast), the gains over Joint may still hold via global audio conditioning plus the query gating; the architecture would not collapse, but the claimed audio-visual synergy would be less interesting.\n\nWho this is for: researchers working on HIREST or query-conditioned multi-modal fusion. It deserves a serious referee. My recommendation: send it to review, but with a requirement to specify the audio temporal alignment and report multiple-seed results.","headline":"A solid incremental contribution to HIREST with a real but fixable gap: the paper never explains how Whisper transcriptions become the Nv-length audio tensor, so the central audio-visual synergy claim is underdetermined as written.","tokens_in":17630,"tokens_out":2468,"would_cite":true,"duration_ms":23129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a query-centric audio-visual network, which aligns audio and video before filtering the fused representation with the text query, achieves state-of-the-art results on HIREST moment retrieval and segmentation and…","keywords":["moment retrieval","moment segmentation","step-captioning","audio-visual representation","query-centric representation","multi-task learning","HIREST","contrastive learning"],"falsifier":"Replace the audio stream with a constant vector repeated on every frame and retrain QUAG on HIREST; if moment-retrieval Recall@0.5 stays within one point of 72.54, the claimed audio-visual synergy is not what drives the result.","tokens_in":16610,"feed_emoji":"🎬","tokens_out":10088,"duration_ms":76741,"temperature":0.7,"pith_summary":"This paper aims to establish that a query-centric audio-visual representation—built by first aligning visual and audio at both global and local levels and only then letting the text query filter the fused content—is a stronger foundation for the three HIREST tasks than the direct element-wise fusion used by the Joint baseline. On the HIREST benchmark, the proposed QUAG network reports the best moment-retrieval results (Recall@0.5 IoU of 72.54, Recall@0.7 IoU of 38.86), the best moment-segmentation results (Recall@0.5 IoU of 39.27, Precision@0.5 IoU of 31.68), and competitive step-captioning (CIDEr of 25.44, entailment of 40.10) against task-specific models. The authors also show the same representation transfers to query-based video summarization on TVSum, where QUAG reaches an average top-5 mAP of 87.0, on par with a task-specific baseline. If the claim holds, it suggests that explicitly structuring modality hierarchies—shallow audio-visual perception before deep query cognition—matters more than the capacity of the fusion operation itself.","feed_headline":"Audio-visual query gating tops HIREST moment retrieval","feed_subtitle":"Modeling audio-visual synergy before querying the video lifts recall and caption quality on the HIREST benchmark.","key_machinery":"The central object is the query-centric audio-visual representation $\\tilde{R}_m$, produced by two modules. Modality-synergistic perception first applies an InfoNCE contrastive loss to the mean-pooled visual and audio features so the two modalities share an embedding space, then exchanges local token-level information through multi-head cross-attention and fuses the two streams with a linear layer. Query-centric cognition then concatenates the text query with the audio-visual representation, projects the result to a temporal attention vector $A_{te}$ and a channel attention vector $A_{ch}$ through sigmoid-gated linear layers, and multiplies their element-wise product into the audio-visual representation; a self-attended version of the query-augmented representation is added back. This filtered, query-centric representation is fed to a transformer encoder and the prediction heads. The load-bearing property is that the query acts as a multiplicative temporal-channel gate on audio-visual content that has already been brought into alignment, rather than being fused with the visual and audio streams from the start.","core_discovery":"The paper's discovery is that replacing undifferentiated multimodal fusion with a two-stage, shallow-to-deep processing pipeline improves all three HIREST tasks. In the modality-synergistic perception stage, an InfoNCE contrastive loss pulls global visual and audio features into a shared space, then multi-head cross-attention exchanges local token-level information between the two modalities before a linear fusion produces an audio-visual representation. In the query-centric cognition stage, the text query is concatenated with that representation, projected to temporal and channel attention maps via sigmoid-gated linear layers, and the element-wise product of these maps filters the audio-visual representation; the filtered output is then combined with a self-attended version of the query-augmented representation. QUAG reports state-of-the-art numbers on HIREST moment retrieval and moment segmentation, and improves step-captioning over the Joint baseline on every metric, with particularly large gains on CIDEr (from 21.19 to 25.44) and entailment (from 35.97 to 40.10). The authors argue this validates that association relations across modalities are best modeled before the query filters content, rather than fused indiscriminately with it.","pith_inferences":["Because the paper never specifies how Whisper transcriptions are mapped to the Nv frame positions, the audio stream may be a single sentence embedding repeated across frames; if that is the case, the MSP cross-attention cannot be doing frame-level audio-visual alignment, and the gains would come mostly from visual features and the query gate. A direct ablation would replace the audio stream with a","The QC2 temporal-channel gate is essentially a learned multiplicative mask; the same mask can be applied to other query-conditioned video tasks such as highlight detection, dense video captioning, or weakly supervised moment grounding, and compared with additive fusion to see whether the gating form, rather than the specific encoders, carries the benefit.","On TVSum the features are clip-level I3D and PANN rather than the EVA-CLIP, Whisper, and MiniLM features used for HIREST, so the reported generalization may reflect the gating structure rather than the audio-visual encoders; testing QUAG with identical feature extractors on both datasets would separate those factors."],"forward_implications":["Moment retrieval on HIREST improves over the Joint baseline by 2.2 points at Recall@0.5 IoU and 4.2 points at Recall@0.7 IoU, and it also exceeds the strong task-specific baselines such as UVCOM and QD-DETR.","Moment segmentation improves by 3.0 points in Recall@0.5 IoU and 3.1 points in Precision@0.5 IoU over the Joint baseline, indicating more accurate step-boundary localization.","Step-captioning benefits from the query-centric representation: CIDEr rises from 21.19 to 25.44, SPICE from 3.02 to 4.49, and entailment from 35.97 to 40.10 compared with Joint.","The same query-centric audio-visual representation transfers to query-based video summarization on TVSum, reaching an average top-5 mAP of 87.0 that is on par with the task-specific TR-DETR (87.1).","Ablations show that the two modules complement each other: with both MSP and QC2, CIDEr improves 20.1 percent and SPICE 48.7 percent over the Joint baseline, whereas each module alone gives smaller or inconsistent gains."],"supporting_citations":[{"why":"Defines the HIREST tasks and the Joint baseline that QUAG is built on and compared against.","marker":"(Zala et al. 2023)"},{"why":"Supplies the InfoNCE loss used for global visual-audio contrastive alignment in MSP.","marker":"(Oord, Li, and Vinyals 2018)"},{"why":"Provides the multi-head cross-attention and self-attention operations used for local interaction and query-centric fusion.","marker":"(Vaswani et al. 2017)"},{"why":"EVA-CLIP pre-trained model is used to extract visual and text query representations.","marker":"(Fang et al. 2023)"},{"why":"Whisper produces the speech transcription that becomes the audio representation.","marker":"(Radford et al. 2023)"},{"why":"MiniLM encodes the transcription into the audio embedding.","marker":"(Reimers and Gurevych 2019)"},{"why":"Round-robin multi-task training paradigm used to jointly optimize retrieval, segmentation, and captioning.","marker":"(Cho et al. 2021)"},{"why":"QD-DETR is a task-specific baseline and the source of the TVSum evaluation setup.","marker":"(Moon et al. 2023)"},{"why":"CLIP4Caption initializes the text decoder for step-captioning.","marker":"(Tang et al. 2021)"}],"fun_headline_variants":["Two-stage audio-visual querying tops HIREST","QUAG: query-aware fusion lifts all HIREST tasks","Audio-visual synergy then query gating: SOTA on HIREST","Shallow-to-deep multimodal filter wins HIREST","Query-centric cognition beats fused baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the audio representation lines up with the video frame by frame, but it never describes how the transcribed speech is laid onto the frame timeline; if each frame just receives the same sentence embedding, the audio-visual interaction has no frame-level timing to work with.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage audio-visual querying tops HIREST","QUAG: query-aware fusion lifts all HIREST tasks","Audio-visual synergy then query gating: SOTA on HIREST","Shallow-to-deep multimodal filter wins HIREST","Query-centric cognition beats fused baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1516,"prompt_tokens":1038,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":654,"tokens_out":478,"duration_ms":5463,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:01:06.497651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the audio stream with a constant vector repeated on every frame and retrain QUAG on HIREST; if moment-retrieval Recall@0.5 stays within one point of 72.54, the claimed audio-visual synergy is not what drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the HIREST tasks and the Joint baseline that QUAG is built on and compared against."},{"cited_title":"N.; Kaiser, .; and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Provides the multi-head cross-attention and self-attention operations used for local interaction and query-centric fusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EVA-CLIP pre-trained model is used to extract visual and text query representations."},{"cited_title":"W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Whisper produces the speech transcription that becomes the audio representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MiniLM encodes the transcription into the audio embedding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Round-robin multi-task training paradigm used to jointly optimize retrieval, segmentation, and captioning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QD-DETR is a task-specific baseline and the source of the TVSum evaluation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP4Caption initializes the text decoder for step-captioning."}],"review_version":1}