{"id":"1c2de01d-6a65-4a8f-a09e-2eae8c36d960","arxiv_id":"2501.10692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"MRNet fuses RGB, optical flow, and depth features with word-, phrase-, and sentence-level query features, and reports improved moment retrieval and highlight detection scores on QVHighlights and Charades-STA.","lead":"This paper introduces MRNet, a model that combines RGB, optical flow, and depth cues to locate video moments and highlights from text queries. It reports gains over prior methods on the QVHighlights and Charades-STA benchmarks, driven by a multi-modal fusion module and a multi-granularity query refinement module.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing issue is not CLIP transfer but the SOTA claim itself: Table 1 compares only UMT, MomentDiff, SeViLA, and Moment-DETR, omitting strong later methods; without a full leaderboard comparison or released code the '+3.41/+3.46' numbers are not established as SOTA.","rationale":"The reader's weakest assumption (CLIP transfer) is not the most load-bearing point. Even if depth/flow features are not text-aligned, MFM uses RGB features as query in cross-attention, so depth/flow only need to supply complementary non-semantic structure; Table 5 actually provides internal evidence that the modalities help. The claim that would falsify the paper's central contribution is the SOTA claim itself, and that claim depends on the baseline set in Tables 1 and 2. Those tables omit several strong methods published between 2022 and the submission date, so the reported margins are against outdated competitors. I would keep the verdict CONDITIONAL because the issue is an unverified empirical assertion, not an internal inconsistency; a full leaderboard comparison plus released code would either validate or refute the headline. This aligns with the reader's overall CONDITIONAL assessment, although not with the specific weakest assumption.","tokens_in":7757,"tokens_out":9259,"duration_ms":90674,"concrete_test":"Compile all publicly available QVHighlights test-split results (official CodaLab leaderboard and papers through Jan 2025), and re-run MRNet on the test split using released code/weights or author-provided reproduction. If any omitted method has MR-mAP@Avg > 39.53 or HD-HIT@1 > 63.45 on official evaluation, or if the reproduced MRNet numbers fall below the reported values, the headline SOTA claim is not established; if no omitted baseline exceeds MRNet, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is 'MRNet outperforms current SOTA methods' (abstract; §4.3). Table 1 supports this with only four baselines: Moment-DETR (2021), SeViLA (2023), MomentDiff (2023), UMT (2022). For a Jan-2025 arXiv paper, this is not the current SOTA set; the QVHighlights leaderboard contains many later supervised methods, and Table 2 likewise has only three baselines on Charades-STA. No code/weights are released, so the missing comparisons cannot be run by a reader. Hence '+3.41 MR-mAP@Avg' and '+3.46 HD-HIT@1' are improvements over UMT, not over the current best model. This is a support gap rather than an internal contradiction: the ablations are directionally consistent and the reported test numbers are plausible. But a SOTA claim is falsified by one omitted stronger baseline, and this paper provides no evidence against that possibility.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MRNet for joint video moment retrieval and highlight detection. MRNet extracts RGB, optical flow, and depth features with a frozen CLIP video encoder, fuses them through a gated multi-modal fusion module, and refines query features via word-, phrase-, and sentence-level aggregation. The architecture removes the transformer decoder of Moment-DETR and uses a cross-attention transformer plus an encoder-only stack. Experiments on QVHighlights and Charades-STA report improvements over UMT and three other baselines, with headline gains of +3.41 MR-mAP@Avg and +3.46 HD-HIT@1 on QVHighlights.","tokens_in":7987,"tokens_out":6305,"duration_ms":60976,"significance":"If the empirical claims are verified, the paper contributes a practical architectural combination: using optical flow and depth alongside RGB through a frozen CLIP encoder, and a multi-granularity query refinement module. The ablations on the validation split are well structured and provide directional support for each module. The paper ships no code or weights, however, and the state-of-the-art claim rests on a very small baseline set; for a January 2025 manuscript, the comparison omits several later methods on the QVHighlights leaderboard. The central idea is plausible and the reported gains are internally consistent, but the headline 'outperforms current SOTA' is not yet established. The contribution is incremental rather than foundational, but it is a reasonable empirical study if the comparison and evaluation rigor are strengthened.","major_comments":[{"comment":"The headline claim that MRNet 'outperforms current SOTA methods' is supported by comparisons against only four baselines: Moment-DETR (2021), SeViLA (2023), MomentDiff (2023), and UMT (2022). For a January 2025 manuscript, this is not the current state-of-the-art set on QVHighlights; several later supervised methods, e.g., QD-DETR and UnLoc, are missing, and Table 2 has only three baselines on Charades-STA. Consequently, the reported margins of +3.41 MR-mAP@Avg and +3.46 HD-HIT@1 are established relative to UMT, not to the current best model. Please expand the baseline comparison to contemporary methods and/or release code and features so the claim can be independently verified.","section":"Section 4.3, Table 1"},{"comment":"All reported metrics are single-run point estimates without error bars, standard deviations, or significance tests. Given that several margins are small (for example, some HD metrics differ by less than 1 point), run-to-run variance could be comparable to the claimed improvements. Please run at least three seeds and report mean and standard deviation, or otherwise justify the stability of the results.","section":"Tables 1-6"},{"comment":"The statement that 'all models were trained from scratch' needs clarification. No details are given for how the baselines were retrained, which backbones or features they used, or whether their released implementations were used. In particular, UMT includes audio features while MRNet does not, so the comparison protocol should be explicit. If the baseline numbers are copied from the original papers rather than obtained under a common protocol, the comparison is not controlled and the 'fairness' claim should be revised.","section":"Section 4.3, comparison protocol"},{"comment":"The method applies a frozen CLIP video encoder to depth maps and optical-flow frames, assuming those feature vectors remain semantically meaningful and text-aligned. Table 5 gives indirect empirical support by showing that adding flow and depth improves over RGB alone, which is a strength. However, the paper does not compare with modality-specific encoders or with fine-tuning the backbone. Please add an ablation or at least a discussion of how the CLIP transfer assumption may limit generalization beyond this specific pipeline.","section":"Section 3.1"}],"minor_comments":[{"comment":"The text refers to a 'classification loss Ncls', but the symbol should presumably be L_cls to match the equation, and the hyperparameters introduced as λ* are not all defined in one place.","section":"Section 3.5, Eq. (2)"},{"comment":"The checkmark alignment in Table 3 is ambiguous; the reader cannot tell which module (MFM, QRM, or CAT) is active in each row. Please reformat the table with clear column labels or explicit 'baseline + X' row labels.","section":"Table 3"},{"comment":"The text claims that 'Row 7 shows optimal performance using all three features', but the RGB+Depth row has a higher R1@0.5 (63.10 vs. 62.00) and the RGB+Flow row has a higher mAP@0.5 (40.10 vs. 40.34 for all three). Qualify the claim or discuss why the aggregated metric favors the three-feature configuration.","section":"Table 5"},{"comment":"The qualitative comparison shows only Moment-DETR versus MRNet. A comparison against a stronger baseline such as UMT would be more informative for assessing the claimed state-of-the-art result.","section":"Figure 5"},{"comment":"Some references are incomplete or inconsistently formatted; for example, [5] gives only an arXiv identifier and [11] misses the volume and page numbers for the IEEE Signal Processing Letters article.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a conference-style technical report than a full journal paper. The main barrier is the unsupported SOTA claim: the baseline set is too small and dated for a January 2025 submission, and no code or weights are provided to let readers verify the numbers independently. I would recommend the editor require an expanded baseline comparison and a reproducibility statement before further consideration. Additionally, the relationship to the authors' own prior work MH-DETR [12] should be clarified, as the decoder-free architecture appears to overlap with that line of work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read of arXiv:2501.10692. The useful core: a clean DETR-style system that adds optical flow and depth to RGB, with a gated fusion mechanism, plus a simple word/phrase/sentence query refinement. The ablations are honest and internally consistent: each module adds a few points on the QVHighlights val split, and the RGB-only baseline is near Moment-DETR. That part is credible and worth a look if you work on temporal grounding.\n\nThe problem is the packaging. The abstract claims 'outperforms current SOTA' with +3.41 MR-mAP@Avg and +3.46 HD-HIT@1, but Table 1 only lists Moment-DETR (2021), SeViLA (2023), MomentDiff (2023), and UMT (2022). That is not the current leaderboard. There are many later supervised methods on QVHighlights that are simply not compared. So the headline number is 'better than UMT', not 'state of the art'. The stress-test note has this right. The CLIP-transfer concern the reader raised is real but secondary: the ablations show depth/flow do help, so within this pipeline the frozen encoder works well enough; the bigger gap is the missing comparisons.\n\nOther soft spots: no code or weights, no multiple seeds or error bars, single-run numbers. The ablation tables are coherent but we can't verify anything without the code. The query refinement module is incremental (1D convs over word features), but the multi-modal fusion ablation shows it does something.\n\nBottom line: this is a decent system paper with an overclaimed headline. If the authors fix the comparisons, add seeds, and release code, it would be a solid contribution to the MR&HD subfield. As is, I'd treat the SOTA claim as unsupported. For peer review, it deserves a serious referee—the method is sound, the writing is clear, and the ablations are useful—but I would recommend major revision before acceptance. Would I cite it? Only if the code appears and the numbers hold up against a fuller baseline set.","headline":"Reasonable multi-modal fusion system with honest ablations, but the SOTA claim rests on a comparison set from 2021–2023.","tokens_in":8541,"tokens_out":1693,"would_cite":false,"duration_ms":16493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fusing RGB, optical flow, and depth through a gated cross-attention fusion, while refining queries at word, phrase, and sentence levels, yields state-of-the-art video moment retrieval and highlight detection.","keywords":["video moment retrieval","highlight detection","multi-modal fusion","optical flow","depth map","query refinement","cross-modal transformer","decoder-free architecture"],"falsifier":"One decisive check: replace the depth and flow inputs to the frozen CLIP encoders with pure noise (or scrambled frames) while keeping the RGB stream and training procedure identical; if MRNet still shows the reported gains, the multi-modal fusion is not doing the claimed work. Another check is a zero-shot probe of CLIP on depth and flow, measuring whether depth-frame and flow-frame features can be matched to captions without any training; near-chance matching would indicate the encoder transfer assumption is unsound.","tokens_in":7563,"feed_emoji":"🎬","tokens_out":5221,"duration_ms":49511,"temperature":0.7,"pith_summary":"This paper proposes MRNet, a network for video moment retrieval and highlight detection that combines three visual signals—RGB, optical flow, and depth—instead of relying on RGB alone. The core idea is that optical flow helps reason about dynamic scenes while depth maps help understand static scenes, and the paper designs a gated fusion mechanism to blend these cues using RGB as the semantic anchor. On the text side, a query refinement module captures word-, phrase-, and sentence-level information to better match human language understanding. Reported experiments on QVHighlights show improvements over prior state-of-the-art methods, including a +3.41 gain in average moment retrieval mAP and a +3.46 gain in highlight detection HIT@1, with consistent gains on Charades-STA.","feed_headline":"Gated flow-depth fusion lifts video retrieval by 3.4 points","feed_subtitle":"MRNet's RGB-guided gating of optical flow and depth maps, plus word- to sentence-level query features, sets a new bar on QVHighlights.","key_machinery":"The load-bearing identity is the gated fusion equation $F_v = Z_r \\odot F_r^f + (1 - Z_r) \\odot F_r^d$, where $Z_r = \\mathrm{Sigmoid}(\\mathrm{FFN}(\\tilde{F}_r))$; this makes the auxiliary modalities conditional on RGB context rather than independent inputs. The query refinement module is a second mechanism: it applies 1D convolutions of kernel sizes 1, 2, and 3 to CLIP text features to capture word- and phrase-level patterns, averages for the sentence level, and concatenates them into tokens of length $N_t+1$. A cross-attention transformer then lets each video moment gather information from these refined text tokens, and a decoder-free transformer encoder with learnable span tokens produces highlight scores and span predictions.","core_discovery":"MRNet establishes that an encoder-only, decoder-free transformer can jointly handle moment retrieval and highlight detection when three visual streams are fused dynamically. The fusion module uses the RGB feature sequence, after self-attention, as the query in separate cross-attention layers over optical flow features and depth features, producing RGB-guided flow and depth features; a gated sigmoid then blends them into a single visual token stream. The query refinement module concatenates word-wise, phrase-wise (convolution kernels of size 2 and 3), and sentence-wise features to give the text tokens a global receptive field. With these modules, the model surpasses the prior state of the art, UMT, across all reported metrics on QVHighlights, and beats Moment-DETR and UMT on Charades-STA. Ablations show that each module contributes and that simply concatenating all multi-modal features is worse than the gated fusion.","pith_inferences":["The gating mechanism implicitly tests modality usefulness: because the fusion weight is learned per token from RGB context, the model can downweight a modality where the frozen encoder produces unreliable features, suggesting the same gate could be applied to other auxiliary cues like audio or saliency without architectural change.","The paper relies on an untested transfer assumption: frozen CLIP encoders trained on RGB images still embed depth maps and optical flow semantically enough to align with text embeddings; probing this with a zero-shot text-to-depth/flow retrieval experiment could reveal whether the reported gains come from genuine multi-modal understanding or from training artifacts.","A direct extension would be to replace the frozen CLIP encoders for depth and flow with encoders pretrained on those modalities; if the gated fusion still improves, it would confirm that the architectural design, not the shared encoder, is the source of the gains."],"forward_implications":["If MRNet's claim holds, adding depth and optical flow through a gated RGB-guided fusion is a working recipe for joint moment retrieval and highlight detection, leaving a consistent gap for RGB-only methods.","The decoder-free encoder-only design, which ablations show improves HD-HIT@1 by about 6 points over a decoder variant, offers a simpler architectural direction for other query-grounded video tasks.","Multi-granularity query features (word, phrase, sentence) provide additive gains on top of multi-modal visual fusion, so text modeling at different scales should be retained in future MR&HD systems.","The transfer of the full method to Charades-STA suggests the improvements are not confined to a single benchmark's distribution."],"supporting_citations":[{"why":"Defines the QVHighlights benchmark, the Moment-DETR baseline architecture, and the span-matching setup that MRNet modifies by removing the decoder.","marker":"[3]"},{"why":"UMT is the prior state-of-the-art model on QVHighlights that MRNet aims to surpass, and it provides the best-performing baseline in the comparison tables.","marker":"[4]"},{"why":"CLIP supplies all frozen visual and textual encoders used to extract RGB, depth, optical flow, and query features.","marker":"[18]"},{"why":"ZoeDepth is the model that generates the depth maps that serve as the third visual modality in the fusion module.","marker":"[17]"},{"why":"GMFlow computes the optical flow between adjacent frames, providing the motion cue in the fusion module.","marker":"[6]"},{"why":"DETR contributes the Hungarian matching and set-based span loss used in Equation (4) for matching predicted spans to ground truth.","marker":"[14]"},{"why":"Charades-STA is the second benchmark dataset used to verify that MRNet transfers beyond QVHighlights.","marker":"[24]"},{"why":"QRNet provides the decoder-free, query-modulated visual-linguistic transformer idea that motivates removing the transformer decoder in MRNet.","marker":"[15]"}],"fun_headline_variants":["Gated multi-modal fusion adds 3.4 MR-mAP and 3.5 HD-HIT","Word-to-sentence queries refine video moment retrieval","Decoder-free transformer fuses three streams for SOTA highlights","Cross-attending to flow and depth boosts video retrieval","RGB-guided gating of depth and flow sets new QVHighlights record"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that a frozen CLIP encoder trained on natural RGB images produces semantically meaningful and text-aligned features when fed depth maps and optical flow frames, a transfer the paper never tests directly.","fun_headline_variants_meta":{"raw":{"variants":["Gated multi-modal fusion adds 3.4 MR-mAP and 3.5 HD-HIT","Word-to-sentence queries refine video moment retrieval","Decoder-free transformer fuses three streams for SOTA highlights","Cross-attending to flow and depth boosts video retrieval","RGB-guided gating of depth and flow sets new QVHighlights record"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1899,"prompt_tokens":906,"completion_tokens":993,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":901}},"tokens_in":522,"tokens_out":993,"duration_ms":9681,"temperature":1.0,"reasoning_tokens":901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:01:37.699851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check: replace the depth and flow inputs to the frozen CLIP encoders with pure noise (or scrambled frames) while keeping the RGB stream and training procedure identical; if MRNet still shows the reported gains, the multi-modal fusion is not doing the claimed work. Another check is a zero-shot probe of CLIP on depth and flow, measuring whether depth-frame and flow-frame features can be matched to captions without any training; near-chance matching would indicate the encoder transfer assumption is unsound.","supporting_citations":[{"cited_title":"2 Related work Most previous MR&HD approaches [5, 11, 12] only em- ploy image and text inputs","cited_arxiv_id":null,"evidence_quote":"Defines the QVHighlights benchmark, the Moment-DETR baseline architecture, and the span-matching setup that MRNet modifies by removing the decoder."},{"cited_title":"Effectiveness of each module in MRNet on QVHigh- lights val split","cited_arxiv_id":null,"evidence_quote":"UMT is the prior state-of-the-art model on QVHighlights that MRNet aims to surpass, and it provides the best-performing baseline in the comparison tables."},{"cited_title":"An empirical study of end-to-end video-language transformers with masked visual modeling,","cited_arxiv_id":null,"evidence_quote":"CLIP supplies all frozen visual and textual encoders used to extract RGB, depth, optical flow, and query features."},{"cited_title":"Localizing moments in video with natural lan- guage,","cited_arxiv_id":null,"evidence_quote":"GMFlow computes the optical flow between adjacent frames, providing the motion cue in the fusion module."},{"cited_title":"How hierarchical is language use?,","cited_arxiv_id":null,"evidence_quote":"DETR contributes the Hungarian matching and set-based span loss used in Equation (4) for matching predicted spans to ground truth."},{"cited_title":"Recognizing American Sign Language Manual Signs from RGB-D Videos","cited_arxiv_id":"1906.02851","evidence_quote":"Charades-STA is the second benchmark dataset used to verify that MRNet transfers beyond QVHighlights."},{"cited_title":"The emergence of hierarchical structure in human language,","cited_arxiv_id":null,"evidence_quote":"QRNet provides the decoder-free, query-modulated visual-linguistic transformer idea that motivates removing the transformer decoder in MRNet."}],"review_version":1}