{"id":"ec20f437-3bdf-4898-bf34-28d71ece83cf","arxiv_id":"2507.12062","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MS-DETR improves moment retrieval and highlight detection by disentangling motion and semantic video features, sharing task information between the two tasks, and training on generated auxiliary captions.","lead":"This paper presents MS-DETR, a video search system that finds the exact moments described by a text query and scores which clips are highlights, by keeping motion and visual details separate as it processes the video. It reports small but consistent accuracy gains over previous best models on four standard benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on auxiliary corpus generation from GT annotations; the paper never states the generation is restricted to training splits, leaving val/test leakage unruled out.","rationale":"The reader's weakest_assumption targets feature disentanglement, and I agree that the paper never verifies the assumed SlowFast-motion / CLIP-semantics separation. That is a real weakness, but even if the two towers are not truly disentangled, the architecture would still be a two-stream attention model whose empirical numbers could stand. The more decisive threat to the central claim is the auxiliary-data protocol: the largest single ablation gain comes from auxiliary data, and the paper never states that corpus generation is train-only. If any val/test video contributed generated pairs, every benchmark table could be invalidated; if the code shows train-only generation, the concern is fully resolved. This is a protocol omission, not an allegation of misconduct, and because the code link is provided the check is cheap. The reader's conditional verdict remains appropriate: accept only after the split restriction and reproducibility are confirmed. If the code confirms train-only generation and reproduces the headline numbers, the empirical claim is credible; if not, the verdict should be downgraded or rejected.","tokens_in":16775,"tokens_out":10600,"duration_ms":128464,"concrete_test":"Clone the released repository, locate the data-preparation/corpus-generation script, and scan the generated pair metadata for val/test video IDs from QVHighlights, TVSum, TACoS, and Charades-STA. If any val/test ID appears, retrain with auxiliary pairs restricted to train IDs and re-run Table 1 val and Table 4; if none appear, reproduce Table 1 val MR mAP 67.19 and TVSum avg mAP 88.43 from the released commands to confirm no hidden contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 describes Motion-Semantics Corpus Generation without ever stating that the generated captions, rewritten queries, and selected video-text pairs are taken only from the training split. The captioning step starts from \"each video clip in the ground truth\" and uses CLIP cosine similarity to assign moment intervals; query rewriting and motion rewriting use the original GT moment intervals to form positive/hard-negative pairs. If any val/test video appears in this corpus, the auxiliary pairs derived from its ground truth leak into training. This is not a marginal concern: ablation Table 5 (d) attributes +3.60 MR mAP (61.61 to 65.21 on QVHighlights val) to auxiliary data alone, and the full-model gain over CG-DETR is +3.12 mAP on the same split. The paper's code link is external, so the submission text itself does not establish the required restriction; the experiments are not independently reproducible from the paper alone. I am not alleging that leakage occurred; I am flagging that the central empirical claim is conditional on a split-restriction fact that is never stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MS-DETR, a DETR-based joint architecture for video moment retrieval (MR) and highlight detection (HD). The encoder uses two cross-modal transformer towers over pre-extracted SlowFast and CLIP features to model temporal-motion and spatial-semantic dimensions separately, and the decoder lets the HD task seed MR queries while MR boundaries refine HD salience. To address annotation sparsity, the authors generate auxiliary captions and rewritten queries, then train with contrastive denoising losses. Experiments on QVHighlights, TACoS, Charades-STA, and TVSum report gains over QD-DETR, CG-DETR, and TR-DETR, with ablations in Table 5 attributing improvements to each proposed component.","tokens_in":17003,"tokens_out":6402,"duration_ms":73159,"significance":"The paper has several strengths: it evaluates on four standard public benchmarks with pre-extracted features, it includes a parameter-matched comparison in Table 6, and the ablations in Table 5 show positive contributions from the proposed modules. The reported improvements over CG-DETR and TR-DETR are plausible and of interest to the MR/HD community. However, the central 'disentangled motion-semantics' explanation is not directly verified, and the auxiliary-data generation procedure leaves a potential train/validation/test leakage ambiguity that is load-bearing because Table 5(d) credits auxiliary data with a large part of the gain.","major_comments":[{"comment":"The corpus-generation text never states that the generated captions, rewritten queries, and selected video-text pairs are derived only from the training split. The captioning step starts from 'each video clip in the ground truth' and assigns intervals by CLIP cosine similarity; if any validation or test video appears in the generated corpus, its ground-truth annotations leak into training. This is not a marginal issue because ablation Table 5(d) attributes +3.60 MR mAP to auxiliary data alone, and the full-model gain over CG-DETR on the same QVHighlights val split is +3.12 mAP. Please state explicitly that all generation is restricted to the training split, and verify that the released code enforces this restriction.","section":"Section 3.4, Motion-Semantics Corpus Generation"},{"comment":"The loss L_enc_cont is not implementable as written. The phrase 'the samples with higher salience scores than n' is undefined because n is a loop index, not a score; the construction of X_pos^n and X_neg^n is therefore ambiguous. In addition, the denominator sums over x in X_pos^n ∪ X_neg^n of exp(S(x_i)/ξ), mixing the summation variable x with x_i. Please specify the exact ranking/partition procedure (for example, rank thresholds within each video, within the ground-truth moments, or across the batch) and rewrite Eq. (13) with consistent variables.","section":"Section 3.4, Eq. (13)"},{"comment":"The claimed disentanglement rests on the assumption that SlowFast features represent temporal motion and CLIP features represent spatial semantics, but the manuscript provides no evidence for this separability. If both pre-trained feature sets contain substantial amounts of both kinds of information, the two towers may reduce to two-stream cross-modal attention without genuinely disentangled dimensions, which would undermine the stated explanation for the gains. Please add a diagnostic test, such as probing each tower's output or ablating by feeding the same feature type to both towers, or temper the disentanglement claim to match what the experiments actually support.","section":"Section 3.2, Eq. (1)"},{"comment":"The training setup is under-specified. The loss weights λ1 and λ2, the margin δ, the temperature ξ, the noise scale δ2, the number of decoder queries K, and the standard training hyperparameters (batch size, learning rate, schedule) are not reported. Since Eq. (14) combines several weighted losses and the auxiliary-data pipeline introduces additional thresholds, reproduction from the paper alone is currently impossible. Please add a training-details subsection or a supplementary document with these values and the selection procedure for the captioning thresholds.","section":"Sections 3.4 and 4"}],"minor_comments":[{"comment":"The notation is inconsistent: the text defines the second tower's query as Q_sv = F_sv, but Eq. (1) writes SSCT(Q_tv, K_t, V_t). Please use Q_sv consistently.","section":"Section 3.2, Eq. (1)"},{"comment":"The sentence 'we replace the verbs in text queries with their corresponding original video moment intervals to produce positive and hard-negative video-text pairs' appears garbled; it should describe replacing verbs with synonyms or antonyms, mirroring the noun rewriting described for the semantics corpus.","section":"Section 3.4, Motion corpus generation"},{"comment":"The sentence 'improving the mAP metric by 2% and 3.77% respectively' does not say which split and which task the percentages refer to. Please specify (e.g., test MR mAP and val HD mAP) so the claim is verifiable.","section":"Section 4.2, Table 1 paragraph"},{"comment":"The claim of outperforming all existing SOTAs should be qualified: on TVSum, MS-DETR is below TR-DETR on several categories (for example DS 76.88 vs. 81.6, and PR 88.52 vs. 88.6). The statement should be phrased in terms of average mAP or the majority of metrics rather than universal superiority.","section":"Section 4.2, Table 4"},{"comment":"It is unclear how Φ(m_noise) is computed when m_noise is a (center, span) pair rather than a scalar reference position. A sentence explaining how noisy moments are converted into decoder position queries would improve reproducibility.","section":"Section 3.3, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of ACM Multimedia and the empirical results are plausible. The main risks are the auxiliary-data split restriction and the under-specified contrastive loss; both are fixable in revision, so I do not recommend rejection. I have no concerns about novelty or citation patterns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the stress-test note. I read the paper with that concern in mind, and I think it lands as a genuine ambiguity rather than proof of leakage.\n\nWhat's new: the paper puts a two-tower cross-attention encoder (TMCT/SSCT) on top of the usual SlowFast+CLIP features, adds a task-collaboration loop between highlight salience and moment queries, and generates auxiliary video-text pairs via LLaVA captions, query rewriting, and verb/noun replacement. The combination of disentangled motion/semantics encoding with contrastive denoising is not in the prior work I know. The ablations are consistent: each component gives a bump, and the parameter-matched comparison in Table 6 addresses the obvious 'more params' confound. The gains over CG-DETR and TR-DETR are real but modest—1-4 points—so 'remarkable margin' is overstatement.\n\nSoft spots, in order of importance:\n\n1. The auxiliary corpus generation in Sec. 3.4 never states that it uses only training-split videos. Since the generated pairs are built from ground-truth moment intervals and CLIP similarity, if any val/test video is included, the reported val/test numbers are inflated. I'm not alleging that happened—the code is linked and the authors likely did the standard thing—but the text should say it explicitly. For a camera-ready, that's a one-paragraph fix.\n\n2. Eq. 13 (L_enc_cont) is not implementable from the text. \"Samples with higher salience scores than n\" doesn't define the iteration or the ranking. The reader's point is right. Similarly, the margin, temperature, noise scale, and loss weights λ1/λ2 are missing. This is less about correctness and more about reproducibility.\n\n3. The disentanglement claim is undersupported. The paper assumes SlowFast=motion, CLIP=semantics, but never probes or ablates the towers to show they capture distinct dimensions. If both towers just learn the same thing, the gains are from extra parameters/attention, not from disentanglement. The parameter-matched ablation weakens that counter, but doesn't fully close it.\n\nThe stress-test note is a fair flag, not a takedown. The central contribution holds up as an incremental but useful step for MR/HD. The paper deserves a serious referee; it's the kind of work that will be accepted after revision if the authors clarify the split restriction and clean up the loss definition. I'd send it to review, and I'd want the revised version to state the training-only generation explicitly.\n\nFor a reading group, it's a reasonable example of applied DETR-style engineering, though not a must-read unless your work touches MR/HD.","headline":"Competent incremental MR/HD paper with a real reproducibility gap over auxiliary-data split restriction; worth reviewing, needs clarification.","tokens_in":17560,"tokens_out":3426,"would_cite":true,"duration_ms":34054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that keeping temporal motion and spatial semantic features in separate cross-attention streams, then exchanging task information between moment retrieval and highlight detection, pushes both tasks past prior…","keywords":["video moment retrieval","highlight detection","DETR","motion-semantics disentanglement","cross-modal transformer","contrastive denoising","SlowFast","CLIP"],"falsifier":"Probe each tower's learned representation with labeled action and scene classifications: if the motion tower classifies scenes as accurately as the semantic tower, or the semantic tower classifies actions as accurately as the motion tower, the features are not disentangled and the central explanation is false. A cheaper check is to mask one tower at test time, since true disentanglement predicts verb-focused queries fail when TMCT is masked while noun-focused queries fail when SSCT is masked.","tokens_in":16552,"feed_emoji":"🎬","tokens_out":5006,"duration_ms":56434,"temperature":0.7,"pith_summary":"This paper tries to establish that jointly solving video moment retrieval and highlight detection is best done by keeping temporal motion and spatial semantic information separate until late in the model, instead of concatenating pre-extracted features up front. The proposed MS-DETR feeds SlowFast motion features and CLIP semantic features into two distinct cross-attention towers, each guided by the text query, and only then fuses them for the encoder; a task-collaborated decoder then lets highlight salience seed moment-retrieval queries and lets retrieved boundaries refine salience scores. To overcome sparse annotations, the authors generate auxiliary video-text pairs from captions and rewrites and train with contrastive denoising. If the results are right, this is a practical recipe: state-of-the-art performance on four benchmarks, with the headline gap on QVHighlights val MR mAP of 67.19 versus 64.07 for the prior CG-DETR.","feed_headline":"Separating motion from appearance sets new moment-retrieval records","feed_subtitle":"MS-DETR keeps motion and semantic features apart until late fusion, beating prior joint highlight and moment models on four benchmarks.","key_machinery":"The load-bearing mechanism is the Motion-Semantics Disentangled Encoder (MSDE), which replaces feature concatenation with two cross-attention towers: TMCT attends from temporal motion features to text, SSCT attends from spatial semantic features to text, and their outputs are concatenated and mapped back to dimension $d$ by a learnable projection (Eq. 1). The Mutual Task-Collaborated Decoder (MTCD) then computes salience scores from a learned salience token via a dot-product (Eq. 3), selects top-$K$ encoder outputs as content queries, turns them into positional queries via sinusoidal encoding, and feeds the two task heads. The auxiliary-data branch generates caption and query pairs and trains with a contrastive denoising loss that adds scaled noise to ground-truth moments before decoding.","core_discovery":"The paper's central claim is that motion-semantics disentanglement, rather than more parameters or a stronger backbone, is what drives the gains. Concretely, it reports that MS-DETR outperforms all compared state-of-the-art joint MR/HD models: on QVHighlights val, MR mAP rises to 67.19 and R1@0.75 to 46.00; on test, MR mAP is 66.41; and on TVSum, highlight mAP reaches 88.43. The ablation in Table 5 attributes the improvement to the three modules in combination, and Table 6 argues the MSDE gain is not simply from added layers.","pith_inferences":["Editorial extension: the architecture's real test is whether TMCT and SSCT capture genuinely different information; probing the two tower outputs with action and scene classifiers would show whether the semantic tower also encodes motion, which would require revising the stated explanation even if the scores stand.","Editorial extension: since the losses rely on LLM-generated captions and rewrites, the method inherits any bias or hallucination in those generators; ablating the generation source (human captions versus generated captions) would separate data richness from data noise.","Editorial extension: replacing SlowFast with another motion feature extractor, or CLIP with another semantic extractor, would reveal how much of the gain comes from the separation principle rather than from the specific pre-trained features."],"forward_implications":["Disentangled motion and semantic streams can be reused as a drop-in encoder upgrade for existing DETR-based MR/HD pipelines, since it only changes how pre-extracted features are combined.","Because HD salience is computed from encoder outputs and MR from decoder outputs, the mutual-collaboration loss should make both tasks improve together; Table 1 shows simultaneous gains on both metrics.","Auxiliary generated pairs act as hard negatives, implying that annotation sparsity can be mitigated at the data level without new human labels.","The method's reported TVSum HD mAP of 88.43 surpasses even audio-augmented models, suggesting motion-plus-semantics video features alone can carry highlight detection when separated properly."],"supporting_citations":[{"why":"Establishes the joint MR/HD task, the QVHighlights dataset, and the baseline DETR-style framework most experiments build on.","marker":"[18]"},{"why":"Supplies the strongest QVHighlights baseline and the architecture used as the starting point for ablations.","marker":"[34]"},{"why":"Provides the task-reciprocal prior joint model that MS-DETR compares against on QVHighlights and TVSum.","marker":"[43]"},{"why":"Supplies the frozen pre-trained CLIP features used as the spatial semantic stream.","marker":"[37]"},{"why":"Supplies the frozen pre-trained SlowFast features used as the temporal motion stream.","marker":"[8]"},{"why":"Generates finer captions for the semantic auxiliary corpus.","marker":"[24]"},{"why":"Provides the language model used to rewrite verbs and nouns for motion and semantic hard-negative pairs.","marker":"[45]"},{"why":"Provides the generalized IoU loss used for temporal moment regression in both tasks.","marker":"[39]"},{"why":"Provides the original DETR encoder-decoder formulation that the whole framework extends.","marker":"[5]"}],"fun_headline_variants":["Separating motion from appearance improves video moment retrieval","Disentangling motion and semantics improves moment and highlight search","MS-DETR: joint motion-semantic learning for moment and highlight detection","Motion and semantics stay apart until fusion, beating prior models","Motion-semantics disentanglement drives moment retrieval gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains are explained as disentanglement, but this relies on the unverified assumption that SlowFast features encode motion only and CLIP features encode appearance only; if both encode mixed information, the two towers are not truly disentangled and the claimed mechanism reduces to extra capacity.","fun_headline_variants_meta":{"raw":{"variants":["Separating motion from appearance improves video moment retrieval","Disentangling motion and semantics improves moment and highlight search","MS-DETR: joint motion-semantic learning for moment and highlight detection","Motion and semantics stay apart until fusion, beating prior models","Motion-semantics disentanglement drives moment retrieval gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001123,"raw_usage":{"total_tokens":4656,"prompt_tokens":914,"completion_tokens":3742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3661}},"tokens_in":530,"tokens_out":3742,"duration_ms":29987,"temperature":1.0,"reasoning_tokens":3661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:55:37.987876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Probe each tower's learned representation with labeled action and scene classifications: if the motion tower classifies scenes as accurately as the semantic tower, or the semantic tower classifies actions as accurately as the motion tower, the features are not disentangled and the central explanation is false. A cheaper check is to mask one tower at test time, since true disentanglement predicts verb-focused queries fail when TMCT is masked while noun-focused queries fail when SSCT is masked.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the joint MR/HD task, the QVHighlights dataset, and the baseline DETR-style framework most experiments build on."},{"cited_title":"TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection","cited_arxiv_id":"2401.02309","evidence_quote":"Provides the task-reciprocal prior joint model that MS-DETR compares against on QVHighlights and TVSum."},{"cited_title":"In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen pre-trained CLIP features used as the spatial semantic stream."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the frozen pre-trained SlowFast features used as the temporal motion stream."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the language model used to rewrite verbs and nouns for motion and semantic hard-negative pairs."}],"review_version":1}