{"id":"52ded791-fce9-4415-8629-682276c51837","arxiv_id":"2502.08544","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper introduces Negative-Aware Video Moment Retrieval, adding rejection of in-domain and out-of-domain irrelevant queries to moment retrieval, and shows a UniVTG adaptation rejects most negatives while retaining most retrieval accuracy.","lead":"Video moment retrieval systems normally return a start and end time for any text query, even when the query describes nothing in the video. This paper adds a positive/negative classification head so models can reject irrelevant queries, and introduces benchmarks to measure that rejection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central rejection numbers depend on a negative benchmark that is neither human-verified nor hardness-controlled; false negatives or an easy-selection bias would inflate every rejection-accuracy figure.","rationale":"The reader's weakest assumption identifies the same point, and I agree. The proposed check is deliberately cheap: a few hundred human labels would settle whether the benchmark contains false negatives. The easy-selection bias is a second, independent reason the reported numbers should not be generalized until a hardness-matched negative set is evaluated. I do not claim the method is wrong; the architecture and losses are plausible and the experiments are internally consistent. But the central claim is empirical, and the empirical instrument used to measure it, the negative test set, has an unvalidated and potentially biased construction. That is exactly the kind of condition that should gate acceptance of the benchmark as a standard evaluation. The reader's CONDITIONAL verdict is therefore the right one, and this stress-test does not move it.","tokens_in":17283,"tokens_out":5697,"duration_ms":65506,"concrete_test":"Randomly sample 200 ID and 200 OOD negative test pairs per dataset (QVHighlights, Charades-STA), have at least two annotators watch the paired video and judge whether the sentence describes any moment in that video, and measure inter-annotator agreement. Remove or relabel false negatives and recompute Rejection Accuracy for UniVTG-NA and the -Thr baselines; also compute RA on an additional ID negative set sampled uniformly from all shuffled pairs, without the 50th-percentile filter. If the false-negative fraction exceeds about 1% or RA drops by more than about 2 points on the corrected or harder set, the released benchmark numbers are not a reliable measure of negative rejection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim, that UniVTG-NA rejects 96.84% of ID and 100.00% of OOD negatives on QVHighlights while retaining 63.48 R1@0.5, is measured entirely against negative sets built in Sec. 3.3. Two properties of that construction are load-bearing. First, ID negatives are not drawn from the natural distribution of irrelevant queries: after shuffling video-sentence pairs, the authors keep only pairs whose CLIP pseudo-similarity is in the lowest 50th percentile for the sentence. This deliberately selects the easiest half of possible negatives under CLIP, so the reported ID rejection accuracy is an upper-bound estimate for realistic use and does not directly test the claimed ability to 'differentiate specific details in videos.' Second, neither ID nor OOD negatives are verified to be true negatives. For ID, the low-similarity filter is only a proxy: a shuffled sentence can still describe a moment in its new video, and CLIP similarity is an imperfect measure of semantic containment. For OOD, Sec. 3.3 selects LLM-generated scenarios that are 'extremely unlikely' to occur, but QVHighlights contains news and vlog footage where competitive sport, animal behaviour, physics and mathematics can in fact appear; no human annotation or automatic check is reported for the 1550 OOD test pairs. If even a small fraction of these queries are accidentally present in their paired videos, 'correct rejection' is scored where the correct answer would be retrieval, corrupting RA on both the baselines and UniVTG-NA. Because all headline rejection numbers derive from this benchmark, the validity of the negative set is the most load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Negative-Aware Video Moment Retrieval (NA-VMR), a task formulation in which a model must both localize moments for positive queries and reject irrelevant (negative) queries. The authors propose a distinction between in-domain (ID) negatives, obtained by shuffling video-sentence pairs, and out-of-domain (OOD) negatives, generated by LLMs from scenarios assumed to be absent from the target datasets. They construct new evaluation benchmarks on QVHighlights and Charades-STA, evaluate three existing moment-retrieval methods (UniVTG, CG-DETR, QD-DETR) with threshold- and SVM-based rejection, and propose a simple modification (adding a classification head over indicator and saliency scores) that yields \"negative-aware\" variants. The main result is that UniVTG-NA achieves high rejection accuracy on the new benchmarks while retaining most moment-retrieval performance; an additional OOD generalization test on 'musician performances' is reported.","tokens_in":17626,"tokens_out":5999,"duration_ms":64729,"significance":"If the benchmarks are accepted as clean, the paper makes a useful contribution: it is the first systematic study of negative-query robustness in video moment retrieval, it introduces a clear task distinction between ID and OOD negatives, and it shows that a lightweight classification head on top of existing indicator and saliency scores provides a practical adaptation that generalizes to an unseen OOD scenario. The code and dataset splits are released, which supports reproducibility. The main caveat is that all reported rejection accuracies rest on negative sets whose validity is not verified by human annotation and whose construction may select artificially easy negatives; as a result, the headline numbers should be treated as upper-bound estimates until the benchmark is validated.","major_comments":[{"comment":"The ID negative set is constructed by keeping only shuffled video-sentence pairs whose CLIP pseudo-similarity falls in the lowest 50th percentile for the sentence. This deliberately removes the harder half of possible shuffled negatives, so the ID rejection accuracies in Tables 1, 2, 3, and 8 are upper bounds for arbitrary irrelevant queries and do not directly support the stated ability to 'differentiate specific details in videos' from Sec. 3.2. No results are reported on a random subset or on the excluded higher-similarity half, and no human verification is provided that the remaining shuffled pairs are truly negative. Because CLIP sentence-sentence similarity is only a proxy for whether a moment is present, a shuffled sentence can still describe a moment in its newly assigned video; if any test negative is actually positive, the reported rejection accuracy is inflated. This is load-bearing for the central claim of high rejection accuracy.","section":"Sec. 3.3 (In-Domain negatives)"},{"comment":"The OOD negative set consists of LLM-generated sentences about competitive sport, animal behaviour, physics laboratory, and mathematics class, under the assumption that these are 'extremely unlikely' to occur in QVHighlights and Charades-STA. However, QVHighlights contains news and vlog footage where such content can appear, and Charades-STA is unscripted home video, not a closed domain. The paper reports no automatic or human check that any of the 1550 OOD test pairs are actually negative, nor for the additional 'musician performances' generalisation set in Table 8. If even a small fraction of these queries are accidentally present in their paired videos, the near-100% OOD rejection accuracies in Tables 2, 3, and 8 overstate the model's true rejection ability and unfairly penalise the baselines. The authors should validate a sample of OOD and ID negatives (e.g., through human annotation) or provide a quantitative analysis of the false-negative rate.","section":"Sec. 3.3 (Out-of-Domain negatives)"}],"minor_comments":[{"comment":"The abstract's claim of 'avg. 98.4%' rejection accuracy is only the average of the ID and OOD numbers for QVHighlights (96.84 and 100.0); on Charades-STA the ID rejection accuracy is 64.11%, and the 'within 3.87% Recall@1' retention is also specific to QVHighlights R1@0.5. The abstract should qualify these numbers by dataset and metric.","section":"Abstract"},{"comment":"The decision threshold for the sigmoid classification output is never stated. The paper should specify the threshold used to convert continuous predictions into accept/reject decisions for computing Rejection Accuracy, and should state whether this threshold is the default 0.5 or a tuned hyperparameter.","section":"Sec. 3.4, Eq. (1)"},{"comment":"The loss weights (lambda_p, lambda_ID, lambda_OOD, lambda_s) are chosen separately for each dataset and each negative type, but no sensitivity analysis or selection criterion is reported. Adding a small ablation over these weights would help assess how robust the reported trade-off between retrieval and rejection is.","section":"Sec. C.3"},{"comment":"There are several typos and formatting issues, e.g., 'incoporates' in the Conclusion, 'QVHighights' in the Appendix, and a missing space in Sec. 4.1 ('methods treat positives and negatives the same way .'). These should be corrected before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is a new benchmark for negative query rejection, and the headline rejection numbers depend entirely on the validity of that benchmark. The main risk is that the negative sets have not been verified for false negatives and the ID set is deliberately filtered to easy negatives; this is fixable with additional annotation or analysis, but it is load-bearing for the core claim. I therefore recommend major revision rather than rejection. The authors should be encouraged to release a human-validated subset and to report results on unfiltered or harder negative splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know upfront. First, this paper gives the VMR community a genuinely new task — negative-aware retrieval with an in-domain/out-of-domain split — and shows that a small classification head on top of UniVTG recovers most rejection ability without destroying retrieval. That is useful, and I expect the benchmarks to get adopted. Second, the headline rejection numbers are measured against negative sets that are deliberately easy, so treat 96.84% ID and 100% OOD as upper bounds, not operating points.\n\nThe core observation — that Moment-DETR-style models confidently localize irrelevant queries — is well demonstrated. The histograms and threshold experiments in Table 1 are convincing. The method itself is simple: add an RNN+MLP head on indicator plus saliency scores, train with BCE and zeroed saliency/indicator losses for negatives. It works. The ablation in Table 3 shows ID and OOD negatives give complementary signal, which is a real result. The OOD generalization test to musician performances is a nice bonus, and shipping code and splits makes the work reproducible.\n\nThe soft spots are in benchmark construction. ID negatives are created by shuffling pairs and keeping only the lowest 50th percentile under a CLIP pseudo-similarity score. That cleans the test set, but it also selects the easiest negatives. So the ID RA numbers overstate how well the model would do on plausible but tricky negatives. The OOD negatives are LLM-generated with no human verification; a small number may accidentally appear in QVHighlights news footage. That is a validity concern, though not a fatal one — even if a few percent are false negatives, the 100% OOD figure only drops to ~97-98%, which does not change the qualitative conclusion.\n\nTwo smaller issues: the abstract's 'avg. 98.4%' applies only to QVHighlights; on Charades-STA the in-domain rejection is 64.11%, which the abstract glosses over. And the decision threshold for the classification head is never stated. Presumably 0.5, but it should be explicit. No error bars either, though that is typical for this area.\n\nOverall the central argument holds up: current models are not negative-aware, and a straightforward adaptation restores most rejection capability. The benchmark design needs refinement before it becomes a standard, but it deserves serious peer review. I would want the authors to add human verification (or at least a sample) of both negative types, report a random-negative baseline, and disclose the threshold.","headline":"Useful new task and benchmarks for negative-aware VMR, but the rejection numbers are measured against easy negatives and should be read as upper bounds.","tokens_in":18159,"tokens_out":3219,"would_cite":false,"duration_ms":33772,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video moment retrieval models can reject irrelevant queries while keeping most retrieval accuracy.","keywords":["video moment retrieval","negative query rejection","negative-aware video moment retrieval","in-domain negatives","out-of-domain negatives","UniVTG","saliency score classification"],"falsifier":"Take a random sample of the constructed in-domain and out-of-domain negative pairs, have human annotators check whether each negative query actually describes a moment visible in its assigned video, and re-measure rejection accuracy on the verified subset; if even a few percent of the 'negative' queries are true positives, the reported rejection numbers would drop accordingly.","tokens_in":17091,"feed_emoji":"🎬","tokens_out":6981,"duration_ms":62228,"temperature":0.7,"pith_summary":"Video moment retrieval models are trained under the assumption that every query sentence has a matching moment in the video, so they always return start and end times even when the query is irrelevant. This paper introduces Negative-Aware Video Moment Retrieval (NA-VMR), which asks models to both localise positive queries and reject negative ones, distinguishing in-domain negatives (plausible but absent) from out-of-domain negatives (from unrelated scenarios). The authors show that current state-of-the-art models cannot separate positive from negative queries using their indicator scores, and propose UniVTG-NA, an adaptation that adds a small classification head over the indicator and saliency scores. With this head and negative-aware training, UniVTG-NA rejects an average of 98.4% of negative queries while losing at most 3.87% Recall@1 on the tested benchmarks. If correct, this means existing Moment-DETR-style video retrieval systems can be made safe for bad queries with a lightweight addition, rather than a full redesign.","feed_headline":"Video moment retrieval can reject irrelevant queries at 98.4%","feed_subtitle":"The model says 'no' to non-matching queries while keeping moment retrieval Recall@1 within 3.87%.","key_machinery":"The load-bearing mechanism is a lightweight binary classification head added to an existing Moment-DETR-style model. It combines the model's indicator scores and saliency scores (by summation for UniVTG, by concatenation for QD-DETR and CG-DETR), processes them through a recurrent layer and a feed-forward layer with a sigmoid output, and is trained with a binary cross-entropy loss alongside negative-specific losses that push indicator and saliency scores toward zero for negative queries. The other half of the machinery is the negative-query construction pipeline: in-domain negatives come from shuffling video-sentence pairs and keeping only pairs in the lowest 50th percentile of a CLIP-based pseudo-similarity score, while out-of-domain negatives are LLM-generated sentences from scenarios chosen not to occur in the datasets.","core_discovery":"The paper's central claim is that current video moment retrieval methods cannot tell positive from negative queries, and that this can be fixed by training with explicitly sampled negatives and a binary classification head. The paper shows that the indicator score used to rank moments is nearly non-separable between positives and negatives, while saliency scores carry more signal but still overlap. Its solution keeps the base model's heads and adds a classification head that takes the sum (for UniVTG) or concatenation (for QD-DETR and CG-DETR) of indicator and saliency scores, passes them through an RNN and a feed-forward layer, and outputs a positive-versus-negative prediction. A binary cross-entropy loss, together with adapted losses that push indicator and saliency scores toward zero for negatives, trains this head. Across QVHighlights and Charades-STA, UniVTG-NA retains most moment retrieval accuracy while rejecting an average of 98.4% of negatives, with out-of-domain rejection near 100% on both datasets. An ablation shows that in-domain and out-of-domain negatives provide complementary training signal: training with only one type leaves a large rejection gap.","pith_inferences":["Editorial inference: if rejection accuracy reflects genuine video-query mismatch, the classification head's score could double as a confidence or abstention signal for downstream video search and question answering, not just a binary reject.","Editorial inference: the ID-negative construction relies on CLIP pseudo-similarity and is never human-verified; a small human audit of the negative pairs would establish whether the benchmark itself is sound.","Editorial inference: the same summation or concatenation head could likely be applied to newer grounding architectures beyond the tested Moment-DETR family, but the head input would need to adapt to models that do not produce saliency scores.","Editorial inference: a natural testable extension is joint training with Video Corpus Moment Retrieval, where rejecting queries absent from the whole corpus is a more difficult version of the same problem."],"forward_implications":["Moment-DETR-based models can be made negative-aware by adding a small classification head and training with negative queries, without changing the base architecture.","Standard video moment retrieval benchmarks overstate practical reliability because they never include queries that do not match the video.","Training with both in-domain and out-of-domain negatives is necessary; either type alone leaves a large gap in rejection accuracy.","The rejection ability transfers to out-of-domain scenarios not seen in training, as shown by high rejection accuracy on the unseen 'musician performance' category.","There is an inherent trade-off: adding rejection lowers moment recall slightly, and datasets without ground-truth saliency scores show weaker in-domain rejection."],"supporting_citations":[{"why":"Supplies the Moment-DETR architecture and the QVHighlights dataset with human-annotated moments and saliency scores that the negative-aware head builds on.","marker":"[21]"},{"why":"UniVTG is the base model adapted into UniVTG-NA; its pretrained weights and indicator/saliency score heads provide the features the classification head combines.","marker":"[23]"},{"why":"Charades-STA provides the second benchmark dataset, whose lack of ground-truth saliency scores tests the method under pseudo-saliency supervision.","marker":"[12]"},{"why":"QD-DETR is both a baseline showing poor threshold-based negative rejection and a model adapted into QD-DETR-NA; it also introduced shuffled video-sentence pairs for training.","marker":"[32]"},{"why":"CG-DETR is the other adapted baseline, providing a comparison point for the classification-head approach on a different Moment-DETR variant.","marker":"[31]"}],"fun_headline_variants":["Video moment retrieval rejects 98.4% of irrelevant queries","Negative-aware video moment retrieval: 98.4% rejection of negatives","Teaching video moment retrieval to reject irrelevant queries","98.4% of negative queries rejected in video moment retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmarks assume that the automatically constructed negative queries really do not match anything in the videos they are assigned to, and this is never verified by human annotation.","fun_headline_variants_meta":{"raw":{"variants":["Video moment retrieval rejects 98.4% of irrelevant queries","Negative-aware video moment retrieval: 98.4% rejection of negatives","Teaching video moment retrieval to reject irrelevant queries","98.4% of negative queries rejected in video moment retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00091,"raw_usage":{"total_tokens":3931,"prompt_tokens":988,"completion_tokens":2943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2874}},"tokens_in":604,"tokens_out":2943,"duration_ms":22655,"temperature":1.0,"reasoning_tokens":2874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:40:28.640169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the constructed in-domain and out-of-domain negative pairs, have human annotators check whether each negative query actually describes a moment visible in its assigned video, and re-measure rejection accuracy on the verified subset; if even a few percent of the 'negative' queries are true positives, the reported rejection numbers would drop accordingly.","supporting_citations":[{"cited_title":"Berg, and Bansal","cited_arxiv_id":null,"evidence_quote":"Supplies the Moment-DETR architecture and the QVHighlights dataset with human-annotated moments and saliency scores that the negative-aware head builds on."},{"cited_title":"Univtg: Towards unified video- language temporal grounding","cited_arxiv_id":null,"evidence_quote":"UniVTG is the base model adapted into UniVTG-NA; its pretrained weights and indicator/saliency score heads provide the features the classification head combines."},{"cited_title":"Tall: Temporal activity localization via language query","cited_arxiv_id":null,"evidence_quote":"Charades-STA provides the second benchmark dataset, whose lack of ground-truth saliency scores tests the method under pseudo-saliency supervision."},{"cited_title":"Query-dependent video representa- tion for moment retrieval and highlight detection","cited_arxiv_id":null,"evidence_quote":"QD-DETR is both a baseline showing poor threshold-based negative rejection and a model adapted into QD-DETR-NA; it also introduced shuffled video-sentence pairs for training."}],"review_version":1}