{"id":"03702aec-6cf3-4c83-9dc5-1aa9811e6c35","arxiv_id":"2504.14553","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Grounding-MD is a grounded video-language pre-training framework that achieves state-of-the-art zero-shot and supervised moment detection via structured prompts and early-late cross-modal fusion.","lead":"A new video-language model, Grounding-MD, detects temporal moments in video from an arbitrary number of natural language queries, unifying temporal action detection and moment retrieval. It pre-trains on large-scale video-text data and reports new state-of-the-art results on four benchmarks in zero-shot and supervised settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"THUMOS14 'unseen' classes are present in the HACS pre-training data, so the open-vocabulary zero-shot gains in Table 4 likely reflect category leakage rather than open-world generalization.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the THUMOS14 'unseen' categories are not truly unseen because the pre-training data HACS overlaps with ActivityNet, and THUMOS14 categories overlap with ActivityNet. This is the central weak point because the paper's headline contribution is open-world, zero-shot moment detection, and Table 4 is the primary evidence for generalizing to unseen action categories. The concern is concrete, evidenced by the paper's own statement in Section 4.3, and directly testable. I agree with the reader's conditional verdict: the supervised fine-tuning results and the zero-shot MR transfer provide independent support for the architecture's basic effectiveness, but the open-vocabulary TAD claim is not yet substantiated. A re-evaluation with a category-disjoint pretraining protocol and, ideally, released code and data filters, is needed before the open-world claim can be accepted at face value. No additional fatal flaw was found; the main issue is evaluation validity rather than internal inconsistency.","tokens_in":14971,"tokens_out":3306,"duration_ms":30727,"concrete_test":"Enumerate the THUMOS14 category list and the HACS/ActivityNet category lists, and count how many of the categories used in the 25%/50% 'unseen' splits appear in HACS. Then retrain G-MD-S(B) on a version of HACS with all THUMOS14-overlapping categories removed (or on FineAction+InternVid-MR alone) and re-run the Section 4.4 protocol. If the unseen-split average mAP falls to roughly the no-pre-training baseline (19.7/11.9), the Table 4 gains are explained by same-category transfer. If no non-overlapping THUMOS14 categories exist, the open-vocabulary claim cannot be tested on this benchmark at all.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 pre-trains on HACS (Table 1), and Section 4.3 states that HACS 'encompasses all categories present in ActivityNet.' THUMOS14 action categories overlap with ActivityNet, so the categories designated 'unseen' in the Section 4.4 open-vocabulary split are very likely present in the HACS pre-training data, complete with temporal annotations. The no-pre-training row in Table 4 (G-MD-S w/o pt: 19.7/11.9 Avg) shows the VideoMAE+BERT combination has little intrinsic open-vocabulary ability; after HACS/InternVid pre-training it jumps to 27.1/26.0. That jump is exactly what one would expect if the model memorized the 'unseen' action categories' visual and textual patterns during pre-training. The paper reports no check that the THUMOS14 unseen split is disjoint from HACS/ActivityNet/FineAction categories, and Table 2 lists G-MD-S(A) pre-trained on HACS alone as the base. Consequently, the central open-world zero-shot claim—generalization to action categories never seen during training—is not supported by Table 4. The MR zero-shot transfer (Table 6) and supervised TAD/MR results remain informative, but they do not establish open-vocabulary TAD generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Grounding-MD, a unified video-language pre-training framework for moment detection that bridges temporal action detection (TAD) and moment retrieval (MR) in an open-world setting. The model accepts a structured prompt containing an arbitrary number of natural-language queries, encodes video and text with VideoMAE and BERT, and uses a Cross-Modality Fusion Encoder, a Query-Wise Pooler, and a Text-Guided Fusion Decoder to align queries with temporal segments. Training combines HACS, FineAction, and a filtered InternVid subset; evaluation covers ActivityNet, THUMOS14, ActivityNet-Captions, and Charades-STA under both zero-shot and supervised protocols. The paper reports state-of-the-art or competitive results in most of these settings and emphasizes the model's efficiency relative to 7B video-LLMs.","tokens_in":15218,"tokens_out":5675,"duration_ms":51682,"significance":"If the claims hold, Grounding-MD would be a valuable step toward open-vocabulary moment detection: it is substantially smaller than video-LLM baselines, shows strong zero-shot transfer on moment retrieval, and produces consistent gains in supervised fine-tuning. The design ideas—early and late cross-modal fusion, query-wise pooling to balance TAD/MR text lengths, and structured prompts with sampled negatives—are clearly presented, and the ablations in Table 8 support the contribution of the proposed modules. However, the central evidence for open-vocabulary temporal action detection (THUMOS14, Table 4) is undermined by category overlap with the HACS pre-training corpus, so that portion of the open-world claim is not currently supported. The external zero-shot MR results (Table 6) and the supervised results remain informative and should be credited.","major_comments":[{"comment":"The claimed open-vocabulary zero-shot evaluation on THUMOS14 is not category-disjoint from pre-training. The model is pre-trained on HACS (Section 3.3, Table 1), and Section 4.3 states that HACS 'encompasses all categories present in ActivityNet.' THUMOS14 action categories are a subset of ActivityNet categories, so the categories marked 'unseen' in the 75/25 and 50/50 splits have very likely been seen with temporal annotations during pre-training. The no-pre-training row (G-MD-S w/o pt) versus the pre-trained row (G-MD-S (B) w/ pt) in Table 4 shows a jump from 19.7 to 27.1 Avg (75/25 split) and from 11.9 to 26.0 Avg (50/50 split), exactly the pattern one would expect from memorizing the visual and textual patterns of the 'unseen' classes. The paper reports no check that the seen/unseen split is disjoint from HACS, FineAction, or InternVid-MR categories. As a result, Table 4 cannot support the paper's open-world zero-shot generalization claim. Please re-run the evaluation on categories verified absent from all pre-training data, or explicitly remove or reword the open-vocabulary TAD claim.","section":"Section 4.4, Table 4; Sections 3.3 and 4.3"},{"comment":"The text states that Grounding-MD 'achieved superior performance compared to Video-LLM-based methods,' but Table 6 shows VideoChat2 achieving 27.8 R@0.5 on ActivityNet-Captions versus 25.3 for G-MD-L; G-MD-L is better on the other three reported metrics. The comparison is therefore selective, and the claim of overall superiority should be qualified to specific metrics and datasets. Please report the full set of metrics with an explicit per-metric comparison, and if variance across evaluation subsets is available, include it.","section":"Section 4.5, Table 6"},{"comment":"The 'zero-shot domain transfer' evaluation on ActivityNet is not an open-vocabulary test: because HACS shares the full ActivityNet category set, no category is unseen. The comparison with AdaTAD* is a valid cross-dataset transfer comparison with seen categories, but it should not be described as evidence for open-world category generalization. The paper should either relabel this setting as seen-category domain transfer or provide an additional evaluation on a dataset whose categories are disjoint from all pre-training data.","section":"Section 4.3, Table 3"}],"minor_comments":[{"comment":"The average results across the 10 random samplings are reported without standard deviation; given the sensitivity of category-split results, please include variance or confidence intervals.","section":"Section 4.4"},{"comment":"The notation in Eq. (9) is ambiguous: the sum over t is divided by T^+, and the role of the indicator P_t in the denominator is unclear. Please clarify the normalization.","section":"Equation 9"},{"comment":"The Query-Wise Pooler relies on a 'query-wise split' of the text features, but the paper does not specify how the token boundaries for each query are determined after the queries are concatenated into a structured prompt. Please clarify this step.","section":"Section 3.2"},{"comment":"The maximum text token length is 512 and the prompt is restricted to 35 queries during training; the phrase 'arbitrary number of open-ended natural language queries' should be qualified accordingly.","section":"Section 4.2"},{"comment":"The TriDet row in Table 5 contains a formatting error ('62.447.469.3') that should be corrected to separate mAP values.","section":"Table 5"},{"comment":"The limitations section discusses computational cost but does not mention the category-overlap issue with THUMOS14 and HACS; this limitation should be disclosed.","section":"Section 4.7"}],"recommendation":"major_revision","confidential_remarks":"The paper's supervised fine-tuning and zero-shot moment retrieval results are solid, but the headline open-vocabulary TAD result on THUMOS14 is compromised by pre-training overlap with HACS. I recommend major revision rather than rejection because the flaw is in the evaluation protocol rather than the architecture; a disjoint-category evaluation could restore the claim. The authors should be asked to provide a category-overlap analysis and to either produce truly unseen-category results or explicitly weaken the open-world TAD claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about Grounding-MD. The framework is real: a grounded video-language pre-training recipe for unified temporal action detection and moment retrieval, with structured prompts that take arbitrary numbers of open-ended queries. Supervised fine-tuning beats AdaTAD and TriDet on THUMOS14, sets state-of-the-art on ActivityNet-Captions and Charades-STA among end-to-end MR methods, and does it with a 0.5B model against 7B video-LLMs. The second thing is that the open-vocabulary zero-shot claim on THUMOS14 does not hold as reported, because the 'unseen' categories were very likely present in the HACS pre-training data.\n\nThe leakage is concrete. Section 4.3 says HACS 'encompasses all categories present in ActivityNet,' and the THUMOS14 action classes are ActivityNet classes. So categories Section 4.4 designates as unseen have full temporal annotations in the pre-training set. The paper's own no-pre-training row makes the problem visible: the same VideoMAE+BERT model goes from 19.7 to 27.1 average mAP on the 75/25 split and 11.9 to 26.0 on the 50/50 split after pre-training. That jump is what category memorization looks like. No disjointness check between the unseen split and the pre-training vocabulary is reported. Table 4 therefore supports same-category transfer, not open-world generalization.\n\nWhat holds up: the zero-shot moment retrieval transfer to ActivityNet-Captions and Charades-STA is genuinely external, and the supervised results across three benchmarks are credible. The ablation is informative—CMFE matters most, and average pooling beating attentive pooling is a plausible, non-obvious finding. The architecture uses standard components, but the combination of grounded pre-training with structured prompts for unified TAD/MR is new relative to UniMD and GLIP, and the efficiency story against video-LLMs is real.\n\nSoft spots, in proportion. Table 6 is selective: VideoChat2 beats them on ActivityNet-Captions R@0.5 (27.8 vs 25.3), yet the text claims 'superior performance' and only highlights the R@0.7 wins against Momentor. Table 4 averages 10 random splits without reporting variance. No code or models despite the abstract's promise. The loss weight λ is fixed at 1 with no sensitivity check.\n\nBottom line: this is a competent, well-engineered paper that overclaims its central evidence. The fix is tractable—verify that the unseen split is disjoint from all pre-training categories, or use genuinely absent categories; report split variance; and make the MR comparison even-handed. It deserves a serious referee, with the open-vocabulary re-evaluation as a condition. If you work on TAD/MR unification or grounded video-language pre-training, the recipe and supervised numbers are worth reading.","headline":"Solid grounded pre-training framework for unified video moment detection with credible supervised gains, but the THUMOS14 open-vocabulary evidence leaks pre-training categories through HACS, so the headline open-world claim needs re-earning.","tokens_in":15808,"tokens_out":6267,"would_cite":true,"duration_ms":51331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a single video-language model can detect any user-described moment in an untrimmed video, unifying temporal action detection and moment retrieval in open-world settings.","keywords":["open-world moment detection","temporal action detection","moment retrieval","video-language pre-training","grounded pre-training","structured prompts","zero-shot temporal localization","cross-modal fusion"],"falsifier":"Remove HACS (or explicitly exclude every 'unseen' THUMOS14 category) from the pre-training corpus and rerun the Table 4 open-vocabulary zero-shot evaluation. If the mAP on unseen categories collapses to the non-pre-trained baseline, the open-world generalization claim fails; if it survives, the claim is supported.","tokens_in":14731,"feed_emoji":"🎬","tokens_out":8121,"duration_ms":67366,"temperature":0.7,"pith_summary":"Grounding-MD argues that temporal action detection and moment retrieval are the same underlying problem: given a video and any number of open-ended natural language queries, output every segment that matches each query, and do so in open-world settings where the queries are not restricted to a predefined label set. The paper builds a video-language pre-training framework that feeds the whole set of queries as one structured prompt, aligns video and text with early fusion in an encoder and late fusion in a decoder, and balances short action labels against long event descriptions with a query-wise pooler. After pre-training on action detection and moment retrieval corpora, it reports new state-of-the-art results in zero-shot and supervised settings on ActivityNet, THUMOS14, ActivityNet-Captions, and Charades-STA. The result matters because the model achieves these numbers with roughly 0.5 billion parameters, claiming zero-shot moment retrieval on par with or better than 7-billion-parameter video-language models.","feed_headline":"Small 0.5B model tops 7B video LLMs at moment search","feed_subtitle":"A grounded video-language detector unifies action spotting and language search, and rivals 7B video LLMs with far fewer parameters.","key_machinery":"The mechanism that carries the argument is the structured prompt paired with early-late cross-modal fusion. A structured prompt concatenates any number of open-ended natural language queries, and a query-wise splitter isolates a per-query text representation after fusion, so each query can be scored against every candidate video moment. The Cross-Modality Fusion Encoder supplies the first interaction between otherwise unrelated VideoMAE video features and BERT text features, while the Text-Guided Fusion Decoder refines the alignment at multiple temporal scales produced by the FPN. A contrastive loss matches video moments to their query representations and a distance regression loss pins down segment boundaries, which together let an unseen query written at test time be grounded without retraining.","core_discovery":"The central claim is that open-world moment detection can be cast as conditional grounding: a structured prompt made of an arbitrary number of open-ended queries acts as the condition, and the model detects temporal segments for every query in a single forward pass. Grounding-MD realizes this with two fusion stages. The Cross-Modality Fusion Encoder first runs self-attention inside each modality and cross-attention in both directions, creating an initial semantic link between video frames and text tokens; the Text-Guided Fusion Decoder then re-attends pooled query representations against multi-scale video features to refine boundaries. The Query-Wise Pooler prevents the longer natural-language event descriptions from overwhelming short action labels during joint training. Large-scale pre-training on HACS, FineAction, and a filtered subset of InternVid supplies the alignment that the VideoMAE and BERT backbones do not natively share. On this basis the paper reports state-of-the-art open-vocabulary zero-shot results on THUMOS14, zero-shot domain transfer on ActivityNet, ActivityNet-Captions, and Charades-STA, and supervised gains across all four benchmarks.","pith_inferences":["The structured-prompt formulation is the temporal analogue of open-vocabulary object detection: the same recipe that made object categories promptable in image grounding could make dense video captioning, action segmentation, and highlight detection promptable in video.","If the efficiency comparison holds under matched pre-training data, video-LLM temporal grounding may shift toward hybrid designs in which a small grounded detector proposes segments and a large language model reranks them, cutting generation cost.","The paper caps prompts at 35 queries during training; behavior beyond that cap is untested, so a natural extension is measuring per-query accuracy as the query count grows.","Section 4.7 concedes that the pre-training phase demands substantial compute and defers masked video modeling to future work, so the open-world advantage is currently demonstrated only at the 0.5B scale with a fixed pre-training budget."],"forward_implications":["A single detector can serve arbitrary user queries at inference time, so video search and action monitoring need no longer train a new classifier head for each closed category set.","Grounding-based detection appears to be a far cheaper route to open-world temporal localization than generative video-LLM pipelines, since a 0.5B-parameter model matches or beats 7B-parameter baselines on zero-shot moment retrieval.","Adding a new action category becomes a purely textual operation: append the query to the structured prompt, and the existing model can localize it without fine-tuning.","The reported gains at high IoU thresholds, including a 3.8-percentage-point improvement over AdaTAD on THUMOS14 at tIoU 0.7, suggest the text-guided decoder sharpens temporal boundaries rather than only improving semantic matching.","The query-wise pooling design makes text-length imbalance a first-class training concern, a lesson that carries over to any multi-query detection or retrieval setting where query lengths vary widely."],"supporting_citations":[{"why":"The closed-set unified moment detection baseline that the paper extends to open-world settings.","marker":"[54]"},{"why":"Source of the grounded pre-training formulation and the contrastive query-moment alignment loss.","marker":"[18]"},{"why":"Supplies the TIA video backbone and the AdaTAD baseline the paper compares against.","marker":"[27]"},{"why":"Supplies the FPN multi-scale transformer encoder and the DIOU regression loss.","marker":"[55]"},{"why":"Source of the InternVid data from which the filtered InternVid-MR pre-training subset is built.","marker":"[46]"},{"why":"HACS is the main temporal action detection pre-training corpus and the basis of the zero-shot ActivityNet baseline.","marker":"[58]"},{"why":"The 7B-parameter Momentor baseline whose zero-shot moment retrieval results the paper claims to match or beat.","marker":"[37]"},{"why":"Establishes the seen/unseen THUMOS14 split and protocol used for open-vocabulary zero-shot evaluation.","marker":"[13]"},{"why":"Open-set grounded detection approach cited as inspiration for the unified open-world moment detection formulation.","marker":"[26]"}],"fun_headline_variants":["Open-world moment detection with a 0.5B model beats 7B LLMs","Cross-modality fusion nails open-vocabulary moment detection","Structured prompts unlock zero-shot moment detection on video","Small grounded model rivals 7B video LLMs for moment search","Grounded video-language pre-training unifies moment detection and search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The open-vocabulary zero-shot evaluation assumes the 'unseen' THUMOS14 action categories were never encountered during pre-training, yet the paper notes that the pre-training dataset HACS covers all categories in ActivityNet, and THUMOS14 categories form a subset of ActivityNet categories, so the reported open-world gains may instead be same-category transfer.","fun_headline_variants_meta":{"raw":{"variants":["Open-world moment detection with a 0.5B model beats 7B LLMs","Cross-modality fusion nails open-vocabulary moment detection","Structured prompts unlock zero-shot moment detection on video","Small grounded model rivals 7B video LLMs for moment search","Grounded video-language pre-training unifies moment detection and search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3358,"prompt_tokens":994,"completion_tokens":2364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2274}},"tokens_in":610,"tokens_out":2364,"duration_ms":15112,"temperature":1.0,"reasoning_tokens":2274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:46:02.072771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove HACS (or explicitly exclude every 'unseen' THUMOS14 category) from the pre-training corpus and rerun the Table 4 open-vocabulary zero-shot evaluation. If the mAP on unseen categories collapses to the non-pre-trained baseline, the open-world generalization claim fails; if it survives, the claim is supported.","supporting_citations":[{"cited_title":"Unimd: Towards unifying moment retrieval and temporal ac- tion detection","cited_arxiv_id":null,"evidence_quote":"The closed-set unified moment detection baseline that the paper extends to open-world settings."},{"cited_title":"Grounded language-image pre-training","cited_arxiv_id":null,"evidence_quote":"Source of the grounded pre-training formulation and the contrastive query-moment alignment loss."},{"cited_title":"End-to-end temporal action detection with 1b parameters across 1000 frames","cited_arxiv_id":null,"evidence_quote":"Supplies the TIA video backbone and the AdaTAD baseline the paper compares against."},{"cited_title":"Hacs: Human action clips and segments dataset for recognition and temporal localization","cited_arxiv_id":null,"evidence_quote":"HACS is the main temporal action detection pre-training corpus and the basis of the zero-shot ActivityNet baseline."},{"cited_title":"Prompting visual-language models for efficient video understanding","cited_arxiv_id":null,"evidence_quote":"Establishes the seen/unseen THUMOS14 split and protocol used for open-vocabulary zero-shot evaluation."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Open-set grounded detection approach cited as inspiration for the unified open-world moment detection formulation."}],"review_version":1}