{"id":"41d497d9-0e0c-4e36-a300-19cde196f409","arxiv_id":"2504.15681","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Vidi uses dense 1 fps sampling and decomposed attention to localize text queries in videos up to an hour or longer, outperforming GPT-4o and Gemini on the new VUE-TR benchmark.","lead":"Vidi is a family of multimodal AI models that locate, in videos up to an hour or longer, the exact time ranges matching a text query. The authors also present VUE-TR, a new benchmark with audio-based queries and manually labeled timestamps, where Vidi reports large gains over GPT-4o and Gemini.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VUE-TR annotations come from the same Section 5 LLM query+caption pipeline that generates Vidi's post-training data; manual curation edits machine-generated queries, so the reported margins over API baselines may reflect distribution overlap rather than general temporal grounding.","rationale":"The strongest claim is comparative performance on VUE-TR. For that claim to hold, the benchmark must be an independent test. The paper's own text creates the opposite expectation: the training data and benchmark are produced by the same LLM query-generation pipeline, with the same caption/subtitle source signals and the same three query formats, and the manual annotation step is a quality filter rather than an independent sample of user queries. This is the least secure link because it directly explains the reported margin: Vidi is fine-tuned to localize caption-like text and to handle queries that mirror VUE-TR, whereas the baselines have no such adaptation. The other major caveat, unequal API constraints, is disclosed and could in principle be removed by giving GPT-4o more frames or Gemini higher resolution; but even a perfectly equalized API evaluation would not fix the benchmark-distribution confound. I agree with the reader's weakest assumption. I do not find an internal inconsistency in the decomposed-attention derivation or the multi-span IoU definition that would change the verdict; those affect all compared models equally and are secondary to representativeness. The proposed held-out human-query experiment is therefore the decisive check. If Vidi's margin survives on independently written queries, the central claim is substantially supported and the paper would mainly need disclosure and artifact release. If the margin collapses, the conditional verdict should be tightened. Since the reader already issued a CONDITIONAL verdict based on essentially this concern, my read does not move the verdict.","tokens_in":16113,"tokens_out":8501,"duration_ms":81808,"concrete_test":"Create a held-out evaluation split from a random subset of VUE-TR videos in which queries are authored from scratch by human annotators who watch the raw video, without access to the Section 5 CoT caption/subtitle pipeline; keep the same three query formats, modality tags, and timestamp annotation rules, and use the same prompting and parsing procedures as the original paper for GPT-4o and Gemini. Rerun the AUC of IoU comparison of Table 3. If Vidi's margin over GPT-4o and Gemini shrinks substantially or disappears on this independent-query split, then the reported superiority reflects training/test distribution overlap rather than general temporal grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Vidi is superior at temporal retrieval rests on the VUE-TR benchmark being an independent, representative test of the task. The paper undermines that condition: Section 2 says the temporal retrieval training data mirror the structure of the VUE-TR benchmark; Section 5 builds training query/timestamp pairs by CoT-prompting an LLM from dense captions and subtitles; and Section 6 says benchmark annotations are manually curated using the same annotation pipeline described in Section 5. The loop is not broken by manual curation, because annotators edit and verify machine-generated queries rather than writing queries independently from raw video. Moreover, Section 4.3 trains Vidi to localize dense captions and sentence-level subtitles, and Section 5 generates benchmark queries as paraphrases of exactly those caption/subtitle signals. Vidi is therefore directly trained on the mapping that the benchmark tests, while GPT-4o and Gemini are not. The comparison is also asymmetric: GPT-4o is limited to 120 frames and Gemini videos are downscaled to 256p in Section 7.2. Those constraints could in principle be equalized, but the pipeline overlap is a confound that advantages Vidi by construction. Without evidence that VUE-TR queries are not near-duplicates of post-training queries and that the benchmark videos are disjoint from the roughly 1 million real training videos, the abstract's claim of remarkable superiority is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Vidi and Vidi-1.5, large multimodal models for temporal retrieval in long videos (up to and beyond one hour), together with a new benchmark, VUE-TR. The model architecture uses the decomposed-attention mechanism of Kuo et al. [13] with a modified fixed-alpha weighting, dense 1 fps visual sampling, and audio input via Whisper. Training proceeds through adapter alignment, synthetic video/audio data with timestamp supervision, real-video dense-caption/subtitle supervision, and application post-training on temporal retrieval queries. VUE-TR contains 428 videos and 1,598 queries spanning 20 seconds to over one hour, with keyword/phrase/sentence formats and vision/audio/vision+audio modality tags, and uses an AUC-of-IoU metric for multispan evaluation. The paper reports large margins over GPT-4o, Gemini-2.0-Flash, and Gemini-2.5-Pro on VUE-TR, especially for long videos.","tokens_in":16422,"tokens_out":4867,"duration_ms":46166,"significance":"If the results hold, VUE-TR is a valuable evaluation resource: it is substantially longer than existing temporal retrieval benchmarks, includes audio-based and vision+audio queries, covers multiple query formats, and proposes a multispan IoU metric. The model's ability to process densely sampled hour-long videos at 1 fps and 16 kHz audio on a single 80 GB GPU is practically meaningful for video editing workflows. The benchmark and model are accompanied by substantial engineering effort, including a multi-stage training pipeline with 30 million timestamped samples. However, the significance of the head-to-head comparison depends on VUE-TR being an independent test; as discussed below, the current design leaves that condition unmet, and the API constraints on baselines further weaken the comparison. The paper's architecture contribution beyond the authors' prior D-Attn work is also modest.","major_comments":[{"comment":"The VUE-TR benchmark is not independent of Vidi's post-training data in the way required to support the abstract's claim. Section 5 describes an annotation pipeline in which LLM-generated queries are derived from dense captions and subtitles and then verified by human annotators; Section 6 states that VUE-TR annotations are \"manually curated using a robust annotation pipeline described in Section 5\"; and Section 2 states that the temporal retrieval training data \"mirror the structure of the VUE-TR benchmark.\" Since Section 4.3 trains Vidi directly on the same dense captions and sentence-level subtitles, Vidi is trained on the exact mapping that VUE-TR evaluates, while the API baselines are not. Please report (a) whether any of the 428 benchmark videos appear in the roughly 1M-video training corpus, (b) the distribution of n-gram or embedding similarity between benchmark queries and post-training queries, and (c) results on a held-out set of queries written by annotators directly from raw video, without LLM generation. Without (a)–(c), the reported margins may reflect distributional overlap rather than general temporal grounding ability.","section":"§5, §6, §2"},{"comment":"The baseline comparison is asymmetric in ways that can only lower the proprietary models' scores. GPT-4o receives at most 120 uniformly sampled frames and no audio, Gemini long videos are downscaled to 256 pixels to satisfy the 100 MB upload limit, and Gemini-2.5-Pro \"often\" produces empty outputs due to content filtering. The paper does not state how empty outputs were parsed or scored, nor does it provide an equalized control (e.g., Vidi with 120 frames, Vidi without audio, Vidi at 256p) to quantify the effect of these constraints. Please report such controls and the empty-output rate and score handling; otherwise the claim that Vidi \"significantly outperforms\" these systems is not established.","section":"Table 3, §7.2"}],"minor_comments":[{"comment":"There are several typos and spacing errors: the abstract says \"video understand editing,\" §7.2 says \"There models\" instead of \"These models,\" and the Section 8 heading appears as \"T emporal Retrieval.\" Please proofread the manuscript.","section":"Abstract, §7.2, §8"},{"comment":"The metric definitions should specify how I(·,·) and U(·,·) are computed for multi-interval predictions and ground truths; the current prose (\"interaction and union function\") is not a formal definition. The summation notation also needs a clear statement of whether it runs over predicted intervals, ground-truth intervals, or all pairwise overlaps.","section":"§6.2, Eq. (5)"},{"comment":"The paper reports a single run per model with no variance or significance measures. Given 1,598 queries, please provide confidence intervals or at least a per-category breakdown of empty and unparseable outputs for the API baselines, especially for Gemini-2.5-Pro.","section":"Table 3"},{"comment":"The statement that the model \"achieves over 80% accuracy on a synthetic evaluation set\" lacks details on how accuracy is computed and on the size and difficulty of that set; please specify the metric and the evaluation protocol.","section":"§4.2"},{"comment":"The paper claims efficient processing of videos exceeding two hours on a single 80 GB GPU, but no runtime, peak memory, or token-count measurements are reported; please add concrete numbers to support this practical claim.","section":"§2, §7.1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim depends on VUE-TR being an independent evaluation, but the benchmark is introduced by the same team and generated through the same LLM-based annotation pipeline used for the model's post-training data. The requested overlap and similarity analyses, plus equalized baseline controls, are therefore essential before the head-to-head margins can be taken as evidence of general temporal grounding ability. I would also advise the editor that the architecture is substantially the authors' own prior D-Attn work; the paper's incremental architectural contribution is the fixed-alpha weighting, which lacks an ablation. A neutral third-party evaluation, or at minimum a benchmark with queries written from scratch by annotators, would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the VUE-TR benchmark is a real contribution: 428 videos up to over an hour, audio and vision+audio queries, three query formats, manual annotation, and a multi-interval IoU metric. It fills a genuine gap, since prior temporal retrieval benchmarks cap at a few minutes and ignore audio. Second, the headline result — Vidi well above GPT-4o and Gemini — is undermined by a design overlap the paper never addresses. The same Section 5 pipeline that produces Vidi's post-training query/timestamp pairs also produces the VUE-TR annotations. Annotators edit and verify machine-generated queries; they do not independently write queries from raw video. So the test set shares distribution and phrasing with the training set. The paper also never states that VUE-TR videos are disjoint from the roughly 1 million real training videos. That confound advantages Vidi by construction.\n\nThe model side is more solid. D-Attn is the authors' own prior work, and applying it to dense 1 fps sampling with audio is reasonable. The fixed alpha weighting is a sensible simplification to avoid the text token being drowned out by long video token counts. The training pipeline is detailed, and the system is practical, running on one 80 GB GPU. No fabricated math; the attention decomposition is derived properly.\n\nThe baseline comparison is also unequal in ways the paper discloses. GPT-4o gets at most 120 frames and no audio; Gemini videos are downscaled to 256p; Gemini-2.5-Pro often returns empty outputs. Those are API constraints, not deliberate sandbagging, but they mean the margin over proprietary APIs is not a clean comparison. Equalized baselines might still lose, but we do not know from this paper.\n\nWhat is missing: a necessary evaluation on an independently built benchmark, or at least a statement that VUE-TR queries were written from scratch by human annotators and that the videos are disjoint from training. Also, release of data and code, or clear access, should be required. For the benchmark alone, the paper deserves peer review. For the claimed superiority, it needs serious revision.\n\nWho this is for: researchers working on long-video grounding and temporal retrieval, and teams building multimodal LLMs for editing. A serious referee should engage, but should focus scrutiny on benchmark validity and baseline fairness. My recommendation: send to peer review with a major-revision decision. The benchmark is worth refereeing; the results need much more careful evaluation.","headline":"VUE-TR is a genuinely useful long-video temporal retrieval benchmark, but the paper's central claim that Vidi beats GPT-4o/Gemini is not established because the test set is generated by the same pipeline as the training data.","tokens_in":17027,"tokens_out":3622,"would_cite":true,"duration_ms":30240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vidi-1.5, a 9B multimodal model, retrieves timestamp ranges matching natural-language queries in videos from 20 seconds to over an hour, outperforming GPT-4o and Gemini on the VUE-TR benchmark.","keywords":["temporal retrieval","long video understanding","multimodal large language models","video editing","audio-visual grounding","benchmark","decomposed attention","moment retrieval"],"falsifier":"Evaluate Vidi on temporal-retrieval datasets whose annotations were created independently of this paper's pipeline (QVHighlights, ActivityNet Captions, Charades-STA) using their standard evaluation protocols, or build a new benchmark with fresh human-written queries from annotators who have never seen Vidi's training data; if Vidi's margin over Gemini and GPT-4o shrinks sharply or reverses, the VUE-TR results would reflect training-test distribution overlap rather than general long-video grounding.","tokens_in":15912,"feed_emoji":"🎬","tokens_out":11807,"duration_ms":87689,"temperature":0.7,"pith_summary":"This paper introduces Vidi, a family of large multimodal models that retrieve the time ranges in videos matching a natural-language query, with emphasis on long, real-world footage from 20 seconds to over an hour. The authors also build the VUE-TR benchmark, 428 videos and 1,598 manually refined queries in keyword, phrase, and sentence form, split across vision-only, audio-only, and vision-plus-audio. On this benchmark, Vidi-7B and Vidi-1.5-9B reach an overall AUC of IoU of 35.4% and 39.6%, while Gemini-2.0-Flash reaches 21.2%, Gemini-2.5-Pro 15.2%, and GPT-4o 13.6%. The paper argues that dense one-frame-per-second sampling plus an efficient attention design can make second-precision temporal grounding practical for hour-long video, a core step in automatic editing.","feed_headline":"Vidi-1.5 beats GPT-4o and Gemini at finding clips in hour-long video","feed_subtitle":"AUC of IoU on VUE-TR: 39.6% for Vidi-1.5 vs 21.2% for Gemini-2.0-Flash, 15.2% for Gemini-2.5-Pro, 13.6% for GPT-4o.","key_machinery":"The load-bearing mechanism is Decomposed Attention (D-Attn), which rewrites the causal self-attention of a pretrained LLM into visual-to-visual self-attention, text-to-text self-attention, and text-to-visual cross-attention. A diagonal variant of the visual-to-visual attention cuts the computational cost from $O(N^2)$ to $O(N)$ in the number of visual tokens, and debiased positional encodings in the cross-attention remove position bias between text and vision. Vidi also fixes the mixture weights to $\\alpha_V = \\alpha_A = \\alpha_T = 1$ so that very long videos, which produce far more visual tokens than text tokens, do not let the multimodal stream drown out the query. Together these components allow the model to ingest one visual frame per second plus a 16 kHz audio track across hour-long inputs while keeping second-level localization.","core_discovery":"Vidi's central claim is that a single multimodal model can localize natural-language queries—keywords, phrases, or sentences—to exact timestamp ranges in videos spanning 20 seconds to more than one hour, including queries that require audio alone or vision plus audio. On the VUE-TR benchmark, both Vidi and Vidi-1.5 beat the best proprietary baselines on every duration bucket for the primary AUC of IoU metric, and the gap widens on long and ultra-long videos: on videos over 60 minutes, Vidi-1.5 scores 32.3% versus 2.9% for Gemini-2.0-Flash and 2.4% for Gemini-2.5-Pro. The authors attribute this to a Decomposed Attention LLM that reduces complexity in the number of visual tokens from $O(N^2)$ to $O(N)$, fixed modality weightings that keep text and multimodal inputs balanced for any video length, and a training pipeline that progresses from synthetic timestamped data to dense-captioned real videos.","pith_inferences":["A natural stress test is to evaluate Vidi on independently annotated short-video benchmarks such as QVHighlights and ActivityNet; if the margins over Gemini and GPT-4o hold there, the gains generalize beyond VUE-TR.","Because 47% of VUE-TR queries mix vision and audio, ablating the audio encoder (removing audio tokens from vision+audio queries) would quantify how much of Vidi's edge comes from the audio channel.","The fixed $\\alpha = 1$ weighting is a simple remedy for the video-length imbalance; an alternative that normalizes attention contributions by token count could retain the same benefit while remaining a principled dynamic weighting scheme."],"forward_implications":["Hour-long raw footage becomes searchable by natural language at second-level precision, making the 'find the good take' step of video editing automatable.","Audio-based queries can be retrieved, not just visual events; the benchmark shows 18% of queries are audio-only and 47% require both vision and audio.","Temporal retrieval accuracy degrades far more slowly as video length grows than it does for the proprietary baselines, whose AUC of IoU collapses to single digits on videos over 30 minutes.","The model runs on a single 80 GB GPU without quantization, so it is deployable in editing products rather than requiring specialized hardware.","The refined IoU metric, which handles multiple predicted and ground-truth time ranges, gives the field a standard for multi-span temporal retrieval."],"supporting_citations":[{"why":"Supplies the Decomposed Attention architecture whose O(N) visual-token cost makes hour-long dense sampling tractable.","marker":"[13]"},{"why":"Mistral-7B-Instruct-v0.3 is the language model backbone for Vidi.","marker":"[11]"},{"why":"Gemma-2-9B is the language model backbone for Vidi-1.5.","marker":"[25]"},{"why":"SigLIP is the visual encoder for Vidi, converting each sampled frame into visual tokens.","marker":"[35]"},{"why":"SigLIP 2 is the upgraded visual encoder for Vidi-1.5, contributing to its performance gain.","marker":"[27]"},{"why":"Whisper large-v3 encodes the 16 kHz audio stream that gives Vidi audio-query ability.","marker":"[22]"},{"why":"Gemini-2.0-Flash and Gemini-2.5-Pro are the proprietary baselines that Vidi outperforms on VUE-TR.","marker":"[1]"},{"why":"GPT-4o is the proprietary image-text baseline, evaluated with 1 fps frames and a custom prompt.","marker":"[10]"},{"why":"LongVALE is the closest prior audio-visual temporal retrieval benchmark, whose short-duration limitation VUE-TR is designed to overcome.","marker":"[7]"},{"why":"QVHighlights supplies a prior standard for multi-range moment retrieval that informs VUE-TR's IoU metric.","marker":"[14]"}],"fun_headline_variants":["Vidi-1.5 beats GPT-4o, Gemini at clip search in hour-long video","Vidi-1.5 locates clips in hour-long videos, surpassing GPT-4o and Gemini","Vidi-1.5 outperforms GPT-4o and Gemini on temporal retrieval in hour-long videos","Vidi-1.5 finds exact timestamps in hour-long videos, beating GPT-4o and Gemini","Vidi-1.5 scores 32.3% vs 2.9% on hour-long video clip retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The VUE-TR benchmark's queries and timestamps come from the same LLM-based generation pipeline that produced Vidi's post-training data, so the benchmark may not be an independent test of general long-video grounding.","fun_headline_variants_meta":{"raw":{"variants":["Vidi-1.5 beats GPT-4o, Gemini at clip search in hour-long video","Vidi-1.5 locates clips in hour-long videos, surpassing GPT-4o and Gemini","Vidi-1.5 outperforms GPT-4o and Gemini on temporal retrieval in hour-long videos","Vidi-1.5 finds exact timestamps in hour-long videos, beating GPT-4o and Gemini","Vidi-1.5 scores 32.3% vs 2.9% on hour-long video clip retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00125,"raw_usage":{"total_tokens":5215,"prompt_tokens":1127,"completion_tokens":4088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":3954}},"tokens_in":743,"tokens_out":4088,"duration_ms":26540,"temperature":1.0,"reasoning_tokens":3954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:20:21.915681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate Vidi on temporal-retrieval datasets whose annotations were created independently of this paper's pipeline (QVHighlights, ActivityNet Captions, Charades-STA) using their standard evaluation protocols, or build a new benchmark with fresh human-written queries from annotators who have never seen Vidi's training data; if Vidi's margin over Gemini and GPT-4o shrinks sharply or reverses, the VUE-TR results would reflect training-test distribution overlap rather than general long-video grounding.","supporting_citations":[{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"SigLIP is the visual encoder for Vidi, converting each sampled frame into visual tokens."},{"cited_title":"Robust speech recognition via large-scale weak supervision","cited_arxiv_id":null,"evidence_quote":"Whisper large-v3 encodes the 16 kHz audio stream that gives Vidi audio-query ability."},{"cited_title":"Berg, and Mohit Bansal","cited_arxiv_id":null,"evidence_quote":"QVHighlights supplies a prior standard for multi-range moment retrieval that informs VUE-TR's IoU metric."}],"review_version":1}