{"id":"c9fd5e77-5c81-4eea-a7f8-ec9797713567","arxiv_id":"2507.18552","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 103K-sample omni-modal video dataset with COT-generated factual, abstract, and intent text layers, plus a 3K benchmark showing foundation models fail at intent-level retrieval.","lead":"VideoMind is a new dataset of 103,000 videos paired with audio and long, layered text descriptions that go from factual summaries to speculative statements about the video creator's intent. It also ships a 3,000-sample benchmark that shows current video models understand factual captions well but struggle with intent-focused queries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Intent-layer ground truth rests on undocumented model-agreement and indirect generation checks; without human reliability metrics, the 35%-vs-80% retrieval gap may not measure deep understanding.","rationale":"The paper's headline contribution is a dataset with a deep-cognitive intent layer and a benchmark demonstrating that current models fail at intent retrieval. Both the dataset's value and the benchmark's conclusion depend on the intent labels being genuinely the intent of the uploader and protagonist. The reader's verdict already identifies this as the weakest assumption, and my stress-test agrees: the validation pipeline in Section 3.2 Step 3 is indirect and under-reported. Pre-validation checks consistency between two models, not correctness; post-validation checks text-to-video plausibility, not correspondence with the source video's actual intent; and the 3K human validation lacks all reliability metrics. This is not an internal inconsistency in the method, but a missing evidentiary basis for the central claim. I considered other potential concerns—such as the 'first deep-cognitive dataset' claim versus IntentQA, or the confound of query length in retrieval—but those are secondary: if the intent labels are wrong, the benchmark measures something else entirely, while if they are right, the resource is valuable even if the 'first' claim is softened. The paper does provide a large, openly released dataset with detailed multi-layer captions and a clear evaluation setup, which is genuine credit. However, the absence of annotation reliability data is a concrete, fixable gap. My proposed test—an independent human re-annotation with agreement metrics—would either vindicate the benchmark or show it is unreliable. Because the reader already requested these missing details as part of a conditional acceptance, my stress-test does not move the verdict; I recommend no change (UNCHANGED) and would ask the authors to release the annotation protocol, agreement scores, and the identity/threshold of the pre-validation model before final acceptance.","tokens_in":10053,"tokens_out":4746,"duration_ms":47219,"concrete_test":"Re-annotate a random 300-sample subset of the VideoMind-3K benchmark with three independent annotators who are blind to the released labels, asking each to judge, from the full video, whether the intent expression correctly states the purpose of the uploader or the protagonist, and to mark it as correct/incorrect/uncertain. Compute pairwise Cohen's kappa and agreement with the released labels. If kappa is below 0.6 or label agreement is below 70%, the intent ground truth cannot support the deep-cognitive claim. As a secondary check, rerun the pre-validation step of Section 3.2 on 1,000 samples using two architecturally distinct mLLMs (e.g., Qwen2.5-Omni and GPT-4o) and report the intent-label change rate; a change rate above 20% would indicate model-dependent labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that VideoMind's intent-layer annotations are reliable ground truth and that the R@1 drop from ~80% (factual) to ~35% (intent) in Table 2 demonstrates current models' lack of deep video understanding—rests entirely on the validity of the intent labels. Section 3.2 Step 3 describes two validation mechanisms, but neither establishes ground truth. Pre-validation computes embedding similarity between the intent term produced by Qwen2.5-Omni and that of a second, unspecified mLLM; this only measures inter-model agreement. No similarity threshold, model identity, or extraction details are reported, and if the second model shares training data or bias with Qwen2.5-Omni, agreement is uninformative. Post-validation uses Wan2.1 to generate a 10-second video from each intent expression and asks two experts to judge whether the generated video is 'reasonable'; this tests the text-to-video model's ability to produce a plausible rendering, not whether the intent is the actual uploader's or protagonist's purpose. The only direct human validation is the 3,000-sample benchmark, described as 'independently validated by three professional annotators,' yet no annotation instructions, selection criteria, or inter-annotator agreement metrics (e.g., Cohen's kappa) are provided. Without such reliability statistics, the benchmark's gold-standard status is unsupported. Furthermore, the retrieval comparison confounds cognitive depth with text length (143 vs. 43 words on average, per Section 3.4) and lexical overlap, so part of the intent-layer drop could stem from query difficulty rather than semantic depth. The load-bearing assumption is therefore the reliability and validity of the intent annotations, and this assumption is not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VideoMind is introduced as a 103K-sample omni-modal video dataset in which each clip is paired with audio and a three-layer textual description (factual, abstract, intent) generated by the mLLM Qwen2.5-Omni in a chain-of-thought procedure, supplemented by 6W tags and a 3,000-sample manually validated benchmark. The paper reports hybrid-cognitive text-to-video and video-to-text retrieval results for several video foundation models, finding that intent-layer queries achieve markedly lower R@1 (~35%) than factual-layer queries (~80%), which the authors interpret as evidence that current models lack deep video understanding. The dataset and benchmark are publicly released.","tokens_in":10421,"tokens_out":3327,"duration_ms":37026,"significance":"If the intent-layer annotations are valid ground truth, VideoMind would be a valuable resource: it is substantially larger and more text-rich than existing video-centric datasets, it is the first to provide explicit intent annotations at scale, and its retrieval results offer a concrete, reproducible falsifiable claim about a limitation of current video foundation models. The authors also deserve credit for releasing the data, code, and model evaluation results, and the internal statistics (103K samples, 22M words, average 225 words) are consistent across the paper. The central empirical finding—a large retrieval gap between factual and intent queries—is interesting and worth investigating, but its interpretation depends entirely on the validity of the intent labels, which is insufficiently established.","major_comments":[{"comment":"The pre-validation step accepts an intent expression only when the embedding similarity between the intent term produced by Qwen2.5-Omni and that of a second, unspecified mLLM exceeds an unreported threshold. This is a measure of inter-model agreement, not of correctness against the true intent of the uploader or protagonist; if the second model shares training data or biases with Qwen2.5-Omni, the agreement is uninformative. The paper must report the second model's identity, the similarity threshold, the extraction procedure, and ideally the agreement of both models with human judgments on a subsample.","section":"§3.2, Step 3 (pre-validation)"},{"comment":"The post-validation step generates a 10-second video from each intent expression using Wan2.1 and asks two expert annotators to judge whether the generated video is 'reasonable'. This tests whether the text-to-video model can produce a plausible rendering of the intent expression, not whether the expression reflects the actual purpose of the video creator or the depicted person. The paper provides no criteria for 'reasonable', no inter-annotator agreement, and no account of how failures were treated (discarded, regenerated, or edited). This is a load-bearing weakness because the post-validation is one of only two mechanisms supporting the claim that the intent labels are valid ground truth.","section":"§3.2, Step 3 (post-validation)"},{"comment":"The paper states that all 3,000 benchmark samples were 'independently validated by three professional annotators', but it gives no annotation instructions, no sample selection criteria, and no inter-annotator reliability statistics such as Cohen's kappa or Krippendorff's alpha. Without such evidence, the gold-standard status of the benchmark is unsupported, and the retrieval results in Tables 2 and 3 cannot be interpreted as measuring performance against reliable intent ground truth.","section":"§3 and §4 (3,000-sample benchmark)"},{"comment":"The retrieval comparison confounds the depth of the described cognitive layer with the length and lexical form of the queries. Factual-layer descriptions average 143 words, while abstract- and intent-layer descriptions average 38 and 43 words, respectively (Section 3.4). Longer, more specific queries are generally easier to match to videos, and the fixed intent template '[subject] aims to [intent] by [action]' may be distributionally distant from the natural captions on which the evaluated models were trained. The reported R@1 drop from ~80% to ~35% may therefore reflect text length and format rather than a lack of deep video understanding. The authors should include matched-length query baselines, retrieval using human-written intent descriptions, or a per-layer human upper bound to separate these factors.","section":"§3.4 and Table 2"},{"comment":"The paper's broader conclusion that 'current foundation models demonstrate limited capability in representing the latent purposes of videos' (Section 4) is only as strong as the validity of the intent labels. Given that the labels are generated and validated by the methods described in Section 3.2, the 35% R@1 result may partly measure how well models can recover synthetic intent guesses rather than genuine human intent. A experiment in which human annotators retrieve videos from intent queries, or in which intent labels are compared against uploader-provided titles/descriptions, would substantially strengthen the claim.","section":"§4 and §5"}],"minor_comments":[{"comment":"The text refers to 'SoRA' in the introduction; this is presumably a typo for 'Sora'.","section":"§1 and Figure 1"},{"comment":"The sentence 'The selected quadruples would go through the text generation process (Figure 2)' is repeated almost verbatim in the same paragraph; one occurrence should be removed.","section":"§3.1 and Figure 2"},{"comment":"The phrase '6W-element tags' is unclear; consider defining '6W' (who, where, when, why, how, what) explicitly at first use.","section":"§3.3"},{"comment":"In the sentence 'These data are strictly restricted to be used only for testing and MUST NOT be involved in any stage of model training', the formatting 'MUST NOT' appears to be a typographical error; use normal capitalization.","section":"§4"},{"comment":"Reference [2] contains a typo: 'Aannual meeting' should be 'Annual meeting'. Also, many references lack DOIs or arXiv IDs; this is acceptable for a preprint but should be cleaned up for the camera-ready version.","section":"References"},{"comment":"The column header 'Len 𝑡𝑒𝑥𝑡' is awkwardly formatted; use 'Text length' or 'Avg. words'.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a strong dataset contribution with a clear and potentially important central claim, but the validity of the intent labels is not sufficiently established. The two validation mechanisms described in Section 3.2 are indirect and under-specified, and the lack of inter-annotator agreement for the 3K benchmark is a serious gap. The retrieval experiments are also confounded by text length and format. These issues are fixable with additional analysis and reporting, so I recommend major revision rather than rejection. I would also encourage the editor to ask the authors to clarify the relationship between the two mLLMs used in pre-validation, since if both are from the Qwen family the agreement evidence is nearly vacuous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: VideoMind is a real resource—103K clips with audio, OCR, ASR, and three-layer text (factual/abstract/intent), averaging 225 words. The intent layer, formatted as [subject] aims to [intent] by [action], is genuinely new relative to VAST, VALOR, InternVid, and IntentQA. The 3K benchmark and the retrieval experiments are a reasonable first pass at measuring deep understanding, and the drop from ~80% R@1 on factual queries to ~35% on intent queries is striking. The paper is transparent about its generation pipeline, which is more than many dataset papers do.\n\nThe soft spots are real and they land on the central claim. The intent labels are produced by Qwen2.5-Omni and validated two ways: embedding similarity to a second, unnamed mLLM's output, and text-to-video generation with Wan2.1 judged by two experts. Neither establishes ground truth. Model-model agreement says nothing about whether the intent matches human judgment, and a plausible text-to-video rendering is not evidence that the intent is correct. The paper reports no inter-annotator agreement for the 3,000 professionally validated samples, no annotation instructions, no similarity threshold, and no identity of the second mLLM. Without those, the benchmark's gold-standard status is unsupported.\n\nThere is also a confound in the retrieval numbers: factual queries average 143 words, intent queries 43 words. Some of the R@1 drop is likely query length rather than cognitive depth. A matched-length baseline would help.\n\nThat said, these are fixable issues. The dataset itself is probably reusable, and the three-layer structure is a useful contribution. I would want to see a revised version with human reliability metrics, direct human validation of intent labels (not via text-to-video), specification of the validation thresholds, and a length-controlled retrieval experiment. The authors should also soften the 'FIRST deep-cognitive' claim—IntentQA already tackles intent reasoning, though not in this layered dataset form.\n\nBottom line: worth sending to a serious referee. The resource has value, and the flaws are about missing details rather than a fundamentally broken design. I would advise the editor to request revision rather than desk-reject. The audience is video-language researchers who need a richly annotated corpus; the benchmark is a bonus, but only after validation is documented. For my own work, I would not cite it yet—I would wait until the validation is documented.","headline":"A genuinely useful omni-modal video dataset with a new intent layer, but the intent ground truth is not yet established, so the headline retrieval gap should be read as provisional.","tokens_in":10965,"tokens_out":3380,"would_cite":false,"duration_ms":32236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VideoMind is a 103K-video dataset that annotates every clip with factual, abstract, and intent descriptions, and shows that today's video models retrieve only about 35% of intent-level queries versus about 80% for factual ones.","keywords":["video understanding","intent grounding","omni-modal dataset","cross-modal retrieval","chain-of-thought","multimodal large language model","deep-cognitive understanding","video benchmark"],"falsifier":"Have independent human annotators write intent expressions for a random subset of the 3,000 benchmark videos without seeing the model-generated labels, then run text-to-video retrieval using those human intents as queries. If the human-annotated queries retrieve the correct videos at a much higher rate than the model-generated ones (or much lower), the model-generated intent labels do not correspond to human-level understanding of intent.","tokens_in":9881,"feed_emoji":"🎬","tokens_out":8828,"duration_ms":84274,"temperature":0.7,"pith_summary":"VideoMind is a new dataset of 103,000 video clips, each paired with audio and a detailed written description organized in three layers: what is factually seen and heard, an abstract summary, and the intent behind the video—why the uploader posted it and why the person in it acts that way. The paper's core claim is that this intent layer, produced by step-by-step Chain-of-Thought prompting, makes VideoMind the first dataset that supports deep-cognitive video understanding rather than surface captioning. On a 3,000-sample human-validated benchmark, the paper reports that leading video foundation models retrieve videos using intent-level text with only about 35% Rank-1 accuracy, compared with about 80% for factual text. That gap is the paper's evidence that current models know what happens in a video but not why it happens. If the annotations are trustworthy, VideoMind provides a training and evaluation resource for making video models reason about purpose and motivation.","feed_headline":"VideoMind: AI video models get intent right only 35%","feed_subtitle":"A 103K-video dataset with intent annotations shows models understand what happens, not why.","key_machinery":"The load-bearing mechanism is a three-stage Chain-of-Thought generation pipeline run on a multimodal large language model. Stage one produces five separate factual descriptions covering visual content, background audio, OCR, ASR, and text; stage two synthesizes an abstract summary; stage three infers intent through two role-playing tasks (uploader and protagonist) using the fixed template '[subject] aims to [intent] by [action].' Each intent is then double-validated: its key terms are compared by embedding similarity against another mLLM's independent speculation, and experts assess a text-to-video reconstruction to judge whether the intent is plausible. The evaluation protocol—separate retrieval with factual, abstract, and intent queries, plus 'hit any layer' and 'hit all layer' video-to-text metrics—is what surfaces the depth gap.","core_discovery":"The paper introduces VideoMind and claims it is the first deep-cognitive omni-modal video dataset. Each of its 103K samples decomposes the video into five non-overlapping factual channels—visual, non-speech audio, OCR, ASR, and raw text—then summarizes those into an abstract layer, and finally speculates on intent using the fixed expression '[subject] aims to [intent] by [action],' generated separately from the uploader's and the protagonist's perspectives. A held-out set of 3,000 samples is manually validated as a gold-standard benchmark. In hybrid-cognitive retrieval experiments, the authors find that five widely used video foundation models achieve about 80% Rank-1 retrieval with factual queries but only about 35% with intent queries, and that retrieving all three layers of ground-truth text within the top 10 is rare, with the lowest-ranked layer often beyond position 500. The paper interprets this drop as evidence that these models lack the deep understanding needed to represent video intent.","pith_inferences":["One testable extension the paper does not run: fine-tune a video foundation model on the 100K training split and measure intent retrieval on the 3K test set; if performance does not improve, the dataset still evaluates but does not teach intent grounding.","The dual-validation design conflates intent with plausibility: a generated video can look reasonable even when the stated intent is wrong, so the benchmark may be more reliable for retrieval than for ground-truth intent semantics.","The observed gap between factual and intent retrieval might be partially explained by distribution shift in text styles, not only by semantic depth; an ablation that matches factual texts for length and format would separate the two.","If other groups apply the same COT pipeline to different base mLLMs and produce different intent labels, the benchmark's stability across generators becomes an important test of whether it measures video intent or model priors."],"forward_implications":["If the intent layer is valid, training on VideoMind could give video foundation models a learning signal that aligns representations with purpose and motivation, improving intent recognition, emotion recognition, and content moderation.","The benchmark provides a standardized test that existing video datasets lack, allowing future models to be compared on deep understanding instead of surface description.","The fixed '[subject] aims to [intent] by [action]' format may allow automatic extraction of subject, action, and intent tags, supporting fine-grained recognition tasks.","The ~35% intent retrieval rate can serve as a baseline; any model that surpasses it on the VideoMind-3K benchmark would demonstrate a measurable improvement in deep video understanding."],"supporting_citations":[{"why":"Baseline model evaluated in hybrid-cognitive retrieval; its intent-layer failure anchors the central claim.","marker":"[19]"},{"why":"Baseline model evaluated in hybrid-cognitive retrieval; shows the same factual-to-intent drop.","marker":"[8]"},{"why":"Baseline model evaluated in hybrid-cognitive retrieval.","marker":"[23]"},{"why":"Baseline model evaluated in hybrid-cognitive retrieval.","marker":"[20]"},{"why":"Omni-modal dataset/model used as the main comparison and as a baseline for retrieval and text generation.","marker":"[3]"},{"why":"Source of video curation methodology (YouTube clips) and an mLLM-generated captioning baseline.","marker":"[18]"},{"why":"Prior video intent reasoning dataset that VideoMind contrasts with as one-sided Q&A descriptions.","marker":"[7]"}],"fun_headline_variants":["Video models get intent only 35% in VideoMind test","VideoMind: AI understands facts 80%, intent 35%","Why AI fails video intent: 35% on new benchmark","VideoMind benchmark: intent accuracy drops to 35%","Video models see actions, not intentions: 35% score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the intent labels being genuine ground truth for the uploader's and protagonist's purposes, even though they are generated by a model and validated only through similarity to another model's output and experts' judgment of a generated video.","fun_headline_variants_meta":{"raw":{"variants":["Video models get intent only 35% in VideoMind test","VideoMind: AI understands facts 80%, intent 35%","Why AI fails video intent: 35% on new benchmark","VideoMind benchmark: intent accuracy drops to 35%","Video models see actions, not intentions: 35% score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1494,"prompt_tokens":1025,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":641,"tokens_out":469,"duration_ms":5198,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:31:47.919384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators write intent expressions for a random subset of the 3,000 benchmark videos without seeing the model-generated labels, then run text-to-video retrieval using those human intents as queries. If the human-annotated queries retrieve the correct videos at a much higher rate than the model-generated ones (or much lower), the model-generated intent labels do not correspond to human-level understanding of intent.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline model evaluated in hybrid-cognitive retrieval; shows the same factual-to-intent drop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline model evaluated in hybrid-cognitive retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline model evaluated in hybrid-cognitive retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Omni-modal dataset/model used as the main comparison and as a baseline for retrieval and text generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior video intent reasoning dataset that VideoMind contrasts with as one-sided Q&A descriptions."}],"review_version":1}