{"id":"da76de5e-d8f7-4115-8be2-5e987232af38","arxiv_id":"2506.09953","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OKCV is a new human-annotated video dialogue dataset where answering questions requires both visual grounding in the video and external knowledge.","lead":"This paper introduces OKCV, a dataset of 2,017 videos with about 6,000 human-annotated conversations that require outside knowledge to answer. It provides a new benchmark that combines video understanding, dialogue management, and external knowledge retrieval.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transcript-only question generation makes 'visually grounded' the least-secure load-bearing property; a text-only ablation could settle whether the video is actually needed.","rationale":"The reader's weakest_assumption already points to the transcript-based generation pipeline and the lack of human verification; my stress-test agrees with that premise but sharpens the failure mode. The issue is not only that the external-knowledge fraction comes from DeepSeek-R1 rather than annotators, but that the question-generation step is video-blind by construction, so the visually grounded property cannot be guaranteed and is not separately measured. This is a correctness risk rather than a novelty objection: the dataset may still be useful, and the paper gives credit where due by releasing the data, collecting temporal certificates and source links, and reporting baselines. But the defining combination 'video + outside knowledge + dialogue' is only distinctive if the video is genuinely needed. Since the paper does not provide any transcript-only control, the central claim is conditional on that validation. The reader's CONDITIONAL verdict is therefore the right outcome; my read does not move it, so the verdict is unchanged.","tokens_in":13213,"tokens_out":5762,"duration_ms":69124,"concrete_test":"Run a transcript-only ablation on a random sample of at least 200 OKCV dialogues. For each dialogue, give a strong text-only LLM (e.g., GPT-4 or Claude 3.5) the question-side turns plus the corresponding QuerYD transcript, with no video frames, and score the generated answers against the human reference using the same BLEURT, BartScore, and Prometheus-2 metrics reported in Section 5. Compare this to the best video-conditioned baseline scores in Table 3. If transcript-only performance is statistically indistinguishable from or close to the video-conditioned baselines on Bleurt/Prometheus, then the video is not necessary for those dialogues and the visual-grounding claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OKCV is a human-annotated video-dialogue benchmark whose dialogues are visually grounded in video segments while requiring outside knowledge. For that claim to hold, the questions must actually depend on visual content. The collection pipeline in Section 3.2, however, generates all question-side dialogue drafts from video transcripts alone: 'we employed a combination of the transcripts and a LLM (GPT-4)' and 'leveraging the video transcription and GPT-4's internal knowledge.' GPT-4 never sees video frames. The prompt in Appendix 7.1 asks it to ensure that questions require multiple frames, but this is a textual instruction that the model cannot verify from a transcript; a question can pass the prompt's surface criteria while being fully answerable from the transcript. Human annotators later watch the video and write answers, and temporal certificates in Section 3.3 record which intervals they thought were needed, but none of these steps establishes that the video, rather than the transcript, is necessary to answer the question. The only quantitative check in Section 4, the DeepSeek-R1 estimate that 57.93% of questions require external knowledge, measures outside knowledge, not visual grounding, and is not human-validated. The paper's Limitations section acknowledges transcription bias but does not address transcript-only answerability. If a strong text-only model can answer a large fraction of dialogues from the transcript alone, the 'visually grounded' component of the contribution is not load-bearing, and OKCV reduces to a transcript-grounded knowledge dialogue dataset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OKCV, a video dialogue dataset intended to combine visual grounding in video with external knowledge. The dataset contains 2,017 videos from QuerYD, with 5,986 human-completed dialogues and 40,954 turns. The collection pipeline uses GPT-4 to generate question-side dialogue drafts from video transcripts, after which crowdworkers provide answers, optionally edit the drafts, list external sources, and mark temporal intervals (temporal certificates) needed to answer. The paper reports dataset statistics, an LLM-based estimate that 57.93% of questions require external knowledge, and zero-shot and fine-tuned baselines using several language and vision-language models with different knowledge augmentations. The authors frame OKCV as the first human-annotated video dialogue benchmark that combines visual grounding with outside knowledge.","tokens_in":13492,"tokens_out":7079,"duration_ms":72158,"significance":"If the defining properties were verified, OKCV would be a valuable resource: it is released publicly, is not restricted to a narrow domain, and its top-down decomposed dialogues with temporal certificates could support research in video dialogue, retrieval, and knowledge integration. The dataset's scale (5,986 dialogues, 40,954 turns) is substantial. However, the evidence that the dialogues actually require both visual grounding and outside knowledge is weak, and one of the paper's own validation numbers partially contradicts the abstract's claim. The baselines, while extensive, do not currently demonstrate that the video modality is necessary for the task. The contribution is therefore promising but requires stronger validation before the central claims can be accepted.","major_comments":[{"comment":"The abstract states that 'questions further require external knowledge that is not visually present,' but Section 4 reports that DeepSeek-R1 classifies only 57.93% of questions as requiring external knowledge. This means the defining property of the dataset fails for roughly 42% of the questions. The dataset should either be filtered to include only questions that meet this criterion, or the abstract and title should be revised to reflect that a majority, not all, questions require outside knowledge.","section":"Abstract; Section 4"},{"comment":"Because GPT-4 drafts the questions from video transcripts alone, the prompt instruction that questions require multiple frames is not verifiable from text, and no later step in the pipeline demonstrates that a question cannot be answered from the transcript alone. Human annotators watch the video to write answers, but the answers could be produced from the transcript as well. A text-only baseline (e.g., an LLM given only the dialogue and transcript) is missing; the zero-shot 'Nothing' results in Table 3 show that language-only models already achieve non-trivial Bleurt scores, which is suggestive but not conclusive. Please add a text-only ablation and, ideally, human judgments of whether each question is visually grounded.","section":"Section 3.2; Appendix 7.1"},{"comment":"The validation that 57.93% of questions require external knowledge is not independent: the questions were generated by GPT-4 under a prompt requiring outside knowledge, and the estimate is produced by DeepSeek-R1, another LLM. There is no human verification of this property. Moreover, the annotation interface in Section 7.2 already asks turkers whether each dialogue requires more visual, commonsense, or factoid reasoning; these human judgments appear not to be analyzed in the paper, even though they could serve as a validation signal. Please report these annotations and/or conduct a human evaluation of a random sample.","section":"Section 4"},{"comment":"The text claims that 'both metrics agree that Llama2-7B, when utilizing both captions and external knowledge, achieves the best performance among the open-source models,' but Table 3 does not support this: Video-Llava with no knowledge achieves Bleurt 0.59, exceeding Llama2-7B's 0.58, and Prometheus-2 ranks Phi-3-Small (Nothing) highest at 3.74. BartScore and Bleurt also disagree with Prometheus-2 on the best model. Please clarify how the overall performance conclusion is derived despite these conflicts.","section":"Section 5; Table 3"}],"minor_comments":[{"comment":"The term 'human-annotated dialogues' is potentially misleading because the question-side turns are generated by GPT-4; consider describing the dataset as 'human-completed' or clarifying the split of labor in the abstract.","section":"Abstract; Section 3.3"},{"comment":"The sentence 'we employed a combination of the transcripts and a LLM (GPT-4)' is vague; the appendix prompt shows only the LLM using the transcript, so please clarify how the combination works.","section":"Section 3.2"},{"comment":"The dialogue count for OKCV is shown as '5.7k' while the text reports 5,986; please make the numbers consistent.","section":"Table 1"},{"comment":"The temporal certificate description asks annotators for the closest begin and end time needed to complete a dialogue, but the interface seems to collect these per dialogue; please clarify whether certificates are per dialogue or per question.","section":"Section 3.3"},{"comment":"The overlap between the two histograms makes the distribution hard to read; consider using separate panels or a log scale.","section":"Figure 2"},{"comment":"The model is called 'Video-Llava' in the results table but 'Video-Llama' in the text (first paragraph of Section 5); please unify the naming.","section":"Section 5"},{"comment":"The mention of a manual check of a randomly sampled subset lacks details; please provide the sample size and any agreement measures.","section":"Section 4"},{"comment":"The reference to 'Sanket Shah and Talukdar, 2019' is formatted inconsistently with the other Shah et al. entries; please check the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset release is a concrete contribution, but I would encourage the editor to weigh the validation gap against the paper's strong framing. The paper's reliance on LLM-generated questions is fine if clearly stated, but the abstract overclaims the 'outside knowledge' property. I also note that the paper cites an email discussion for fair use of YouTube videos; this is not a technical issue, but the legal status of the dataset is worth verifying given the source. The lack of analysis of the reasoning-type annotations collected from turkers is a missed opportunity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OKCV is a concrete and useful resource: a released, human-annotated video dialogue dataset with 2,017 videos, 5,986 dialogues, temporal certificates, and source links. The combination it targets—video grounding plus outside knowledge plus dialogue—does not exist in the prior datasets the authors cite, and that is a genuine contribution. The data collection pipeline is described transparently, and the human annotators did real work in answering questions, editing drafts, and marking which video segments were needed. That part of the paper is solid and deserves credit.\n\nThe soft spots are real but fixable. The biggest one is that the 'visually grounded' property is asserted rather than demonstrated. GPT-4 generated the questions from transcripts alone; it never saw the video. The prompt tells it to make questions require multiple frames, but a text model cannot verify that, and the human annotators were not asked to check whether the video was necessary. So the central characterization in the abstract—that the dialogue context is visually grounded—rests on an untested assumption. A cheap text-only baseline (transcript plus dialogue, no video) would settle this, and I am surprised the authors did not run it. If a strong LLM can answer most turns from transcripts, the dataset is still interesting but it is a knowledge-grounded conversation dataset, not a video-grounded one.\n\nSecond, the external-knowledge claim is also weakly validated. The 57.93% figure comes from DeepSeek-R1, a model; no human verification is reported, and by that estimate 42% of questions do not require outside knowledge. That makes the dataset's name an over-promise. What saves the paper is that it is clear about its pipeline, and the Limitations section does acknowledge transcription bias, though it does not address the transcript-only answerability concern.\n\nThe baselines are noisy: Gemini was evaluated on only 45% of videos, the three metrics disagree on which model wins, and there are no significance tests. As a benchmark paper, that is acceptable for a first pass but limits what you can conclude about model capability gaps.\n\nBottom line: this is a worthwhile dataset that deserves a serious referee, but the referee should ask for a text-only ablation and a human-validated subsample for the knowledge and grounding properties. I would cite the dataset, and I would bring the stress-test discussion to a reading group.","headline":"A useful, released video-dialogue dataset with a novel knowledge+conversation combination, but its central 'visually grounded' property is under-validated because questions are drafted from transcripts alone.","tokens_in":14010,"tokens_out":2512,"would_cite":true,"duration_ms":28578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces OKCV, a benchmark of 2,017 videos with 5,986 human-annotated dialogues that require a model to ground answers in specific video segments while drawing on knowledge not present in the video.","keywords":["OKCV dataset","outside knowledge VQA","video dialogue","visual grounding","human-annotated dialogues","temporal certificates","question decomposition","video understanding"],"falsifier":"Take a random sample of about 200 OKCV dialogues and have one group of annotators answer after watching the video while a separate group answers from the transcript alone; if the transcript-only group achieves comparable correctness and information content, the visual-grounding claim collapses. Alternatively, have expert annotators judge whether each question genuinely needs outside knowledge; if the human-verified proportion falls well below 57.93%, the outside-knowledge claim fails.","tokens_in":13043,"feed_emoji":"🎬","tokens_out":3729,"duration_ms":44451,"temperature":0.7,"pith_summary":"The paper tries to establish that a conversational agent watching a video must do three things at once: locate the relevant moments in the video, bring in world knowledge that is not visible in those moments, and keep track of an ongoing dialogue. It introduces OKCV, a dataset of 2,017 videos with 5,986 human-annotated dialogues totaling 40,954 interleaved turns, where each turn is visually grounded in a specific video segment but the questions require outside knowledge. If the dataset works as claimed, it fills a gap no previous benchmark covers: video-language models are currently tested either on video question answering, on knowledge-based image QA, or on visually grounded dialogue, but not on the combination. The paper also provides baselines showing that open-source video-language models score low on standard similarity metrics, while a closed-source model scores high on one metric and low on another, suggesting the task is not yet solved.","feed_headline":"Video dialogue benchmark adds outside-knowledge questions","feed_subtitle":"OKCV pairs visual grounding with world knowledge across 5,986 human-annotated conversations.","key_machinery":"The central mechanism is a human-in-the-loop collection pipeline. GPT-4 receives a video transcript and a few-shot prompt that requires each generated question to be related to the video, to demand outside knowledge, and to need multiple frames, and it is asked to decompose one high-level question into dependent sub-questions; human annotators then watch the video, answer and edit the questioner's side, and record temporal certificates and any external sources they used. The temporal certificate, borrowed from EgoSchema, is an interval or set of intervals of the video that suffices to answer the dialogue, and it is what lets the dataset claim genuine visual grounding. The DeepSeek-R1-based classification of questions into external-knowledge versus vision-only is the quantitative support for the outside-knowledge property.","core_discovery":"The central claim is that OKCV is the first human-annotated dataset requiring a conversational agent to combine temporal visual grounding, external knowledge, and dialogue context in a single task. The dataset contains 2,017 videos drawn from QuerYD, with 5,986 human-completed dialogues averaging 6.8 turns; each dialogue is built from a GPT-4-generated chain of top-down decomposed questions that annotators answer, may edit, and supplement with temporal certificates marking the video segments needed. An analysis using DeepSeek-R1 estimates that 57.93% of the questions cannot be answered from visual information alone, supporting the outside-knowledge property. Baseline experiments show open-source video-language models produce low similarity scores on Bleurt and BartScore, while Gemini-1.5-pro reaches a high Bleurt score but the lowest BartScore, which the paper interprets as evidence of a large capability gap and of metric disagreement on what counts as a good dialogue answer.","pith_inferences":["Because the questions are drafted from transcripts rather than from the video pixels, a likely risk is that some 'visually grounded' questions are actually transcript-grounded; a natural test is to compare answer quality when models are given the transcript alone versus the full video.","The top-down decomposition of questions into dependent sub-questions may let a model answer later turns by exploiting earlier answers alone, without watching the video; an ablation that hides the dialogue history could measure how much genuine multi-turn reasoning is required.","The temporal certificates provide a supervised signal for video moment retrieval, so the dataset could double as a benchmark for localizing the segments that support each dialogue turn.","The annotator-supplied sources (641 sources across 550 dialogues) could support a retrieval-oriented extension where models must pull knowledge from those exact references rather than from parametric memory."],"forward_implications":["If the benchmark holds, video dialogue systems will need joint spatiotemporal grounding, external knowledge retrieval, and dialogue state tracking, rather than treating video QA and dialogue as separate modules.","The 57.93% external-knowledge estimate implies that roughly two-fifths of OKCV questions might be answerable from visual or transcript information alone, so the dataset tests a mix of skills rather than a pure outside-knowledge setting.","The large gap between Gemini-1.5-pro's Bleurt score and its BartScore shows that reference-based dialogue metrics disagree strongly, meaning evaluation methodology itself is a bottleneck for progress on this task.","Fine-tuning improves Bleurt and BartScore for most open-source baselines but does not consistently improve Prometheus-2 scores, suggesting that fine-tuning helps surface similarity without clearly improving rubric-based answer quality."],"supporting_citations":[{"why":"Supplies the QuerYD video corpus with transcripts, which is the source of all 2,017 videos in OKCV.","marker":"Oncescu et al., 2021"},{"why":"Defines the outside-knowledge VQA task that OKCV extends to the video-dialogue setting.","marker":"Marino et al., 2019"},{"why":"GPT-4 is the model used to generate the questioner-side dialogue drafts from transcripts.","marker":"OpenAI, 2023"},{"why":"Provides the temporal certificate method that OKCV uses to characterize which video segments are needed to answer each dialogue.","marker":"Mangalam et al., 2024"},{"why":"DeepSeek-R1 is the model used to classify what fraction of questions require external knowledge.","marker":"Guo et al., 2025"},{"why":"Defines the visually grounded dialogue paradigm that OKCV extends by adding outside knowledge and video grounding.","marker":"Das et al., 2017"},{"why":"KnowIT-VQA is the closest prior video knowledge-QA benchmark, restricted to sitcoms, which OKCV contrasts by being open-domain.","marker":"Garcia et al., 2020"}],"fun_headline_variants":["First dataset for video dialogue with outside knowledge","OKCV: benchmark for knowledge-hungry video dialogues","Video dialogue challenge needs external knowledge 58% of time","New benchmark tests video dialogue beyond visual content","Human-annotated video dialogues demand outside knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4-generated questions, drafted from transcripts, really do require watching the video and really do require outside knowledge; the paper does not verify this at scale with human annotators, since the 57.93% figure comes from a model-based classification.","fun_headline_variants_meta":{"raw":{"variants":["First dataset for video dialogue with outside knowledge","OKCV: benchmark for knowledge-hungry video dialogues","Video dialogue challenge needs external knowledge 58% of time","New benchmark tests video dialogue beyond visual content","Human-annotated video dialogues demand outside knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1235,"prompt_tokens":917,"completion_tokens":318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":245}},"tokens_in":533,"tokens_out":318,"duration_ms":3885,"temperature":1.0,"reasoning_tokens":245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:36:13.740267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 OKCV dialogues and have one group of annotators answer after watching the video while a separate group answers from the transcript alone; if the transcript-only group achieves comparable correctness and information content, the visual-grounding claim collapses. Alternatively, have expert annotators judge whether each question genuinely needs outside knowledge; if the human-verified proportion falls well below 57.93%, the outside-knowledge claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the QuerYD video corpus with transcripts, which is the source of all 2,017 videos in OKCV."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the temporal certificate method that OKCV uses to characterize which video segments are needed to answer each dialogue."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KnowIT-VQA is the closest prior video knowledge-QA benchmark, restricted to sitcoms, which OKCV contrasts by being open-domain."}],"review_version":1}