{"id":"be497182-799b-41b3-9a14-1f00156bfe96","arxiv_id":"2411.19941","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper presents a new hour-long video question-answering benchmark with 70 long-context questions, plus challenge leaderboard results showing large gaps between models and humans.","lead":"This paper reports the 2024 Perception Test challenge results and introduces 1h-walk VQA, a small benchmark of 70 manually-written questions over hour-long city tour videos. The benchmark is meant to test whether video models can reason across long temporal contexts, where current models score 44% versus a 99.6% human baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's long-context claim rests on unquantified manual review; Appendix Q1/Q14/Q41 look answerable from a single frame or brief clip, so the hour-long property is not established.","rationale":"The reader's weakest_assumption identifies exactly the point on which the paper's contribution hinges. If the 70 questions can be answered from a short clip, the benchmark is not an hour-long reasoning benchmark, regardless of video length. I agree with this assessment. The paper's Section 2 describes a two-iteration human annotation process, but the final selection criterion is asserted, not demonstrated. The appendix even contains items that appear to refute the criterion, e.g., Q1 is a static count. This makes the concern concrete rather than hypothetical. My proposed short-clip control would settle it. Other issues—the swapped best/runner-up in Table 2, the 'HOTA' header in Table 9, and the underspecified human baseline timing—are real but secondary; they affect presentation and leaderboard reliability, not the existence of the benchmark artifact. Because the reader already conditions acceptance on validating the long-context property and releasing the data, my read does not move the verdict. I set verdict_should_be to UNCHANGED and agreement to agree. The appendix's full question list is a positive: it makes the item-level audit feasible and the proposed test directly executable.","tokens_in":11228,"tokens_out":6172,"duration_ms":57430,"concrete_test":"For each of the 70 questions, construct a matched short-clip input: for questions mentioning a specific landmark or timestamp, extract a 5-second clip centered on that event (e.g., the frame containing the gate, the bell ring, the Caffé Nero audio); for other questions, use the single most informative frame according to an independent annotator. Run a strong zero-shot VLM (e.g., Gemini 1.5 Pro or Qwen2-VL-7B) on the 5-option question with only this short clip, and separately have 3 fresh annotators answer with only the short clip. Compare accuracy to the 20% chance level with a binomial test. If either model or human accuracy is significantly above chance on more than, say, 20% of items, the long-context claim is falsified for those items; report the fraction and re-run the leaderboard excluding them. If the fraction is low, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 asserts that the 70 QA pairs were manually reviewed and selected to \"cannot be answered from a single frame or a very short clip,\" yet no quantitative check, baseline, or release of the selection protocol is provided. This is the load-bearing condition for the paper's central claim that 1h-walk VQA measures hour-long temporal reasoning. Several questions in the appendix appear to violate the stated criterion: Q1 asks to count statue figures above a gate (a single frame suffices); Q14 asks how many dogs were encountered (some encounters are visible in short clips, and the counting operation itself does not require hour-long integration); Q41 asks what sound is heard at Caffé Nero (a short audio clip suffices). Without a short-clip control, we cannot distinguish genuine long-range integration from item-level localization that a strong VLM could solve from a few frames. The paper provides no inter-annotator agreement on this property, only the authors' manual review, which is not independently checkable from the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on the Second Perception Test challenge, organized as a workshop at ECCV 2024, summarizing the setup, baselines, and results of seven tracks: object tracking, point tracking, temporal action localization, temporal sound localization, multiple-choice video QA, grounded video QA, and the newly introduced hour-long video QA track. The main novel contribution is 1h-walk VQA, a small benchmark built from the Walking Tours dataset, consisting of 70 manually curated five-way question-answer pairs over ten hour-long city tour videos, with a 3-video/11-question validation split and a 7-video/59-question test split. The paper describes the two-stage annotation procedure, reports human and random baselines, and presents the top two submissions per track, highlighting that the winning hour-long video QA system reaches 44% accuracy versus a human baseline of 99.64%.","tokens_in":11512,"tokens_out":4158,"duration_ms":37568,"significance":"The proposed benchmark addresses a real gap in video QA evaluation, since most existing datasets permit answers from very short clips or single frames; a public benchmark that genuinely requires hour-long temporal reasoning would be a valuable community resource. The challenge results provide a useful snapshot of progress across multiple video understanding tasks, including a new long-video track, and the paper makes its question list available in the appendix. However, the central claim that 1h-walk VQA measures hour-long temporal reasoning is not yet convincingly established, and the statistical and tabular inconsistencies reduce the reliability of the reported results. If the long-context property is later verified with a short-clip control and the reporting errors are corrected, the benchmark could serve as a seed for larger-scale efforts, as the discussion itself acknowledges.","major_comments":[{"comment":"The paper asserts that questions were manually reviewed and selected because they 'cannot be answered from a single frame or a very short clip,' but it provides no quantitative check of this property, no inter-annotator agreement, and no details of the selection protocol. The appendix contains questions that appear to violate the stated criterion: Q1 ('How many statue figures were there above the gate seen just before the DIESEL fashion store?') can be answered from a single frame, and Q41 ('When passing by Caffé Nero, what sound can be heard?') from a short audio clip. A short-clip control or an analysis showing the temporal spread of the evidence needed for each question is required to support the load-bearing claim that 1h-walk VQA measures hour-long understanding.","section":"Section 2"},{"comment":"The human baseline of 99.64% is not consistent with the described annotation protocol. The text says each question was answered by 10 participants and each participant received 27 questions; with 70 questions this gives 700 total answers in the simplest reading, and an accuracy of 99.64% corresponds to about 0.36 errors, which is not an integer. The paper should report the exact total number of human answers and the number of errors, or revise the protocol description to match the reported accuracy.","section":"Section 4.7, Table 9"},{"comment":"There are clear reporting errors in the results tables. In Table 2, the 'Best' entry (NJUST-THU) is reported with IoU 0.734, which is lower than the 'Runner-up' entry (FAUgeddaboudit) with IoU 0.813; as ranked, the best should have the highest score. In Table 9, the metric column is labeled 'HOTA' although Section 4.7 explicitly defines the metric for this track as top-1 accuracy. These errors call into question the reliability of the reported challenge results and must be corrected.","section":"Tables 2 and 9"},{"comment":"Even if a question is not answerable from a single frame or a very short clip, it does not follow that the question requires hour-long temporal reasoning. Several questions, such as Q13 (two appearances of a clocktower), Q15 (comparing two marketplaces at timestamps 33:20 and 51:00), and Q16 (time between two bridge crossings), can be answered by retrieving two localized short clips rather than integrating information across the entire hour. The paper should provide evidence—for example, an analysis of the temporal distance between relevant segments, or an evaluation comparing full-video access with access only to short windows—that the questions indeed require long-range video understanding.","section":"Section 2"}],"minor_comments":[{"comment":"The caption states 'Average video length in our newly-proposed 1h-walk VQA benchmark compared to existing benchmarks,' but the figure does not show axis labels or units; please add them so the comparison is interpretable.","section":"Figure 2"},{"comment":"The description of the human baseline is under-specified: 'Each participant received 27 questions' does not make clear how the 70 questions were distributed across participants or how the total number of answers was derived; please provide the exact counts.","section":"Section 4.7"},{"comment":"The 'Train' row uses dashes for both columns; consider stating explicitly that no training data or training questions are provided, which is already implied by the zero-shot evaluation intent.","section":"Table 1"},{"comment":"The appendix lists the unique questions, but the paper does not provide a direct pointer to the actual benchmark files (e.g., question JSON, annotations, or a dataset URL); adding this would make the benchmark fully reproducible and usable by the community.","section":"Section 2"},{"comment":"Some references have formatting issues, such as the trailing period in the URL of the 'A simple recipe' entry and inconsistent page ranges; please standardize according to the journal style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop challenge summary and introduces a small benchmark. The benchmark's central claim about hour-long reasoning is not yet verified, and the reporting errors in Tables 2 and 9 plus the human baseline inconsistency are concerning but fixable. I would encourage the editor to treat this as a major revision requiring a short-clip baseline or an equivalent evidence that the questions cannot be solved from short windows, and a careful correction of the statistical and tabular reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the new 1h-walk VQA benchmark is a real first in hour-long video QA, and the 44% vs 99.6% human gap is dramatic. But the paper's central claim—that the questions actually require hour-long context—is unsupported. Several appendix questions look answerable from a frame or short clip, and the challenge tables have mechanical errors.\n\nWhat's genuinely new: no prior benchmark offers hour-long videos with questions designed to span multiple segments and modalities. The manual curation process with two iterations is described, and the choice of non-narrated Walking Tours videos is sound. The split is small (70 questions total) but honest about it. The human baseline also tells you something: humans get essentially all questions right, so the gap is real.\n\nThe soft spot is load-bearing. The paper says questions were selected to be unanswerable from a single frame or short clip, but that selection is only supported by the authors' own manual review. There is no inter-annotator agreement, no short-clip baseline, no control. The appendix makes this concrete: Q1 counts statue figures above a gate (single frame), Q41 asks for a sound at a specific café (short audio clip), Q14 asks how many dogs were encountered (detection plus counting, not necessarily hour-long integration). If a meaningful fraction of the 70 questions are like this, the benchmark does not measure what it claims. That needs fixing before people build on it.\n\nAlso, Table 2 inverts best and runner-up (0.813 vs 0.734), and Table 9 is titled HOTA but contains accuracy. With only 59 test questions, there are no confidence intervals, so the leaderboard differences should be read cautiously. The appendix lists questions but not the answer options or ground truth, so the benchmark is not fully usable from the paper.\n\nI would still send this to peer review—new long-horizon benchmarks are scarce and the challenge drew real submissions—but I'd ask the authors to release the full QA set with answers and provide a short-clip baseline. For my own work, I wouldn't cite it as evidence about hour-long understanding until that validation exists.","headline":"New hour-long video QA benchmark with a real human-model gap, but the long-context claim is unvalidated and the tables have errors.","tokens_in":11905,"tokens_out":3637,"would_cite":false,"duration_ms":30176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces 1h-walk VQA, a benchmark for hour-long video question answering, and reports that even the best model is far below human accuracy.","keywords":["perception","evaluation","hour-long video understanding","video question answering","long temporal context","multimodal","benchmark"],"falsifier":"Give a capable video-QA model the same 1h-walk VQA questions under two conditions: once with the full hour-long video and once with only a short clip around the answer-relevant frames. If the short-clip condition matches or nearly matches full-video accuracy, or if full-video accuracy is no better than chance for human raters seeing only short clips, the benchmark is not measuring hour-long understanding.","tokens_in":11055,"feed_emoji":"🎬","tokens_out":5811,"duration_ms":49273,"temperature":0.7,"pith_summary":"The paper’s central contribution is 1h-walk VQA, a small benchmark for hour-long video question answering built from ten un-narrated city-tour videos. It adds 70 manually curated five-way questions that are meant to require reasoning over video and audio across temporally separated moments, with many questions spanning more than thirty minutes. The motivation is that existing video-QA benchmarks can often be answered from a single frame or a short clip, so they do not test long-range temporal understanding. In the first challenge run on this benchmark, the best model scored 44.07% top-1 accuracy while a zero-shot human baseline scored 99.64%, leaving a wide gap to close.","feed_headline":"Hour-long video QA: top models 44%, humans 99%","feed_subtitle":"New 1h-walk VQA benchmark asks questions spanning a full city walk, and top models still trail people badly.","key_machinery":"The load-bearing object is the 1h-walk VQA benchmark itself: ten hour-long, un-narrated city-walk videos paired with 70 five-way multiple-choice questions. The curation process is what makes it target long context: raters first proposed questions that needed one, two, or more separated video segments or audio, and a second annotation round used event detection to localise repeated events such as bridge crossings, stair climbs, and clock-tower views before writing questions around those timestamps. A final manual review kept only questions that cannot be answered from a single frame or a very short clip.","core_discovery":"The paper claims that 1h-walk VQA measures hour-long understanding because its questions were selected through iterative annotation and manual review to require more than one video segment, or both video and audio, to answer. The benchmark contains 70 question-answer pairs over the ten Walking Tours videos, split into 3 validation videos with 11 questions and 7 test videos with 59 questions, and is deliberately zero-shot with no training data provided. The paper’s reported results show that the strongest submitted system, a sliding-window chain-of-thought method, still falls far short of human accuracy, which it takes as evidence that hour-long video reasoning remains an open problem.","pith_inferences":["My inference: the no-short-clip property is the one assumption most worth stress-testing; an input-length ablation could shrink the benchmark’s claimed difficulty if many questions are solvable locally.","My inference: the event-detection pipeline could be automated to scale beyond 70 curated questions, replacing expensive human annotation with detector-guided template questions.","My inference: with only ten source videos, per-city and per-walk idiosyncrasies may dominate results, so the benchmark’s numbers should be read as provisional evidence about hour-long understanding generally."],"forward_implications":["The best 2024 hour-long-VQA submission reached 44.07% versus 99.64% for human raters, so the benchmark leaves substantial headroom.","Because the benchmark is zero-shot by design, strong performance has to come from pretraining and prompting rather than fine-tuning on its 11 validation questions.","The same event-detector-then-question pipeline used to create 1h-walk VQA could be re-run on other long videos to build larger hour-long benchmarks.","Across all seven tracks, the 2024 winners improved over the 2023 winners, with grounded video QA showing the largest jump."],"supporting_citations":[{"why":"Supplies the ten long, un-narrated city-tour videos that 1h-walk VQA is built on.","marker":"Venkataramanan et al., 2024"},{"why":"Documents that typical video-QA questions can be answered from short clips, motivating the long-context curation.","marker":"Papalampidi et al., 2023"},{"why":"Defines the Perception Test benchmark and the human zero-shot baseline protocol used in the challenge.","marker":"Pătrăucean et al., 2023"}],"fun_headline_variants":["New hour-long video QA benchmark: AI 44%, humans 99%","AI trails humans badly on long video QA: 44% vs 99%","Hour-long video understanding unsolved: AI 44%, humans 99%","1h-walk VQA: top AI models score 44%, humans 99%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark’s value depends on the manual-review judgment that no selected question can be answered from a single frame or a short clip, and the paper provides no quantitative verification of that property.","fun_headline_variants_meta":{"raw":{"variants":["New hour-long video QA benchmark: AI 44%, humans 99%","AI trails humans badly on long video QA: 44% vs 99%","Hour-long video understanding unsolved: AI 44%, humans 99%","1h-walk VQA: top AI models score 44%, humans 99%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2295,"prompt_tokens":862,"completion_tokens":1433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1346}},"tokens_in":478,"tokens_out":1433,"duration_ms":9717,"temperature":1.0,"reasoning_tokens":1346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:38:31.065476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a capable video-QA model the same 1h-walk VQA questions under two conditions: once with the full hour-long video and once with only a short clip around the answer-relevant frames. If the short-clip condition matches or nearly matches full-video accuracy, or if full-video accuracy is no better than chance for human raters seeing only short clips, the benchmark is not measuring hour-long understanding.","supporting_citations":[],"review_version":1}