{"id":"8c4892ad-54eb-46e2-be60-b3547c2a68d3","arxiv_id":"2507.14743","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new traffic-intersection VideoQA benchmark containing roughly 28,800 human-verified GPT-seeded QA pairs, with evaluations showing fine-tuning improves three video-language models.","lead":"This paper introduces InterAct VideoQA, a benchmark dataset of 8 hours of traffic-intersection video with tens of thousands of question-answer pairs. It tests and fine-tunes three video-language models, finding that domain-specific fine-tuning improves traffic question answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth QA reliability is unquantified; all 28,800 pairs are claimed human-verified but no inter-annotator agreement, error rate, or audit sample is reported, so the benchmark's validity and fine-tuning improvements rest on an unsupported premise.","rationale":"The paper is transparent about discovering GPT hallucinations and claims a manual cleanup, but the absence of any quantitative quality metrics makes the cleanup unfalsifiable. Because every downstream result—benchmark scores, fine-tuning improvements, and the dataset's value as a testbed—depends on label correctness, this is the single point of failure. The reader's weakest assumption correctly identifies this. The concrete test would settle it: if independent re-annotation matches the released labels, the concern is resolved; if not, the benchmark's evaluations and fine-tuning claims are not interpretable. The dataset-size and taxonomy inconsistencies reinforce the need for direct verification but are secondary. I therefore keep the reader's CONDITIONAL verdict: the paper should be accepted only if the authors supply the missing quality metrics and reconcile the reported numbers.","tokens_in":16093,"tokens_out":6630,"duration_ms":69798,"concrete_test":"Release a random sample of 500 QA pairs with their source clips; have two fresh annotators, blinded to the released answers, independently answer the questions from the videos using the paper's rubric. Compute exact-match agreement with the released answers and per-category factual error rate (where the released answer contradicts the video). If exact-match agreement is below 95% or the factual error rate exceeds 5%, the ground-truth reliability claim in §4.1 is quantitatively refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that InterAct VideoQA is a reliable benchmark and that fine-tuning on it improves VideoQA models. This requires the QA labels to be correct and unbiased. Section 4.1 states that human evaluators 'cross-referenced every question-answer pair against its video footage' and removed GPT hallucinations, but it provides no inter-annotator agreement, no number of annotators or audited clips, no per-category error rates, and no protocol for resolving disagreements. The supplementary prompt (Figure 5) reveals that QA pairs were generated by GPT-o3 with a prompt that asks for exactly ten QAs per clip and, contradicting Section 8.1's claim, instructs the model to 'show internal reasoning' before outputting the markdown table. If the model followed this instruction, the generated QAs may contain reasoning traces or template-style phrasings that survived the human pass; if the human pass was lighter than described, systematic biases (e.g., yes/no questions, object-presence checks, counterfactual negations) could be baked into the labels. Either way, the reported BLEU/ROUGE/METEOR/CIDEr/SPICE scores and fine-tuning deltas in Tables 2-3 would measure fit to a flawed ground truth, not genuine traffic reasoning. The dataset-size inconsistency (abstract 'over 25,000', §3.4 '28,800', Table 1 '~2.9k/34.6k') is a symptom of the same problem: the paper's quantitative claims are not aligned with each other, making it impossible to verify the benchmark's contents without direct inspection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces InterAct VideoQA, a new benchmark dataset for video question answering in urban traffic intersection monitoring. The dataset consists of 8 hours of real-world traffic footage segmented into 10-second clips with question-answer pairs generated by GPT-o3 and subsequently human-verified. The QA pairs are organized into five reasoning categories: attribution, counting, event reasoning, reverse reasoning, and counterfactual inference. The authors evaluate three existing VideoQA models (VideoLLaMA2, LLaVA-NeXT-Video, Qwen2-VL) using standard NLG metrics (BLEU, ROUGE, METEOR, CIDEr, SPICE) and report improvements after fine-tuning on the new dataset. The paper also includes a conceptual framework with equations for spatio-temporal reasoning, multi-agent interaction, and counterfactual inference, though these are not used in the experiments.","tokens_in":16395,"tokens_out":8936,"duration_ms":82406,"significance":"InterAct VideoQA addresses an underexplored area: traffic-specific VideoQA at intersections with dense multi-event annotations. If the dataset is reliable, it could be a valuable contribution to the ITS community, especially because it uses real-world footage rather than synthetic data. The paper's strengths include the public release of the dataset, the use of standard evaluation metrics, and the explicit effort to include human verification. However, the validity of the central claims is currently undermined by unquantified human verification, inconsistent dataset statistics, and unexplained fine-tuning results (including identical before/after rows). The significance of the contribution is conditional on addressing these validation gaps.","major_comments":[{"comment":"The dataset size is reported inconsistently: the abstract states 'over 25,000 QA pairs', §3.4 states '28,800 question-answer pairs', and Table 1 lists '∼2.9k/34.6k' for InterAct VideoQA. Given 8 hours of 10-second clips, roughly 2,880 clips are expected, so 28,800 pairs implies exactly 10 per clip as in the prompt, but 34.6k pairs would require about 12 per clip. The authors should specify the exact number of clips and QA pairs, and reconcile these figures.","section":"Abstract / §3.4 / Table 1"},{"comment":"Section 4.1 claims that human evaluators 'cross-referenced every question-answer pair against its video footage' and eliminated hallucinations, but no inter-annotator agreement, number of annotators, audited sample size, or residual error rate is reported. The benchmark's ground truth and the fine-tuning gains in Tables 2–3 are only as trustworthy as this verification step, so the authors must provide quantitative validation statistics, including agreement measures and the percentage of QA pairs discarded or corrected.","section":"Section 4.1"},{"comment":"There is a direct contradiction between the description of the generation prompt and its actual text: Section 8.1 says the prompt 'prohibits revealing internal reasoning', but Figure 5 instructs the model to 'Think step-by-step, show internal reasoning; output only the final markdown table'. Additionally, the prompt's taxonomy uses 'Basic Understanding' and 'Attribution' (with counting under Attribution), while the paper elsewhere (Figure 1, §3.3, Table 2) treats 'Counting' as a separate category. This inconsistency affects dataset construction and evaluation rows, and the authors must clarify the actual category taxonomy.","section":"Section 8.1 / Figure 5"},{"comment":"The fine-tuning results are presented without error bars, confidence intervals, or significance tests, and several before/after rows in Table 2 are identical (e.g., CounterFactual for VideoLLaMA2 and LLaVA-NeXT-Video: BLEU -, ROUGE 31.25, METEOR 31.20, CIDEr -, SPICE 18.18). Identical scores across all five metrics before and after fine-tuning are implausible unless the model weights did not change, and the table caption does not explain this. Furthermore, Table 3 refers to 'Qwen2-VL-7B-Instruct' while Table 2 and §5.1 use different names ('Qwen2-VL-7B-hf' / 'Qwen2-VL-7B-Instruct'), so the model identity must be corrected. The central claim of 'notable performance improvements' requires statistical support and an explanation of the unchanged rows.","section":"Tables 2 and 3"},{"comment":"Equations (1)–(3) in Section 5.3 are introduced as part of a 'Conceptual Evaluation Approach', but the paper never reports how these quantities were computed from the video data or what results were obtained. No experimental measurements are tied to these equations, so the framework is untestable as written. The authors should either remove this section or describe the concrete procedures and results for each equation.","section":"Section 5.3"}],"minor_comments":[{"comment":"The sentence 'The models were selected based on a previous study conducted [?]' contains a missing citation placeholder; reference [42] should be properly cited here or the claim removed.","section":"Section 5.1"},{"comment":"There are typos in metric names: 'ROGUE' should be 'ROUGE', and 'BLUE score' should be 'BLEU score' in the discussion of reverse reasoning improvements.","section":"Section 5.2"},{"comment":"The dataset name is rendered inconsistently as 'InterAct VideoQA', 'InterAct-Video', and 'InterAct videoQA'; please standardize the spelling and capitalization.","section":"Throughout"},{"comment":"Some example QA pairs appear mismatched or ungrammatical, e.g., Figure 11 asks 'What is the color of the sedan in the middle?' and answers 'No, there is no red sedan in the video.' Such samples undermine confidence in the claimed human verification and should be corrected or replaced with cleaner examples.","section":"Figures 9–14"},{"comment":"The FCA score formula uses undefined variables Fdiv, Cdup, and Aexact; please define these terms and explain how the rejection threshold of 0.70 was determined.","section":"Section 8.3.2"},{"comment":"The phrase 'to hallucinate events' in the description of counterfactual questions is unclear; it should be rephrased, e.g., 'to test whether the model hallucinates events that do not occur.'","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a promising benchmark idea but currently suffers from internal inconsistencies and insufficient validation detail. The dataset size discrepancy, the contradiction in the generation prompt, the lack of human-agreement statistics, and the unexplained identical fine-tuning rows are load-bearing issues that the authors must resolve. I would look favorably on a revised version that corrects these problems and adds a proper analysis of annotation reliability. The paper's scope fits a computer vision or multimodal venue, but its current depth of validation is more appropriate for a workshop paper than a main-conference or journal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper introduces InterAct-VideoQA, a new traffic-intersection VideoQA benchmark with 8 hours of roadside footage, over 25k QA pairs, and a five-category reasoning taxonomy. That combination is genuinely new relative to existing traffic QA datasets like SUTD-TrafficQA and TUMTraffic, which are either isolated-event focused or lack intersection density. The dataset is released and the fine-tuning results show plausible gains, so the resource itself is likely to be useful to the traffic-VideoQA and ITS communities.\n\nThe paper is weaker where it counts most: the reliability of the ground-truth QA labels. Section 4.1 says human evaluators cross-referenced every pair against video and removed GPT hallucinations, but it reports no inter-annotator agreement, no annotator count, no audited sample size, and no remaining error rate. That is a load-bearing omission because the benchmark's value and the fine-tuning conclusions rest on label correctness. The stress-test note also caught a real contradiction: the prompt in Figure 5 tells the model to 'show internal reasoning' before outputting the table, while Section 8.1 claims the prompt prohibits revealing internal reasoning. If the model followed that instruction, reasoning traces or template phrasings may have survived into the final QA pairs.\n\nThere are also mechanical inconsistencies that should be fixed: the abstract says 'over 25,000' QA pairs, Section 3.4 says 28,800, and Table 1 says ~2.9k/34.6k; Tables 2 and 3 report no error bars and several rows are unchanged before/after fine-tuning; the conceptual equations in Section 5.3 are disconnected from the experiments. A missing citation placeholder appears in Section 5.1. None of these are fatal on their own, but together they make the paper feel less careful than a benchmark proposal should be.\n\nBottom line: the dataset is a plausible contribution and deserves a serious referee who can inspect the actual files and request the missing quality metrics. I would not desk-reject it, but I would require the authors to report annotation statistics, reconcile the size figures, and either remove or operationalize the conceptual equations. Who is this for? Anyone building or evaluating traffic VideoQA models, and the ITS community looking for a dense intersection benchmark. A careful reader can get value from the resource even while treating the fine-tuning claims as provisional.\n\nRecommendation: send to peer review with major revision. The work is worth refereeing, but the current evidence does not support the reliability claims as written.","headline":"A plausible new intersection-focused VideoQA benchmark whose ground-truth QA reliability is unquantified and whose internal counts disagree; worth refereeing, but only after the authors supply annotation statistics and fix the inconsistencies.","tokens_in":16951,"tokens_out":2165,"would_cite":false,"duration_ms":22796,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a traffic-specific VideoQA benchmark with 8 hours of intersection footage and over 25,000 QA pairs, and shows that fine-tuning on it improves state-of-the-art video-language models.","keywords":["video question answering","traffic monitoring","intersection dataset","spatiotemporal reasoning","multi-event traffic scenes","fine-tuning video-language models","benchmark dataset","intelligent transportation systems"],"falsifier":"Independently re-annotate a random sample of released clips with expert annotators who do not see the original labels and measure agreement; if agreement is low or many original answers are judged incorrect, the fine-tuning gains could be artifacts of noisy labels rather than real understanding.","tokens_in":15867,"feed_emoji":"🚦","tokens_out":7229,"duration_ms":72746,"temperature":0.7,"pith_summary":"This paper introduces InterAct VideoQA, a benchmark dataset for testing whether video question-answering models can make sense of real urban intersections. It compiles eight hours of roadside footage into 10-second clips and pairs them with over 25,000 questions about vehicle interactions, traffic incidents, counting, and counterfactual scenarios. The authors show that current state-of-the-art VideoQA models answer these questions poorly, and that fine-tuning them on the dataset produces measurable accuracy gains. If correct, the work establishes that traffic monitoring needs domain-specific VideoQA benchmarks rather than generic video datasets, and that such benchmarks can double as training data.","feed_headline":"A traffic video benchmark exposes VideoQA blind spots","feed_subtitle":"Eight hours of real intersection footage and 25,000+ QA pairs show where models fail and why fine-tuning helps.","key_machinery":"The central object is the InterAct VideoQA dataset itself: a collection of 10-second real-world intersection clips with dense QA annotations generated by a human-plus-LLM pipeline. The pipeline records and segments traffic footage, uses a multimodal LLM to extract per-frame metadata and draft questions, then human annotators verify, correct, or discard pairs before release. The dataset is organized around five question types adapted from earlier traffic QA work (attribution, counting, event reasoning, reverse reasoning, counterfactual inference). The evaluation machinery is standard captioning metrics (BLEU, ROUGE, METEOR, CIDEr, SPICE) measured before and after LoRA fine-tuning of three video-language models.","core_discovery":"The paper's central claim is that real-world traffic intersections are too dense and multi-event for current VideoQA models, and that a purpose-built dataset can both reveal and reduce that weakness. InterAct VideoQA consists of eight hours of roadside footage from urban intersections, cut into 10-second clips and annotated with over 25,000 QA pairs (reported as 28,800) across five question types: attribution, counting, event reasoning, reverse reasoning, and counterfactual inference. Evaluated on this benchmark, three state-of-the-art video-language models show clear weaknesses in spatiotemporal and multi-agent reasoning; after fine-tuning on the dataset, their BLEU, ROUGE, METEOR, CIDEr, and SPICE scores improve, with the largest gains in reasoning-heavy categories. The author concludes that domain-specific, densely annotated traffic data is necessary to make VideoQA useful for intelligent transportation systems.","pith_inferences":["Inference beyond the paper: the reported fine-tuning gains were measured against models trained on general video data, so an ablation comparing equal-size generic versus traffic-specific fine-tuning would be needed to prove the dataset, not just extra data, drives the improvement.","Inference beyond the paper: because the QA pairs were drafted by an LLM and then corrected, template regularities from the generator could remain in the final labels; probing with rephrased questions would test whether fine-tuned models improved at understanding or at pattern-matching the question format.","Inference beyond the paper: if independent annotation confirms label quality, the counterfactual and reverse-reasoning categories could be repurposed as training signal for causal reasoning in autonomous driving and traffic-safety systems."],"forward_implications":["Fine-tuning VideoQA models on InterAct VideoQA improves their performance on traffic scenes, with the largest gains in event reasoning and reverse reasoning.","The benchmark provides a reusable testbed for measuring spatiotemporal and multi-agent reasoning in dense intersection footage.","General-purpose VideoQA datasets do not capture overlapping, concurrent traffic events; intersection-specific data is needed to evaluate and train deployable models.","Traffic monitoring applications such as incident detection and vulnerable-road-user safety can be evaluated in structured QA form rather than through detection-only metrics.","Public release of the dataset lets other groups reproduce the evaluations and extend the benchmark with new sites and seasons."],"supporting_citations":[{"why":"Supplies the five-question taxonomy and the traffic VideoQA baseline that InterAct VideoQA extends.","marker":"[48]"},{"why":"Supplies the generative model used to draft candidate question-answer pairs before human correction.","marker":"[30]"},{"why":"VideoLLaMA2 is one of the three video-language models evaluated and fine-tuned on the dataset.","marker":"[7]"},{"why":"Qwen2-VL is one of the three video-language models evaluated and fine-tuned on the dataset.","marker":"[45]"},{"why":"LLaVA-NeXT-Video is one of the three video-language models evaluated and fine-tuned on the dataset.","marker":"[52]"},{"why":"BLEU is one of the five metrics used to score model answers before and after fine-tuning.","marker":"[31]"},{"why":"CIDEr is one of the five metrics used to score model answers before and after fine-tuning.","marker":"[41]"},{"why":"TUMTraffic-VideoQA is the closest roadside traffic VideoQA benchmark and motivates the intersection-specific focus.","marker":"[53]"}],"fun_headline_variants":["New traffic QA dataset exposes VideoQA reasoning gaps","InterAct benchmark: 25K QA pairs train VideoQA for real traffic","Why VideoQA fails at busy intersections and how fine-tuning fixes it","Traffic videos: 25,000 questions reveal VideoQA blind spots","New dataset trains video QA to handle real intersection chaos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is trustworthy: the 25,000+ question-answer labels are correct and unbiased, even though the paper reports no inter-annotator agreement or residual error rate.","fun_headline_variants_meta":{"raw":{"variants":["New traffic QA dataset exposes VideoQA reasoning gaps","InterAct benchmark: 25K QA pairs train VideoQA for real traffic","Why VideoQA fails at busy intersections and how fine-tuning fixes it","Traffic videos: 25,000 questions reveal VideoQA blind spots","New dataset trains video QA to handle real intersection chaos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2534,"prompt_tokens":960,"completion_tokens":1574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1487}},"tokens_in":576,"tokens_out":1574,"duration_ms":15287,"temperature":1.0,"reasoning_tokens":1487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:48:49.119183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample of released clips with expert annotators who do not see the original labels and measure agreement; if agreement is low or many original answers are judged incorrect, the fine-tuning gains could be artifacts of noisy labels rather than real understanding.","supporting_citations":[{"cited_title":"Sutd-trafficqa: A question answering benchmark and an efficient network for video rea- soning over traffic events","cited_arxiv_id":null,"evidence_quote":"Supplies the five-question taxonomy and the traffic VideoQA baseline that InterAct VideoQA extends."},{"cited_title":"Goucher, Adam Perelman, and Aditya Ramesh et al","cited_arxiv_id":null,"evidence_quote":"Supplies the generative model used to draft candidate question-answer pairs before human correction."},{"cited_title":"Llava- next: A strong zero-shot video understanding model, April","cited_arxiv_id":null,"evidence_quote":"LLaVA-NeXT-Video is one of the three video-language models evaluated and fine-tuned on the dataset."}],"review_version":1}