{"id":"3e4fb9b7-18d3-4466-b0e2-6bfb452b1f07","arxiv_id":"2512.05277","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"TAD, a 5,861-question benchmark, shows VLMs score far below humans on temporal understanding of driving videos, and an ego-trajectory text summary (TCogMap) substantially boosts their scores.","lead":"A new benchmark (TAD) measures whether vision-language models can understand time in driving videos, using 5,861 questions on 150 NuScenes clips. Current models average about 52% accuracy versus 75% for humans; adding an ego-motion text map (TCogMap) lifts open models by up to 18 points, but part of that gain comes from privileged trajectory information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TCogMap's headline gain may be an artifact of hand-set thresholds and oracle-like access to ground-truth ego poses; a held-out tuning test is needed before the 17.72% claim is accepted.","rationale":"The reader's weakest assumption—that TCogMap's empirically set thresholds lack a held-out tuning protocol—is real and important. My read extends it: the thresholds are not just arbitrary, they convert ground-truth ego poses into the same action vocabulary used to build TAD, making TCogMap an oracle-like label injector for ego-centric tasks. This is a more specific mechanism for why the headline gain may be optimistic. However, the benchmark itself (5,861 QAs, human verification, multi-model evaluation) remains a legitimate contribution, and the large VLM-vs-human gap is robust. The reader's CONDITIONAL verdict already requires addressing the overfitting concern, so my read does not move the verdict; it sharpens the condition. The concrete split-half tuning experiment would settle whether the TCogMap gain survives honest tuning. I also note the STSBench claim in the abstract is absent from the full text, an additional missing-support issue already flagged by the reader; it should be corrected or removed regardless of the threshold experiment.","tokens_in":25113,"tokens_out":6570,"duration_ms":74606,"concrete_test":"Split the 150 NuScenes videos into two disjoint sets by scene/location (e.g., 75/75). Use one set to tune the six Algorithm 1 thresholds via a small grid search, freeze the selected thresholds, and recompute all TCogMap rows from Table 3 on the held-out set, reporting ego-only and non-ego-only gains separately. If the held-out gain over the baseline + ego pose variant falls below a pre-specified margin (e.g., 5pp) or is not consistently positive across the six open-source models, then the reported 17.72% improvement is threshold overfitting rather than a robust method gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—TCogMap improves TAD accuracy by up to 17.72%—is not yet established as a genuine gain in VLM temporal reasoning. In Sec. 3.1, TAD's action labels were created by giving annotators BEV visualizations that include ego trajectories. In Sec. 4.2 (Algorithm 1), TCogMap consumes exactly those NuScenes ego poses and converts them, using hand-set thresholds from Sec. 5.1, into the same eight action categories used as TAD ground truth. For ego questions (53.7% of all questions, plus all of Action Duration and Relative Temporal Action Localization), the injected 'motion summary' is therefore close to a ground-truth action string. The baseline + raw ego pose control does not remove this problem, because the raw pose is not parsed into action labels. No held-out tuning protocol is reported for the six thresholds, so the up-to-17.72pp improvement in Table 3 may reflect threshold selection on the test set plus oracle-like label access rather than improved VLM temporal understanding. Scene-CoT is less affected, but the largest headline gain comes from TCogMap. Separately, the abstract claims up to 10.35% improvement on STSBench, but no STSBench experiment or result appears in the full text; that claim is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TAD, a video QA benchmark for temporal understanding in autonomous driving, built on 150 NuScenes validation videos with 5,861 QA pairs across seven segment- and scene-level tasks and 4,481 vehicle-action annotations. The authors evaluate 9 closed- and open-source generalist and AD-specialist VLMs, reporting a large gap between model performance (~52% average) and human performance (74.72%). They also propose two training-free inference-time methods: Scene-CoT, a chain-of-thought video partitioning and LLM-QA pipeline, and TCogMap, which injects an ego-vehicle motion summary computed from NuScenes ego poses into the VLM prompt. The abstract claims average accuracy improvements on TAD of up to 17.72% and up to 10.35% on STSBench.","tokens_in":25510,"tokens_out":4814,"duration_ms":47846,"significance":"The TAD benchmark is a potentially valuable community resource: it is the first AD video benchmark combining segment-level action recognition and scene-level temporal tasks, it was constructed with human verification, and it includes a blind text-only control indicating that questions are not answerable from language alone. The release of data and evaluation code is a concrete strength. If the method-evaluation concerns below are addressed, the benchmark could support meaningful progress in VLM temporal reasoning for driving. The proposed methods, however, are currently presented as the paper's headline contribution, and the evidence for TCogMap in particular is confounded by oracle-like access to ground-truth ego trajectories.","major_comments":[{"comment":"The central claim that TCogMap improves temporal reasoning by up to 17.72% is not established by the current protocol. Algorithm 1 converts NuScenes ego poses into the same eight action categories used to construct TAD, using thresholds reported in §5.1 as 'empirically set' with no tuning protocol or held-out split. For the 53.7% of questions about the ego, and in particular the ego-only Action Duration (124 questions) and Relative Temporal Action Localization (92 questions) tasks, the motion summary supplied to the VLM is effectively a ground-truth action string. The 'baseline + textual ego pose' control in §5.1 does not remove the confound, because raw pose is not parsed into the benchmark's action vocabulary. A proper control would tune the thresholds on a held-out NuScenes split, report non-ego performance separately, and/or include a condition in which the motion summary comes from","section":"§4.2, §5.1, Table 3"},{"comment":"The abstract states that the methods improve average accuracy by 'up to 10.35% on STSBench,' but no STSBench experiment, configuration, or numerical result appears anywhere in the main text or the supplementary material. STSBench is only cited in Table 1 and Related Work. This claim is unsupported as written and must either be removed or substantiated with an actual experimental comparison.","section":"Abstract; full text"},{"comment":"Scene-CoT is presented as a general improvement, but for the larger open-source models it decreases average accuracy: Qwen2.5-VL-32B drops from 55.20 to 48.78, InternVL3-14B from 52.35 to 51.81, and InternVL3-38B from 56.49 to 53.95. The text in §5.3 acknowledges that 'Scene-CoT does not appear to offer accuracy gains' for larger models, but the abstract and introduction present both proposed methods as improving average accuracy. The claims should be qualified to state that Scene-CoT helps only smaller models, while TCogMap provides more consistent gains, subject to the confound in the first major comment.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The text reports a '17.74%' improvement for Qwen2.5-VL-7B, but Table 3 shows 62.18 - 44.46 = 17.72. Please make the numbers consistent.","section":"§5.3"},{"comment":"The blind VLM test is performed only with InternVL3-8B. The conclusion that 'the benchmark questions cannot be answered without visual input' should be qualified as demonstrated for one model; running the blind condition for at least one additional model, particularly a closed-source one, would strengthen the claim.","section":"Figure 4"},{"comment":"The text states that Scene-CoT requires '42 seconds per question' for segment-description generation, while Table 10 lists an average inference time of 47.10 seconds for Scene-CoT. Clarify whether the 42 s figure excludes the LLM QA time and reconcile the numbers.","section":"§11.3, Table 10"},{"comment":"The empirical choices of the five-second segment length, 50 m distance cutoff, and 50% overlap are described only briefly, and the LLM selection for Scene-CoT is relegated to the supplement. Since these choices affect benchmark difficulty and method performance, a sentence in the main text summarizing the selection procedure would improve reproducibility.","section":"§3.1, §11.4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a solid contribution and the data/code release is commendable. However, the TCogMap results as presented should not be published as evidence of improved temporal understanding without a non-oracle control; if the authors cannot provide such control, the method claims should be substantially weakened. The missing STSBench results and the overstated Scene-CoT claims also need attention. I would not reject the paper, but the current revision requires additional experiments before the central quantitative claims can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is the honest part of this paper. TAD is the first AD temporal QA set that mixes segment- and scene-level questions, with human-verified labels and a blind test showing the questions aren't answerable from text alone. The human-model gap (74.7 vs ~52) is a useful target for the field. The 4,481 action annotations are a standalone asset. I'd cite the benchmark and expect others to use it.\n\nThe methods are where I part ways. TCogMap's motion summary is built from ground-truth ego poses and hand-set thresholds that produce exactly the same action categories used as ground truth. For the 53.7% of questions about the ego, the injected text is effectively the answer. The baseline + raw ego pose control does not fix this: raw poses are not parsed into action labels, so the VLM can't exploit them the way TCogMap does. The blind test actually confirms the problem — TCogMap-only, without frames, beats image-only on several tasks. That's not temporal reasoning; that's label leakage. The thresholds in Algorithm 1 are described as \"empirically set\" with no held-out tuning protocol, so the up-to-17.72% claim in the abstract should not be taken at face value.\n\nScene-CoT is less problematic but also less impressive. It helps small models, hurts larger ones, and the authors admit this. The abstract also claims up to 10.35% improvement on STSBench, but no STSBench experiment appears anywhere in the paper. That claim should be removed or substantiated. There are also no error bars anywhere, which matters for a 5.8k-question benchmark where several tasks have only 80–100 questions.\n\nNone of this kills the benchmark. The benchmark construction, annotation quality, and evaluation of nine models are solid. But the paper currently overstates the methods, and the central quantitative claim about TCogMap is not yet credible. A serious referee should ask for a held-out tuning split, a control that converts poses to labels without hand-tuned thresholds, and removal or verification of the STSBench result. I'd send it to review, but I'd expect major revision before acceptance.","headline":"The TAD benchmark is a real contribution, but the headline TCogMap gain is likely inflated by oracle-like access to ego-action labels, and the STSBench claim is unsupported.","tokens_in":25973,"tokens_out":1155,"would_cite":true,"duration_ms":14236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temporal understanding in driving videos improves sharply when a vision-language model is handed a text summary of the ego vehicle's motion, and a new benchmark measures the gap to humans.","keywords":["TAD benchmark","temporal understanding","autonomous driving","vision-language models","temporal cognitive map","chain-of-thought reasoning","NuScenes","vehicle action recognition"],"falsifier":"Re-run the TAD experiments with TCogMap after splitting the 150 videos into a tuning subset and a held-out test subset, and tune the six thresholds on the tuning subset only; if the average accuracy gain over baseline drops substantially (e.g., from 17.72% to near zero) or becomes inconsistent across models, the threshold setting is overfit to the benchmark.","tokens_in":25075,"feed_emoji":"🚗","tokens_out":1727,"duration_ms":19142,"temperature":0.7,"pith_summary":"This paper argues that vision-language models (VLMs) fail at temporal reasoning in autonomous driving because they lack explicit access to ego-vehicle motion, and that this deficit is measurable and fixable. It introduces TAD, a benchmark of 5,861 question-answer pairs over 150 driving videos, spanning both short segment-level action recognition and longer scene-level temporal tasks. On TAD, state-of-the-art VLMs average roughly 52% accuracy while humans reach about 74.7%. The paper proposes two training-free interventions—Scene-CoT, a chain-of-thought reasoning wrapper, and TCogMap, a module that converts the ego vehicle's trajectory into a per-segment textual motion summary—and shows that TCogMap lifts average accuracy on TAD by up to 17.72 percentage points across several VLMs.","feed_headline":"Ego-motion text summaries lift driving-VLM accuracy by up to 18%","feed_subtitle":"A new driving benchmark shows VLMs lag humans by 22 points; a trajectory-derived text map narrows the gap without retraining.","key_machinery":"TCogMap's ego-vehicle motion classifier (Algorithm 1) is the load-bearing component. It consumes a sequence of ego poses (translation, quaternion, timestamp), computes finite-difference velocities, transforms them into the ego's local frame, and applies a hierarchical rule set with fixed thresholds to label each five-second segment as Stopped, Starting, Stopping, Straight, Turn left/right, or Change lane left/right. The resulting per-segment labels are formatted as strings like 'Motion summary for Frame1 to Frame8: The ego-vehicle is moving forward at a relatively constant speed' and concatenated into the VLM prompt. This converts high-frequency kinematic data into a compact, temporally inde","core_discovery":"The central claim is that temporal understanding in autonomous driving is a distinct, under-measured capability, and that a large part of the gap between VLMs and humans comes from missing explicit temporal structure rather than from missing visual information. The paper demonstrates this by building TAD, the first driving benchmark that mixes segment-level and scene-level temporal questions, and by showing that simply appending a textual temporal cognitive map of ego motion—derived from the vehicle's pose sequence, not from vision—improves accuracy substantially on both ego and non-ego questions. This works even when the VLM receives no additional frames beyond the baseline set, and it outp","pith_inferences":["A natural extension is to build similar temporal cognitive maps for surrounding vehicles—the paper mentions this as future work—and one would predict further gains on non-ego tasks if the map carries reliable object-level motion labels.","The threshold values in Algorithm 1 could likely be tuned per scene type or learned from data; if gains persist after proper held-out tuning, the method's practical value increases, but if not, the reported margins may shrink.","The benchmark's reliance on NuScenes' 20-second clips means it does not test very long-horizon temporal reasoning; a version with longer drives or event ordering across minutes might reveal different failure modes.","Scene-CoT's high inference cost (about 47 seconds per question) suggests that its benefits may be superseded by more efficient prompting or by TCogMap-style structured context in deployed systems."],"forward_implications":["If TCogMap's gains hold, any existing driving VLM can improve its temporal reasoning by plugging in a trajectory-analysis module—no fine-tuning required, just a text prompt addition.","The TAD benchmark provides a reusable testbed for isolating temporal understanding from static scene understanding in driving, allowing future methods to be compared on a common ground.","Ego-motion summaries appear to supply contextual cues that also help non-ego object reasoning, suggesting that self-motion is a globally useful prior for scene dynamics.","The finding that raw ego pose text does not help as much as the classified summary implies that VLMs benefit from symbolic abstraction of kinematics, not just more data."],"fun_headline_variants":["New benchmark exposes driving-VLM temporal blind spot; text map fixes it","VLMs fail driving temporal tests; ego-motion text summary boosts accuracy 18%","Temporal blind spot in driving VLMs closed by trajectory-derived text map"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The accuracy gains from TCogMap depend on empirically chosen thresholds in the motion classifier (e.g., 0.2 m/s for stopped, 10 degrees for turn, plus a 1.5x multiplier) that were not validated on a separate tuning set, so the reported improvements may be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes driving-VLM temporal blind spot; text map fixes it","VLMs fail driving temporal tests; ego-motion text summary boosts accuracy 18%","Temporal blind spot in driving VLMs closed by trajectory-derived text map"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2652,"prompt_tokens":888,"completion_tokens":1764,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1699}},"tokens_in":632,"tokens_out":1764,"duration_ms":11771,"temperature":1.0,"reasoning_tokens":1699,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:24:09.934571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the TAD experiments with TCogMap after splitting the 150 videos into a tuning subset and a held-out test subset, and tune the six thresholds on the tuning subset only; if the average accuracy gain over baseline drops substantially (e.g., from 17.72% to near zero) or becomes inconsistent across models, the threshold setting is overfit to the benchmark.","supporting_citations":[],"review_version":1}