{"id":"39774fa9-eb3e-4b8e-8629-e201e0ce616c","arxiv_id":"2506.06218","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.","lead":"This paper introduces STSBench, a tool that automatically turns annotated driving data into multiple-choice tests of a model's ability to understand traffic situations over time and across camera views. It uses the tool to build STSnu from the NuScenes dataset and finds that current vision-language driving models perform poorly, especially on interactions between other vehicles and pedestrians.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that VLMs lack spatio-temporal reasoning is not settled: Table 2 compares VLMs on raw images to LLMs on ground-truth trajectories, so the gap could stem from perception, prompt matching, or model scale rather than from reasoning.","rationale":"I examined the paper's central claim and the reader's verdict. The reader identified the comparison between LLMs with ground-truth trajectories and VLMs with raw images as the weakest assumption. I agree: the conclusion that VLMs lack spatio-temporal reasoning requires that the performance gap be attributed to reasoning rather than to perception, prompt design, or scale. This is the single most load-bearing concern, since the benchmark's utility as a reasoning test depends on this attribution. The paper's own Table 13 is an incomplete control because it only covers expert VLMs and yields mixed results. The proposed experiment cleanly separates perception from reasoning by feeding VLMs the same objective data given to LLMs, and it controls for scale with a small LLM. If the experiment shows that VLMs can reason well from trajectories, the original conclusion would be an overreach; if not, the conclusion stands. Accordingly, I recommend no change to the reader's CONDITIONAL verdict, which appropriately awaits this control.","tokens_in":1174,"tokens_out":837,"duration_ms":91743,"concrete_test":"Run a three-condition evaluation on STSnu for InternVL 2.5 8B (best off-the-shelf VLM) and DriveMM (best expert VLM): (1) image/video input only, as in the paper; (2) the same images plus the exact ground-truth trajectory data that was provided to LLMs in the LLM prompts; and (3) trajectory text only, with a fixed blank image in place of visual input, to engage the language backbone without vision. If accuracy in conditions (2) and (3) approaches GPT-4o's 57% level, the Table 2 gap is largely perceptual or prompt-driven, and the 'no spatio-temporal understanding' conclusion must be weakened. If accuracy stays near the image-only result, the reasoning-deficit claim is supported. Include a scale-matched small LLM (e.g., Llama 3.2 3B) with the same trajectory prompts to control for model scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion—that driving VLMs 'do not have a spatio-temporal understanding of dynamic traffic scenes'—rests on the comparison in Table 2. In this comparison, LLMs receive exact GPS positions and agent trajectories while VLMs receive only raw images with prompts that are adapted per model (Appendix E). This setup conflates three factors with spatio-temporal reasoning: (1) perception quality, because VLMs must infer object states from pixels; (2) prompt compatibility, because each model gets a bespoke prompt that may not match its training distribution; and (3) model scale, because the best LLMs (GPT-4o, DeepSeek V3) are far larger than the 1B–8B VLMs. Without a condition that feeds a VLM the same ground-truth trajectory information given to LLMs, the observed accuracy gap cannot be uniquely attributed to a deficit in reasoning. The paper's Appendix D, Table 13 does provide ground-truth information to three expert VLMs, but the results are mixed (DriveMM improves from 39.5 to 48.5, OmniDrive stays flat, Senna collapses), and the experiment does not cover off-the-shelf VLMs or the exact LLM prompt format. Thus the strongest claim in the paper is underdetermined by the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STSBench, a framework for automatically mining defined traffic scenarios (ego, agent, ego-to-agent, and agent-to-agent) from datasets with rich ground-truth annotations, together with a lightweight human-verification interface and automatic multiple-choice question generation. Applied to the NuScenes validation split, the framework yields STSnu, comprising 43 scenario types and 971 human-verified multiple-choice questions. The authors evaluate three LLMs, three off-the-shelf VLMs, and three driving-expert VLMs on STSnu, and report that LLMs given ground-truth trajectories substantially outperform the visual models. They interpret this as evidence that current driving VLMs lack spatio-temporal understanding of dynamic traffic scenes, particularly for third-party agent interactions.","tokens_in":46343,"tokens_out":4709,"duration_ms":44533,"significance":"The framework addresses a genuine evaluation gap: most existing driving-VLM benchmarks focus on single-image or monocular ego-centric tasks, whereas STSnu targets multi-view video, third-party interactions, and holistic spatio-temporal reasoning. The release of code and data, the reporting of inter-reviewer agreement (85.6% positive agreement and 20.8% disagreement on negatives), the analysis of multiple-choice letter distribution, and the ablations of query frame and chain-of-thought for OmniDrive are concrete strengths that make the benchmark resource potentially useful to the community. If the benchmark is adopted, the per-scenario results in Tables 8-11 will be a valuable reference. However, the paper's headline conclusion goes beyond what the current experimental design can establish, because the central comparison between LLMs and VLMs is confounded by differences in the information given to each model class, in prompt formatting, and in model scale.","major_comments":[{"comment":"The claim that VLMs 'do not have a spatio-temporal understanding of dynamic traffic scenes' is not established by the reported comparison, because the LLM baseline receives ground-truth trajectories (GPS positions, LiDAR coordinates, velocities) while the VLMs receive raw images with per-model adapted prompts; the accuracy gap can therefore be explained by perception quality, prompt compatibility, or model scale rather than by reasoning ability alone. The authors' own control experiment in Table 13, which feeds ground-truth information to three expert VLMs, yields mixed results (DriveMM improves from 39.5 to 48.5, OmniDrive stays flat at 28.4, Senna collapses to 3.2), and it does not cover the off-the-shelf VLMs or the exact LLM prompt format, so it does not resolve the confound. I recommend adding an oracle-perception or text-input condition for the same VLMs (e.g., providing the same trajectory and GPS text used for the LLMs) and/or softening the conclusion to the supported claim that current driving VLMs perform poorly on these questions under their native input formats.","section":"Section 4, Table 2, and Appendix D, Table 13"},{"comment":"No confidence intervals or significance tests are reported, despite small per-scenario sample sizes (e.g., Table 8 shows scenario groups with 10-37 questions) and differences between model accuracies that are often small relative to the implied sampling noise (e.g., Table 2: InternVL 2.5 8B at 46.07% vs. DriveMM at 39.51% vs. OmniDrive at 29.33%). The phrase 'outperforms ... by a significant margin' in Section 4 requires a formal test, such as McNemar's test or bootstrap confidence intervals; without this, the model ranking and the 'critical shortcomings' narrative are not quantitatively supported.","section":"Section 4 and Tables 8-11"},{"comment":"The evaluation protocol adapts prompts and input formats per model 'in order to get better performance' (Appendix A.4), which makes cross-model accuracy differences at least partly attributable to how well each prompt matches the model's training distribution rather than to spatio-temporal reasoning. I recommend reporting results with a shared, minimally-adapted prompt as the primary protocol, with the per-model optimized prompts as a secondary analysis, and documenting the variance induced by prompt changes, for example by running each model with two or three variants on a subset of the benchmark.","section":"Section 4 and Appendix A.4/E"}],"minor_comments":[{"comment":"The caption of Table 9 says 'for ego-to-agent scenarios', but the table lists single-agent scenarios such as jaywalking, walking, standing, crossing, left turn, and overtaking ego; the caption should read 'agent scenarios' to match the main text and the scenario categories.","section":"Table 9 caption"},{"comment":"There are several typos in the figures and appendix prompts, including 'U-Tuen' in Figure 2, 'caputred' in Figures 27-34, 'assistent' and 'specilized' in Figures 35-38, and 'whic is a pedestrian' in Figure 42; these should be corrected for a polished final version.","section":"Figures and Appendix E"},{"comment":"Reference [15] (HiLM-D) uses the same arXiv identifier 2308.12966 as reference [4] (Qwen-VL), which appears to be an incorrect duplicate; please verify and replace it with the correct identifier for the HiLM-D technical report.","section":"References"},{"comment":"The verification protocol accepts positive samples by majority voting but keeps only negatives with full agreement across all reviewers; this asymmetry in quality thresholds is not analyzed, and a brief discussion of its potential effect on benchmark difficulty and on the false-negative rate would improve transparency.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction is not circular: scenarios are mined from NuScenes ground truth and human verification, independent of the evaluated models, and no parameters are fitted to produce the findings. The main risk is the strength of the central claim: the LLM-vs-VLM comparison in Table 2 is confounded, and the control experiment in Table 13 does not fully address this. I would advise the editors that the paper is likely acceptable after the authors add a controlled condition (or correspondingly weaken the claim) and add basic statistical support for the cross-model comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about STSBench. First, the artifact is genuinely useful: an automatic pipeline that mines traffic scenarios from ground-truth annotations, verifies them with a lightweight human-in-the-loop tool, and emits multiple-choice questions. STSnu itself—43 scenarios, 971 human-verified questions across four categories, with agent-to-agent interactions as a new axis—is the kind of shared resource this subfield needs. Code and data are public. Second, the paper's central empirical claim, that driving VLMs 'do not have spatio-temporal understanding,' is not settled by the evidence in Table 2. The comparison gives LLMs ground-truth trajectories while VLMs get raw images with per-model prompt adaptations, so the gap could reflect perception, prompt mismatch, or model scale rather than reasoning.\n\nWhere the paper earns credit: the scenario mining is described carefully, with heuristic thresholds and subsampling criteria; verification by three licensed drivers with reported agreement is more rigorous than most benchmark papers; and the agent-to-agent category is a real gap in prior benchmarks. The multiple-choice letter distribution analysis and the option-count scaling study are thoughtful touches.\n\nSoft spots, in proportion. The evaluation section is the weak part. There are no confidence intervals or significance tests; the reported numbers are point estimates on a 971-question set, and many fine-grained cells are tiny. The LLM baseline is a perfect-perception upper bound by design, so the observed margin is expected and uninformative about the reasoning gap. The paper does try to address this in Appendix D, where expert models get ground-truth information; DriveMM improves substantially while OmniDrive stays flat and Senna collapses. That is useful, but it is a partial check on three expert models only, not a controlled comparison for the off-the-shelf VLMs or the central claim. The abstract and conclusion are more confident than the experiments warrant.\n\nWho is this for: researchers building or evaluating VLMs for driving. The benchmark is worth adopting as a shared measure, and the framework's design is a good template for other datasets. The paper deserves a serious referee; the evaluation claims need tightening and the reporting needs error bars, but the contribution is substantial. I would send it to review.","headline":"A well-built benchmark that fills a real gap, but the headline claim that driving VLMs lack spatio-temporal reasoning outruns the confounded evaluation.","tokens_in":46864,"tokens_out":2302,"would_cite":true,"duration_ms":23868,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Driving vision-language models fail at spatio-temporal scene understanding, a new 971-question benchmark argues.","keywords":["autonomous driving","vision-language models","spatio-temporal reasoning","scenario benchmark","multiple-choice question answering","nuScenes","multi-view video","end-to-end driving models"],"falsifier":"Give the best driving-expert VLM perfect perception (for example, replace its visual input with ground-truth object tracks and rendered bounding boxes while keeping the language model frozen) and re-run STSnu; if its agent-to-agent accuracy jumps to the level of the trajectory-fed LLM, the paper's conclusion that the model lacks spatio-temporal reasoning would collapse, because the failure would trace to the vision-to-language interface rather than to reasoning.","tokens_in":45902,"feed_emoji":"🚗","tokens_out":7640,"duration_ms":71868,"temperature":0.7,"pith_summary":"STSBench is a framework that turns annotated driving datasets into multiple-choice benchmarks about traffic scenarios: it mines pre-defined events such as lane changes, overtakes, pedestrian crossings, and agent-to-agent interactions from ground-truth annotations, verifies them with a lightweight human interface, and generates questions. Applied to NuScenes, it yields STSnu, 971 human-verified questions across 43 scenarios in four categories: ego maneuvers, agent actions, ego-to-agent interactions, and agent-to-agent interactions. The paper's central claim is that current vision-language models, especially driving-expert VLMs trained for end-to-end driving, do not have reliable spatio-temporal understanding of dynamic traffic scenes. The evidence is that LLMs given ground-truth trajectories answer the same questions far more accurately than VLMs given raw images or multi-view video, and the largest failures occur where holistic spatio-temporal integration is required. If the paper is right, open-loop planning metrics on standard driving datasets substantially overstate what these models understand about the world.","feed_headline":"AI driving models fail to grasp dynamic traffic scenes, benchmark shows","feed_subtitle":"A 971-question test finds driving VLMs struggle most with agent-to-agent interactions and ego maneuvers.","key_machinery":"The load-bearing mechanism is the STSBench scenario catalog: 43 textual scenario definitions (e.g., lane change, overtake, wait for pedestrian to cross), each paired with negative scenarios that do not occur, so that every question has distractors requiring discrimination of close alternatives. Mining heuristics use ground-truth 3D bounding boxes, tracks, class labels, ego-motion, and HD maps to detect these patterns automatically; a lightweight human verification interface lets drivers confirm positives and reject false negatives; verified samples are then turned into multiple-choice questions of the form 'which of the following best describes ...' with one correct answer among at least four choices. The evaluation protocol compares three model families under adapted prompts: LLMs receive ground-truth trajectories, off-the-shelf VLMs receive single-view image sequences with camera metadata, and driving-expert VLMs receive full multi-view video, making the comparison hinge on how each model integrates spatial and temporal evidence.","core_discovery":"The paper's central claim is that spatio-temporal reasoning is the capability currently missing in driving vision-language models. Using STSnu, the paper shows that when an LLM is given perfect trajectories it can identify ego maneuvers and interactions at 57.08% average accuracy (GPT-4o), while the best driving-expert VLM (DriveMM) reaches only 39.51%; agent-to-agent interactions, where neither participant is the ego vehicle, are hardest for all models. The paper interprets this gap as evidence that visual models have not learned to jointly reason over spatially distributed multi-view inputs and temporally extended dynamics. It therefore argues that end-to-end driving models need architectural mechanisms that explicitly model spatio-temporal relationships, not just better perception heads or larger training sets.","pith_inferences":["A plausible extension is to use STSnu as a training signal: fine-tune a driving VLM with a trajectory-infusion module and re-measure; the paper's data predict a large gain if the gap is mainly perceptual rather than reasoning-based.","The multiple-choice format may reward elimination strategies rather than true identification; a follow-up variant that asks the model to justify its choice or to detect two simultaneous scenarios in one scene would test whether the apparent reasoning is robust.","Because NuScenes is recorded in Boston and Singapore and contains mostly lawful behavior, applying STSBench to more diverse or adversarial recordings could reveal even larger failures than the 57% ceiling suggests."],"forward_implications":["Any claim that an end-to-end driving VLM understands a scene should be backed by interaction-level questions like those in STSnu; waypoint or ego-action accuracy alone is insufficient.","Training data and objectives for driving VLMs should explicitly include third-party interactions, since agent-to-agent scenarios are where all evaluated models drop hardest.","Injecting perception-derived trajectories or 3D object states into the language model may be a more direct route to spatio-temporal reasoning than asking the vision encoder to infer them from raw pixels.","STSBench can be re-instantiated on other datasets with ground-truth annotations, producing comparable interaction benchmarks across different sensor setups without per-dataset manual annotation.","Benchmark distractors should remain semantically close (e.g., overtake versus pass) because the results show models frequently confuse such distinctions."],"supporting_citations":[{"why":"Supplies the NuScenes validation split with the ground-truth tracks, ego-motion, and HD maps from which all STSnu scenarios are mined.","marker":"[6]"},{"why":"Senna-VLM is one of the evaluated driving-expert VLMs; its low ego-scenario accuracy anchors the claim that experts lack spatio-temporal understanding.","marker":"[29]"},{"why":"OmniDrive is an evaluated expert model whose BEV-based pipeline performs poorly on ego and agent-to-agent scenarios, supporting the representation-bottleneck reading.","marker":"[60]"},{"why":"DriveMM is the best-performing driving-expert model; its partial success on ego and ego-to-agent scenarios defines the expert upper bound in the comparison.","marker":"[22]"},{"why":"GPT-4o is the top LLM baseline given ground-truth trajectories; its margin over VLMs is the main evidence for the spatio-temporal gap.","marker":"[46]"},{"why":"InternVL 2.5 1B and 8B are off-the-shelf VLM baselines whose relatively strong agent-to-agent results complicate the expert-model comparison.","marker":"[9]"},{"why":"Qwen2.5-VL 7B is an off-the-shelf VLM baseline used in the multi-model comparison.","marker":"[5]"},{"why":"NuInstruct is an existing multi-view video benchmark without human verification, used to motivate STSBench's verified multi-view design.","marker":"[16]"}],"fun_headline_variants":["Driving AI fails at spatio-temporal reasoning, benchmark reveals","Agent-to-agent traffic interactions trip up driving VLMs, test shows","New benchmark exposes driving AI's blind spot: dynamic scenes","Benchmark: driving VLMs struggle with multi-agent traffic dynamics","Driving AI needs explicit spatio-temporal modeling, 971-question test finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the accuracy gap between LLMs fed ground-truth trajectories and VLMs fed raw images is caused by differences in spatio-temporal reasoning ability rather than by perception quality, prompt formatting, or model scale.","fun_headline_variants_meta":{"raw":{"variants":["Driving AI fails at spatio-temporal reasoning, benchmark reveals","Agent-to-agent traffic interactions trip up driving VLMs, test shows","New benchmark exposes driving AI's blind spot: dynamic scenes","Benchmark: driving VLMs struggle with multi-agent traffic dynamics","Driving AI needs explicit spatio-temporal modeling, 971-question test finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1427,"prompt_tokens":955,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":571,"tokens_out":472,"duration_ms":4504,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:57:56.928068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the best driving-expert VLM perfect perception (for example, replace its visual input with ground-truth object tracks and rendered bounding boxes while keeping the language model frozen) and re-run STSnu; if its agent-to-agent accuracy jumps to the level of the trajectory-fed LLM, the paper's conclusion that the model lacks spatio-temporal reasoning would collapse, because the failure would trace to the vision-to-language interface rather than to reasoning.","supporting_citations":[{"cited_title":"Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom","cited_arxiv_id":null,"evidence_quote":"Supplies the NuScenes validation split with the ground-truth tracks, ego-motion, and HD maps from which all STSnu scenarios are mined."},{"cited_title":"Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models","cited_arxiv_id":null,"evidence_quote":"NuInstruct is an existing multi-view video benchmark without human verification, used to motivate STSBench's verified multi-view design."}],"review_version":1}