{"id":"c69c8ea4-f2c9-405d-8645-73b4bab13924","arxiv_id":"2502.02449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new roadside-traffic video benchmark unifies multiple-choice QA, referred object captioning, and spatio-temporal grounding, with a Qwen baseline showing open models still struggle on spatial reasoning.","lead":"TUMTraffic-VideoQA is a new benchmark with 1,000 roadside traffic videos, 85,000 multiple-choice video QA pairs, 2,300 referred-object captions, and 5,700 spatio-temporal grounding annotations. It unifies video question answering, object captioning, and grounding in one evaluation, and ships a Qwen-based baseline to probe how well current vision-language models reason about traffic scenes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spatial QA labels come from an unspecified pseudo-3D projection (Appendix A.2) with no quantitative validation; if the projected point or calibration is biased, the benchmark's core spatial-reasoning numbers are not measuring 3D understanding.","rationale":"The reader identified label trustworthiness as the weakest assumption, and I agree; my concern sharpens that assumption to a specific, under-specified step in the pipeline. The no-vision ablation in Table 6 is a serious, self-reported validity threat for the MQA track, but the pseudo-3D projection issue is more load-bearing because it also affects the ST-OG task's spatial references and the benchmark's unique claim of evaluating fine-grained spatial reasoning. I do not see an internal inconsistency in the paper, and the benchmark may still be useful for V-ROC and as a resource. However, the central claim of a unified spatio-temporal understanding benchmark requires the projection step to be specified and validated; without that, the reported positioning accuracies and the conclusion that models leave 'large gaps in spatial reasoning' are not firmly grounded. This does not change the reader's CONDITIONAL verdict, because the condition should exactly be the release and validation of the spatial-label pipeline and a no-vision control in the main paper.","tokens_in":19485,"tokens_out":9996,"duration_ms":110165,"concrete_test":"Take 200 randomly sampled positioning questions from the validation split, with release of the projection code, camera matrices, and raw videos. For each object pair, independently annotate the ground-contact point in the relevant frames, then recompute the Eq. 8 label two ways: (i) using the bounding-box center as implied by the tuple representation and (ii) using the bottom-center ground-contact point. Also have two human annotators classify the relative position directly from the video. If the two projection variants disagree on more than 10% of labels, or if human agreement with the released labels is below 95%, the spatial ground truth is not reliable enough to support the benchmark's claimed measurement of 3D spatial reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.2 describes how all positioning labels are produced: 2D detections are projected to pseudo-3D under a planar-ground assumption, motion directions are computed from consecutive 3D coordinates, and relative positions are binned by the angle θ (Eq. 8). The manuscript never states which image point is projected. For a roadside camera with nonzero pitch, projecting the bounding-box center instead of the bottom-center ground-contact point produces a systematic, distance-dependent error in the 3D position; that error propagates into the motion direction and into all eight relative-position bins. This is not a hypothetical inconvenience: the same pseudo-3D coordinates are the basis for the ST-OG spatial references, and the only stated quality control is a 'manual quality assurance' pass on metadata, not a quantitative agreement study on the final spatial labels. The paper's own no-vision control (Table 6) shows 70.2% positioning accuracy with no video input, so the MQA track currently cannot separate text-template priors from visual 3D reasoning. Because the benchmark's central value is measuring fine-grained spatio-temporal understanding, an under-specified and unvalidated projection is the most load-bearing assumption in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces TUMTraffic-VideoQA, a roadside traffic video-language benchmark built from 1,000 videos with 85,000 multiple-choice QA pairs, 2,300 referred-object captioning annotations, and 5,700 spatio-temporal grounding annotations. It proposes a tuple-based object expression (c, fn, x, y), three evaluation tasks (MQA, V-ROC, ST-OG), and a Qwen-2 based baseline, TUMTraffic-Qwen, with four visual token sampling strategies. Experiments report fine-tuned baseline results and zero-shot results for three open-source VLMs, along with ablations on frame count and token sampling.","tokens_in":19695,"tokens_out":6934,"duration_ms":68147,"significance":"Should the annotation pipeline be validated, the dataset would be a genuinely useful resource: it targets an under-served roadside perspective, unifies three tasks under one evaluation protocol, and comes with a public release and a reproducible baseline. The paper also contains a commendable no-vision ablation (Table 6) and an honest discussion of text-pattern exploitation. This transparency does not, however, dissolve the concern that the current MQA numbers substantially reflect language priors and that spatial ground truth rests on an unvalidated pseudo-3D projection.","major_comments":[{"comment":"The spatial labels—both the eight relative-position classes in MQA and the references used in ST-OG—are derived from camera-based pseudo-3D coordinates, but the manuscript never states which 2D image point is projected (bounding-box center, bottom-center, or other), nor does it provide calibration reprojection error or a validation of the planar-ground assumption. For roadside cameras with nonzero pitch, projecting the box center instead of the ground-contact point produces a systematic, distance-dependent error that propagates into motion direction and all relative-position bins. Please specify the exact projection procedure, report intrinsic/extrinsic calibration accuracy, and include a quantitative agreement study between the automatically derived spatial labels and human-verified spatial relations on a labeled subset.","section":"Appendix A.2, Eq. (8)"},{"comment":"The no-vision baseline reaches 70.18% (0.5B) and 73.32% (7B) positioning accuracy and 67.59% / 71.01% overall MQA accuracy with no video input. This shows that the MQA track is answerable to a large degree from the question text, option statistics, and template priors, so the current evaluation numbers do not isolate spatio-temporal visual understanding. The paper acknowledges the effect in the appendix, but the abstract and Section 5.1 still frame the benchmark primarily as a measure of spatio-temporal understanding. Please report per-type and per-difficulty gaps between no-vision and vision inputs, analyze answer/option distribution biases, and either add a suitable control protocol to the benchmark or temper the central claims so they are commensurate with the measured visual signal.","section":"Section B.1, Table 6"},{"comment":"QA pairs and captions are generated by GPT-4o-mini from templates, with only a 'selective quality evaluation' whose sample size, pass rates, and agreement statistics are not reported; the manual quality assurance step is described for metadata, not for the final QA or grounding labels. Without quantitative verification, label noise is uncontrolled and could inflate or deflate model scores in unknown ways. Please provide the number of QA pairs filtered, inter-annotator agreement on a random sample, per-question-type human error rates, and a clear description of the human verification protocol for the released annotations.","section":"Section 3.1"},{"comment":"The relative-position bins are asymmetric in angular width: 'front left' and 'front right' each cover 60 degrees, while 'left' and 'right' cover only 30 degrees (and 'front' covers 30 degrees). This asymmetry is not justified in the text and can systematically skew the label distribution of positioning questions. Either justify the convention with reference to traffic-scene semantics or make the bins symmetric, and report the resulting label distribution per bin.","section":"Appendix A.2, Eq. (8)"}],"minor_comments":[{"comment":"Table 1 reports 87.3k QAs/captions while the abstract and Section 3.3 report 85k QA pairs; clarify that the 87.3k figure includes the 2.3k captioning annotations.","section":"Table 1"},{"comment":"There is a mismatched parenthesis in Eq. (7) in the definition of p(X_a | S_v(Z_v), X_q); please fix the typesetting.","section":"Eq. (7)"},{"comment":"The phrase 'The video is uniformly divided into 100 segments, including the first and last frames, resulting in a total of N = 101 frames' is confusing; 'sampled uniformly at 101 timestamps' would be clearer.","section":"Section 4.1"},{"comment":"METEOR and SPICE are reported only for the TUMTraffic-Qwen baseline, while open-source models show '-' without explanation; state whether these metrics were computed and omitted for space, or add them.","section":"Table 5"},{"comment":"The row label 'without' for frame count should be '0 frames' for readability.","section":"Section B.1, Table 6"},{"comment":"Training hyperparameters (batch size, learning rate schedule, warmup, number of steps, data augmentation) are not given; please add them for reproducibility.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The no-vision ablation is placed in the supplementary material even though it materially affects the interpretation of the main benchmark results; during revision it should be moved or at least summarized in the main text. The paper's novelty claim hinges on the label pipeline being trustworthy, so I would ask the editor to insist on the validation experiments described in the major comments before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xingcheng and colleagues have built something genuinely useful: the first roadside-traffic video-language benchmark that unifies multiple-choice QA, referred-object captioning, and spatio-temporal grounding. The 1,000 videos with diverse weather and anomalies fill a real gap, and the tuple (c, f_n, x, y) is a natural, workable extension of DriveLM and ELM. The baseline models and token-sampling study are solid engineering, and I was pleasantly surprised to see the no-vision ablation in Table 6. That is the right experiment to run, and the authors report it honestly rather than bury it. Related work is thorough, and the self-citations are mostly to the authors' earlier TUM surveys, which is not a problem here.\n\nThe problem is what that ablation reveals. A 0.5B model with no video input gets 70.18% on positioning questions; 7B gets 73.32%. With 101 frames the gain is only 1.82-3.24 points. So the MQA positioning track is mostly measuring language priors, not visual 3D reasoning. The authors acknowledge this and call it future work, which is fair, but it undercuts the claim that the benchmark tracks fine-grained spatio-temporal understanding.\n\nThe load-bearing soft spot is the pseudo-3D projection in Appendix A.2. The paper never says which 2D point is projected. For a roadside camera with nonzero pitch, projecting the bounding-box center instead of the bottom-center ground-contact point produces a systematic, distance-dependent error in the 3D coordinates. Those coordinates are used for both the motion-direction angles and the relative-position bins in Eq. 8, so the error propagates directly into the spatial QA labels. There is no quantitative validation against known 3D positions or GPS, only a 'manual quality assurance' pass on metadata. That is not enough for the benchmark's central spatial claims.\n\nOther issues are smaller: ST-OG checks only start and end frames, which is a deliberate simplification; the QA generation uses GPT-4o-mini with selective human filtering, and no inter-annotator agreement numbers are given; and the dataset and code are not inspectable in the preprint. For a dataset paper, that last point is critical.\n\nWho should read it: anyone building roadside traffic VLMs or using video-language benchmarks for intelligent transportation. It deserves a serious referee, but the review must force a precise specification and validation of the projection, plus a leakage analysis for each question type. If those hold up, this becomes a standard resource. For now, treat the numbers as provisional.","headline":"A genuinely new roadside video-language benchmark with an honest no-vision ablation, but the spatial-QA labels rest on an unvalidated pseudo-3D projection and the QA track leaks heavily to language priors.","tokens_in":20299,"tokens_out":5647,"would_cite":false,"duration_ms":49246,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a roadside video-language benchmark that unifies multi-choice video QA, referred object captioning, and spatio-temporal object grounding, and a fine-tuned baseline that clearly beats zero-shot open models.","keywords":["video question answering","spatio-temporal grounding","referred object captioning","traffic scene understanding","roadside surveillance","vision-language models","benchmark dataset","intelligent transportation systems"],"falsifier":"Take a random sample of validation videos, have independent human annotators label the relative spatial directions and the first and last visible frames of referred objects, and compare their labels with the benchmark's ground truth; if agreement is low, the reported accuracy and grounding errors do not measure true spatio-temporal understanding.","tokens_in":19255,"feed_emoji":"🚦","tokens_out":10223,"duration_ms":89055,"temperature":0.7,"pith_summary":"This paper is trying to establish that a single benchmark can measure how well vision-language models understand roadside traffic videos in space and time. It introduces a dataset of 1,000 videos with 85,000 multiple-choice QA pairs, 2,300 referred-object captions, and 5,700 spatio-temporal grounding annotations, using a tuple object expression $(c, f_n, x, y)$ as a common interface. The paper also trains a unified baseline and reports that it reaches roughly 82% mean accuracy on multi-choice QA, far above the best zero-shot open model at about 56%, while positioning questions remain the hardest for every model. As a shared resource, it is intended to give intelligent transportation research a common testbed for fine-grained video reasoning under realistic roadside conditions.","feed_headline":"New benchmark unifies QA, captioning, and grounding in traffic video","feed_subtitle":"1,000 roadside videos and 85,000 question-answer pairs give traffic AI a common test for fine-grained spatial reasoning.","key_machinery":"The load-bearing mechanism is the four-element tuple $(c, f_n, x, y)$ for referring to any object at any moment in a video: it gives one standardized language interface that the three benchmark tasks all consume. The benchmark's spatial QA labels are produced by projecting 2D detections to pseudo-3D positions under a planar ground assumption and binning relative directions into eight angular regions, so that positional relationships can be phrased as natural-language questions.","core_discovery":"The central claim is that this is the first roadside, third-person-perspective video-language benchmark to unify three tasks in one evaluation framework: multiple-choice question answering, referred object captioning, and spatio-temporal object grounding. Object references are written as tuples $(c, f_n, x, y)$, where $c$ is a persistent object identifier, $f_n$ is a normalized frame timestamp, and $(x,y)$ are normalized image coordinates, so the same object can be pointed to across the whole video and grounding is expressed as a start/end pair of tuples. The paper reports that its fine-tuned 7B baseline reaches about 81-82% mean accuracy on multi-choice QA versus at most 56% for zero-shot open-source vision-language models, cuts spatio-temporal error from roughly 0.8 down to about 0.24, and substantially improves captioning metrics, while positioning questions focusing on 3D spatial relationships remain the most difficult for all models tested. The dataset covers accidents, rescue operations, congestion, and adverse weather, and is released publicly.","pith_inferences":["If the planar-ground pseudo-3D labels generalize, the same eight-angle binning pipeline could auto-generate spatial QA for other roadside camera networks without manual 3D annotation.","The tuple interface could become a common interchange format for referring to objects across video datasets in surveillance and embodied settings, not only traffic.","The no-visual-input ablation suggests part of the QA score may come from language priors; a counterfactual benchmark with balanced answer distributions would reveal how much of the apparent understanding is genuinely visual.","A testable extension is to replace the discrete 45-degree angular bins with continuous angles to see whether the remaining positioning errors come from bin granularity or from depth ambiguity in the pseudo-3D projection."],"forward_implications":["A task-specific fine-tune on this benchmark currently gives a large boost over zero-shot open models, so a unified roadside video model needs in-domain training, not just a strong generic backbone.","Positioning questions are the hardest dimension for every model, so progress on the benchmark will depend on improving genuine 3D spatial reasoning rather than object recognition.","Multi-resolution visual token sampling improves multi-choice QA and captioning but slightly hurts spatio-temporal grounding, revealing a trade-off between frame-level detail and cross-frame consistency.","Increasing the number of input frames helps all tasks, with most of the gain coming by 11 frames and little additional gain at 101 frames, so efficient video token representation is a key bottleneck.","Because grounding is evaluated as start/end tuples with normalized coordinates, the benchmark directly measures cross-frame object association with a single numeric error, making it usable for tracking and grounding research."],"supporting_citations":[{"why":"Supplies the multi-choice QA design with easy/hard single-hop vs multi-hop splits that the benchmark adapts.","marker":"[18]"},{"why":"Introduces the structured tuple representation <c, CAM, x, y> that the paper extends to a spatio-temporal tuple.","marker":"[20]"},{"why":"Provides the base architecture and pre-aligned projector weights that the baseline model is built from.","marker":"[9]"},{"why":"Serves as one of the zero-shot open-source vision-language models the benchmark compares against.","marker":"[1]"},{"why":"Serves as another zero-shot open-source video-LLM baseline, including frame-count ablations.","marker":"[3]"},{"why":"Provides the language-prompt referred multi-object tracking formulation that the grounding task contrasts with.","marker":"[26]"},{"why":"Is the LLM backbone used in the fine-tuned baseline model.","marker":"[29]"},{"why":"Is the visual encoder used to produce frame-level visual tokens for the baseline.","marker":"[30]"}],"fun_headline_variants":["Traffic video benchmark unifies QA, captioning, and grounding","1,000 videos, 85k QAs: new benchmark for traffic scene AI","First roadside video benchmark to fuse QA, captioning, and grounding","85k QA pairs tie traffic video tasks into one benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark scores are trustworthy only if the automatically produced labels are accurate: spatial answers come from projecting 2D detections to pseudo-3D positions under a planar ground assumption and binning directions into eight angles, while most questions and captions are written by a large language model with only selective human checks.","fun_headline_variants_meta":{"raw":{"variants":["Traffic video benchmark unifies QA, captioning, and grounding","1,000 videos, 85k QAs: new benchmark for traffic scene AI","First roadside video benchmark to fuse QA, captioning, and grounding","85k QA pairs tie traffic video tasks into one benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001324,"raw_usage":{"total_tokens":5393,"prompt_tokens":953,"completion_tokens":4440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":4363}},"tokens_in":569,"tokens_out":4440,"duration_ms":24858,"temperature":1.0,"reasoning_tokens":4363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:05:28.144465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of validation videos, have independent human annotators label the relative spatial directions and the first and last visible frames of referred objects, and compare their labels with the benchmark's ground truth; if agreement is low, the reported accuracy and grounding errors do not measure true spatio-temporal understanding.","supporting_citations":[{"cited_title":"Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-choice QA design with easy/hard single-hop vs multi-hop splits that the benchmark adapts."},{"cited_title":"Drivelm: Driving with graph visual question answering","cited_arxiv_id":null,"evidence_quote":"Introduces the structured tuple representation <c, CAM, x, y> that the paper extends to a spatio-temporal tuple."},{"cited_title":"Llava-onevision: Easy visual task transfer, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the base architecture and pre-aligned projector weights that the baseline model is built from."},{"cited_title":"Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms, 2024","cited_arxiv_id":null,"evidence_quote":"Serves as another zero-shot open-source video-LLM baseline, including frame-count ablations."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Is the visual encoder used to produce frame-level visual tokens for the baseline."}],"review_version":1}