{"id":"9d0a960c-5d09-4696-ba84-9760e6ea6027","arxiv_id":"2507.00525","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Box-QAymo introduces a box-referring VQA benchmark for autonomous driving, with hierarchical binary, attribute, and motion reasoning questions built from Waymo data and crowd-sourced labels.","lead":"Box-QAymo is a new driving dataset where users ask questions about a specific object by drawing a red box around it. It tests whether vision-language models can identify, describe, and track the motion of that object across scenes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Motion QA answers are derived from 3D tracks with linear interpolation, but the benchmark's interpretation as visual/temporal reasoning requires those answers to be inferable from the supplied frames; if not, the reported motion deficits partly measure privileged-metadata access.","rationale":"The paper is a well-structured dataset contribution with crowdsourced semantic labeling, negative sampling, temporal consistency checks, and a clear evaluation protocol. The central claim of being the first box-referred driving VQA benchmark is plausible and the reported numbers are informative. The reader's weakest assumption names the right load-bearing point: Section 3.2.3 and Section 3.3 generate motion answers from Waymo 3D tracks with linear interpolation, but the benchmark interpretation as visual/temporal reasoning requires those answers to be visually inferable from the one or two frames presented to the model. This is not an internal inconsistency, so the paper is not fatally flawed; it is a benchmark-validity concern that a human visual baseline can settle. If the concern lands, the correct fix is to filter or re-annotate motion questions and/or report visual-answerability statistics, not to discard the dataset. The conditional verdict therefore stands, and the reader's concern is the same one I would raise.","tokens_in":11431,"tokens_out":4615,"duration_ms":58553,"concrete_test":"Sample 100 motion QA pairs stratified across the six motion subcategories, and give human annotators exactly the model input (single frame or two frames with the red box, but no 3D tracks or text about interpolation) plus the same MCQ options, with at least three judgments per item. If human agreement with the dataset's ground truth on Speed Assessment, Relative Motion Direction, Trajectory Analysis, and Path Conflict Detection is at or near chance, or markedly below human performance on binary sanity items, then those labels are not visually grounded, and the benchmark should be re-run after excluding or re-annotating such items to see whether the hierarchy and the two-frame degradation survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evaluation claim is that current VLMs show systematic failures in localized, intent-driven perception, especially temporal motion reasoning (motion F1 37.6%). That inference presupposes that ground-truth answers to Speed Assessment, Relative Motion Direction, Trajectory Analysis, and Path Conflict Detection are visually determinable from the exact inputs the model sees. Section 3.2.3 says implicit questions use a single frame and infer motion from visual cues, while explicit questions provide frame sequences. Section 3.3 then says trajectory-based questions are constructed by 'linearly interpolating object positions to handle missing object information in Waymo' and by requiring movement patterns over multiple consecutive frames. These are LiDAR/3D track operations, not image-derived facts. A 100-ms two-frame gap can make displacement sub-pixel for distant objects, and single-frame speed/relative-motion questions are often impossible from a static RGB image unless visible cues such as motion blur, wheel angle, or road markings are present and resolvable. Moreover, interpolation can place an object at a position where it is occluded or outside the camera frustum, so the label is not recoverable from pixels at all. If many labels cannot be recovered from the provided images, then low VLM F1 on motion tasks is expected even for a perfect visual reasoner, and the benchmark partially tests access to privileged metadata rather than visual reasoning. This concern does not invalidate the dataset as a resource, but it directly affects the paper's headline interpretation and the two-frame-degradation result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Box-QAymo is a box-referring VQA dataset and benchmark built from Waymo Open Dataset scenes. The authors crowd-source fine-grained object classes and visual attributes, generate a hierarchy of QA pairs (binary sanity checks, attribute prediction, and motion/spatio-temporal reasoning), and evaluate LLaVA-1.5, Qwen-VL, and Senna in zero-shot and LoRA-finetuned settings. The main reported results are average F1 of 66.1% on binary, 18.3% on attribute, and 37.6% on motion questions, with finetuning improving most tasks and two-frame inputs degrading performance.","tokens_in":11632,"tokens_out":4671,"duration_ms":52148,"significance":"The contribution is timely and potentially useful: a public dataset that grounds driving VQA in user-drawn boxes would fill a real gap, and the hierarchical protocol plus difficulty balancing are sensible design choices. The paper's strengths include the crowdsourced semantic expansion beyond Waymo's coarse labels, explicit negative sampling, temporal-consistency checks, and the observation that current VLMs often fail on localized, intent-driven queries. However, the benchmark's central interpretation—that motion scores measure visual spatio-temporal reasoning—depends on the assumption that trajectory-derived answers are inferable from the RGB frames shown to the model; this assumption is not currently verified.","major_comments":[{"comment":"The motion question answers are derived from Waymo 3D tracks, including linear interpolation of object positions and smooth velocity calculations using temporal derivatives. The paper interprets low motion F1 (e.g., 37.6% average, Table 4) as a deficit in visual spatio-temporal reasoning, but this interpretation requires that the ground-truth answers be recoverable from the supplied camera frames. For single-frame implicit questions about speed or relative motion, distant objects with sub-pixel displacement over 100 ms may not provide any visual evidence; for interpolated trajectories, objects may be occluded or outside the camera frustum at the interpolated position. The authors should provide a human-visual oracle or a per-question answerability audit demonstrating that the labels can be inferred from the exact inputs given to the models, and should either remove or re-flag questions that fail this audit. Without this, the reported motion deficits could partly measure privileged-metadata access rather than visual reasoning.","section":"3.2.3 and 3.3"},{"comment":"The crowdsourced semantic labeling procedure is described at a high level, but no inter-annotator agreement statistics are reported. Since the attribute questions (Table 3) are evaluated against these crowd labels, annotation noise is conflated with model error. The authors should report per-task agreement metrics such as Cohen's kappa or Fleiss' kappa, the number of annotators per item, and how disagreements were adjudicated.","section":"3.1"},{"comment":"All F1, precision, and recall numbers are reported as point estimates with no confidence intervals or significance tests. Given the relatively small validation split (101 scenes) and uneven question counts across subcategories, the performance gaps that drive the paper's conclusions—e.g., binary vs. attribute vs. motion, and finetuning gains such as LLaVA† Trajectory Analysis 98.90 vs. Qwen-VL 17.03—should be accompanied by bootstrap confidence intervals or a significance test. The authors should also report the exact number of questions per subcategory in Tables 2-4.","section":"4.2, Tables 2-4"}],"minor_comments":[{"comment":"The evaluated model is listed as 'Qwen-7B' but the citation [4] is the Qwen2.5-VL technical report; please clarify the exact model name and version used in the experiments.","section":"4.1"},{"comment":"Senna is reported with 0.00 F1 on all attribute tasks, but Section 4.3 notes that Senna produces only 34% valid responses overall. Please clarify whether these zeros are entirely due to invalid response parsing or also reflect incorrect answers, since this affects how the attribute failures should be interpreted.","section":"Table 3"},{"comment":"The negative sampling description is difficult to follow: 'We find samples that do not fit the criteria for the ground truth answer and create additional samples from these criteria.' Please rephrase to specify how negative examples are constructed and how their labels are verified.","section":"3.3"},{"comment":"The paper states that approximately 50% of objects are labeled and that objects with poor visibility are skipped, but it does not explain how this 50% was selected or how 'poor visibility' was operationalized. This matters for potential selection bias in the attribute and motion splits.","section":"3.1"},{"comment":"The 'Improvement (%)' column reports negative values for decreases (e.g., -5.42 for LLaVA two-frame), which is clearer if labeled 'Change (%)' or 'Difference (%)'. Also, the 100-ms interval between frames may be too short to produce visible motion in many driving scenes; a brief discussion of this limitation would help interpret the two-frame results.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the motion portion of the benchmark may partly measure access to privileged 3D-track metadata rather than visual reasoning. The revised version should include a human-visual oracle or explicit answerability analysis; otherwise the headline 'motion reasoning gap' claim will not be fully supported. For a dataset paper, the authors should also consider adding annotation-cost and release-documentation details, which are not currently present."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Box-QAymo is a real contribution: the first driving VQA benchmark I know of that uses user-drawn bounding boxes to ground questions in specific objects, with a sensible four-tier protocol from binary sanity checks to attribute prediction to motion reasoning. The crowd-sourced fine-grained labels (following Argoverse taxonomy) and color annotations add genuine value over Waymo's coarse classes, and the quality-control steps—negative sampling, temporal consistency checks, difficulty balancing—show care. The evaluation is honest: three VLMs, zero-shot and LoRA-finetuned, with per-category F1, and the findings (two-frame inputs hurt motion reasoning; Senna collapses on structured Q&A) are interesting and likely useful for the community.\n\nThe soft spots are real but manageable. The stress-test concern about motion labels is the one I'd push on. Section 3.2.3 says implicit questions use a single frame to infer motion from visual cues, while Section 3.3 describes linearly interpolating object positions from Waymo 3D tracks to construct trajectory questions. Those interpolated positions can place an object where it is occluded or outside the camera frustum, and a 100 ms frame gap can make displacement sub-pixel for distant objects. So some motion answers may not be visually recoverable from the exact inputs the model sees. That means the low motion F1 (37.6%) and the two-frame degradation result are partially confounded with privileged-metadata access, not pure visual/temporal reasoning. The authors should validate a sample of motion labels by showing they are pixel-inferable (e.g., through human visual-only annotation or by checking occlusion/frustum status at interpolated positions). This doesn't sink the dataset as a resource—the box-referring and attribute parts are on firmer ground—but it changes how strongly I'd cite the motion numbers.\n\nOther soft spots are minor: no error bars or significance tests, no inter-annotator agreement, and no head-to-head comparison with existing driving VQA benchmarks like LingoQA or DriveLM on a common subset. I'd like to see those before calling Box-QAymo a standard evaluation protocol.\n\nBottom line: a solid dataset paper that deserves serious refereeing. I'd send it to review with a request for the motion-label validation study. If the authors can show the motion answers are visually grounded, this becomes a very useful benchmark. If not, the attribute and box-referring portions still stand, but the motion claims need to be scoped down.","headline":"Useful box-referring driving VQA benchmark, but motion labels partly rely on 3D-track interpolation and need visual-grounding validation before the motion scores are taken at face value.","tokens_in":12217,"tokens_out":2521,"would_cite":true,"duration_ms":27923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Box-QAymo grounds driving questions in user-drawn boxes and shows current VLMs score 66.1% F1 on binary checks, 18.3% on attributes, and 37.6% on motion.","keywords":["visual question answering","autonomous driving","box-referring","spatiotemporal reasoning","benchmark","vision-language models","object grounding","dataset"],"falsifier":"Give a held-out set of motion questions to a VLM while removing or blurring the target object's trajectory cues in the frames; if accuracy barely drops, the questions are answerable without temporal visual information and the benchmark is not measuring what it claims.","tokens_in":11197,"feed_emoji":"🚗","tokens_out":5817,"duration_ms":57307,"temperature":0.7,"pith_summary":"Box-QAymo is a box-referring visual question answering dataset for autonomous driving, built on Waymo scenes. Its central claim is that grounding a question in a user-drawn bounding box, the way a passenger points at something, is a natural and necessary format for testing whether vision-language models actually perceive what a specific person is asking about. The paper argues that existing driving VQA benchmarks only test full-scene descriptions or planning, and therefore miss failures in localized, intent-driven perception. On this benchmark, three VLMs score 66.1% F1 on binary sanity checks, 18.3% on attributes, and 37.6% on motion questions, and even a driving-specific model often fails to produce usable answers. If the benchmark is right, progress on human-vehicle communication is being overestimated by scene-level evaluations.","feed_headline":"User-drawn boxes test driving AI: 66% on basics, 18% on attributes","feed_subtitle":"Grounding a question in a user-drawn box exposes how much self-driving AI models miss about specific objects.","key_machinery":"The central mechanism is the box-referring question format: a red bounding box overlaid on a camera frame, paired with a question about the object inside that box (e.g., 'What color is the object highlighted in red?'). This turns a user's pointing gesture into a visual prompt that the VLM must attend to, letting the benchmark test whether a model can separate a referred instance from scene-level statistics. Around this, the dataset builds a three-level protocol, binary sanity checks, attribute prediction, and motion reasoning, with motion answers derived from Waymo 3D tracks, negative sampling for balance, and temporal consistency checks.","core_discovery":"The paper's central claim is that Box-QAymo is the first driving VQA benchmark that answers questions about a specific object the user has highlighted with a bounding box, and that this format exposes a hierarchy of capabilities in current VLMs. The benchmark moves from binary sanity questions (movement status, orientation) to attribute prediction (fine-grained class, color, facing direction) to motion reasoning (speed, movement direction, trajectory, path conflict, relative motion). Across LLaVA-1.5, Qwen-VL, and the driving-specific Senna, mean F1 falls from 66.1% on binary, to 18.3% on attributes, to 37.6% on motion; the driving-specialized model outputs parseable answers only 34% of the time and scores zero on all attribute tasks. Finetuning with LoRA lifts performance, but providing two consecutive frames instead of one consistently hurts motion reasoning, suggesting current VLMs do not truly integrate short-term temporal information.","pith_inferences":["If the motion answers are not visually inferable from the images alone, part of the reported motion gap may measure access to privileged 3D-track information rather than visual reasoning; a human-perception study on the same frames would settle this.","The red-box grounding result hints that overlaying a user-specified region could improve deployed systems at little cost, but the effect may not generalize to models with weaker visual encoders.","The two-frame degradation points to a training-data mismatch, suggesting that pretraining VLMs on short frame pairs rather than single images or long videos could directly address the temporal reasoning gap.","A natural extension is to ask the same box-referring questions with the target object occluded or removed, turning the benchmark into a test of whether models can flag uncertainty when they cannot see the referred object."],"forward_implications":["Current general-purpose VLMs are unreliable for localized, user-driven perception in driving, and finetuning on the dataset partially closes the gap.","Providing two frames instead of one does not help current VLMs understand motion, so the bottleneck appears to be temporal integration rather than input length.","A driving-specialized model trained mainly for planning can score zero on structured attribute questions, meaning narrow domain training does not transfer to perception communication.","Box-referring questions are a workable interface: Qwen-VL consistently improves when the red box is present, though the gain is architecture-dependent.","Hierarchical evaluation separates basic scene understanding from fine-grained and temporal reasoning, allowing future benchmarks to measure progress at each level."],"supporting_citations":[{"why":"Supplies the base scenes and 3D object tracks from which the Box-QAymo dataset is constructed.","marker":"[25]"},{"why":"Provides the fine-grained semantic taxonomy that crowd-sourced labelers follow for object classes.","marker":"[29]"},{"why":"Shows that simple visual prompts like red circles can guide VLM attention, the design basis for red bounding box grounding.","marker":"[23]"},{"why":"One of the general-purpose VLMs evaluated and the base model for LoRA finetuning experiments.","marker":"[15]"},{"why":"The strongest zero-shot model in the evaluation, used to measure box-grounding effectiveness.","marker":"[4]"},{"why":"The domain-specific driving VLM whose format failure and zero attribute scores motivate the benchmark's design.","marker":"[12]"},{"why":"The low-rank adaptation method used to finetune models on the dataset.","marker":"[11]"}],"fun_headline_variants":["Driving AI stumbles on user-pointed objects: 18% attribute accuracy","Box-drawn questions reveal VLM blind spots in driving scenes","New dataset: VLMs get 66% on basic, 18% on detailed driving queries","User box queries expose AI's weak object understanding in driving","Two frames hurt motion reasoning in self-driving VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Motion answers are derived from 3D tracks, so the benchmark's validity rests on the assumption that the speed and trajectory questions can be answered from the supplied images alone, rather than requiring hidden track data.","fun_headline_variants_meta":{"raw":{"variants":["Driving AI stumbles on user-pointed objects: 18% attribute accuracy","Box-drawn questions reveal VLM blind spots in driving scenes","New dataset: VLMs get 66% on basic, 18% on detailed driving queries","User box queries expose AI's weak object understanding in driving","Two frames hurt motion reasoning in self-driving VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1530,"prompt_tokens":1041,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":657,"tokens_out":489,"duration_ms":5312,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:12:23.403635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a held-out set of motion questions to a VLM while removing or blurring the target object's trajectory cues in the frames; if accuracy barely drops, the questions are answerable without temporal visual information and the benchmark is not measuring what it claims.","supporting_citations":[{"cited_title":"Scal- ability in perception for autonomous driving: Waymo open dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the base scenes and 3D object tracks from which the Box-QAymo dataset is constructed."},{"cited_title":"What does clip know about a red circle? visual prompt engineer- ing for vlms","cited_arxiv_id":null,"evidence_quote":"Shows that simple visual prompts like red circles can guide VLM attention, the design basis for red bounding box grounding."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"One of the general-purpose VLMs evaluated and the base model for LoRA finetuning experiments."}],"review_version":1}