{"id":"9d0e4fa2-0cc8-41ab-a46c-d0c79ee97204","arxiv_id":"2412.06324","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An instruction-guided token selection and cross-attention module improves MLLM performance on autonomous driving QA and planning benchmarks, trained with a new GPT-generated object-level risk assessment dataset.","lead":"This paper proposes a multi-modal large language model for autonomous driving that selects instruction-relevant visual tokens before feeding them to the LLM, along with a GPT-generated risk assessment dataset. The method reports improved accuracy on driving QA benchmarks and lower collision rates in open-loop planning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ORA scores may not measure perception-limited reasoning: GPT-4o labels were generated from ground-truth object lists that include occluded objects, while at inference the model sees only images.","rationale":"Reader's verdict is CONDITIONAL, and I agree with the reader's weakest-assumption identification. The ORA benchmark is the only evidence aimed directly at reasoning in perception-limited regions; if its labels are produced with privileged ground-truth object information and evaluated without that information, the claimed 'world knowledge-enhanced reasoning' is not established. Other experiments (NuScenes-MQA, OmniDrive-NuScenes, NuInstruct, planning) show general capability gains, but they do not isolate the occluded-scene world-knowledge claim, and Table 4 in particular compares only OPT-1.3B/6.7B rather than current MLLM SOTA. The proposed visibility-split test is feasible with nuScenes annotations and would settle whether ORA scores contain signal beyond visual evidence. Given the architecture improvements are plausible and reproducible in principle, a conditional verdict remains appropriate; the benchmark-validity condition should be added to acceptance criteria. Hence no change to the reader's CONDITIONAL verdict.","tokens_in":16211,"tokens_out":7627,"duration_ms":79751,"concrete_test":"Use nuScenes 3D boxes and camera frusta to raycast each risk target's box into the six input views; label each ORA test question as 'visible' if the object has any pixel evidence in any view, otherwise 'fully occluded.' Report model accuracy on the two splits against the majority-class baseline. If fully-occluded accuracy is at or near baseline while visible accuracy is high, the benchmark is not measuring perception-limited reasoning, and the central claim fails; if fully-occluded accuracy is significantly above baseline, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the ORA benchmark measures 'world-knowledge-enhanced reasoning' in perception-limited scenes. The construction pipeline (Section 'Object-level Risks Assessment (ORA)', prompts in Tables 14 and 15) provides GPT-4o with the full nuScenes object list, including category, direction, and distance, and asks it to enumerate risks for 'the current scene contains the following objects...'. The resulting QA pairs are then used as both training data and test labels. At inference, however, Eq. (1) and Figure 2 show the model receives only multi-view video and language instructions; the privileged object list is never provided. Thus a model can score well on ORA by imitating GPT-4o's answer distribution and wording, or by exploiting common risk priors (e.g., 'close car in front = collision possibility'), without performing any inference from the image about occluded regions. The benchmark therefore cannot distinguish a system that truly integrates world knowledge with perception from one that has learned to parrot the teacher's output style. Because ORA is the only dedicated evaluation of the paper's central claim, this is a construct-validity problem, not a minor metric issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-modal large language model framework for autonomous driving that centers on an instruction-guided interactor module. The interactor selects the top-k visual tokens most relevant to the language instruction and then applies cross-attention between those selected tokens and the full visual feature set, thereby pre-fusing multi-view images and BEV features before they enter the LLM. The authors also present a three-stage training pipeline (single-view alignment, multi-view alignment, task-specific instruction tuning) and collect a large fine-tuning corpus of roughly 2M QA pairs and 1.7M grounding samples. To evaluate reasoning under perception-limited conditions, they introduce an object-level risk assessment (ORA) dataset of 200K QA pairs generated with GPT-4o and GPT-4o-mini. The reported experiments show improvements over OPT-based baselines on NuScenes-MQA, over OmniDrive on OmniDrive-NuScenes, and strong results on NuInstruct and open-loop planning, including a new state-of-the-art collision rate.","tokens_in":16474,"tokens_out":6655,"duration_ms":64812,"significance":"If the results hold, the instruction-guided interactor is a simple, plug-and-play component that could make multi-view and high-resolution driving inputs tractable for MLLMs without sacrificing reasoning quality, which would be broadly useful. The scale of the collected dataset and the public-benchmark improvements provide a useful engineering contribution. However, the central claim that the method integrates world knowledge to reason under perception-limited conditions rests almost entirely on the ORA benchmark, whose construction includes privileged object information not available at inference. The public-benchmark comparisons also do not yet establish the claimed state of the art because the baseline sets are narrow and backbone choices are not controlled. The paper includes no error bars, and the ORA labels are not human-validated, so the central empirical claims need additional support.","major_comments":[{"comment":"The ORA benchmark, which is the only dedicated evaluation of the paper's central claim of world-knowledge-enhanced reasoning in perception-limited regions, is constructed with privileged object information that the evaluated model never receives at inference. The generation prompt in Table 14 gives GPT-4o the full nuScenes object list, including objects that may be occluded in the camera views, together with their categories, directions, and distances, and asks for per-object risk enumeration; Table 15 then converts these into QA pairs used as both training and test labels. At test time the model receives only multi-view video and an instruction (Eq. 1, Figure 2), never the object list. Consequently, a model could score well by imitating GPT-4o's answer distribution and by exploiting generic risk priors (e.g., a close car ahead implies collision possibility) without performing any image-based inference about occluded areas. The paper provides no evidence that ORA actually measures occlusion-driven reasoning: there is no human validation of GPT-generated labels, no control condition that ablatesthe object-list input, and no analysis of whether the test questions depend on objects that are invisible in the provided images. Because the ORA results (Table 3) underpin the paper's key contribution, this construct-validity issue is load-bearing and must be addressed, for instance with a human-annotated subset, a comparison of models with and without the privileged object list, and an analysis of how performance varies with the degree of occlusion.","section":"Object-level Risks Assessment (ORA), Tables 14-15"},{"comment":"The claims of \"10.6% improvement over previous SoTA\" on NuScenes-MQA and \"51.4% improvement in CIDEr\" on OmniDrive-NuScenes are not supported by the baseline choices. Table 4 compares only against OPT-1.3B and OPT-6.7B, which are small, dated language models and are not representative of current driving MLLMs; none of the recent SoTA driving models (e.g., OmniDrive variants, BEV-InMLLM, DriveLM, DriveMLLM) are included. Table 5 compares only against OmniDrive, and the comparison does not control for backbone differences: the paper uses LLaMA3-8B and EVA-02, while OmniDrive uses different settings. The reported gains could be largely attributable to the choice of base LLM and visual encoder rather than to the proposed interactor. The authors should either add strong, code-available baselines with matched backbones and training data or substantially qualify the SoTA statements.","section":"Tables 4 and 5"},{"comment":"No error bars, confidence intervals, or statistical significance tests are reported anywhere in the paper, even though several headline improvements are small. For example, Table 8 shows ACC improving from 74.1 to 74.4 on NuScenes-MQA and CIDEr from 101.2 to 103.9 on OmniDrive-NuScenes, and Table 3 contains differences under 2 points across many metrics. LLM fine-tuning is stochastic, and these differences may fall within run-to-run variability. The authors should report the mean and standard deviation over at least three independent training runs for the main tables, or otherwise demonstrate that the observed differences are statistically reliable.","section":"Tables 3, 8, 10"}],"minor_comments":[{"comment":"There are several typographical errors: \"Matrics\" should be \"Metrics\", \"Comparsions\" should be \"Comparisons\", \"open-loos\" should be \"open-loop\", \"commends\" should be \"commands\", and in Figure 2c \"ego cat\" should likely be \"ego car\".","section":"General"},{"comment":"The similarity operator ℕ in Eq. 2 is not defined at its first use; although Section \"Implementation\" states that cosine similarity is used, this should be stated in the method section. The top-k operator K is also not formally defined; please provide precise definitions for both.","section":"Eq. 2 and Implementation"},{"comment":"The baseline \"Bunny-Llama3\" is not described anywhere in the text. Please provide its configuration, training setup, or a citation so that readers can understand what it represents.","section":"Table 3"},{"comment":"The abstract states the model uses \"2 million natural language QA pairs, 1.7 million grounding task data\", but Table 1 reports 1.7M grounding and 100K caption pre-training samples, while Table 2 lists about 2.3M fine-tuning pairs. The relationship between these numbers should be clarified and made consistent.","section":"Abstract and Section 'Dataset Construction'"},{"comment":"The Discussion section honestly notes that closed-loop evaluation and 3D grounding are not performed, yet the Conclusion states that \"Extensive experiments across tasks such as VQA, open-loop driving, and detection demonstrate the effectiveness\". Given that open-loop L2 error is comparable but not superior to baselines, the conclusion should be more measured.","section":"Discussion and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a useful modular architecture and a large dataset collection effort, but the central scientific claim about world-knowledge-enhanced perception-limited reasoning currently rests on an unvalidated synthetic benchmark. The public-benchmark comparisons are too narrow to support the SoTA statements. I would encourage the authors to strengthen the empirical case with the controls described in the major comments; with those additions, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid engineering paper with a modest but useful module, and a central claim that outruns its evidence. The instruction-guided interactor—top-k selection by similarity to the instruction, then cross-attention with the original features—is a clean, plausible extension of Q-Former-style token reduction to multi-view driving inputs. The ablations show consistent gains on NuScenes-MQA, OmniDrive, and NuInstruct, and the planning results (collision rate 0.18 avg vs 0.30 for the best comparable) are interesting. I also give them credit for retraining SparseBEV without future frames or validation data and for the sky-masking redundancy experiment; that is the kind of honest checking that makes me trust the public-benchmark numbers.\n\nThe soft spot is the ORA dataset, and it is load-bearing. The stress-test note is right: GPT-4o was given the full nuScenes object list, including occluded objects, and asked to enumerate risks. The model at inference only sees images. So a high ORA score can come from imitating GPT-4o's answer style and exploiting common risk priors, not from making inferences about occluded regions. The paper's key phrase 'perception-limited reasoning' is exactly what this benchmark does not isolate. That is not a minor metric issue—it is a construct-validity problem for the paper's central thesis.\n\nSome smaller issues: Table 4 compares only against OPT baselines, not current driving MLLMs, so the 10.6% SoTA claim is overstated. No error bars anywhere. No code or dataset release, which makes the ORA evaluation impossible to scrutinize. The writing has typos and some sections are hard to follow (e.g., the arrow in Table 8 suggests mAP should have an up arrow, and Table 10 has odd metric shifts).\n\nWho should read this: anyone working on token-efficient vision-language models for driving, or on the methodology of GPT-generated driving benchmarks. I would bring it to a reading group as a case study in benchmark validity, but I would not cite the ORA benchmark in my own work.\n\nRecommendation: send it to peer review. The module deserves scrutiny, and the authors should be pushed to either validate ORA against human judgments or a visible-object-only baseline, release the data and code, and soften the 'world knowledge' claim until the benchmark actually supports it. As is, it is a conditional accept at best.","headline":"Useful token-reduction module for driving MLLMs, but the ORA benchmark does not support the world-knowledge claim as designed.","tokens_in":16987,"tokens_out":2658,"would_cite":false,"duration_ms":26030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that selecting only the instruction-relevant visual tokens before they reach the language model lets an autonomous-driving system reason about occluded hazards, and reports state-of-the-art results on several benchmarks.","keywords":["autonomous driving","multi-modal large language models","world knowledge","occluded object reasoning","instruction-guided interactor","visual token selection","object-level risk assessment","open-loop planning"],"falsifier":"Have human annotators verify a held-out subset of ORA where the risky object is fully occluded in every camera view, and compare the model's accuracy against a no-image text baseline on that subset; if the model does not beat the baseline, the claim that it reasons about perception-limited regions is not supported.","tokens_in":16039,"feed_emoji":"🚗","tokens_out":10901,"duration_ms":93703,"temperature":0.7,"pith_summary":"Autonomous-driving language models answer questions well when everything is visible, but they struggle when a bus or a building hides a pedestrian, because the long, redundant streams of tokens from six cameras crowd out the world knowledge already stored in the language model. The paper claims that this can be fixed at the interface: an instruction-guided interactor scores every visual and bird's-eye-view token against the user's question, keeps only the most relevant $k$ tokens, and fuses them back with the original features so detail survives compression. Around this module the authors build a three-stage training pipeline, a refined set of roughly 2 million QA pairs and 1.7 million grounding samples, and a new 200K-pair object-level risk assessment benchmark generated from nuScenes with GPT models. They report state-of-the-art or near-state-of-the-art results on six-view QA, captioning, grounding, VQA, risk assessment, and open-loop planning, including a 10.6% accuracy gain on NuScenes-MQA, a 51.4% CIDEr gain on OmniDrive-NuScenes, and the best collision rate in planning. The pith, stated sympathetically, is that instruction-relevance is the right pressure to put on multi-view video: compressing it that way lets a frozen LLM actually use its world knowledge for driving.","feed_headline":"Instruction-chosen tokens improve driving AI's occluded-risk reasoning","feed_subtitle":"It shrinks six camera views to the tokens a query needs, improving QA and collision rate.","key_machinery":"Instruction-guided interactor: a plug-and-play module that selects and pre-fuses multimodal tokens before they enter the LLM. It uses the instruction token sequence as a query to score all multi-view image tokens and BEV tokens by cosine similarity, keeps only the top $k$ (image $k=90$, BEV $k=300$), and runs two-layer cross-attention between those selected tokens and the full original features, producing a compact instruction-relevant representation for the language model. The three-stage training pipeline (single-view visual-language alignment, multi-view alignment with frozen BEV encoder, then task-specific LoRA fine-tuning) and the curated datasets supply the alignment and supervision; the interactor is the load-bearing mechanism that makes multi-view input tractable and, on the paper's account, lets world knowledge enter the reasoning.","core_discovery":"The paper's central claim is that the bottleneck in perception-limited driving is not the visual encoder but the token sequence handed to the LLM: when six camera views and BEV features are concatenated directly, the input is too long and too undifferentiated for the model's world knowledge to engage, especially for occluded hazards. Its proposed fix is the instruction-guided interactor, a plug-and-play module that computes cosine similarity between instruction tokens and every visual token, selects the top $k$ most similar tokens ($k=90$ for images, $k=300$ for BEV), and then lets those selected tokens attend to the full feature map through cross-attention so that dropped details are recoverable. The authors argue that this 'pre-fusion' produces highly aggregated visual representations that fit within LLM sequence limits while remaining responsive to the query, and they support the claim with a three-stage training strategy and a large multi-view dataset. On their reported experiments, the complete model improves over prior driving MLLMs on NuScenes-MQA, OmniDrive-NuScenes, NuInstruct, NuScenes-QA, the new ORA benchmark, and open-loop planning, where it reaches the state of the art in collision rate.","pith_inferences":["The interactor's recipe (score tokens by instruction relevance, keep top-$k$, re-attend to the full feature set) is modality-agnostic, so the same module could consume lidar, radar, or map tokens; the paper only demonstrates camera and BEV inputs, leaving that extension as a testable prediction.","Because ORA labels are generated by GPT-4o from ground-truth object lists that include occluded objects, part of the reported risk-assessment performance may reflect imitation of GPT's annotation style rather than visual inference; a human-labeled subset would separate the two effects.","If instruction-relevance is the correct compression criterion, the same idea could serve as a queryable memory system for real-time driving, materializing only the scene tokens a current decision needs instead of processing all frames.","The collision-rate gains without ego status raise the possibility that token selection acts as a structural regularizer on the planner, forcing it to rely on scene layout; the paper does not test this hypothesis directly."],"forward_implications":["With the interactor, a frozen 8-billion-parameter LLM can take six-view video plus BEV as input without exceeding sequence limits, so multi-view reasoning becomes a token-selection problem rather than a scaling problem.","The reported gains on NuScenes-MQA (+10.6% ACC), OmniDrive-NuScenes (+51.4% CIDEr), and NuInstruct (+98.44% BLUE4) indicate the module transfers across QA, captioning, grounding, and reasoning tasks, not just one benchmark.","The ORA dataset, with its four object-level risk categories (view obstruction, collision possibility, traffic-rule violation, potential risk) and six question types, gives the driving-MLLM community a way to train and measure perception-limited reasoning separately from normal perception.","In open-loop planning the method reaches the best collision rate even without ego status and high-level commands, which the authors attribute to better scene understanding rather than to trajectory bias."],"supporting_citations":[{"why":"Supplies the cross-attention mechanism that the interactor's interaction operation is built on.","marker":"(Li et al. 2023)"},{"why":"Provides the BEV encoder whose global semantic features are fused with multi-view images.","marker":"(Liu et al. 2023a)"},{"why":"Contributes the OmniDrive-NuScenes benchmark and baseline; the paper reports a 51.4% CIDEr improvement.","marker":"(Wang et al. 2024a)"},{"why":"Provides NuScenes-MQA, the benchmark on which the paper claims a 10.6% accuracy gain.","marker":"(Inoue et al. 2024)"},{"why":"Provides NuInstruct, the multi-view benchmark on which the method reports large MAE, ACC, and BLUE4 gains.","marker":"(Ding et al. 2024)"},{"why":"Provides NuScenes-QA, used for the VQA evaluation and comparison.","marker":"(Qian et al. 2024)"},{"why":"Supplies the open-loop planning baselines and the ego-status encoding scheme used in the planning experiments.","marker":"(Li et al. 2024)"},{"why":"Provides nuScenes, the source data for the ORA benchmark and for the refined training datasets.","marker":"(Caesar et al. 2019)"},{"why":"Used to extract significant objects from single-view images for the grounding data.","marker":"(Liu et al. 2023b)"},{"why":"Its pretraining recipe is followed for optimizer, learning rate, and batch size in the alignment stages.","marker":"(Liu et al. 2024a)"}],"fun_headline_variants":["Pick the right tokens: New method beats prior on occluded driving hazards","Query selects 90 tokens from six camera views for safer autonomous driving","Token pruning by query: Driving AI improves occluded-risk reasoning","Six views, top-k tokens: Driving reasoning improves on occluded risks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the GPT-4o-generated risk labels on the ORA benchmark measure a model's ability to reason about occluded objects, even though those labels are built from ground-truth object lists the model never sees at inference.","fun_headline_variants_meta":{"raw":{"variants":["Pick the right tokens: New method beats prior on occluded driving hazards","Query selects 90 tokens from six camera views for safer autonomous driving","Token pruning by query: Driving AI improves occluded-risk reasoning","Six views, top-k tokens: Driving reasoning improves on occluded risks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001378,"raw_usage":{"total_tokens":5606,"prompt_tokens":991,"completion_tokens":4615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":4539}},"tokens_in":607,"tokens_out":4615,"duration_ms":34107,"temperature":1.0,"reasoning_tokens":4539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:46:22.331290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators verify a held-out subset of ORA where the risky object is fully occluded in every camera view, and compare the model's accuracy against a no-image text baseline on that subset; if the model does not beat the baseline, the claim that it reasons about perception-limited regions is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NuScenes-QA, used for the VQA evaluation and comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NuScenes-MQA, the benchmark on which the paper claims a 10.6% accuracy gain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NuInstruct, the multi-view benchmark on which the method reports large MAE, ACC, and BLUE4 gains."}],"review_version":1}