{"id":"1ec5ae2a-3b9c-46d5-bc8b-30910f9beba5","arxiv_id":"2411.12980","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A driving VQA model prunes image tokens using the text query and enhances the survivors with cross-frame attention, reporting 168x token reduction and gains over its baseline.","lead":"LaVida Drive is a vision-language system for autonomous driving that keeps high-resolution camera views for detail and low-resolution video for motion, then selects only image tokens relevant to the driver's question. It reports better or comparable question-answering scores on DriveLM and NuScenes-QA while using far fewer visual tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 / Eq. (6) in §3.2 does not define a usable per-token selection score as printed: the softmax is over text tokens, W sums token embeddings, and M mixes non-conforming objects, so the 168× compression claim cannot be reproduced without a corrected specification.","rationale":"I reviewed the paper in good faith. The component ablations in Tables 2 and 3 give some internal evidence that a token-selection and enhancement pipeline can match or exceed the low-resolution baseline, so I do not claim the approach is obviously false or that the experimental gains are fabricated. However, the printed definition of the token selection module is not executable: the normalization in Eq. (6), the construction of S_sum, and the definition of W in Algorithm 1 do not combine into a valid per-token relevance score. Because this module is the stated source of the 168-fold compression, the central claim cannot currently be verified from the manuscript alone. The reader's weakest assumption about CLIP token-level alignment is a real and related concern, and it becomes especially important once the pseudocode is corrected, since the enhancement module may not recover high-resolution details that are discarded by selection; hence I mark partial agreement rather than full agreement. I recommend keeping the existing CONDITIONAL verdict rather than escalating to rejection, because the issue is correctable: the authors can provide a corrected algorithm, release runnable code, and demonstrate that the top-k selection actually retrieves query-relevant regions. Additional reporting gaps, such as the missing single-frame single-view row in Table 4 despite being described in the text and the inconsistent compression claims in the abstract versus introduction, further support conditioning acceptance on a code release and a clarified specification.","tokens_in":10300,"tokens_out":12008,"duration_ms":122967,"concrete_test":"Run the printed Algorithm 1 on one DriveLM sample with m image tokens and n text tokens: compute p_i via Eq. (6), then S_sum, W, and M, and check shapes and values. The concern lands if S_sum is constant across image tokens (because p_i is normalized over text tokens) or if M's addition is dimensionally invalid. If the authors clarify that the intended softmax is over image tokens, ask for the corrected equation and then verify whether the top-k selected patch tokens contain the ground-truth answer region on DriveLM (e.g., IoU with answer-relevant boxes) at the reported 168× ratio, with random top-k retrieval as the null model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism that produces the claimed 168-fold compression is the Query-aware Token Selection module, but its specification in Section 3.2 / Algorithm 1 is not computationally coherent as written. Equation (6) defines p_i(x) = exp(s(I', T_i)/τ) / Σ_{j=1}^N exp(s(I', T_j)/τ), normalizing each image token's similarity over the text tokens; summing this over text tokens, as Algorithm 1 does for S_sum, yields a constant for every image token, so the selection map carries no ranking signal. The subsequent step W ← Σ_i I'_i sums d-dimensional image-token embeddings into a single d-vector, and M ← (1−α)·S_sum + α·W then adds objects of incompatible shape or meaning. Even if the intended softmax was over image tokens rather than text tokens, the paper does not state or implement that; the pseudocode cannot be run as printed. This matters because the headline result is specifically attributed to query-aware selection: if the selection score is not a well-defined function of the query, the reported improvements cannot be traced to the proposed mechanism, and the method is not reproducible without a corrected algorithm. This concern is logically prior to, and independent of, the reader's CLIP-alignment concern: even if CLIP tokens are perfectly aligned, the printed scoring rule does not provide a valid per-image-token relevance ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LaVida Drive, a vision-language model framework for visual question answering in autonomous driving. The method uses a frozen CLIP image/text encoder pair plus a TimeSformer video encoder, with two trainable modules: a query-aware token selection module that prunes visual tokens based on cosine similarity between image and text tokens, and a spatial-temporal token enhancement module that uses cross-attention to recover context from support branches. Experiments on DriveLM and NuScenes-QA report improvements over EM-VLM4AD baselines and competitive results against larger models, with a claimed 168-fold token compression. The paper includes component ablations and an input-type ablation.","tokens_in":10556,"tokens_out":4736,"duration_ms":45321,"significance":"If the central claims hold, LaVida Drive would be a useful efficiency-oriented contribution to autonomous-driving VQA: it is plausible that query-aware token pruning plus cross-attention-based context restoration can reduce token counts while keeping or improving accuracy. The paper provides comparisons on standard public benchmarks, a component-wise ablation, and an input-type robustness study, which are strengths. The main significance is undermined, however, by (i) a formally incoherent specification of the token-selection algorithm, (ii) overclaimed 'optimal performance' that is not supported by the tables, and (iii) inconsistent compression-ratio statements between the abstract and the introduction. These issues prevent the reader from reproducing the central mechanism and from assessing the true magnitude of the reported gains.","major_comments":[{"comment":"The Query-aware Token Selection module is not specified in a computationally coherent way. Equation (6) defines p_i(x) = exp(s(I', T_i)/tau) / sum_{j=1}^N exp(s(I', T_j)/tau), which normalizes over text tokens for a fixed image token. Algorithm 1 then computes S_sum <- sum_{k=1}^K p_i(x); for a fixed image token, summing these normalized values over text tokens yields a constant (approximately 1), so S_sum carries no per-token ranking signal. The subsequent step W <- sum_i I'_i collapses all image-token embeddings into a single d-dimensional vector, and M <- (1-alpha)*S_sum + alpha*W then attempts to add objects of incompatible shapes/meanings. As printed, the pseudocode cannot be executed and the top-k selection is not a well-defined function of the query. The authors must specify whether the softmax should be over image tokens (the usual interpretation) and provide a corrected per-token selection score, or otherwise the reported improvements cannot be traced to the proposed mechanism.","section":"Section 3.2, Eq. (6) and Algorithm 1"},{"comment":"The compression-ratio claims are inconsistent. The abstract states 'an impressive 168-fold token compression,' while Section 1 states the method 'reduces visual tokens by 50% to 84%,' which corresponds to a 2x to 6.25x reduction. Table 2 'fixing the overall reduction ratio to 168' suggests that 168x is measured relative to the high-resolution patch token count (49*6*28 tokens), not to the baseline low-resolution token count. The paper should state clearly the reference token count for the 168x claim and reconcile it with the 50-84% statement in Section 1.","section":"Abstract and Section 1"},{"comment":"The claim of 'optimal performance' in the abstract is not supported by the reported results. On DriveLM, LaVida Drive achieves BLEU-4 of 51.3, which is below DriveLM-Agent's 53.1; on NuScenes-QA, it achieves Exist accuracy of 78.0, which is below NusceneQA-Agent's 84.8. The method is best or second-best on several metrics, but it is not uniformly optimal. The authors should temper the abstract and conclusion to 'competitive or state-of-the-art on most metrics' and discuss the trade-offs where the method is not the best.","section":"Table 1 and Table 2"},{"comment":"No error bars, standard deviations, or significance tests are reported for any experiment. Many of the claimed improvements are small (e.g., CIDEr 3.32 vs 3.20, ROUGE-L 73.9 vs 72.0 in Table 1; several +0.5 to +0.6 increments in Table 3), and without variance information it is impossible to judge whether the differences are meaningful or within run-to-run noise. The authors should report means and standard deviations over at least three random seeds, especially for the main comparison and the ablation tables.","section":"All experimental tables"},{"comment":"The query-aware selection mechanism rests on the assumption that cosine similarity between CLIP image patch tokens and text tokens reliably ranks answer-relevant regions, but the paper provides no quantitative validation of this ranking. The qualitative example in Figure 4 is illustrative only. A concrete test would be to replace the query-aware selection with random selection or a fixed grid selection at the same token budget and compare performance; if the query-aware version does not clearly outperform these baselines, the claim that selection is query-aware is not substantiated.","section":"Section 3.2 and Figure 4"}],"minor_comments":[{"comment":"Typographical and spacing issues appear throughout, e.g., 'T ext Encoder', 'T op-K', 'T ext', and 'NusceneQA' (inconsistent with 'NuScenes-QA'). Please proofread carefully.","section":"Section 3.2"},{"comment":"The row label 'High Solution Patch' should read 'High Resolution Patch'. Also, the delta format '+4.0' vs '+0.9' is inconsistent: some deltas are given as absolute points and some as percentage points; please make this uniform.","section":"Table 3"},{"comment":"The implementation detail 'each image divided into 4x7 patches of size 224x224' is ambiguous: a 224x224 patch on a typical camera image would imply a very large input resolution (e.g., 896x1568). Please specify the exact input resolution and how the '4x7' grid is derived.","section":"Section 4.1"},{"comment":"The algorithm uses 'p_img_i(x)' but never defines x, and the index i in the softmax numerator is not introduced as an image-token index. Please rewrite the notation so that each quantity is defined precisely.","section":"Section 3.2, Algorithm 1"},{"comment":"Several references appear to be incomplete or non-verifiable, e.g., [4] 'Journal of Autonomous Systems', [8] 'Proceedings of CVPR, 2024', [23] 'IEEE Transactions on Vehicular Technology', [27] 'IEEE Access', [28] 'IEEE Transactions on Robotics', [29] 'Proceedings of CVPR, 2024', and [31] 'IEEE Transactions on Intelligent Vehicles' lack arXiv identifiers, DOIs, or author-page details that are standard for verification. Please provide complete, verifiable citations.","section":"References"},{"comment":"The paper claims improved 'inference efficiency' and 'computational efficiency' but reports no runtime, FLOPs, or latency measurements. Token count is only a proxy for efficiency; please include actual throughput or latency numbers for the final model versus the baselines.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central technical argument is plausible but the specification of Algorithm 1 is not executable as printed, which is a load-bearing reproducibility issue. In addition, the abstract's 'optimal performance' and '168-fold' claims are inconsistent with the tables and the introduction. The reference list contains several entries that appear to be placeholder-like or non-verifiable; I recommend the editor check the citations carefully before any acceptance decision. These issues are fixable in revision, so I recommend major_revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for the architecture, but the central algorithm as printed doesn't compute what the paper claims. In Eq. (6), p_i^img is defined by normalizing an image token's similarity over text tokens; summing that over text tokens, as Algorithm 1 does for S_sum, yields a constant for every image token, so the selection map carries no ranking signal. Then W sums image tokens into a single vector, and M adds a scalar to that vector. That is not a minor notation slip; the paper's main claim is that query-aware selection produces the 168x compression, and the selection score is not a well-defined per-token function of the query. Even if the intended softmax was over image tokens, the paper never states or implements that. This alone justifies a major revision.\n\nWhat is good: the overall design is sensible — keep high-resolution spatial patches, use low-resolution temporal frames, prune tokens by query relevance, and restore context with cross-attention. Table 1 shows consistent gains over EM-VLM4AD on most DriveLM and NuScenes-QA metrics, and the component ablations in Table 3 line up with the design story. That is real evidence, even if the exact mechanism is underspecified.\n\nThe soft spots beyond the algorithm: the abstract claims \"optimal performance\" while the paper's own tables show it is not best on every metric; the compression figure is stated as 168-fold in the abstract but 50–84% in the introduction; the single-frame single-view ablation is described in the text but missing from Table 4; no error bars or code are provided; and token-selection baselines are not cited. The CLIP alignment assumption in Section 3.2 is also load-bearing — if token-level cosine similarity does not rank relevant image regions, the selection cannot recover the answer. That concern is secondary to the broken pseudocode, but worth the authors addressing.\n\nMy take: the empirical results are suggestive, not conclusive, and the central mechanism is not reproducible as printed. This paper deserves a serious referee, but only with a demand for a corrected algorithm, code, and consistent claims. If the algorithm is fixable and the results hold in re-runs, it could be a useful subfield contribution.","headline":"Promising driving VQA idea with a real empirical hook, but the printed token-selection algorithm cannot be run as written, so the headline result is not reproducible.","tokens_in":11123,"tokens_out":2315,"would_cite":false,"duration_ms":69785,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LaVida Drive claims a 168-fold reduction in visual tokens for autonomous-driving VQA while matching or beating prior benchmarks on most metrics.","keywords":["visual question answering","autonomous driving","vision-language models","query-aware token selection","temporal modeling","token compression","high-resolution perception","spatial-temporal enhancement"],"falsifier":"On a held-out set of DriveLM examples with bounding boxes around the objects a question asks about, measure whether the patches containing those objects fall in the top-k selected tokens for the model's best configuration; if they frequently do not, while the model still answers correctly, then the query-aware selection is not the mechanism carrying performance.","tokens_in":10060,"feed_emoji":"🚗","tokens_out":6450,"duration_ms":60572,"temperature":0.7,"pith_summary":"LaVida Drive tackles a bottleneck in vision-language models for autonomous driving: high-resolution multi-view input is necessary for fine-grained questions, but it floods the language model with tokens. The paper's proposal is to keep high-resolution patches for spatial detail, add a lower-resolution video stream for motion, and then use the user's text query to select only the most relevant visual tokens before they reach the language model. A second module recovers context lost in selection by letting selected tokens attend, through cross-attention, to the full image-encoder and video-encoder outputs. On DriveLM and NuScenes-QA the model reaches the best or second-best scores on most metrics while claiming roughly a 168-fold token compression, which is the pay-off: much cheaper inference without giving away accuracy. The central bet is that semantic similarity between CLIP image tokens and text tokens is a reliable guide to which image regions answer the question.","feed_headline":"168x fewer tokens keeps driving VQA near the top","feed_subtitle":"Query-aware token selection plus spatial-temporal recovery preserves answers while slashing compute.","key_machinery":"The mechanism that carries the argument is the pair of modules around which the architecture is built. Query-aware Token Selection computes a token-wise similarity matrix between the MLP-aligned image patch tokens and the CLIP text tokens of the user's question, normalises it over image tokens with a temperature $\\tau$, sums the per-text-token relevance scores, mixes in a weighted token-importance term, and keeps the top-k image tokens; this is what produces the dramatic token reduction. Spatial-Temporal Token Enhancement then applies token-wise cross-attention, $\\mathrm{Softmax}(QK^{\\top}/\\sqrt{d_k})V$, with the selected tokens as queries and the full image-encoder output (spatial restoration) and TimeSformer video-encoder output (temporal enhancement) as keys and values, merging the two enhanced representations through an MLP. That second module is the paper's answer to the objection that dropping tokens destroys spatial context: it lets the language model still access scene-wide and motion information without increasing the token count.","core_discovery":"The paper's central claim is that a VQA model for driving can have both high-resolution spatial understanding and temporal awareness without paying the full token cost, by making token pruning query-dependent and then restoring what pruning removed. Concretely, LaVida Drive processes high-resolution patches with a frozen CLIP image encoder, aligns those patch tokens to the question's text tokens with an MLP and a temperature-scaled cosine-similarity matrix, and keeps only the top-k image tokens ranked by summed relevance to the text. Those surviving tokens are enhanced by token-wise cross-attention that uses the full image-encoder output for spatial restoration and a TimeSformer video-encoder output for temporal enhancement, while the LLM (T5-medium) receives only the compact, enhanced token set plus the query text. In the paper's experiments on DriveLM, this configuration beats the baselines it is compared with on METEOR, ROUGE-L, and CIDEr, and comes second on BLEU-4; on NuScenes-QA it is best on Object, Status, and Comparison categories, second on Exist, all without 3D detector outputs. The 168-fold figure is the overall reduction from the high-resolution patch token count, with ablations showing that the best accuracy comes from splitting that reduction between top-k selection and an MLP compression stage rather than using either alone.","pith_inferences":["If CLIP's token-level alignment is imperfect, the top-k selection may discard the very patch containing the answer; a natural stress test is to check selection recall against human-annotated relevant regions, since the paper does not report such a test.","The same query-aware selection mechanism could transfer to other vision-language tasks where a text query determines which image regions matter, such as referring expression comprehension or visual grounding, though the paper only evaluates VQA in driving.","The enhancement module's reliance on frozen full-image and video encoders means the real computational savings land only inside the language model; the paper does not account for the cost of computing full-resolution image features before selection, so end-to-end savings may be lower than 168-fold.","A harder temporal benchmark with fast-moving objects and dense multi-frame queries would test whether the low-resolution video stream can actually supply motion information the high-resolution stream lacks."],"forward_implications":["If the compression figure holds, high-resolution multi-view driving VQA becomes feasible on a single GPU at inference, since the LLM sees roughly 1/168 of the raw patch tokens.","The query dependence of selection means the system can allocate visual tokens per question, which is a step toward running richer perception only when needed.","Because the selected tokens are a sparse, query-relevant subset, the model produces a natural visual explanation of its answer (the patches it attended to), which could support debugging of perception failures.","The ablation across select/compress ratios implies there is a tunable cost-accuracy frontier: practitioners can dial the reduction while watching the four VQA metrics.","The input-type ablation shows the design degrades gracefully when some views or frames are missing, suggesting robustness to sensor dropout."],"supporting_citations":[{"why":"Supplies the EM-VLM4AD protocols, training/eval splits, and the primary baseline LaVida Drive must beat on DriveLM.","marker":"[6]"},{"why":"Supplies the DriveLM dataset and the DriveLM-Agent baseline that slightly beats LaVida Drive on BLEU-4.","marker":"[19]"},{"why":"Supplies the NuScenes-QA benchmark and its four accuracy categories used for the second evaluation.","marker":"[14]"},{"why":"Provides the frozen CLIP text and image encoders whose shared semantic space the token-selection similarity assumes.","marker":"[15]"},{"why":"Provides the TimeSformer video encoder whose output feeds the temporal token enhancement.","marker":"[2]"},{"why":"Provides the T5-medium language model that consumes the selected and enhanced tokens.","marker":"[17]"},{"why":"Provides the BLEU-4 metric used to evaluate DriveLM answers.","marker":"[12]"},{"why":"Provides the ROUGE-L metric used to evaluate DriveLM answers.","marker":"[10]"},{"why":"Provides the METEOR metric used to evaluate DriveLM answers.","marker":"[1]"},{"why":"Provides the CIDEr metric used to evaluate DriveLM answers.","marker":"[21]"}],"fun_headline_variants":["Ask the query before picking tokens for driving VQA","168x fewer tokens, same driving VQA answers","Token-efficient driving VQA that restores what it prunes","High-res driving VQA without the token bill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that cosine similarity between CLIP image-patch tokens and the question's text tokens reliably ranks which image regions are needed to answer the question, so that the top-k selection does not throw away the tokens that contain the answer.","fun_headline_variants_meta":{"raw":{"variants":["Ask the query before picking tokens for driving VQA","168x fewer tokens, same driving VQA answers","Token-efficient driving VQA that restores what it prunes","High-res driving VQA without the token bill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3262,"prompt_tokens":1077,"completion_tokens":2185,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":2121}},"tokens_in":693,"tokens_out":2185,"duration_ms":15297,"temperature":1.0,"reasoning_tokens":2121,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:58:23.166760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of DriveLM examples with bounding boxes around the objects a question asks about, measure whether the patches containing those objects fall in the top-k selected tokens for the model's best configuration; if they frequently do not, while the model still answers correctly, then the query-aware selection is not the mechanism carrying performance.","supporting_citations":[{"cited_title":"Multi-frame, lightweight & efficient vision- language models for question answering in autonomous driving, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the EM-VLM4AD protocols, training/eval splits, and the primary baseline LaVida Drive must beat on DriveLM."},{"cited_title":"Nuscenes-qa: A multi-modal vi- sual question answering benchmark for autonomous driving scenario, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the NuScenes-QA benchmark and its four accuracy categories used for the second evaluation."},{"cited_title":"Learning trans- ferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP text and image encoders whose shared semantic space the token-selection similarity assumes."},{"cited_title":"Is space-time attention all you need for video under- standing? In Proceedings of the International Confer- ence on Machine Learning (ICML) , 2021","cited_arxiv_id":null,"evidence_quote":"Provides the TimeSformer video encoder whose output feeds the temporal token enhancement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the T5-medium language model that consumes the selected and enhanced tokens."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"Provides the BLEU-4 metric used to evaluate DriveLM answers."},{"cited_title":"Rouge: A package for automatic eval- uation of summaries","cited_arxiv_id":null,"evidence_quote":"Provides the ROUGE-L metric used to evaluate DriveLM answers."},{"cited_title":"Meteor: An auto- matic metric for mt evaluation with improved correla- tion with human judgments","cited_arxiv_id":null,"evidence_quote":"Provides the METEOR metric used to evaluate DriveLM answers."},{"cited_title":"Cider: Consensus-based image descrip- tion evaluation","cited_arxiv_id":null,"evidence_quote":"Provides the CIDEr metric used to evaluate DriveLM answers."}],"review_version":1}