{"id":"145b0b29-cbab-441a-8762-ca3e8284f015","arxiv_id":"2501.04671","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A retrieval-based interleaved visual chain-of-thought method, RIV-CoT, improves VLM answer accuracy by 3.1 points and reasoning accuracy by 4.6 points on a new driving theory VQA benchmark.","lead":"This paper introduces DrivingVQA, a new visual question answering benchmark built from French driving theory exams, with expert explanations and annotated image regions. The authors then propose RIV-CoT, a prompting method that feeds detected image crops back into a vision-language model's chain-of-thought, improving answer and reasoning accuracy over standard chain-of-thought prompting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reasoning-accuracy gain of RIV-CoT rests on an LLM judge validated on only 50 pairs; format bias from interleaved bounding boxes could explain the 4.6-point gap.","rationale":"The reader's weakest_assumption points to the validity of the GPT-4o-mini reasoning judge, validated on 50 pairs and applied without uncertainty or format-bias analysis. I agree this is the most load-bearing concern. I considered other candidates: (i) that the augmented explanations used for training RIV-CoT might leak extra information relative to CoT; this is controlled by QP-IEA, which scores 56.4, essentially tied with CoT, so it does not explain the answer gain; (ii) that the answer-accuracy gain itself might be fragile; it is not, given five seeds and reported standard deviations; (iii) that the A-OKVQA scaling results depend on unvalidated pseudo-labels; this is a limitation but not the central claim. The remaining soft spot is the reasoning-accuracy headline of 4.6 points in Figure 8, which is exactly what supports the paper's claim that interleaved visual crops ground VLM reasoning. Because the judge is cheap to rerun and a format-bias check is straightforward, I would keep the reader's CONDITIONAL verdict and ask for this additional evidence before treating the reasoning-accuracy magnitude as established. Hence verdict_should_be is UNCHANGED relative to the reader's verdict, with the same condition attached.","tokens_in":22981,"tokens_out":5114,"duration_ms":49770,"concrete_test":"Sample roughly 200 test reasoning outputs stratified across CoT, QP-RB-RV-EA, QP-IBEA, and RIV-CoT. Have 2-3 human annotators judge correctness using the rubric in Appendix C.3, and run the GPT-4o-mini judge on both original outputs and stripped outputs with all bounding-box coordinates and image-token markers removed. The concern is settled if (a) human labels reproduce a RIV-CoT-over-CoT reasoning gap close to 4.6 points, or (b) stripped judge scores still show the gap while original judge scores are similar; it is confirmed if human labels show a substantially smaller gap, or if stripping erases the gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The answer-accuracy results in Sec. 5.3 (Table 1) are well supported: five seeds, reported standard deviations, and a 3.1-point gap over CoT that is several standard errors wide. The load-bearing weak point is the reasoning-accuracy metric used to support the claim that interleaved visual evidence improves reasoning quality, not just answer selection. In Sec. 5.4.2 and Figure 8, GPT-4o-mini is validated against human labels on only 50 reasoning pairs (F1 0.82) and then applied to the full test set, with no variance estimate, no per-strategy agreement, and no precision/recall breakdown. RIV-CoT outputs differ stylistically from CoT outputs: they contain bounding-box coordinates and, in the full pipeline, interleaved image-crop tokens. The judge prompt asks for argument-level comparison, so the presence of exact coordinates and explicit entity references may be treated as evidence of grounded reasoning even when the conceptual content is wrong or incomplete. With only 50 validation pairs, a systematic format bias of the size needed to create or inflate a 4.6-point gap would be very hard to detect. The paper's claim of improved visual reasoning is therefore not yet established at the claimed magnitude; the answer-accuracy gain is robust, but the reasoning-accuracy gain needs stronger human-grounded validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DrivingVQA, a visual question answering dataset of 3,931 multiple-choice driving-theory problems with expert explanations and human-annotated relevant entities (bounding boxes). It proposes RIV-CoT, a training/inference framework in which the VLM generates chain-of-thought text interleaved with bounding-box coordinates and, at inference, with retrieved image crops associated with predicted entities. The main empirical claims are that on DrivingVQA, RIV-CoT improves answer accuracy by 3.1 points over vanilla CoT prompting (59.3% vs. 56.2%, Table 1) and reasoning accuracy by 4.6 points (Sec. 5.4.2 and Fig. 8), and that the approach transfers to A-OKVQA using automatically generated pseudo-labels (Table 4).","tokens_in":23257,"tokens_out":9869,"duration_ms":86632,"significance":"If the claims hold, DrivingVQA would be a valuable real-world benchmark for visually grounded reasoning, and RIV-CoT would demonstrate a practical way to force VLMs to attend to specific image regions during multi-step reasoning. The paper's strengths include the public release of code and dataset, five-seed fine-tuning with standard deviations, ablations that isolate each component (entity names, bounding boxes, visual crops, interleaving), an oracle-entity condition, and an AnyRes-based token-count control. The A-OKVQA experiment further shows the pipeline can be transferred when gold entity annotations are unavailable. However, the headline 'best exam score' claim is challenged by the AnyRes baseline in Table 3, and the reasoning-accuracy claim rests on a thinly validated LLM judge. The answer-accuracy gain over CoT is well supported; the broader claims need revision.","major_comments":[{"comment":"The Image Split row reports QPR−EA† at 60.7% (±0.9) exam score, which is higher than RIV-CoT's 59.3% (±1.0) from Table 1. This contradicts the abstract's and Sec. 5.3's claim that RIV-CoT achieves the best exam score, and it undercuts the conclusion in Sec. 5.4.1 that carefully selected visual patches are superior to generic image splitting: that conclusion is only supported for oracle entities (QPRV−EA, 62.0%), not for the deployed RIV-CoT system. The authors should either include the AnyRes condition in the main comparison table and discuss why RIV-CoT is preferred despite the lower score, or demonstrate with a statistical test and a clear statement of comparability that the AnyRes result is not directly comparable.","section":"Table 3, Sec. 5.4.1"},{"comment":"The reasoning-correctness metric is estimated with a single GPT-4o-mini judge that was validated against human labels on only 50 reasoning pairs (F1=0.82) and then applied to the entire test set without variance estimates, per-strategy agreement, or precision/recall. RIV-CoT outputs contain bounding-box coordinates and interleaved image-crop tokens that CoT outputs do not, so the judge's argument-comparison criterion may be systematically easier to satisfy for RIV-CoT even when the conceptual reasoning is wrong. The 4.6-point reasoning-accuracy gain is therefore not established at the claimed magnitude; I recommend validating the judge on a larger, strategy-stratified human sample and reporting a format-controlled condition (e.g., stripping coordinates and crop tokens from generated reasoning before judging).","section":"Sec. 5.4.2, Fig. 8"},{"comment":"The comparison used to support the 'carefully selected visual patches provide a stronger advantage' claim is between the AnyRes baseline (QPR−EA†, 60.7%) and the oracle-entity model (QPRV−EA, 62.0%). Since the actual retrieval-based system (RIV-CoT) obtains 59.3%, the conclusion as stated is misleading for the proposed method. The claim should be re-scoped to oracle visual patches, or the paper should explain why the predicted-entity setting is the intended operating point and what the AnyRes gap means for it.","section":"Sec. 5.4.1, Table 3"}],"minor_comments":[{"comment":"The notation for conversation formats (QP−EA, QP−RB−RV−EA, QPR−EA†, etc.) is not defined consistently; in Table 3 the 'R' in QPR−EA† appears to denote image patches rather than the 'relevant entity labels' defined in Sec. 5.1. A summary table of all variants and their input/output components would improve readability.","section":"Sec. 5.1"},{"comment":"The statement 'as the maximum number of entities per sample in DRIVING VQA is five, we use N=4' does not justify matching tokens in the AnyRes baseline; the number of visual tokens should be matched per sample, not only at the maximum, and reporting per-sample token counts would strengthen the fairness of this control.","section":"Sec. 5.4.1"},{"comment":"The judge prompt instructs GPT-4o-mini to output '1' or '0'; please report the precision and recall of the judge on the 50 validation pairs, not only the F1 score, and provide per-strategy agreement because the validation set is small and the strategies produce stylistically different outputs.","section":"Appendix C.3"},{"comment":"The 50% bounding-box expansion factor is a free parameter; a sensitivity analysis (e.g., 0%, 50%, 100%) would strengthen the claim that the crop strategy, rather than this specific value, drives the measured gains.","section":"Sec. 4.2"},{"comment":"The sentence reporting 'an inter-annotator agreement score of 0.95 (Krippendorff's alpha, Fleiss' kappa, and Cohen's kappa)' is ambiguous; these are three different coefficients, and the text should state which coefficient(s) were computed and on what unit (sample-level filtering decisions?).","section":"Sec. 3.1"},{"comment":"Add confidence intervals or error bars to the reasoning-correctness points; currently the figure presents only point estimates from the LLM judge, which makes it impossible to assess the significance of the 4.6-point gap.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a valuable dataset, but the AnyRes baseline in Table 3 is a direct threat to the headline 'best exam score' claim and must be addressed head-on. The reasoning-correctness evaluation is currently too thin for the strength of the conclusions drawn from it; a larger human-validated judge sample and format-controlled analysis are needed. The answer-accuracy comparison against CoT is robust and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about visual grounding in VLMs. The paper introduces DrivingVQA, a driving-theory VQA benchmark with 3,931 real exam questions, expert-written explanations, and multiple entity bounding boxes per image, plus RIV-CoT, a method that interleaves image crops into the chain-of-thought during decoding. The headline answer-accuracy gain of 3.1 points over vanilla CoT on a 7B LLaVA-OneVision is solid: five seeds, standard deviations, ablations of each component, an oracle-entity control, and an AnyRes comparison that rules out token count alone. The method is genuinely new relative to Visual CoT and CogCOM: it pauses generation to crop and insert predicted regions, rather than selecting one region upfront. The A-OKVQA transfer with pseudo-labeled entities is a nice extra.\n\nThe main soft spot is the reasoning-correctness metric. It relies on GPT-4o-mini as a judge, validated on only 50 human-evaluated pairs (F1 0.82), then applied to the whole test set with no variance or per-strategy agreement. RIV-CoT outputs include bounding-box coordinates and explicit entity mentions, so the judge may be cueing on format rather than conceptual content. The 4.6-point reasoning gain over CoT could be partly inflated. That said, this doesn't sink the paper: the answer-accuracy gain is robust and the two metrics move together. Strengthening the judge validation with a larger human set and an error analysis would settle it.\n\nMinor quibbles: the A-OKVQA pseudo-label quality isn't analyzed, and 4k samples is modest, but the dataset fills a real gap. The paper deserves a serious referee and, with the reasoning evaluation tightened, acceptance. I'd bring it to a reading group.","headline":"Useful new driving-VQA benchmark and a genuinely interleaved visual-CoT method; answer-accuracy gains are solid, but the reasoning-accuracy metric rests on a judge validated on only 50 examples.","tokens_in":23798,"tokens_out":3353,"would_cite":true,"duration_ms":30117,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that interleaving image crops retrieved from predicted bounding boxes into generated reasoning improves VLM answer accuracy by 3.1 points and reasoning accuracy by 4.6 points over vanilla chain-of-thought, reaching an…","keywords":["DrivingVQA","visual chain-of-thought","vision-language models","visual question answering","retrieval-based generation","grounded reasoning","bounding boxes","driving theory exam"],"falsifier":"Have human raters blind to model identity judge a few hundred reasoning outputs from RIV-CoT and vanilla CoT, with the bounding boxes and image tokens removed from the text, and compare their labels to the GPT-4o-mini judge's labels; if agreement on RIV-CoT outputs is substantially lower than the reported 0.82 F1, the 4.6-point reasoning gain is an artifact of format rather than a real improvement in reasoning.","tokens_in":22806,"feed_emoji":"🚗","tokens_out":4787,"duration_ms":39414,"temperature":0.7,"pith_summary":"Vision-language models often answer from text and memorized knowledge instead of actually looking at the image. This paper introduces DrivingVQA, a 3,931-question benchmark drawn from French driving theory exams, with expert-written explanations and human-annotated bounding boxes for the entities that matter. It then proposes RIV-CoT, a prompting and fine-tuning method that makes the model pause while writing its explanation, crop the image at a predicted bounding box, insert the crop as an image token, and continue reasoning. On the test set, RIV-CoT reaches 59.3% exam score, 3.1 points above vanilla chain-of-thought in answer accuracy and 4.6 points above it in judged reasoning accuracy. The headline claim is that giving the model visual crops of the entities it is talking about, interleaved with the text, is a more effective way to ground visual reasoning than coordinates or descriptions alone.","feed_headline":"Image crops in chain-of-thought lift VLM reasoning by 4.6 points","feed_subtitle":"Grounding each step in actual image patches beats text-only chain-of-thought on driving-theory exam questions.","key_machinery":"The engine is multi-step retrieval-based generation. During training, each expert explanation is rewritten so that every reasoning step starts by naming a relevant entity and its bounding box; the model is trained on multi-turn conversations where the expected output for each turn ends in coordinates and the next turn contains the image crop at those coordinates, expanded by 50%. At inference the model generates until it emits a bounding box, the generation pauses, the crop is encoded by the vision encoder and inserted as an image token, and generation resumes; this pause-retrieve-insert loop repeats until the final answer. The interleaved explanations are produced by GPT-4o with two demonstrations, then cleaned with regex heuristics.","core_discovery":"On the paper's own terms, the central discovery is that retrieval-based interleaved visual chain-of-thought—where the model repeatedly predicts a relevant entity's bounding box, retrieves the corresponding image crop, and continues its explanation with that crop in context—is the best tested way to make a VLM reason about real-world driving scenes. Adding entity names alone (57.0%) adds little; adding bounding box coordinates (57.7%) helps; adding visual crops as a separate step (58.4%) helps more; and interleaving those crops inside the explanation (59.3%) helps most. The same recipe transfers to A-OKVQA using automatically generated pseudo-labels, improving multiple-choice accuracy from 80.6% with CoT to 84.2%. A comparison with oracle entity crops (62.0%) indicates the remaining bottleneck is entity detection, not the interleaving format.","pith_inferences":["The GPT-4o-mini judge may reward the interleaved entity-mention format regardless of genuine evidential use, so the 4.6-point reasoning gain should be stress-tested with a larger human sample.","The same retrieval-interleaving recipe could be tested on other visual reasoning benchmarks where explanations reference named objects, such as science diagrams or medical images; if the gain replicates, visual grounding through crops is a general lever.","Because the method needs only predicted boxes at inference, it could be combined with stronger open-set detectors or trained jointly with detection heads, potentially approaching the oracle-entity performance."],"forward_implications":["Interleaving image crops into explanations gives VLMs a stronger visual grounding signal than writing bounding box coordinates or entity names alone; each added component (names, boxes, crops, interleaving) raises exam score on DrivingVQA.","The best score with oracle entity crops (62.0% vs 59.3% predicted) implies that improving entity detection would further close the gap, pointing to detector quality as a bottleneck.","The method transfers to A-OKVQA using automatically extracted pseudo-labels, so it can be applied without human bounding-box annotations.","Reasoning correctness improved more than answer accuracy (4.6 vs 3.1 points), suggesting the format changes how the model reasons, not just what it answers."],"supporting_citations":[{"why":"introduces chain-of-thought prompting, the baseline that RIV-CoT is compared against and extends.","marker":"[51]"},{"why":"closest prior visual-CoT work; supplies the visual-patch cropping baseline that QP-RB-RV-EA resembles.","marker":"[41]"},{"why":"A-OKVQA dataset used to test scaling of RIV-CoT with automatically generated pseudo-labels.","marker":"[40]"},{"why":"GroundingDINO detector that localizes relevant entities and produces pseudo-labels when annotations are unavailable.","marker":"[25]"},{"why":"LLaVA-OneVision-7B, the VLM fine-tuned in all experiments and the backbone of the reported gains.","marker":"[17]"},{"why":"establishes the LLM-as-a-judge paradigm used by the GPT-4o-mini reasoning correctness evaluator.","marker":"[59]"}],"fun_headline_variants":["Visual crops in chain-of-thought lift VLM reasoning by 4.6%","Interleaving image crops into reasoning beats text-only CoT on driving QA","Grounding VLM steps in image patches improves driving QA by 4.6%","Retrieval-based visual CoT: crops in explanations boost reasoning 4.6%","Crop-by-crop interleaving raises VLM reasoning accuracy 4.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-mini judge used to measure reasoning correctness on the full test set, validated on only 50 human-evaluated pairs, remains fair across all outputs and does not systematically favor the interleaved-entity format that RIV-CoT produces.","fun_headline_variants_meta":{"raw":{"variants":["Visual crops in chain-of-thought lift VLM reasoning by 4.6%","Interleaving image crops into reasoning beats text-only CoT on driving QA","Grounding VLM steps in image patches improves driving QA by 4.6%","Retrieval-based visual CoT: crops in explanations boost reasoning 4.6%","Crop-by-crop interleaving raises VLM reasoning accuracy 4.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000983,"raw_usage":{"total_tokens":4154,"prompt_tokens":908,"completion_tokens":3246,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":3135}},"tokens_in":524,"tokens_out":3246,"duration_ms":21632,"temperature":1.0,"reasoning_tokens":3135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:26:24.781222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human raters blind to model identity judge a few hundred reasoning outputs from RIV-CoT and vanilla CoT, with the bounding boxes and image tokens removed from the text, and compare their labels to the GPT-4o-mini judge's labels; if agreement on RIV-CoT outputs is substantially lower than the reported 0.82 F1, the 4.6-point reasoning gain is an artifact of format rather than a real improvement in reasoning.","supporting_citations":[{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"introduces chain-of-thought prompting, the baseline that RIV-CoT is compared against and extends."},{"cited_title":"A-okvqa: A benchmark for visual question answering using world knowl- edge","cited_arxiv_id":null,"evidence_quote":"A-OKVQA dataset used to test scaling of RIV-CoT with automatically generated pseudo-labels."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023","cited_arxiv_id":null,"evidence_quote":"establishes the LLM-as-a-judge paradigm used by the GPT-4o-mini reasoning correctness evaluator."}],"review_version":1}