Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that interleaving image crops retrieved from predicted bounding boxes into generated reasoning improves VLM answer accuracy by 3.1 points and reasoning accuracy by 4.6 points over vanilla chain-of-thought, reaching an…

desk verdict Useful new driving-VQA benchmark and a genuinely interleaved visual-CoT method; answer-accuracy gains are solid, but the reasoning-accuracy metric rests on a judge validated on only 50 examples. read the letter →

arxiv 2501.04671 v2 pith:EZR5GXKG submitted 2025-01-08 cs.CV cs.AI

Charles Corbière , Simon Roburin , Syrielle Montariol , Antoine Bosselut , Alexandre Alahi This is my paper · ORCID
classification cs.CVcs.AI
keywords DrivingVQAvisualchain-of-thoughtvision-languagemodelsquestionansweringretrieval-basedgenerationgroundedreasoningboundingboxestheoryexam
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models often answer from text and memorized knowledge instead of actually looking at the image. This paper introduces DrivingVQA, a 3,931-question benchmark drawn from French driving theory exams, with expert-written explanations and human-annotated bounding boxes for the entities that matter. It then proposes RIV-CoT, a prompting and fine-tuning method that makes the model pause while writing its explanation, crop the image at a predicted bounding box, insert the crop as an image token, and continue reasoning. On the test set, RIV-CoT reaches 59.3% exam score, 3.1 points above vanilla chain-of-thought in answer accuracy and 4.6 points above it in judged reasoning accuracy. The headline claim is that giving the model visual crops of the entities it is talking about, interleaved with the text, is a more effective way to ground visual reasoning than coordinates or descriptions alone.

What carries the argument

The engine is multi-step retrieval-based generation. During training, each expert explanation is rewritten so that every reasoning step starts by naming a relevant entity and its bounding box; the model is trained on multi-turn conversations where the expected output for each turn ends in coordinates and the next turn contains the image crop at those coordinates, expanded by 50%. At inference the model generates until it emits a bounding box, the generation pauses, the crop is encoded by the vision encoder and inserted as an image token, and generation resumes; this pause-retrieve-insert loop repeats until the final answer. The interleaved explanations are produced by GPT-4o with two demonstrations, then cleaned with regex heuristics.

What would settle it

Have human raters blind to model identity judge a few hundred reasoning outputs from RIV-CoT and vanilla CoT, with the bounding boxes and image tokens removed from the text, and compare their labels to the GPT-4o-mini judge's labels; if agreement on RIV-CoT outputs is substantially lower than the reported 0.82 F1, the 4.6-point reasoning gain is an artifact of format rather than a real improvement in reasoning.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that retrieval-based interleaved visual chain-of-thought—where the model repeatedly predicts a relevant entity's bounding box, retrieves the corresponding image crop, and continues its explanation with that crop in context—is the best tested way to make a VLM reason about real-world driving scenes. Adding entity names alone (57.0%) adds little; adding bounding box coordinates (57.7%) helps; adding visual crops as a separate step (58.4%) helps more; and interleaving those crops inside the explanation (59.3%) helps most. The same recipe transfers to A-OKVQA using automatically generated pseudo-labels, improving multiple-choice accuracy from 80.6% with CoT to 84.2%. A comparison with oracle entity crops (62.0%) indicates the remaining bottleneck is entity detection, not the interleaving format.

Load-bearing premise

The load-bearing premise is that the GPT-4o-mini judge used to measure reasoning correctness on the full test set, validated on only 50 human-evaluated pairs, remains fair across all outputs and does not systematically favor the interleaved-entity format that RIV-CoT produces.

Editorial extensions

If this is right

  • Interleaving image crops into explanations gives VLMs a stronger visual grounding signal than writing bounding box coordinates or entity names alone; each added component (names, boxes, crops, interleaving) raises exam score on DrivingVQA.
  • The best score with oracle entity crops (62.0% vs 59.3% predicted) implies that improving entity detection would further close the gap, pointing to detector quality as a bottleneck.
  • The method transfers to A-OKVQA using automatically extracted pseudo-labels, so it can be applied without human bounding-box annotations.
  • Reasoning correctness improved more than answer accuracy (4.6 vs 3.1 points), suggesting the format changes how the model reasons, not just what it answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The GPT-4o-mini judge may reward the interleaved entity-mention format regardless of genuine evidential use, so the 4.6-point reasoning gain should be stress-tested with a larger human sample.
  • The same retrieval-interleaving recipe could be tested on other visual reasoning benchmarks where explanations reference named objects, such as science diagrams or medical images; if the gain replicates, visual grounding through crops is a general lever.
  • Because the method needs only predicted boxes at inference, it could be combined with stronger open-set detectors or trained jointly with detection heads, potentially approaching the oracle-entity performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DrivingVQA, a visual question answering dataset of 3,931 multiple-choice driving-theory problems with expert explanations and human-annotated relevant entities (bounding boxes). It proposes RIV-CoT, a training/inference framework in which the VLM generates chain-of-thought text interleaved with bounding-box coordinates and, at inference, with retrieved image crops associated with predicted entities. The main empirical claims are that on DrivingVQA, RIV-CoT improves answer accuracy by 3.1 points over vanilla CoT prompting (59.3% vs. 56.2%, Table 1) and reasoning accuracy by 4.6 points (Sec. 5.4.2 and Fig. 8), and that the approach transfers to A-OKVQA using automatically generated pseudo-labels (Table 4).

Significance. If the claims hold, DrivingVQA would be a valuable real-world benchmark for visually grounded reasoning, and RIV-CoT would demonstrate a practical way to force VLMs to attend to specific image regions during multi-step reasoning. The paper's strengths include the public release of code and dataset, five-seed fine-tuning with standard deviations, ablations that isolate each component (entity names, bounding boxes, visual crops, interleaving), an oracle-entity condition, and an AnyRes-based token-count control. The A-OKVQA experiment further shows the pipeline can be transferred when gold entity annotations are unavailable. However, the headline 'best exam score' claim is challenged by the AnyRes baseline in Table 3, and the reasoning-accuracy claim rests on a thinly validated LLM judge. The answer-accuracy gain over CoT is well supported; the broader claims need revision.

major comments (3)
  1. [Table 3, Sec. 5.4.1] The Image Split row reports QPR−EA† at 60.7% (±0.9) exam score, which is higher than RIV-CoT's 59.3% (±1.0) from Table 1. This contradicts the abstract's and Sec. 5.3's claim that RIV-CoT achieves the best exam score, and it undercuts the conclusion in Sec. 5.4.1 that carefully selected visual patches are superior to generic image splitting: that conclusion is only supported for oracle entities (QPRV−EA, 62.0%), not for the deployed RIV-CoT system. The authors should either include the AnyRes condition in the main comparison table and discuss why RIV-CoT is preferred despite the lower score, or demonstrate with a statistical test and a clear statement of comparability that the AnyRes result is not directly comparable.
  2. [Sec. 5.4.2, Fig. 8] The reasoning-correctness metric is estimated with a single GPT-4o-mini judge that was validated against human labels on only 50 reasoning pairs (F1=0.82) and then applied to the entire test set without variance estimates, per-strategy agreement, or precision/recall. RIV-CoT outputs contain bounding-box coordinates and interleaved image-crop tokens that CoT outputs do not, so the judge's argument-comparison criterion may be systematically easier to satisfy for RIV-CoT even when the conceptual reasoning is wrong. The 4.6-point reasoning-accuracy gain is therefore not established at the claimed magnitude; I recommend validating the judge on a larger, strategy-stratified human sample and reporting a format-controlled condition (e.g., stripping coordinates and crop tokens from generated reasoning before judging).
  3. [Sec. 5.4.1, Table 3] The comparison used to support the 'carefully selected visual patches provide a stronger advantage' claim is between the AnyRes baseline (QPR−EA†, 60.7%) and the oracle-entity model (QPRV−EA, 62.0%). Since the actual retrieval-based system (RIV-CoT) obtains 59.3%, the conclusion as stated is misleading for the proposed method. The claim should be re-scoped to oracle visual patches, or the paper should explain why the predicted-entity setting is the intended operating point and what the AnyRes gap means for it.
minor comments (6)
  1. [Sec. 5.1] The notation for conversation formats (QP−EA, QP−RB−RV−EA, QPR−EA†, etc.) is not defined consistently; in Table 3 the 'R' in QPR−EA† appears to denote image patches rather than the 'relevant entity labels' defined in Sec. 5.1. A summary table of all variants and their input/output components would improve readability.
  2. [Sec. 5.4.1] The statement 'as the maximum number of entities per sample in DRIVING VQA is five, we use N=4' does not justify matching tokens in the AnyRes baseline; the number of visual tokens should be matched per sample, not only at the maximum, and reporting per-sample token counts would strengthen the fairness of this control.
  3. [Appendix C.3] The judge prompt instructs GPT-4o-mini to output '1' or '0'; please report the precision and recall of the judge on the 50 validation pairs, not only the F1 score, and provide per-strategy agreement because the validation set is small and the strategies produce stylistically different outputs.
  4. [Sec. 4.2] The 50% bounding-box expansion factor is a free parameter; a sensitivity analysis (e.g., 0%, 50%, 100%) would strengthen the claim that the crop strategy, rather than this specific value, drives the measured gains.
  5. [Sec. 3.1] The sentence reporting 'an inter-annotator agreement score of 0.95 (Krippendorff's alpha, Fleiss' kappa, and Cohen's kappa)' is ambiguous; these are three different coefficients, and the text should state which coefficient(s) were computed and on what unit (sample-level filtering decisions?).
  6. [Figure 8] Add confidence intervals or error bars to the reasoning-correctness points; currently the figure presents only point estimates from the LLM judge, which makes it impossible to assess the significance of the 4.6-point gap.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RIV-CoT's gains are measured on a held-out test split with objective answer scoring, and no fitted constant or self-citation chain forces the reported improvements.

full rationale

The paper's central derivation chain is self-contained. The DrivingVQA dataset is constructed from external driving-theory exams with human annotations, and the main answer-accuracy results (Table 1) are evaluated on a held-out 20% test split using exact-match exam scoring, repeated over five seeds with reported standard deviations. No parameter is fitted to the test set, and the RIV-CoT method's retrieval of image crops at inference depends on the model's own predicted bounding boxes and the input image, not on a fitted constant. The reasoning-correctness metric uses GPT-4o-mini as an LLM judge that is human-validated on 50 pairs (F1 0.82); while this is a validity and format-bias risk, it is not circular because the judge is not constructed from RIV-CoT's outputs and the comparison against the CoT baseline is not forced by definition. Training explanations for both CoT and RIV-CoT derive from the same human expert explanations, so the relative gain is not an artifact of using the evaluation target as training signal for only one method. There are no load-bearing self-citations or imported uniqueness theorems; the related-work comparisons are standard and the ablations (QP-IEA vs QP-IBEA) explicitly test component contributions. Overall, the paper does not reduce its claims to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method is empirical; the only hand-set constants are the crop expansion factor and the A-OKVQA entity cap. The main assumptions are about the reliability of LLM-generated training signals and the LLM judge, both of which are explicitly stated in the paper.

free parameters (2)
  • Bounding box expansion factor = 1.5x (50% expansion)
    Sec. 4.2: crops are expanded by 50% to add context; this constant is chosen by hand, not swept or fitted.
  • Max entities retained for A-OKVQA pseudo-labels = 5
    Sec. 5.5: the top 5 confidence detections are kept; a hand-set cap matching the maximum in DrivingVQA.
assumptions (3)
  • domain assumption GPT-4o-generated interleaved explanations preserve the reasoning content of the expert-written explanations
    Sec. 4.1 uses GPT-4o to rewrite explanations with bounding boxes; if the rewriting corrupts reasoning, the training signal degrades.
  • domain assumption GPT-4o-mini reasoning judge agrees with human judgment on the full test set
    Sec. 5.4.2 validates the judge on 50 pairs (F1 0.82) and then applies it to all test predictions without further validation or uncertainty reporting.
  • domain assumption GroundingDINO provides adequate open-set localization for pseudo-labeling
    Sec. 3.2 and Sec. 5.5 rely on GroundingDINO to localize entities; errors are only partially corrected by humans on DrivingVQA and not corrected on A-OKVQA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios." pith.science (2026). https://pith.science/paper/EZR5GXKG

@misc{pith2026250104671,
  author       = {Pith},
  title        = {Pith review of: Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZR5GXKG}},
  note         = {Machine review of arXiv:2501.04671}
}
read the original abstract

While chain-of-thought (CoT) prompting improves reasoning in large language models, its effectiveness in vision-language models (VLMs) remains limited due to over-reliance on textual cues and memorized knowledge. To investigate the visual reasoning capabilities of VLMs in complex real-world scenarios, we introduce DrivingVQA, a visual question answering dataset derived from driving theory exams, which contains 3,931 multiple-choice problems with expert-written explanations and grounded entities relevant to the reasoning process. Leveraging this dataset, we propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables VLMs to reason using visual crops corresponding to these relevant entities. Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting. Furthermore, we demonstrate that our method effectively scales to the larger A-OKVQA reasoning dataset by leveraging automatically generated pseudo-labels, outperforming CoT prompting.

Figures

Figures reproduced from arXiv: 2501.04671 by the authors.

Figure 1
Figure 1. Illustration of retrieval-based interleaved visual chain-of-thought in DRIVINGVQA. Successfully answering the question requires detecting relevant entities (e.g., the truck, the car in the rear-view mirror), recognizing their attributes (e.g., the car signaling to overtake), and reasoning spatially to determine whether overtaking is safe. The interleaved explanation provides step-by-step reasoning aligned with visua… view at source ↗
Figure 2
Figure 2. DRIVINGVQA example with a multiple-choice ques￾tion, a set of relevant entities with their coordinates, and an expert￾written explanation describing the situation step by step. reasoning in visual elements. GQA-CoT [41] contains real￾world images paired with their corresponding relevant en￾tities but relies on synthetic explanations and restricts its annotations to a single entity per image. In the field of autonomo… view at source ↗
Figure 4
Figure 4. Illustration of multi-step retrieval-based generation. During inference, starting with a tokenized question and an image tokenized by the adapter on the output of the Vision Encoder, the Large Language Model generates output until it predicts a bounding box. At this point, the generation process pauses to extract the corresponding image crop based on the predicted coordinates. The image crop is encoded and adapted i… view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Conversation formats for different fine-tuning strate￾gies. The model is given an image, a question and possible answer choices and is asked to predict relevant entities labels (QP-REA), or to predict labels and bounding box coordinates, before reason￾ing and answering…
Figure 7
Figure 7. Figure 7: Comparison of zero-shot model performance on the DRIVINGVQA test set. The dashed lines indicate the random baseline (19.8%) and the human baseline (79.6%). epochs, following hyperparameters used by the authors of LLaVA-OV [17]. Evaluation metrics. The performance on th…
Figure 8
Figure 8. Figure 8: Reasoning correctness vs. Exam score for each fine￾tuning strategy. In shades of green are methods using bounding boxes; in shades of red, using image crops. DRIVINGVQA’s ground truth explanations. We use GPT￾4o-mini as an evaluator, leveraging the LLM-as-a-judge parad…
Figure 9
Figure 9. Figure 9: Prompt for relevant entity extraction. B. Examples This section presents four representative examples from DRIVINGVQA dataset. Each example includes an egocen￾4 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Prompt for evaluation of model reasoning. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects

    cs.RO 2025-05 conditional novelty 4.0 of 10

    A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.

Reference graph

Works this paper leans on

70 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015. 2

    work page 2015

  3. [3]

    Response wide shut: Surprising observations in ba- sic vision language model capabilities

    Shivam Chandhok, Wan-Cyuan Fan, and Leonid Si- gal. Response wide shut: Surprising observations in ba- sic vision language model capabilities. arXiv preprint arXiv:2408.06721, 2024. 1

    arXiv 2024

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1, 2, 7

    arXiv 2023

  5. [5]

    Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns

    Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. arXiv preprint arXiv:2403.13315 ,

    arXiv

  6. [6]

    Gpt-4 technical report, 2024

    OpenAI Contributors. Gpt-4 technical report, 2024. 3

    work page 2024

  7. [7]

    Making the V in VQA matter: El- evating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: El- evating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2

    work page 2017

  8. [8]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xue- hao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. 8

    arXiv 2024

Show all 70 references
  1. [9]

    Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  2. [10]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 2

    2023

  3. [11]

    Gqa: A new dataset for real-world visual reasoning and composi- tional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  4. [12]

    Emma: End-to-end multimodal model for autonomous driving

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262,

    arXiv

  5. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  6. [14]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  7. [15]

    Textual explanations for self-driving vehicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In European Conference on Computer Vision (ECCV), 2018. 3, 13

    2018

  8. [16]

    Scaffolding coordinates to promote vision-language coordination in large multi-modal models

    Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. Scaffolding coordinates to promote vision-language coordination in large multi-modal models. arXiv preprint arXiv:2402.12058, 2024. 1, 2

    2024 arXiv

  9. [17]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and 9 Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5, 6

    2024 arXiv

  10. [18]

    From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 13

    2022

  11. [19]

    Chain-of-region: Visual language models need details for diagram analysis

    Xue Li, Yiyou Sun, Wei Cheng, Yinglun Zhu, and Haifeng Chen. Chain-of-region: Visual language models need details for diagram analysis. In The Thirteenth International Con- ference on Learning Representations, 2025. 2

    2025

  12. [20]

    Enhancing advanced visual reasoning ability of large language models

    Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, and Weidong Cai. Enhancing advanced visual reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1915–1929, 2024. 1, 2

    2024

  13. [21]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 1

    2024

  14. [22]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 7

    2024

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 2024. 1

    2024

  16. [24]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 1

    2024 arXiv

  17. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4, 8, 3

    2023 arXiv

  18. [26]

    Aligning with hu- man judgement: The role of pairwise preference in large lan- guage model evaluators

    Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vuli´c, Anna Korhonen, and Nigel Collier. Aligning with hu- man judgement: The role of pairwise preference in large lan- guage model evaluators. arXiv preprint arXiv:2403.16950,

    arXiv

  19. [27]

    Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning

    Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, et al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning. arXiv preprint...

    2025 arXiv

  20. [28]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 2

    2025

  21. [29]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  22. [30]

    Lingoqa: Visual question an- swering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Lingoqa: Visual question an- swering for autonomous driving. In European Conferen...

    2024

  23. [31]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  24. [32]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024. 1, 2

    2024

  25. [33]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 1, 6

    2024

  26. [34]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. 6

    2024

  27. [35]

    Enhancing visual question answering through question-driven image captions as prompts

    ¨Ovg¨u ¨Ozdemir and Erdem Akag ¨und¨uz. Enhancing visual question answering through question-driven image captions as prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1562– 1571, 2024. 1, 2

    2024

  28. [36]

    Piaget’s theory of intelligence

    Jean Piaget. Piaget’s theory of intelligence . Englewood Cliffs, NJ: Prentice Hall, 1978. 1

    1978

  29. [37]

    Cogcom: Train large vision-language models diving into details through chain of manipulations

    Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large vision-language models diving into details through chain of manipulations. arXiv preprint arXiv:2402.04236, 2024. 2

    2024 arXiv

  30. [38]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario,

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario,

  31. [39]

    Prism: A framework for decoupling and assessing the capabilities of vlms

    Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms. Advances in Neural Information Processing Systems, 37:111863–111898, 2025. 2

    2025

  32. [40]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European Conference on Computer Vision (ECCV),

  33. [41]

    Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. arXiv preprint arXiv:2403.16999, 2024. 1, 2, 3, 7, 13

    2024 arXiv

  34. [42]

    Drivelm: Driving with graph visual ques- tion answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual ques- tion answering. In European Conference on Computer Vision (ECCV), 2024. 3, 13

    2024

  35. [43]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE 10 Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  36. [44]

    To cot or not to cot? chain-of-thought helps mainly on math and sym- bolic reasoning

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and sym- bolic reasoning. arXiv preprint arXiv:2409.12183, 2024. 1

    2024 arXiv

  37. [45]

    Vipergpt: Visual inference via python execution for reasoning

    D ´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 2

    2023

  38. [46]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  39. [48]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 5

    2024 arXiv

  40. [49]

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M. Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning, 2024. 3

    2024

  41. [50]

    Drive anywhere: Generalizable end-to-end autonomous driving with multi-modal foundation models,

    Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus. Drive anywhere: Generalizable end-to-end autonomous driving with multi-modal foundation models,

  42. [51]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022. 1

    2022

  43. [52]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 2024. 3

    2024

  44. [53]

    List items one by one: A new data source and learning paradigm for multimodal llms

    An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. List items one by one: A new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404.16375, 2024. 2

    2024 arXiv

  45. [54]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  46. [55]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 5

    2023

  47. [56]

    Multimodal chain-of-thought rea- soning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought rea- soning in language models. Transactions on Machine Learn- ing Research (TMLR), 2024. 1

    2024

  48. [57]

    Multimodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, George Karypis, Alex Smola, et al. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Re- search, 2024. 1, 2

    2024

  49. [58]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023. 2

    2023

  50. [59]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 8 11 Retriev...

    2023

  51. [60]

    Convert the original reasoning into a clear, step- by-step reasoning that makes use of all entities in the list

  52. [61]

    If the original reasoning does not mention one of the entities at all, add a short sentence refer- encing that entity and link it with the reasoning steps

  53. [62]

    Replicate bound- ing box coordinates exactly as provided in the list

    Refer to each relevant entity in the format: **en- tity name** [x1, x2, y1, y2]. Replicate bound- ing box coordinates exactly as provided in the list

  54. [63]

    Keep all other wording as close to the original reasoning as possible

  55. [64]

    1 Demonstration 1 Question: [”I am turning right at the next intersec- tion

    The entities should always be mentioned at the beginning of the sentences. 1 Demonstration 1 Question: [”I am turning right at the next intersec- tion. My attention is focused on: pedestrians”, ”I am turning right at the next intersection. My atten- tion is focused on: cyclist...

  56. [65]

    A driving theory test question

  57. [66]

    A list of possible answer options

  58. [67]

    correct reasoning

    The official “correct reasoning”

  59. [68]

    Step-by-Step Instructions:

    A student’s reasoning for the same question Your task: Assess if the student’s reasoning matches the correct reasoning. Step-by-Step Instructions:

  60. [69]

    • For each argument, briefly state whether it is correct or not, given the provided correct reasoning

    Identify Student’s Arguments • List each key argument or step in the student’s reasoning. • For each argument, briefly state whether it is correct or not, given the provided correct reasoning

  61. [70]

    • List important points or steps from the correct reasoning that the student omits or directly contradicts

    Check for Missing or Contradictory Points • Look at the official correct reasoning. • List important points or steps from the correct reasoning that the student omits or directly contradicts. • Minor omissions or differences in wording/style are acceptable

  62. [71]

    1” if you judge the student’s reasoning is overall correct, “0

    Decide on Overall Correctness • If the student’s reasoning is mostly consistent with the correct reasoning and hasno major factual errors, then it is considered correct. • If the student’s reasoning contains significant logical or factual errors, or omits critical steps from t...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.