Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Visually Interpretable Subtask Reasoning for Visual Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VISTAR fine-tunes a multimodal large language model on structured Subtask-of-Thought rationales so that answering a visual question produces an ordered chain of sub-task operations, intermediate answers, and object bounding boxes—and this…

desk verdict VISTAR is a clear, modest step beyond VPD—train an MLLM to emit bbox-grounded Subtask-of-Thought rationales without external execution—but the interpretability claim is weakened by same-generator evaluation and answer-conditioned rationale generation. read the letter →

arxiv 2505.08084 v1 pith:UZKKZVFG submitted 2025-05-12 cs.CV

Yu Cheng , Arushi Goel , Hakan Bilen This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringinterpretabilitysubtaskreasoninginstructiontuningmultimodallargelanguagemodelgroundingchain-of-thoughtGQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VISTAR asks whether a vision-language model can explain itself step by step without losing accuracy, and answers yes. The paper proposes training an MLLM to output a Subtask-of-Thought rationale—naming each object it selects, each attribute it filters, and each relation it checks, with a bounding box for every object—instead of relying on external program execution. The training data is synthesized by an LLM from GQA scene graphs, sub-task operation sequences, and ground-truth answers, then used to fine-tune NVILA-8B on the full rationale. On GQA test the fine-tuned model reaches 65.1% accuracy versus 64.0% for the base model, and on a zero-shot CRIC sample it reaches 61.1% versus 60.8%. A sympathetic reader would care because this is evidence that interpretability can be a training target rather than a post-hoc add-on that degrades performance.

What carries the argument

The central object is the Subtask-of-Thought (SoT) rationale, an ordered set $S=\{(op_1,r_1),\dots,(op_n,r_n)\}$ in which each $op_i$ is a sub-task operation and each $r_i$ is its intermediate result, with the final result $r_n$ being the answer. The machinery that carries the argument is an LLM data-synthesis step that converts GQA's scene graphs and sub-task operation sequences into SoT annotations, followed by visual instruction tuning in which the MLLM is trained, under the prompt 'Explain the rationale by decomposing into subtasks to answer the question,' to predict the entire SoT. This design does the work: reasoning steps and bounding boxes become supervised output tokens, so interpretability is learned inside the model rather than obtained by executing external programs.

What would settle it

Train VISTAR on rationales generated without the ground-truth answer in the LLM prompt, or with a deliberately wrong answer, and compare final accuracy and bounding-box IoU. If accuracy and grounding survive, the rationales carry real visual evidence; if they collapse or the boxes drift while answers stay correct, the supervision is answer-conditioned narration rather than visual reasoning.

Watch

Extended reading notes

Core claim

The central claim is that an MLLM can be trained to produce both the final answer and a full visual-textual rationale, and that doing so improves compositional VQA accuracy. A Subtask-of-Thought is an ordered set of reasoning steps, each pairing a sub-task operation (select, relate, filter, verify, query, and, or, ...) with an intermediate result that is an object name plus a normalized bounding box, an attribute value, or a boolean. The training signal is generated by an LLM prompted with the question, the scene graph, the GQA sub-task operation sequence, and the ground-truth answer, followed by filtration of rationales whose final answer mismatches the ground truth or whose format is malformed. The paper then fine-tunes NVILA-8B with a next-token-prediction loss over the full rationale, and reports a consistent accuracy gain on GQA (65.1% versus 64.0%), a smaller zero-shot gain on CRIC (61.1% versus 60.8%), and improved object grounding measured by IoU, precision, and recall.

Load-bearing premise

The load-bearing premise is that the LLM-generated rationales, produced from a scene graph with the correct answer included in the prompt, are faithful traces of visual reasoning; if the LLM only writes plausible-sounding steps consistent with the known answer, the interpretability the paper measures is not independent evidence of genuine visual reasoning.

Editorial extensions

If this is right

  • VISTAR answers with an auditable trace: each intermediate answer and its bounding box can be inspected, so a wrong final answer can be traced to a specific sub-task failure.
  • Because the full SoT is generated by the model itself, the method removes the per-query cost of running object detectors or code interpreters at inference time.
  • The reported GQA and CRIC results imply that structured sub-task supervision can improve accuracy while adding explanation, reversing the usual explanation penalty observed in base MLLMs.
  • The same next-token training recipe can in principle be applied to any MLLM backbone, so the interpretability gain is not tied to a specific architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the rationales are faithful, the answer-conditioned generation of training data is the main bottleneck; a variant that removes the ground-truth answer from the LLM prompt would test whether the model learns genuine visual search or answer-consistent narration.
  • The bounding-box tokens may act as an implicit grounding regularizer, so one could ablate coordinates only (keeping object names) to see whether visual grounding and final accuracy move together.
  • Because the paper notes that OCR operations were absent from GQA training, the method's boundary can be probed by adding OCR-style sub-task operations to a synthetic dataset and checking whether VISTAR-style training picks them up.
  • An untested scaling hypothesis is that longer, more detailed SoT rationales improve accuracy the way longer chain-of-thought traces do in text-only LLMs; measuring accuracy as a function of rationale length would connect this work to that pattern.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VISTAR, a framework that fine-tunes multimodal large language models (MLLMs) on GQA-derived "Subtask-of-Thought" (SoT) rationales. SoT rationales are generated by LLaMA-3.1-70B from GQA scene graphs, operation sequences, and ground-truth answers, and are formatted as sequences of sub-task operations with intermediate textual answers and bounding boxes. VISTAR then fine-tunes NVILA-8B to produce such rationales at inference, reporting 65.1% accuracy on GQA test versus 64.0% for the base model, and 61.1% versus 60.8% in zero-shot evaluation on CRIC. The paper also reports interpretability metrics: object-level bounding-box precision/recall for visual explanations, and GPT-4-based operation accuracy and logical accuracy for textual rationales, plus a human evaluation on about 300 samples.

Significance. The paper makes a useful practical contribution: it offers a concrete recipe for turning scene-graph-derived rationales into an instruction-tuning dataset, and it shows that an MLLM can be trained to emit structured, bounding-box-grounded reasoning steps without external program execution at inference. The final-answer accuracy claims are grounded in external GQA and CRIC labels, so they are not circular. The bounding-box evaluation is also partly independent because it is scored against GQA scene-graph annotations. The main risk is that the textual "reasoning" component is validated against rationales generated by the same LLM and prompt that created the training targets, so the interpretability claim is only partially supported. In addition, the reported accuracy gains are small and are not accompanied by variance or significance estimates.

major comments (3)
  1. [Section 3.2 and Section 5.3] The interpretability evaluation of the textual rationales is partly circular. In Section 3.2, the SoT generation prompt includes the ground-truth answer y "to guide the expected reasoning process," and in Section 5.3 the predicted SoTs are compared against validation SoTs produced by the same LLM and prompt. Therefore, the reported Logical Acc (90.3) and the GPT-4-based semantic similarity scores largely measure agreement with the answer-conditioned generator's style, not whether the intermediate textual answers are the steps a visual reasoner would take from the image alone. The bounding-box scores are more independent because they are compared with GQA scene-graph boxes, but the intermediate textual answers, which constitute the "reasoning" part of the SoT, are not independently validated. I would ask for a human faithfulness study in which annotators judge whether each intermediate answer is supported by the image without being shown the final answer, or for an independent source of reasoning annotations.
  2. [Section 5.2, Tables 2 and 3] The accuracy improvements are small (+1.1 on GQA, +0.3 on CRIC) and are reported without error bars, multiple seeds, or significance tests. Given this scale, the abstract's statement that VISTAR "consistently improves reasoning accuracy" is not yet supported. The CRIC comparison is also only against NVILA-8B. Please report variance over at least three fine-tuning seeds and/or bootstrap confidence intervals, and include additional zero-shot baselines on CRIC if the generalization claim is to be made.
  3. [Section 5.3, Human evaluation on SoT] The human evaluation is based on the stated assumption that when the final predicted answer is correct, the sub-task operations and intermediate answers should also be correct. This assumption checks internal consistency between the final answer and the reasoning steps, but it does not establish that the intermediate answers are faithful to the image. The reported 85.2% consistency ratio is therefore not evidence of visually interpretable reasoning in the sense claimed in the title. In addition, the annotation protocol for the roughly 300 samples is not described; please report the exact instructions given to annotators, the number of annotators, and inter-annotator agreement.
minor comments (5)
  1. [Table 2] The row "PaLI-3-VPD (55B) [14]" cites reference [14] (GQA), but this result should cite the VPD paper [12]; please correct the citation.
  2. [Figure 2 caption] The caption contains a typo: "Overview of VISATR" should be "Overview of VISTAR."
  3. [Appendix E, Figure 8 and Section 3.2 example] The output text uses "coach" where the intended object is "couch"; please fix this typo consistently in both the figure and the main text example.
  4. [Appendix B] The prompt description refers to LLaMA-3.1-70B-Instruct but cites reference [33], which is the original LLaMA paper; a citation for LLaMA-3.1 is needed.
  5. [Table 1] The row for "Test-dev-balanced" lists "0.3K" under images and "12.5K" under questions, which is unclear; please clarify the units and the relationship to the test-dev split used in Table 2.

Circularity Check

2 steps flagged · score 4.0 of 10

Final-answer accuracy is externally grounded, but the textual-reasoning interpretability claim is partly circular: SoT rationales are generated with the ground-truth answer in the prompt and then validated against the same generator's outputs.

  1. self definitional [Section 3.2 (SoT generation), prompt design; Figure 2 caption]
    "we obtain the corresponding SoT by predicting the missing intermediate answers {r_i}_{i=1}^{N-1} where r_N = y ... The ground truth answer y: To guide the expected reasoning process. ... VISTAR uses an LLM to generate faithful SoTs via in-context learning."

    The intermediate answers are produced by an LLM that is explicitly given the ground-truth answer y and the scene graph, with y included 'to guide the expected reasoning process', and the filtration stage removes generated SoTs whose final answer does not match y. The surviving rationales are therefore answer-conditioned by construction: they are post hoc chains built to lead to y, not independently verified visual reasoning. The paper calls these 'faithful SoTs' and trains the MLLM to reproduce them, so the claimed visual interpretability of the textual steps rests on labels whose faithfulness was never independently established.

  2. fitted input called prediction [Section 4 (GQA-SoT) and Section 5.3 (Evaluation on interpretability)]
    "we generate 249K SoT annotations on the GQA dataset for fine-tuning, and 132K for validation ensuring quality evaluation of predicted SoT from the fine-tuned model ... we evaluate the semantic similarity, operation accuracy without arguments and logical consistency of our model's predicted SoT with the ground truth annotations, also leveraging GPT-4-turbo."

    The validation 'ground truth' SoTs are generated by the same LLaMA-3.1-70B prompt pipeline, including the ground-truth answer, that created the training targets. Logical Acc and semantic-similarity scores therefore measure how closely the model imitates the answer-conditioned generator, not whether the intermediate reasoning steps are independently correct for the image. The sub-task operation labels come from GQA and the bounding boxes from the scene graph, so Op. Acc and IoU retain independent content; but the textual reasoning claim is evaluated against self-generated labels, making that part of the interpretability result circular.

full rationale

The paper's headline accuracy results are not circular: GQA test accuracy (65.1 vs 64.0) and zero-shot CRIC accuracy (61.1 vs 60.8) are scored against external answer labels, and the bounding-box targets are GQA scene-graph annotations, so the visual grounding numbers have independent content. The circularity is confined to the textual SoT interpretability evaluation. Section 3.2 constructs the SoT training data by prompting LLaMA-3.1-70B with the ground-truth answer 'to guide the expected reasoning process'; the resulting rationales are answer-conditioned post hoc explanations. Section 4 then uses the same generation pipeline to create the 132K validation SoTs that Section 5.3 treats as ground truth for Logical Acc and semantic similarity. Thus the logical-accuracy and intermediate-answer-agreement metrics largely measure fit to the same answer-conditioned generator. The human study (Section 5.3) explicitly assumes that a correct final answer implies correct sub-task steps, so it checks internal consistency rather than faithfulness to the image. No self-citation chain or imported uniqueness theorem is load-bearing. Overall, the central accuracy claim and the object-localization results are independent; only the claim that the textual step-by-step reasoning is visually faithful is partially circular, warranting a moderate score of 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The method relies on GQA's scene graphs and operation sequences, on LLM-generated rationales that are filtered only by answer agreement, and on GPT-4 as a judge. None of these are independently verified, and the validation rationales are generated by the same pipeline as the training rationales, which limits the interpretability claims.

free parameters (3)
  • SoT filtration length threshold
    Used in Section 3.2 step 2 to discard overly long rationales; the threshold is not specified and affects dataset composition.
  • Question sampling balance for GQA-SoT
    Section 4 says questions are 'selectively sampled to ensure balanced coverage', but the exact sampling distribution or weights are not reported; this determines which 249K of 994K question-answer pairs become training rationales.
  • Fine-tuning hyperparameters = lr=1.5e-5, warmup=0.3, batch=128, 1 epoch
    Standard hand-chosen training settings reported in Section 5.1; they are not fitted to the benchmarks but still influence the final model behavior.
assumptions (4)
  • domain assumption GQA scene graphs and 145 sub-task operation sequences are accurate and sufficient to derive correct rationales.
    SoT generation in Section 3.2 depends entirely on these annotations; any incompleteness or noise propagates into training data and validation targets.
  • ad hoc to paper LLM-generated SoTs that survive filtration are faithful reasoning chains.
    Filtration only checks final-answer match, argument format, and length (Section 3.2, step 2), not the correctness of intermediate bounding boxes or operation arguments.
  • domain assumption GPT-4-turbo similarity and logical consistency judgments are valid proxies for reasoning quality.
    Textual interpretability metrics in Table 4 rely on GPT-4-turbo without reporting prompt details, calibration, or human agreement rates.
  • domain assumption NVILA-8B is a representative strong base model and fine-tuning on GQA-SoT preserves its general abilities.
    Only GQA and a CRIC subset are evaluated; no standard VQA or grounding benchmarks are reported to check for regression outside these sets.
invented entities (2)
  • Subtask-of-Thought (SoT) rationales
    purpose: A structured reasoning format in which each VQA step has an operation, an intermediate text answer, and a bounding box; used as training and evaluation targets.
    No independent human-verified grounding for the format; the validation rationales are produced by the same LLM pipeline as the training data.
  • GQA-SoT dataset
    purpose: The filtered LLM-generated SoT annotations used to fine-tune NVILA-8B.
    The dataset is not released at the time of review and was not independently validated; its quality rests on the GQA scene graphs and LLM prompt in Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visually Interpretable Subtask Reasoning for Visual Question Answering." pith.science (2026). https://pith.science/paper/UZKKZVFG

@misc{pith2026250508084,
  author       = {Pith},
  title        = {Pith review of: Visually Interpretable Subtask Reasoning for Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZKKZVFG}},
  note         = {Machine review of arXiv:2505.08084}
}
read the original abstract

Answering complex visual questions like `Which red furniture can be used for sitting?' requires multi-step reasoning, including object recognition, attribute filtering, and relational understanding. Recent work improves interpretability in multimodal large language models (MLLMs) by decomposing tasks into sub-task programs, but these methods are computationally expensive and less accurate due to poor adaptation to target data. To address this, we introduce VISTAR (Visually Interpretable Subtask-Aware Reasoning Model), a subtask-driven training framework that enhances both interpretability and reasoning by generating textual and visual explanations within MLLMs. Instead of relying on external models, VISTAR fine-tunes MLLMs to produce structured Subtask-of-Thought rationales (step-by-step reasoning sequences). Experiments on two benchmarks show that VISTAR consistently improves reasoning accuracy while maintaining interpretability. Our code and dataset will be available at https://github.com/ChengJade/VISTAR.

Figures

Figures reproduced from arXiv: 2505.08084 by the authors.

Figure 1
Figure 1. Comparison of standard MLLMs, programmatic reasoning and VISTAR across the output.(a) Existing methods either provide no explanations, lack visual grounding, or have high computational costs. (b) Performance degradation of standard MLLMs (LLaVA-1.5-7B [21] and NVILA-8B [23]) when forced to generate explanations alongside answers. ‘Exp-GPT-4’ evaluates semantic similarity using GPT-4-turbo [1]. (c) VISTAR effectively… view at source ↗
Figure 2
Figure 2. Overview of VISATR. VISTAR uses an LLM to generate faithful SoTs via in-context learning. Given the input including the query, ground truth, scene graph in an image and sub-task operation sequence inside the dataset (top), LLaMA-3.1-70B-Instruct would output a SoT to answer the query. The generated SoT is then used to fine-tune an MLLM, enabling it to produce both visual (object-level bounding boxes) and textual exp… view at source ↗
Figure 3
Figure 3. Quantitative results of visual explanations. (a) Comparison of answer correctness and object localization accuracy (measured by IoU when the answer is correct). (b-d) Precision & recall at different IoU thresholds (0.5, 0.75, 0.95). VISTAR consistently outperforms NVILA-8B in both accuracy and object-level visual grounding, demonstrating improved interpretability and localization quality. evaluate the quality of tex… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples of successful and failed cases on human evaluation. We present some successful and failed cases for our SoT prediction based on VISTAR. operation sequences, including arguments and intermediate results, should also be high. Conversely, when the final an￾swer i…
Figure 5
Figure 5. Figure 5: Consistency evaluation between answers and reasoning steps. (a) Ablation results without intermediate answer supervision, evaluated by GPT-4-turbo over operation and arguments with answers (b) Results from the full training procedure, evaluated by GPT-4- turbo over ope…
Figure 6
Figure 6. Figure 6: Examples of LLM-generated Subtask-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Instruction for bounding box prediction during inference E. Example of converting sub-task operations into subtask-of-thought In the GQA [14] dataset, they provide sub-task operation sequences, consisting of the sub-task operation and the correspond￾ing arguments. We r…
Figure 8
Figure 8. Figure 8: Example of sub-task steps in GQA [14] dataset and our subtask-of-thought formate. For the operations of the given query, it means first we need to identify the garland in the image, and then find the curtain positioned to the right of the garland. Next, we can locate f…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 44 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 1, 7

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayi- heng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng ...

    arXiv 2023

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz L...

    arXiv 2005

  4. [4]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde De Oliveira Pinto, Jared Kaplan, Harri Ed- wards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv, abs/2107.03374, 2021. 3

    arXiv 2021

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2

    2023

  6. [6]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, pages 49250–49267. Curran Associates, Inc., 2023. 2, 5, 6

    work page 2023

  7. [7]

    Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. ArXiv, abs/2411.14432, 2024. 2

    arXiv 2024

  8. [8]

    Cric: A vqa dataset for compositional reasoning on vision and commonsense

    Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. Cric: A vqa dataset for compositional reasoning on vision and commonsense. IEEE TPAMI, 45(5):5561–5578, 2023. 5, 6

    work page 2023

Show all 63 references
  1. [9]

    Omnifusion technical report

    Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, and An- drey Kuznetsov. Omnifusion technical report. ArXiv, abs/2404.06212, 2024. 1

    2024 arXiv

  2. [10]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In CVPR, 2017. 8

    2017

  3. [11]

    Visual pro- gramming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual pro- gramming: Compositional visual reasoning without training. CVPR, pages 14953–14962, 2022. 2, 3

    2022

  4. [12]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. CVPR, pages 9590–9601, 2023. 2, 3, 5, 6

    2023

  5. [13]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. CVPR, pages 14271–14280, 2023. 1, 2

    2023

  6. [14]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, pages 6693–6702, 2019. 3, 5, 6, 8, 11

    2019

  7. [15]

    Seeing the unseen: Visual metaphor cap- tioning for videos

    Abisek Rajakumar Kalarani, Pushpak Bhattacharyya, and Sumit Shekhar. Seeing the unseen: Visual metaphor cap- tioning for videos. ArXiv, abs/2406.04886, 2024. 1

    2024 arXiv

  8. [16]

    Hydra: A hyper agent for dynamic compositional visual reasoning

    Fucai Ke, Zhixi Cai, Simindokht Jahangard, Weiqing Wang, Pari Delir Haghighi, and Hamid Rezatofighi. Hydra: A hyper agent for dynamic compositional visual reasoning. InECCV, pages 132–149. Springer, 2024. 3

    2024

  9. [17]

    Exploring question decomposition for zero-shot vqa

    Zaid Khan, Vijay Kumar BG, Samuel Schulter, Manmohan Chandraker, and Yun Fu. Exploring question decomposition for zero-shot vqa. NeurIPS, 36:56615–56627, 2023. 3

    2023

  10. [18]

    A survey on benchmarks of multi- modal large language models.ArXiv, abs/2408.08632, 2024

    Jian Li and Weiheng Lu. A survey on benchmarks of multi- modal large language models.ArXiv, abs/2408.08632, 2024. 2

    2024 arXiv

  11. [19]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 2, 3

    2023

  12. [20]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. CVPR, pages 10955–10965, 2021. 3

    2021

  13. [21]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CVPR, pages 26286–26296, 2023. 1, 2, 5, 6

    2023

  14. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892–34916, 2023. 1, 2

    2023

  15. [23]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yux- ian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vish- wesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xia...

    2024 arXiv

  16. [24]

    Sae-v: Interpreting multimodal models for enhanced align- ment

    Hantao Lou, Changye Li, Jiaming Ji, and Yaodong Yang. Sae-v: Interpreting multimodal models for enhanced align- ment. ArXiv, abs/2502.17514, 2025. 2

    2025 arXiv

  17. [25]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiao- juan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In ECCV, pages 417–

  18. [26]

    Task navigator: Decomposing complex tasks for multimodal large language models

    Feipeng Ma, Yizhou Zhou, Yueyi Zhang, Siying Wu, Zheyu Zhang, Zilong He, Fengyun Rao, and Xiaoyan Sun. Task navigator: Decomposing complex tasks for multimodal large language models. CVPRW, pages 2248–2257, 2024. 3

    2024

  19. [27]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2

    2019

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2

    2021

  21. [29]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 44:1623–1637, 2019. 3

    2019

  22. [30]

    Reid, and Silvio Savarese

    Seyed Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese. General- ized intersection over union: A metric and a loss for bound- ing box regression. CVPR, pages 658–666, 2019. 6

    2019

  23. [31]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InCVPR, pages 8317–8326, 2019. 8

    2019

  24. [32]

    Vipergpt: Visual inference via python execution for reasoning

    D’idac Sur’is, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. ICCV, pages 11854–11864, 2023. 2, 3, 5, 6

    2023

  25. [33]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur´elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langu...

    2023 arXiv

  26. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  27. [35]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Lian zi Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tieju...

    2024 arXiv

  28. [36]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. ICCV, pages 11941–11952, 2023. 2

    2023

  29. [37]

    Visual question decomposition on multimodal large language mod- els

    Haowei Zhang, Jianzhe Liu, Zhen Han, Shuo Chen, Bailan He, V olker Tresp, Zhiqiang Xu, and Jindong Gu. Visual question decomposition on multimodal large language mod- els. ArXiv, abs/2409.19339, 2024. 3

    2024 arXiv

  30. [38]

    Ferret-v2: An improved baseline for refer- ring and grounding with large language models

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for refer- ring and grounding with large language models. ArXiv, abs/2404.07973, 2024. 1, 2

    2024 arXiv

  31. [39]

    Mllms know where to look: Training-free per- ception of small visual details with multimodal llms

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Mllms know where to look: Training-free per- ception of small visual details with multimodal llms. ArXiv, abs/2502.17422, 2025. 2

    2025 arXiv

  32. [40]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304.10592, 2023. 2, 3, 5, 6 A. Examples Outputs of Subtask-of-Thought We present some examples of LLM-gene...

    2023 arXiv

  33. [41]

    select: { Input: ([Object]), Output: [Object 1 (location), Object 2 (location), ...] }

  34. [42]

    exist: { Input: ([Object_List]), Output: [Boolean/None] }

  35. [43]

    relate: { Input: (Subject, Relation, Object)/(Subject, Relation) /(Relation, Object), Output: [Object 1 (location), Object 2 (location), ...] }

  36. [44]

    filter: { Input: ([Object_List], Attribute), Output: [Object 1 (location), Object 2 (location), ...] }

  37. [45]

    verify: { Input: ([Object_List], Attribute), Output: [Boolean] },

  38. [46]

    verify rel: { Input: (Subject, Relation, Object), Output: [Boolean] },

  39. [47]

    choose name, hposition (means horizontal position), vposition (means vertical position), material, location, color, place and other attribute: { Input: (Object, Choice1|Choice2), Output: [Right Choice] }

  40. [48]

    choose rel: { Input: (Subject, Choice1|Choice2)/(Choice1|Choice2, Object), Output: [Right Choice] }

  41. [49]

    and: { Input: (Attribute, Attribute), Output: [Boolean] }

  42. [50]

    or: { Input: (Attribute, Attribute), Output: [Boolean] }

  43. [51]

    common: { Input: (Object, Object), Output: [Attribute] }

  44. [52]

    query: { Input: ([Object_List], Attribute), Output: [Attribute] }

  45. [53]

    compare: { Input: (Object, Object, Attribute), Output: [Attribute] }

  46. [54]

    same color, shape, material, attr: { Input: (Object, Object), Output: [Boolean] }

  47. [55]

    same: { Input: ([Object_List], Attribute), Output: [Boolean] }

  48. [56]

    different color, shape, material, attr: { Input: (Object, Object), Output: [Boolean] }

  49. [57]

    operation

    different: { Input: ([Object_List], Attribute), Output: [Boolean] } Examples: Question: Do the bananas to the left of the plantains look large and yellow? Operation: [ {"operation": "select", "dependencies": [], "argument": "plantains (681259)"}, {"operation": "relate", "depen...

  50. [58]

    operation

    Important: { "operation":"choose rel","dependencies":[0], "argument":"rice,to the left of|to the right of,s (681260)" } represents ’rice’ is the Subject (681260) of Relation ’to the left of|to the right of’, should be { "choose rel(rice, to the left of|to the right of, [’The O...

  51. [59]

    Important: For all operations involving selection, like ’select(person)’, Answer should include all related objects for the target, including singular and plural in naming or attributes, When the question or operation specifies a relationship or context, such as Operation ’rel...

  52. [60]

    Important: If the Object does not exist, the answer should be [None]

  53. [61]

    Noticeably, For Operation ’choose rel’ in the last step, keeping identical with Final Answer

    Important: The final answer should always match the logical conclusion drawn from the operations and their reasoning steps. Noticeably, For Operation ’choose rel’ in the last step, keeping identical with Final Answer. For example, choose rel(Subject, to the left of|to the righ...

  54. [62]

    Important: Answer should align with Given Relations and Locations in Object Description

  55. [63]

    attribute value

    Important: For Operation ’select(scene)’ should return ’there are’ with key object IDs without bounding boxes, which are necessary to answer the question like ’there are [#2, #3, #5]’ where #2, #3, #5 are key objects ID to answer the question and should be the argument for the...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.