{"id":"9542af6f-e0e0-407a-8856-de1e3cc645ec","arxiv_id":"2505.08084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a visual question answering model on LLM-generated subtask rationales with bounding boxes improves GQA accuracy from 64.0 to 65.1 percent and adds grounded explanations.","lead":"VISTAR is a training method that teaches a multimodal AI model to answer visual questions by first writing out step-by-step subtask rationales with bounding boxes. This paper reports small accuracy gains on two question-answering benchmarks and argues that transparent, step-by-step explanations do not have to come at the cost of accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-conditioned SoT generation plus same-generator evaluation leaves the faithfulness of the textual reasoning steps unestablished, even though bbox grounding is independently measured.","rationale":"The reader's weakest assumption identifies the right vulnerability. VISTAR's distinctive contribution is joint accuracy and interpretability via structured rationales; the accuracy gains are small (+1.1 GQA, +0.3 CRIC) and the CRIC gain alone is within typical run-to-run noise, but the more decisive issue is that the reasoning-chain evidence is partly circular: the intermediate answers are generated by an LLM that knows the final answer, and the validation labels come from that same generator. I would not move to REJECT because the method is clearly specified, the bbox results use gold GQA scene-graph coordinates, and the operation-sequence labels are gold, so parts of the interpretability claim have independent support. The proposed experiment directly tests whether answer-conditioning changes the rationales and whether the accuracy benefit depends on it; combined with the code/data release and significance testing the reader already requested, it would either support or refute the central claim. A direct-answer fine-tuning control would also strengthen attribution of the accuracy gain to the SoT format rather than to in-domain fine-tuning. Hence the verdict stays CONDITIONAL.","tokens_in":18478,"tokens_out":13754,"duration_ms":137783,"concrete_test":"On a random 2K-question subset of GQA-SoT, regenerate SoTs with LLaMA-3.1-70B while withholding the ground-truth answer from the prompt (keeping Q, SG, and the gold operation sequence). Measure how often the LLM's final answer still matches y and how often the intermediate answers match the original answer-conditioned SoTs. Then fine-tune NVILA-8B on this answer-agnostic subset and compare GQA test accuracy and human faithfulness ratings (annotators mark each intermediate answer as supported or unsupported by the image) against answer-conditioned VISTAR. If the answer-agnostic model keeps the accuracy gain but its rationales diverge from the original labels, the answer-conditioned rationales are style artifacts; if it loses accuracy, the reported gain depends on answer-conditioned data. Either outcome resolves the circularity concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central interpretability claim requires that the intermediate answers in each Subtask-of-Thought (SoT) are the steps a visual reasoner would actually take from the image. In Section 3.2, LLaMA-3.1-70B is prompted with the ground-truth answer y (\"to guide the expected reasoning process\") and the scene graph, then asked to fill in the missing intermediate answers; this setup is prone to producing post hoc rationalizations that merely lead to y rather than chains the image alone would support. In Section 5.3, predicted SoTs are validated against ground-truth SoTs produced by the same LLM and prompt; Op. Acc (98.5) is anchored to GQA's gold operation labels and is informative, but Logical Acc (90.3) and intermediate-answer agreement largely measure consistency with the generator's answer-conditioned style, not independent visual correctness. The bbox targets are GQA scene-graph annotations, so the visual grounding result is partially independent; however, the intermediate textual answers—the \"reasoning\" part of SoT—are not independently validated. The human study samples roughly 300 cases and, by the authors' stated assumption, only checks consistency between sub-task steps and a correct final answer, not faithfulness to the image. This leaves the core \"visually interpretable reasoning\" claim unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VISTAR, a framework that fine-tunes multimodal large language models (MLLMs) on GQA-derived \"Subtask-of-Thought\" (SoT) rationales. SoT rationales are generated by LLaMA-3.1-70B from GQA scene graphs, operation sequences, and ground-truth answers, and are formatted as sequences of sub-task operations with intermediate textual answers and bounding boxes. VISTAR then fine-tunes NVILA-8B to produce such rationales at inference, reporting 65.1% accuracy on GQA test versus 64.0% for the base model, and 61.1% versus 60.8% in zero-shot evaluation on CRIC. The paper also reports interpretability metrics: object-level bounding-box precision/recall for visual explanations, and GPT-4-based operation accuracy and logical accuracy for textual rationales, plus a human evaluation on about 300 samples.","tokens_in":18764,"tokens_out":3361,"duration_ms":33505,"significance":"The paper makes a useful practical contribution: it offers a concrete recipe for turning scene-graph-derived rationales into an instruction-tuning dataset, and it shows that an MLLM can be trained to emit structured, bounding-box-grounded reasoning steps without external program execution at inference. The final-answer accuracy claims are grounded in external GQA and CRIC labels, so they are not circular. The bounding-box evaluation is also partly independent because it is scored against GQA scene-graph annotations. The main risk is that the textual \"reasoning\" component is validated against rationales generated by the same LLM and prompt that created the training targets, so the interpretability claim is only partially supported. In addition, the reported accuracy gains are small and are not accompanied by variance or significance estimates.","major_comments":[{"comment":"The interpretability evaluation of the textual rationales is partly circular. In Section 3.2, the SoT generation prompt includes the ground-truth answer y \"to guide the expected reasoning process,\" and in Section 5.3 the predicted SoTs are compared against validation SoTs produced by the same LLM and prompt. Therefore, the reported Logical Acc (90.3) and the GPT-4-based semantic similarity scores largely measure agreement with the answer-conditioned generator's style, not whether the intermediate textual answers are the steps a visual reasoner would take from the image alone. The bounding-box scores are more independent because they are compared with GQA scene-graph boxes, but the intermediate textual answers, which constitute the \"reasoning\" part of the SoT, are not independently validated. I would ask for a human faithfulness study in which annotators judge whether each intermediate answer is supported by the image without being shown the final answer, or for an independent source of reasoning annotations.","section":"Section 3.2 and Section 5.3"},{"comment":"The accuracy improvements are small (+1.1 on GQA, +0.3 on CRIC) and are reported without error bars, multiple seeds, or significance tests. Given this scale, the abstract's statement that VISTAR \"consistently improves reasoning accuracy\" is not yet supported. The CRIC comparison is also only against NVILA-8B. Please report variance over at least three fine-tuning seeds and/or bootstrap confidence intervals, and include additional zero-shot baselines on CRIC if the generalization claim is to be made.","section":"Section 5.2, Tables 2 and 3"},{"comment":"The human evaluation is based on the stated assumption that when the final predicted answer is correct, the sub-task operations and intermediate answers should also be correct. This assumption checks internal consistency between the final answer and the reasoning steps, but it does not establish that the intermediate answers are faithful to the image. The reported 85.2% consistency ratio is therefore not evidence of visually interpretable reasoning in the sense claimed in the title. In addition, the annotation protocol for the roughly 300 samples is not described; please report the exact instructions given to annotators, the number of annotators, and inter-annotator agreement.","section":"Section 5.3, Human evaluation on SoT"}],"minor_comments":[{"comment":"The row \"PaLI-3-VPD (55B) [14]\" cites reference [14] (GQA), but this result should cite the VPD paper [12]; please correct the citation.","section":"Table 2"},{"comment":"The caption contains a typo: \"Overview of VISATR\" should be \"Overview of VISTAR.\"","section":"Figure 2 caption"},{"comment":"The output text uses \"coach\" where the intended object is \"couch\"; please fix this typo consistently in both the figure and the main text example.","section":"Appendix E, Figure 8 and Section 3.2 example"},{"comment":"The prompt description refers to LLaMA-3.1-70B-Instruct but cites reference [33], which is the original LLaMA paper; a citation for LLaMA-3.1 is needed.","section":"Appendix B"},{"comment":"The row for \"Test-dev-balanced\" lists \"0.3K\" under images and \"12.5K\" under questions, which is unclear; please clarify the units and the relationship to the test-dev split used in Table 2.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the textual interpretability claim rests on an evaluator that shares the generator with the training data. This is fixable with additional experiments, but without them the central interpretability claim is not established. The small accuracy gains also need statistical support. I would not recommend acceptance in the current form, but the approach is promising and the bounding-box evaluation is a strong point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable, clearly written paper that moves past VPD by training an MLLM to emit bbox-grounded Subtask-of-Thought (SoT) rationales directly, with no external program execution. The accuracy gains are small but real, and the method description is good enough to reproduce in principle. The interpretability story, however, is weaker than the paper claims, and the stress-test concern lands.\n\nWhat is new and done well: the GQA-SoT dataset (249K training SoTs with bounding boxes), the LLM-prompted generation pipeline, and the instruction tuning that lets NVILA-8B produce structured reasoning plus visual grounding. The bbox grounding is assessed against GQA scene-graph annotations, which are independent of the LLM generator, and the Op. Acc metric (98.5) is anchored to GQA's gold operation labels. Those are real contributions. The writing is honest about the method's limitations, including the inability to handle unseen operations like OCR.\n\nWhere the soft spots are: Section 3.2 explicitly feeds the ground-truth answer into the LLM prompt 'to guide the expected reasoning process,' so the intermediate steps are prone to post hoc rationalization rather than being steps the image alone would support. Section 4 then validates predicted SoTs against ground-truth SoTs produced by the same LLM and prompt; Logical Acc and intermediate-answer agreement mostly measure consistency with the generator's style, not independent visual correctness. The human study (Section 5.3, ~300 samples) only checks coherence between sub-task steps and a correct final answer, not faithfulness to the image. Accuracy gains are +1.1 on GQA and +0.3 on CRIC with no error bars or significance tests, and code/data are promised but not yet available. These are real weaknesses, though not fatal: the bbox output is a concrete grounding signal tied to scene-graph annotations, and the final-answer numbers are externally anchored.\n\nI would not dismiss the paper. The method is sensible, the resource is useful, and the overclaim is narrowing: what is established is 'structured rationales consistent with the generator and reasonably grounded in bboxes,' not full 'visually interpretable reasoning.'\n\nThis paper is for people working on interpretable VQA and MLLM reasoning. It deserves serious peer review, with the expectation of heavy revision: independent validation of the textual rationales, significance testing, and an ablation that removes the answer-conditioning from rationale generation. If those are addressed, I would be comfortable seeing it in a main track. As submitted, it is a conditional accept with major revisions, not a reject.","headline":"VISTAR is a clear, modest step beyond VPD—train an MLLM to emit bbox-grounded Subtask-of-Thought rationales without external execution—but the interpretability claim is weakened by same-generator evaluation and answer-conditioned rationale generation.","tokens_in":19293,"tokens_out":1840,"would_cite":true,"duration_ms":19330,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VISTAR fine-tunes a multimodal large language model on structured Subtask-of-Thought rationales so that answering a visual question produces an ordered chain of sub-task operations, intermediate answers, and object bounding boxes—and this…","keywords":["visual question answering","interpretability","subtask reasoning","instruction tuning","multimodal large language model","visual grounding","chain-of-thought","GQA"],"falsifier":"Train VISTAR on rationales generated without the ground-truth answer in the LLM prompt, or with a deliberately wrong answer, and compare final accuracy and bounding-box IoU. If accuracy and grounding survive, the rationales carry real visual evidence; if they collapse or the boxes drift while answers stay correct, the supervision is answer-conditioned narration rather than visual reasoning.","tokens_in":18263,"feed_emoji":"🧩","tokens_out":9122,"duration_ms":81095,"temperature":0.7,"pith_summary":"VISTAR asks whether a vision-language model can explain itself step by step without losing accuracy, and answers yes. The paper proposes training an MLLM to output a Subtask-of-Thought rationale—naming each object it selects, each attribute it filters, and each relation it checks, with a bounding box for every object—instead of relying on external program execution. The training data is synthesized by an LLM from GQA scene graphs, sub-task operation sequences, and ground-truth answers, then used to fine-tune NVILA-8B on the full rationale. On GQA test the fine-tuned model reaches 65.1% accuracy versus 64.0% for the base model, and on a zero-shot CRIC sample it reaches 61.1% versus 60.8%. A sympathetic reader would care because this is evidence that interpretability can be a training target rather than a post-hoc add-on that degrades performance.","feed_headline":"An image Q&A model that shows its work also answers better","feed_subtitle":"The fine-tuned model beats its base on two visual reasoning benchmarks while explaining every step with boxes.","key_machinery":"The central object is the Subtask-of-Thought (SoT) rationale, an ordered set $S=\\{(op_1,r_1),\\dots,(op_n,r_n)\\}$ in which each $op_i$ is a sub-task operation and each $r_i$ is its intermediate result, with the final result $r_n$ being the answer. The machinery that carries the argument is an LLM data-synthesis step that converts GQA's scene graphs and sub-task operation sequences into SoT annotations, followed by visual instruction tuning in which the MLLM is trained, under the prompt 'Explain the rationale by decomposing into subtasks to answer the question,' to predict the entire SoT. This design does the work: reasoning steps and bounding boxes become supervised output tokens, so interpretability is learned inside the model rather than obtained by executing external programs.","core_discovery":"The central claim is that an MLLM can be trained to produce both the final answer and a full visual-textual rationale, and that doing so improves compositional VQA accuracy. A Subtask-of-Thought is an ordered set of reasoning steps, each pairing a sub-task operation (select, relate, filter, verify, query, and, or, ...) with an intermediate result that is an object name plus a normalized bounding box, an attribute value, or a boolean. The training signal is generated by an LLM prompted with the question, the scene graph, the GQA sub-task operation sequence, and the ground-truth answer, followed by filtration of rationales whose final answer mismatches the ground truth or whose format is malformed. The paper then fine-tunes NVILA-8B with a next-token-prediction loss over the full rationale, and reports a consistent accuracy gain on GQA (65.1% versus 64.0%), a smaller zero-shot gain on CRIC (61.1% versus 60.8%), and improved object grounding measured by IoU, precision, and recall.","pith_inferences":["If the rationales are faithful, the answer-conditioned generation of training data is the main bottleneck; a variant that removes the ground-truth answer from the LLM prompt would test whether the model learns genuine visual search or answer-consistent narration.","The bounding-box tokens may act as an implicit grounding regularizer, so one could ablate coordinates only (keeping object names) to see whether visual grounding and final accuracy move together.","Because the paper notes that OCR operations were absent from GQA training, the method's boundary can be probed by adding OCR-style sub-task operations to a synthetic dataset and checking whether VISTAR-style training picks them up.","An untested scaling hypothesis is that longer, more detailed SoT rationales improve accuracy the way longer chain-of-thought traces do in text-only LLMs; measuring accuracy as a function of rationale length would connect this work to that pattern."],"forward_implications":["VISTAR answers with an auditable trace: each intermediate answer and its bounding box can be inspected, so a wrong final answer can be traced to a specific sub-task failure.","Because the full SoT is generated by the model itself, the method removes the per-query cost of running object detectors or code interpreters at inference time.","The reported GQA and CRIC results imply that structured sub-task supervision can improve accuracy while adding explanation, reversing the usual explanation penalty observed in base MLLMs.","The same next-token training recipe can in principle be applied to any MLLM backbone, so the interpretability gain is not tied to a specific architecture."],"supporting_citations":[{"why":"Supplies GQA images, questions, scene graphs, sub-task operation sequences, and ground-truth answers that the SoT generation pipeline builds on.","marker":"[14]"},{"why":"Provides the NVILA-8B backbone that VISTAR fine-tunes and the main accuracy baseline it must beat.","marker":"[23]"},{"why":"Prior instruction-tuning-with-distillation approach that VISTAR extends and contrasts with, since VPD still relies on program execution.","marker":"[12]"},{"why":"Visual programming framework whose external modular execution VISTAR replaces with in-model reasoning.","marker":"[11]"},{"why":"Zero-shot ViperGPT baseline compared in GQA evaluation, representing program-execution interpretability.","marker":"[32]"},{"why":"The LLM used in Section 3.2 to generate the SoT rationales from scene graphs and operation sequences.","marker":"[33]"},{"why":"GPT-4-turbo is used to score semantic and logical quality of generated textual rationales.","marker":"[1]"},{"why":"CRIC serves as the unseen compositional reasoning benchmark for the zero-shot generalization claim.","marker":"[8]"}],"fun_headline_variants":["Teaching MLLMs to show reasoning steps improves VQA","VISTAR: Showing reasoning steps improves VQA accuracy","Interpretable reasoning steps lead to better VQA answers","Fine-tuned MLLM explains steps and improves visual reasoning","Reasoning steps improve VQA accuracy and transparency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated rationales, produced from a scene graph with the correct answer included in the prompt, are faithful traces of visual reasoning; if the LLM only writes plausible-sounding steps consistent with the known answer, the interpretability the paper measures is not independent evidence of genuine visual reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Teaching MLLMs to show reasoning steps improves VQA","VISTAR: Showing reasoning steps improves VQA accuracy","Interpretable reasoning steps lead to better VQA answers","Fine-tuned MLLM explains steps and improves visual reasoning","Reasoning steps improve VQA accuracy and transparency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001071,"raw_usage":{"total_tokens":4477,"prompt_tokens":926,"completion_tokens":3551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":3472}},"tokens_in":542,"tokens_out":3551,"duration_ms":21717,"temperature":1.0,"reasoning_tokens":3472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:04:30.278577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VISTAR on rationales generated without the ground-truth answer in the LLM prompt, or with a deliberately wrong answer, and compare final accuracy and bounding-box IoU. If accuracy and grounding survive, the rationales carry real visual evidence; if they collapse or the boxes drift while answers stay correct, the supervision is answer-conditioned narration rather than visual reasoning.","supporting_citations":[{"cited_title":"Hudson and Christopher D","cited_arxiv_id":null,"evidence_quote":"Supplies GQA images, questions, scene graphs, sub-task operation sequences, and ground-truth answers that the SoT generation pipeline builds on."},{"cited_title":"Visual program distillation: Distilling tools and programmatic reasoning into vision-language models","cited_arxiv_id":null,"evidence_quote":"Prior instruction-tuning-with-distillation approach that VISTAR extends and contrasts with, since VPD still relies on program execution."},{"cited_title":"Visual pro- gramming: Compositional visual reasoning without training","cited_arxiv_id":null,"evidence_quote":"Visual programming framework whose external modular execution VISTAR replaces with in-model reasoning."},{"cited_title":"Vipergpt: Visual inference via python execution for reasoning","cited_arxiv_id":null,"evidence_quote":"Zero-shot ViperGPT baseline compared in GQA evaluation, representing program-execution interpretability."},{"cited_title":"Cric: A vqa dataset for compositional reasoning on vision and commonsense","cited_arxiv_id":null,"evidence_quote":"CRIC serves as the unseen compositional reasoning benchmark for the zero-shot generalization claim."}],"review_version":1}