Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that training a multimodal LLM on chain-of-thought data whose key numbers and terms are grounded in self-verified bounding boxes makes the model adapt to specialized visual formats like charts and receipts with as few as…

desk verdict Solid few-shot adaptation paper; the self-verification loop is less circular than the stress-test says, but it needs an explicit comparison to the prior Grounded CoT work and reproducible hyperparameters. read the letter →

arxiv 2507.02859 v1 pith:CNWATLJH submitted 2025-07-03 cs.CV

Jiaer Xia , Bingkui Tong , Yuhang Zang , Rui Shao , Kaiyang Zhou This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelschain-of-thoughtvisualgroundingboundingboxself-verificationfew-shotadaptationchartunderstandingdocumentbootstrapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that chain-of-thought (CoT) data distilled from a third-party multimodal model fails to help a model adapt to specialized visual formats mainly because the reasoning steps contain factual errors—misread prices, counts, or labels—even when the final answer is correct. To fix this, it proposes Grounded Chain-of-Thought (GCoT): each key number or term in the distilled CoT is augmented with a bounding box, and every box is self-verified by cropping the region, reading its content, and keeping only boxes whose content matches the claimed text. Training on the filtered, augmented GCoT data outperforms both fine-tuning on question-answer pairs and distillation on raw CoT across five datasets covering charts, tables, receipts, and financial reports, using just 8 to 128 labeled examples. If the claim holds, it gives a low-cost path for adapting generic multimodal models to text-heavy specialized documents without large-scale retraining.

What carries the argument

The load-bearing mechanism is a bootstrapping self-verification loop. For each distilled CoT sample, NLTK extracts meaningful nouns and numbers; each extracted target becomes a sub-question of the form "Where is <target>?", and the MLLM proposes bounding boxes for it. The model then crops each proposed box, reads the enclosed content with a "The content in this image is:" prompt, and compares it with the target string; boxes that match are kept, others are discarded, and the surviving boxes fine-tune the model for the next iteration. After several rounds, the accepted boxes are spliced into the CoT as coordinate tuples right after the corresponding targets, forming GCoT; the model is fine-tuned on this data, then prompted to generate several fresh GCoT candidates per question, selecting only those whose final answer and bounding boxes both pass verification.

What would settle it

A human audit of a few hundred GCoT training samples from each dataset, labeling whether every boxed number matches the image and whether the surrounding reasoning step is factually correct, would settle the central claim; if the box-verified steps contain frequent mismatches on any dataset, the filter is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery is that grounding information, injected as bounding boxes, makes chain-of-thought training data faithful to the image and thereby unlocks data-efficient adaptation of multimodal LLMs. Concretely, with only 8 labeled examples the GCoT-trained model already outperforms zero-shot inference, and at 128 samples it consistently beats two training baselines built on the same backbone and LoRA: fine-tuning on the original question-answer pairs and distillation on CoT generated by LLaMA 3.2. The ablation on TabMWP shows the mechanism: removing augmentation costs 1.63 to 2.36 accuracy points, but removing box verification costs 5.94 to 10.36 points, so the gain comes mainly from filtering reasoning steps by checking that each localized number or term actually appears in the image. The authors also show GCoT improves over distillation regardless of which teacher (LLaMA 3.2, Claude 3.5, GPT-4o, Qwen2-VL, Gemini 1.5-Pro) produces the initial CoT, reducing variance across sources.

Load-bearing premise

The entire filtering pipeline assumes the model's self-verification step—cropping a predicted bounding box, reading what is inside, and comparing it to the target string—is a reliable test of whether the reasoning step is correct; the only direct evidence is a manual check of 64 ChartQA samples.

Editorial extensions

If this is right

  • With only 8 labeled examples, the GCoT-trained model beats zero-shot inference by about 2% on average, and its lead over fine-tuning and distillation grows as the sample budget reaches 128.
  • On TabMWP, removing box verification costs 5.94 to 10.36 accuracy points, while removing augmentation alone costs 1.63 to 2.36 points, so verifying intermediate grounded steps is the main driver of improvement.
  • The improvement holds across five different CoT teachers—LLaMA 3.2, Claude 3.5, GPT-4o, Qwen2-VL, and Gemini 1.5-Pro—so the method does not depend on a specific distillation model and reduces cross-teacher variance.
  • Training on 128 ChartQA samples with GCoT also improves accuracy on TabMWP and DVQA, showing the grounded reasoning transfers to other chart and table benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation statement predicts GCoT will work best where content is text and numbers; a direct test of that boundary is applying the same pipeline to diagrams built from lines, icons, or plots without readable tokens.
  • The box-verification stage checks only that a localized region contains the claimed string, not that the arithmetic or logical steps combining those strings are valid, so a natural extension is a second verification stage that re-checks each derived step against the grounded values.
  • If the self-verification loop genuinely improves localization, the number of true boxes should keep rising with more iterations and the gains should appear without any human annotation; both are testable predictions of the paper's mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Grounded Chain-of-Thought (GCoT), a bootstrapping method that adds bounding-box grounding to Chain-of-Thought (CoT) reasoning data distilled from a pretrained MLLM. Given a small labeled set (8-128 samples), GCoT distills CoT from a third-party LLM, extracts key terms, asks VisCoT-7B to propose bounding boxes for those terms, crops the image, reads the crop with the same model, and retains boxes whose content matches the target term. The resulting grounded CoT data is used for LoRA fine-tuning, and the model is then prompted to generate additional verified GCoT samples. The method is evaluated on ChartQA, TabMWP, SROIE, DVQA, and TAT-QA against zero-shot, fine-tuning, and distillation baselines. The main claim is that GCoT significantly outperforms these baselines under data-limited regimes, with the ablation in Table 2 attributing the gains primarily to box verification.

Significance. If the findings hold, GCoT provides a simple, annotation-light recipe for adapting MLLMs to specialized visual formats such as charts, tables, invoices, and reports, which is practically valuable given the difficulty of collecting large grounded datasets for these domains. The paper's strengths include evaluation on five standard benchmarks, three random samplings with reported standard deviations, an ablation that isolates the box-verification component, and a robustness study across five different distillation sources. These design choices make the empirical comparisons more reliable than a single-run few-shot evaluation. However, the central mechanism—self-verification of bounding boxes—is performed by the same model that proposes the boxes and reads the crops, so the improvement attributed to box verification could partly stem from self-confirmation rather than factual grounding. Independent evidence for the verifier's accuracy is currently thin, which is the main risk to the paper's central claim.

major comments (4)
  1. [Section 3.2 and Appendix A.2, Table 6] The load-bearing assumption is that the consistency check, where VisCoT-7B reads its own crop of a box it proposed and compares the text to a target string, is a valid proxy for factual grounding. Because the same model generates the box, reads the crop, and performs the comparison, the filter can certify its own hallucinated readings; for example, if the model's prior from the distilled CoT leads it to read '$1.24' from a crop of '$0.72', the check passes. The only independent evidence is the manual audit of 64 ChartQA samples in Table 6, which does not report the false-accept rate, does not cover SROIE, TabMWP, DVQA, or TAT-QA, and counts 'acc' in a way that does not distinguish reasoning-step accuracy from final-answer accuracy. Since the largest ablation effect in Table 2 (5.94-10.36 points) is attributed to box verification, this circularity is a correctness risk for the central claim. Please provide a larger, cross-dataset evaluation of the verifier's precision/recall (e.g., with a different MLLM or OCR as an independent check), or replace the self-verification with an external verifier.
  2. [Section 3.2 and Section 4.4] The bootstrapping loop is described as iterating for a 'pre-determined number of iterations', but neither the number N nor the stopping criterion is reported. The consistency check is also underspecified: no threshold, matching rule, or prompt template is given that would allow reproduction of the filter. The temperature for GCoT generation is 0.8, but the temperature for bootstrap box proposals is not stated. Please report these values explicitly, or state clearly if they were fixed constants or tuned per dataset.
  3. [Section 4.2 and Fig. 3] The paper says 'three independent random samplings' were performed for each sample size, but it does not specify whether samples are drawn from the official training splits, whether the same sampled examples are used across fine-tuning, distillation, and GCoT, or whether any validation-based selection was performed. These details are needed to interpret the statistical comparisons and the reported standard deviations.
  4. [Section 4.6, Table 2] The ablation that isolates the effect of box verification is conducted on a single dataset (TabMWP). Since the central claim is that self-verified box grounding is the key ingredient, the absence of at least one additional ablation dataset (e.g., ChartQA or SROIE) leaves open the possibility that the observed gains are dataset-specific. Please extend the ablation to at least one or two other datasets.
minor comments (5)
  1. [Section 3.1] The text states 'We first pre-train a base MLLM using visual grounding data' and then says 'In our implementation, we directly adopt the VisCoT-7B model developed by Shao et al.'; please clarify whether any additional pre-training was actually performed or whether the existing VisCoT-7B checkpoint was used as-is.
  2. [Appendix A.2, Table 6] The column label 'Acc/All' is ambiguous: please define what is counted as 'Acc' (e.g., number of samples whose entire reasoning chain is factually correct) and explain why the denominator is 64 for the first three columns but 192 for the final column.
  3. [References] Reference [30] (Grounded Chain-of-Thought for Multimodal Large Language Models, arXiv:2503.12799) appears in the bibliography but I could not find a citation to it in the main text; either add a discussion where it is relevant or remove it from the reference list.
  4. [Figure 3] In the provided PDF, the subfigure titles in Figure 3 render as garbled tokens (e.g., '/uni...' sequences); please check the font embedding so the plot labels are readable.
  5. [Table 5] The caption says 'The model is trained on 128 ChartQA datasets'; this should read '128 training samples from ChartQA' to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is evaluated on external held-out test sets, and the self-verification loop is an empirical design choice whose reliability is a correctness risk, not a definitional reduction.

full rationale

The paper's central claim—that fine-tuning an MLLM on grounded chain-of-thought data enables data-efficient adaptation—is tested on held-out test splits of five public benchmarks (ChartQA, TabMWP, SROIE, DVQA, TAT-QA), so the reported gains are not forced by the construction of the GCoT data. The bootstrapping loop in Sec. 3.2 uses the MLLM to propose bounding boxes, crop the regions, read them with the prompt 'The content in this image is:', and compare the reading against targets extracted from the distilled CoT. This is a self-referential filtering step whose reliability is a legitimate correctness risk, but it is not circular in the derivation-chain sense: the filter can fail, the paper does not define the evaluation metric in terms of the filter's decisions, and the manual audit in Appendix A.2 (Table 6) attempts an independent check. The backbone VisCoT-7B is adopted from external prior work [26], and no uniqueness theorem or load-bearing result is imported from the present authors' prior papers. The only same-author citation in the body appears in the Limitations section as a future-work pointer ([31]), so it is not load-bearing. The paper is therefore self-contained against external benchmarks rather than circular by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of self-verification and the transferability of visual grounding, not on a fitted mathematical model. There are no explicit free parameters fitted to the test benchmarks; the listed quantities are implementation choices. No invented entities appear.

free parameters (4)
  • bootstrap iterations N = not specified
    The loop repeats 'for a pre-determined number of iterations' (Section 3.2), but N is never given; the results depend on this stopping choice.
  • GCoT samples per question = 3
    Section 3.2 and 4.4: three verified GCoT samples are selected per question; augmentation strength is a hand-chosen design choice.
  • sampling temperature = 0.8
    Section 4.4: set to encourage diversity when generating GCoT; affects data distribution.
  • LoRA rank and alpha = rank 16, alpha 32
    Section 4.4: fixed without sensitivity study; central comparisons share these settings, so the absolute effects may depend on them.
assumptions (4)
  • domain assumption Self-verification via cropped-box content consistency is a reliable proxy for factual correctness.
    Sections 3.2 and 3.3 use the MLLM's own reading of the cropped box to accept or reject reasoning steps; no external oracle is used.
  • domain assumption NLTK noun and number extraction identifies all task-relevant entities in the distilled CoT.
    Section 3.2: targets are selected with NLTK; missed entities never get grounded or verified.
  • domain assumption VisCoT-7B's visual grounding pretraining transfers to charts, tables, receipts, and reports.
    Section 3.1 and 4.4: the backbone is adopted, not trained for these formats; its localization ability is assumed to be sufficient on specialized images.
  • domain assumption Text-only LLMs can produce useful reasoning structure for visual questions despite not seeing the image.
    Section 3.2: LLaMA3.2 distills CoT without access to the image; the method relies on that reasoning being worth grounding rather than discarded wholesale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation." pith.science (2026). https://pith.science/paper/CNWATLJH

@misc{pith2026250702859,
  author       = {Pith},
  title        = {Pith review of: Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNWATLJH}},
  note         = {Machine review of arXiv:2507.02859}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in interpreting images using natural language. However, without using large-scale datasets for retraining, these models are difficult to adapt to specialized vision tasks, e.g., chart understanding. This problem is caused by a mismatch between pre-training and downstream datasets: pre-training datasets primarily concentrate on scenes and objects but contain limited information about specialized, non-object images, such as charts and tables. In this paper, we share an interesting finding that training an MLLM with Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in specialized vision tasks, especially under data-limited regimes. However, we identify a critical issue within CoT data distilled from pre-trained MLLMs, i.e., the data often contains multiple factual errors in the reasoning steps. To address the problem, we propose Grounded Chain-of-Thought (GCoT), a simple bootstrapping-based approach that aims to inject grounding information (i.e., bounding boxes) into CoT data, essentially making the reasoning steps more faithful to input images. We evaluate our approach on five specialized vision tasks, which cover a variety of visual formats including charts, tables, receipts, and reports. The results demonstrate that under data-limited regimes our approach significantly improves upon fine-tuning and distillation.

Figures

Figures reproduced from arXiv: 2507.02859 by the authors.

Figure 1
Figure 1. Comparison between normal and grounded Chain￾of-Thought (CoT). The normal CoT contains factual errors, de￾spite giving the correct answer. In addition to reasoning, the grounded CoT (GCoT) uses bounding boxes to ground key infor￾mation in the image, which can be self-verified by the model and help minimize factual errors. arXiv:2507.02859v1 [cs.CV] 3 Jul 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of GCoT generation process. For each sampled training data point, we start by generating a distilled CoT using a third-party model. Key information from the CoT is then extracted to form a set of sub-questions, which are fed into a bootstrapping loop. This loop iteratively generates bounding boxes and filters out the correct ones to improve the model’s grounding capability. The finalized filtered bounding b… view at source ↗
Figure 3
Figure 3. Main results on the five specialized vision datasets. Fine-tuning means directly fine-tuning the model with simple question￾answer pairs. Distillation means training the model with LLaMA distilled CoT data. All training-based methods are based on the same backbone and the LoRA method. Overall, the performance of GCoT surpasses the fine-tuning and distillation methods across all datasets and sample sizes. The shadow … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Result of analyzing the impact of bootstrapping loop [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Self-generated GCoT data eliminating external language discrepancies and aligning closely with the model’s linguistic tendencies. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Example of GCoT applied to ChartQA dataset, illustrating its ability to generate reliable and diverse training CoT data. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Example of GCoT applied to TAT-QA dataset, illustrating its ability to generate reliable and diverse training CoT data. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Example of GCoT applied to SROIE dataset, illustrating its ability to generate reliable and diverse training CoT data. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Answer-conditioned CoT distillation lets a 3B VLM outperform direct LoRA and sometimes GPT-4.1 on industrial few-shot classification using 18–30 labeled images.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Lion: Empowering multimodal large language model with dual-level visual knowledge

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26540–26550, 2024. 2

    2024

  2. [2]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 2

    arXiv

  3. [3]

    Train- ing verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 3

    arXiv

  4. [4]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  5. [5]

    Chartllama: A mul- timodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A mul- timodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023. 2, 3

    arXiv 2023

  6. [6]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In- ternational Conference on Learning Representations , 2021. 5

    work page 2021

  7. [7]

    Icdar2019 compe- tition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 compe- tition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. 5

    2019

  8. [8]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

Show all 39 references
  1. [9]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information pro- cessing systems, 35:22199–22213, 2022. 3

    2022

  2. [10]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  3. [11]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  4. [12]

    Deductive verification of chain-of-thought reasoning

    Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. Advances in Neural Informa- tion Processing Systems, 36, 2024. 3

    2024

  5. [13]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2, 4

    2024

  6. [14]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  7. [15]

    Nltk: The natural language toolkit

    Edward Loper and Steven Bird. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Ef- fective Tools and Methodologies for Teaching Natural Lan- guage Processing and Computational Linguistics, pages 63– 70, 2002. 4

    2002

  8. [16]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 5

    2019

  9. [17]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. The Eleventh In- ternational Conference on Learning Representations , 2022. 5

    2022

  10. [18]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 5

    2022

  11. [19]

    Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild

    Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild. arXiv preprint arXiv:2407.04172, 2024. 3

    2024 arXiv

  12. [20]

    Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning

    Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.02384, 2024. 2, 3

    2024 arXiv

  13. [21]

    Selfcheck: Using llms to zero-shot check their own step-by-step reason- ing

    Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using llms to zero-shot check their own step-by-step reason- ing. The Twelfth International Conference on Learning Rep- resentations, 2023. 3

    2023

  14. [22]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011. 2

    2011

  15. [23]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  16. [24]

    Aligning large and small language models via chain-of-thought reasoning

    Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain-of-thought reasoning. In Proceedings of the 18th Conference of the European Chap- ter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812–1827, 2024. 3

    2024

  17. [25]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Robert Kaczmarczyk, Aran Komat- suzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Je- nia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Workshop Datacentric AI . J ¨ulich Su...

    2021

  18. [26]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv e-prints, pages arXiv–2403, 2024. 4, 5

    2024

  19. [27]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  20. [28]

    Mome: Mixture of multimodal experts for generalist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. In Advances in Neural Information Processing Systems , pages 42048– 42070, 2024. 2

    2024

  21. [29]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 2, 3

    2022

  22. [30]

    Grounded chain-of-thought for multimodal large language models

    Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models. arXiv preprint arXiv:2503.12799, 2025. 2

    2025 arXiv

  23. [31]

    Visionary-r1: Mitigating shortcuts in vi- sual reasoning with reinforcement learning

    Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-r1: Mitigating shortcuts in vi- sual reasoning with reinforcement learning. arXiv preprint arXiv:2505.14677, 2025. 8

    2025

  24. [32]

    Falcon: Resolv- ing visual redundancy and fragmentation in high-resolution multimodal large language models via visual registers

    Renshan Zhang, Rui Shao, Gongwei Chen, Miao Zhang, Kaiwen Zhou, Weili Guan, and Liqiang Nie. Falcon: Resolv- ing visual redundancy and fragmentation in high-resolution multimodal large language models via visual registers. In Proceedings of the IEEE/CVF International Conferenc...

    2025

  25. [33]

    Tat-qa: A question answering benchmark on a hybrid of tab- ular and textual content in finance

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tab- ular and textual content in finance. InProceedings of the 59th Annual Meeting of the Association for Computa...

    2021

  26. [34]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 4995–5004, 2016. 4 A. Appendix A.1. Evaluation of Generalization Ability Theo...

    2016

  27. [36]

    First, let's understand the problem: The actor was told how many fan letters he received

  28. [37]

    Potential fan letters: - The number of fan letters [0.411, 0.195, 0.998, 0.271] could be 204 received on Thursday [0.000, 0.455, 0.281, 0.531] - 271 received on Monday [0.000, 0.715, 0.255, 0.871]

  29. [38]

    The table shows the number of fan letters for each day

    Total fan letters calculation: - 204 + 271 = 475 *Answer*: 475 GCoT (Claude) We need to find the total number of fan letters received on Thursday and Monday. The table shows the number of fan letters for each day. The actor received 204 fan letters on Thursday [0.095, 0.455, 0...

  30. [39]

    Opening Balance.\

    The fan letter count on Monday was 271 [0.617, 0.793, 0.751, 0.859]. Thus, the total number of fan letters received on Thursday and Monday is: 204 + 271 = 475. *Answer*: 475. GCoT (Qwen) TabMWP Figure 5. Self-generated GCoT data eliminating external language discrepancies and ...

  31. [204]

    Thus, the total number of fan letters received on Thursday and Monday is: 204 + 271 = 475

    The fan letter count on Monday [0.019, 0.819, 0.275, 0.895] was 271 [0.619, 0.819, 0.719, 0.865]. Thus, the total number of fan letters received on Thursday and Monday is: 204 + 271 = 475. *Answer*: 475. GCoT (LLaMA)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.