Pith. sign in

REVIEW 3 major objections 4 minor 11 cited by

ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReFocus claims that letting a multimodal LLM edit the input image between reasoning steps—masking, boxing, or highlighting regions—improves its answers on tables and charts, and that the resulting visual traces are better training data…

desk verdict A useful visual-editing prompting framework for tables and charts, but the headline gains are likely inflated because the ReFocus prompt also feeds the model OCR header/coordinate text the baseline never sees. read the letter →

arxiv 2501.05452 v1 pith:MGYOWXK3 submitted 2025-01-09 cs.CV cs.CL

Xingyu Fu , Minqian Liu , Zhengyuan Yang , John Corring , Yijuan Lu , Jianwei Yang , Dan Roth , Dinei Florencio
show 1 more author
Cha Zhang
This is my paper · ORCID
classification cs.CVcs.CL
keywords visualchain-of-thoughtmultimodallargelanguagemodelsquestionansweringimageeditingselectiveattentiontableschartsstructuredunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a multimodal language model can do better multi-step visual reasoning on tables and charts if, between reasoning steps, it edits the image itself: masking out irrelevant columns, drawing boxes around relevant rows, or highlighting bars and subplots. The authors build ReFocus, which gives GPT-4o a small set of Python image-editing tools and prompts it to generate code that refocuses the image before answering. On six benchmarks the edited-image loop outperforms plain GPT-4o by an average of 11.0 points on table tasks and 6.8 points on chart tasks, and it does so without adding external information. The same loop also produces a 14k training set whose visual chain-of-thought records—including the bounding boxes the model chose to focus on—teach a smaller model (Phi-3.5-vision) more effectively than the same question-answer pairs or text-only CoT. If this holds, intermediate visual edits, not just text reasoning, are a viable way to steer large multimodal models on structured images.

What carries the argument

The load-bearing mechanism is the ReFocus loop. The multimodal LLM is prompted to alternate THOUGHT and ACTION, where an ACTION is Python code calling one of a fixed set of image-editing tools—highlight, mask-out, or draw-box applied to columns, rows, bars, or subplots. The code runs, the edited image replaces the model's visual input, and the loop terminates when the model outputs an answer. Coordinates come from OpenCV contour detection for table rows and columns and chart subplots, plus dataset-provided axis coordinates for bar charts. The physical edit is the point: it removes or marks the distracting regions before the next perceptual step, so the model re-reads a simplified image instead of re-attending to the original one.

What would settle it

Run ReFocus on a table or chart benchmark with the automatic OpenCV or dataset coordinates replaced by ground-truth boxes, and separately with boxes deliberately shifted by 5 to 10 percent of image width. If the shifted-box version matches the correct-coordinate version, the gains are not actually caused by isolating the intended region; if the ground-truth version beats the automatic version, the coordinate detector is the bottleneck.

Watch

Extended reading notes

Core claim

The paper's central claim is that selective attention on structured images can be operationalized as executable visual edits: when a multimodal LLM is allowed to iteratively call code that masks columns, draws red boxes over rows, or highlights bars and subplots, its accuracy on visual QA over tables and charts rises consistently. ReFocus does not add outside knowledge; it only returns a modified version of the input image. The authors attribute the gains to improved visual grounding, OCR, and counting, and they show the edited images also transfer: open-source models that never saw such edits improve when given ReFocus-edited images. When the edited-image traces are distilled into about 14k supervised fine-tuning examples, a Phi-3.5-vision model trained with the visual chain-of-thought data beats the same model trained on QA pairs by 8.0 points on average and on CoT text by 2.6 points.

Load-bearing premise

The whole pipeline rests on the coordinate detector drawing accurate boxes around rows, columns, bars, and subplots; if those boxes are wrong, the edits hide or emphasize the wrong region and the claimed gains should shrink or reverse.

Editorial extensions

If this is right

  • The largest reported gains are on VWTQ and horizontal bar charts, with near-zero gain on VTabFact, so the benefit is strongest where the question forces the model to pick among many visually similar rows, columns, or bars.
  • ReFocus with figure-only input beats GPT-4o given both gold text and the figure on most tasks, meaning visual editing can substitute for perfectly transcribed table or chart content.
  • The edited images transfer: LLaVA-NeXT and Phi-3-vision, which were not trained on visual prompts, mostly improve when fed ReFocus-edited images.
  • The visual chain-of-thought data is a better SFT supervision signal than QA pairs by 8.0 points on average and than text-only CoT by 2.6 points, so the focus-area information, not just the reasoning text, carries training signal.
  • Mask-out, draw-box, and highlight produce similar results, indicating the benefit comes from refocusing itself rather than from one specific edit style.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not test: the same edit loop should transfer to other dense visual layouts such as forms, maps, or diagrams, since the mechanism is not table- or chart-specific.
  • The SFT result suggests focus-area bounding boxes are a cheap supervision signal: any correct GPT-4o run can be recorded as visual chain-of-thought data, so the data-generation pipeline could scale to more benchmarks without human annotation.
  • The bounding-box dependency points to a testable improvement: replacing OpenCV coordinate detection with a model that predicts layout, or with OCR-derived boxes, could remove the main source of brittle edits and improve gains on unusual table and chart styles.
  • One could also make ReFocus iterative at training time, letting the model decide when to stop editing and compare answers across multiple edited views; the paper's loop is fixed, so the error cases where it stops too early or over-masks are unexplored.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ReFocus, a framework that lets a multimodal LLM (GPT-4o) generate Python code to edit the input image—masking, highlighting, or drawing boxes around rows, columns, bars, or subplots—as intermediate 'visual thoughts' before answering. The authors report consistent accuracy gains over vanilla GPT-4o on table (VWTQ, VWTQ_syn, VTabFact) and chart (CharXiv, ChartQA horizontal/vertical bar) benchmarks, and they collect a 14k training set that, when used for SFT of Phi-3.5-vision, outperforms QA-pair and textual-CoT supervision. The paper includes detailed prompts, tool descriptions, and analyses of which editing operations help.

Significance. If the causal attribution holds, the idea of using executable visual edits as a chain of thought is a useful and transferable contribution to structured image understanding. The paper is clearly written, provides full prompts and tool code, and the released SFT data is a potentially valuable resource. The cross-checkpoint consistency (two GPT-4o snapshots) and the open-source model transfer experiment (Table 2) strengthen the empirical case. However, the main experimental comparison is confounded by additional textual information injected only into the ReFocus prompt, which undermines the central claim that visual editing itself drives the gains; a control experiment is needed to isolate the effect.

major comments (3)
  1. [§3.3, Appendix C, Table 1] The central comparison in Table 1 is confounded. In the ReFocus system prompt (Appendix C), the model receives 'USER Bounding Box Info' listing column headers, row headers, and their bounding-box coordinates for tables, and x/y value names for charts. The vanilla GPT-4o baseline does not receive this textual information. Thus the reported gains (+11.0 table, +6.8 chart) could arise from the OCR-style header/coordinate hints that improve attention or reduce OCR errors, rather than from the visual edits themselves. The claim in §4.3 that 'REFOCUS does not introduce any additional information' is contradicted by the prompt design. The paper needs a control condition that gives GPT-4o the same textual hints but forbids editing (or, conversely, applies the edits without the textual hints). Without such a control, the causal claim of §3 is underdetermined.
  2. [§5, Table 7] The SFT comparison between 'SFT w/ REFOCUS VCoT' and 'SFT w/ REFOCUS CoT' is not cleanly controlled. Table 7 reports 'include edited image in input' as 'No' for the VCoT model and 'Yes' for the CoT model, while the text says the only difference between the two data types is the presence of refocus bounding-box coordinates. If CoT training includes the edited image and VCoT training does not, then the 2.6% average gain attributed to bounding boxes could instead reflect the difference in visual input, or the comparison could be accidentally favorable in a different way. Please clarify the exact input formats and, ideally, run a matched control where the only varying factor is the bounding-box coordinates.
  3. [§3.1, §4.3] The coordinate-acquisition heuristics (longest vertical/horizontal contours for tables; top-k contours with k=10 for CharXiv subplots) are not evaluated for accuracy. If the detected bounding boxes are wrong, the mask/draw/highlight edits could hide or emphasize the wrong regions. The paper reports no detection-quality metrics or failure cases, so the robustness of the pipeline across the diverse table/chart styles is unclear. Reporting the fraction of cases where the acquired boxes match the intended row/column/bar regions would substantially increase confidence in the mechanism.
minor comments (4)
  1. [Abstract vs. §1 vs. §5] The training-set size is given as 14k in the abstract and §5, but §1 (contribution 3) states 'we curate a 21k training set.' Please align these numbers.
  2. [Table 3] The CharXiv row for 'Text input' is shown as '\' (missing). Since this is a key comparison, a value or an explicit statement that CharXiv text input is unavailable would improve the table.
  3. [Figure 5 caption] The caption says the model 'correctly recognize the characters as "Partenvia"' but the correct spelling appears to be 'Partenavia' (as in the original and the model's first attempt). This seems like a typo and should be fixed.
  4. [§4.3, Table 4] The analysis of editing tools (mask vs. draw vs. highlight) compares only edited conditions; adding a no-edit baseline with the same prompt structure would make the tool comparison more interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ReFocus is an externally benchmarked empirical prompting and SFT study; no load-bearing claim reduces to its inputs by construction.

full rationale

ReFocus's central claims are measured gains on external benchmarks (TableVQA, ChartQA, CharXiv), not derived predictions. The reported +11.0/+6.8 figures are direct accuracy comparisons between GPT-4o with and without the ReFocus prompt, and the SFT results compare Phi-3.5 trained on VCoT, CoT, and QA versions of the same 14k examples. The VCoT-versus-CoT contrast isolates the presence of focus-area bounding boxes, and the QA baseline uses the same QA pairs, so the comparison is controlled rather than self-definitional. The citations to Visual Sketchpad and BLINK are related-work comparisons used to position the method, not load-bearing premises; no uniqueness theorem is imported, and no ansatz is smuggled in via self-citation. The most plausible concern is a confound rather than circularity: the ReFocus prompt injects structured textual cues such as 'USER Bounding Box Info', column headers, and x/y value coordinates that the vanilla GPT-4o baseline does not receive, so part of the gain could come from these hints rather than from the visual edits. That threatens causal attribution and is a correctness or validity criticism, but it does not make the result equivalent to its inputs by construction. No equation in the paper reduces to itself, and no fitted parameter is later renamed as a prediction. The 'no additional information' claim is presented as an interpretive statement about external knowledge, not as the formal basis of the measured gains. Therefore no circular steps are present and the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method relies heavily on OCR-derived coordinate maps and header text that are injected into the prompt. These are not free parameters in a fitting sense, but they are important unstated inputs. The k=10 contour choice is a hand-set parameter. No new physical or conceptual entities are invented.

free parameters (1)
  • k (10) for top-k contours in CharXiv subplot detection = 10
    Section 3.1: 'we take the top k longest contours', k=10 chosen by hand without sensitivity analysis.
assumptions (3)
  • ad hoc to paper Longest vertical and horizontal contours in table images correspond to row and column boundaries.
    Section 3.1 coordinate acquisition relies on this heuristic with no validation of its accuracy across the diverse table styles.
  • domain assumption GPT-4 as an answer judge behaves consistently with the benchmarks' original accuracy metrics.
    Appendix A.2 states all evaluations use GPT-4 to judge correctness; WTQ and ChartQA originally use exact-match or evaluator-based metrics, so the judge may be more lenient and affect absolute numbers.
  • ad hoc to paper The OCR-extracted headers and bounding boxes in the prompt do not contribute to the observed gains.
    The paper claims improvements come from visual editing 'without introducing additional information' (Section 4.3), but the system prompts in Appendix C provide column/row headers and bounding-box coordinates that the baseline does not receive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding." pith.science (2026). https://pith.science/paper/MGYOWXK3

@misc{pith2026250105452,
  author       = {Pith},
  title        = {Pith review of: ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGYOWXK3}},
  note         = {Machine review of arXiv:2501.05452}
}
read the original abstract

Structured image understanding, such as interpreting tables and charts, requires strategically refocusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, we introduce ReFocus, a simple yet effective framework that equips multimodal LLMs with the ability to generate "visual thoughts" by performing visual editing on the input image through code, shifting and refining their visual focuses. Specifically, ReFocus enables multimodal LLMs to generate Python codes to call tools and modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas, thereby enhancing the visual reasoning process. We experiment upon a wide range of structured image understanding tasks involving tables and charts. ReFocus largely improves performance on all tasks over GPT-4o without visual editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth analysis of the effects of different visual edits, and reasons why ReFocus can improve the performance without introducing additional information. Further, we collect a 14k training set using ReFocus, and prove that such visual chain-of-thought with intermediate information offers a better supervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over CoT.

Figures

Figures reproduced from arXiv: 2501.05452 by the authors.

Figure 1
Figure 1. Overview of REFOCUS. REFOCUS performs visual chain of thought via input-image editing on an example data from TableVQA [16]. Given an image and question pair, REFOCUS equips GPT-4 with editing tools (details in §3), and GPT-4 generates pseudo code if an edit action is needed. REFOCUS then executes the editing actions, and feeds GPT-4 with the new image until an answer is reached. In the above example, mask_column an… view at source ↗
Figure 2
Figure 2. Example of how REFOCUS + GPT-4o solves previously unsolvable problem in ChartQA dataset [24] through improved visual grounding. Given the original horizontal bar image (left), GPT-4o grounds to the wrong bars and thus gets the wrong answer. REFOCUS eliminates such possibility through editing, guiding the model to the correct answer (right). pad [13], advances visual reasoning by providing various computer vision too… view at source ↗
Figure 3
Figure 3. REFOCUS equips GPT-4 with selective attention. Above is an example of how REFOCUS + GPT-4o solves previously unsolvable problem in ChartXiv dataset [34]. Specifically, REFOCUS edits upon the original image by masking out all irrelevant information – the other three subplots that could be distracting. As a result, GPT-4o is able to conduct better reasoning with the edited image, and reach the correct answer. come fro… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: REFOCUS unleashes better visual grounding and counting abilities for GPT-4 as in ChartQA [24] Vertical Bar problems. What’s total wins by belgian riders❓ ACTION: Focus with Mask using Python Code THOUGHT: Focus on columns Country and Wins # Find coordinates of the colu…
Figure 5
Figure 5. Figure 5: REFOCUS unleashes better OCR for GPT-4. In this example from TableVQA [16], REFOCUS + GPT-4 conducts the edit action highlight_column. With this simple action, GPT-4 can focus more on the important subarea, and recognize the characters better. bar figures and vertical …
Figure 6
Figure 6. Figure 6: Training set collection using REFOCUS on ChartQA dataset. 640 137 107 139 78 127 750 250 250 143 191 222 0 200 400 600 800 VWTQ VWTQ_syn VTabFact CharXiv Horizontal Bar Vertical Bar edited image all image [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Statistics of how often visual editing are performed. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Phi-3.5-vision finetuned with REFOCUS visual chain of thought data outputs the areas to focus on. For illustration purposes, we draw these areas in red boxes, and compare with the REFOCUS + GPT-4o prompting output. 10 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning

    cs.CV 2025-05 reject novelty 7.0 of 10

    Point-RFT uses point-grounded chain-of-thought with GRPO reinforcement to improve chart reasoning, reporting 90.04% on ChartQA, though internal tables and the OOD setup weaken the claim.

  2. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.

  3. FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A self-judging multi-agent VLM injects process-image helpfulness verdicts into tool observations and scales tool rewards by the helpful-call ratio, improving accuracy and tool faithfulness.

  4. Fully Spiking Neural Networks with Target Awareness for Energy-Efficient UAV Tracking

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A two-stage pure RL method with an information-gap global view and hierarchical grounding loss makes MLLMs truly rely on precise crops and sets SOTA on high-res VQA under tight token budgets.

  5. Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

    cs.CV 2025-07 conditional novelty 6.0 of 10

    RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.

  6. TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding

    cs.AI 2025-06 conditional novelty 6.0 of 10

    TableMoE is a multimodal table model whose role-aware router sends table tokens to HTML, JSON, and code experts and reports state-of-the-art results on its own WildStruct benchmarks and MMMU-Table.

  7. CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new 100-image benchmark shows that state-of-the-art multimodal models detect subtle, socially meaningful visual clues at near-chance levels and hallucinate accusations on innocent images.

  8. ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.

  9. MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    MedLVR interleaves latent visual reasoning segments in autoregressive decoding and uses two-stage training to raise average medical VQA accuracy from 48.3% to 53.4% over a Qwen2.5-VL-7B backbone on OmniMedVQA and five...

  10. Empowering Nanoscale Connectivity through Molecular Communication: A Case Study of Virus Infection

    cs.NI 2025-08 unverdicted novelty 4.0 of 10

    A position paper proposing molecular communication as the link layer for epidemic-control bio-nano networks, with an ORF3a-based mutation identification simulation; the provided manuscript body does not match this abstract.

  11. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

39 extracted references · 18 canonical work pages · cited by 11 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 6, 7, 8

    arXiv 2024

  2. [2]

    Vip- llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip- llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12914– 12923, 2024. 3

    work page 2024

  3. [3]

    Bigtable: A distributed storage system for structured data

    Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C Hsieh, Deborah A Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E Gruber. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS), 26(2):1–26, 2008. 1

    work page 2008

  4. [4]

    Tabfact: A large-scale dataset for table-based fact verification

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019. 4

    arXiv 1909

  5. [5]

    Hybridqa: A dataset of multi-hop question answering over tabular and textual data

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347, 2020. 1

    arXiv 2004

  6. [6]

    Visual chain- of-thought prompting for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, and Chuang Gan. Visual chain- of-thought prompting for knowledge-based visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 1254–1262, 2024. 1

    work page 2024

  7. [7]

    Selective attention and the organization of visual information

    John Duncan. Selective attention and the organization of visual information. Journal of experimental psychology: Gen- eral, 113(4):501, 1984. 1

    work page 1984

  8. [8]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390,

    arXiv

Show all 39 references
  1. [9]

    The cambridge structural database

    Colin R Groom, Ian J Bruno, Matthew P Lightfoot, and Suzanna C Ward. The cambridge structural database. Struc- tural Science, 72(2):171–179, 2016. 1

    2016

  2. [10]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 1, 2, 6

    2023

  3. [11]

    Chartllama: A mul- timodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A mul- timodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023. 1, 2

    2023 arXiv

  4. [12]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 8

    2022

  5. [13]

    Visual sketchpad: Sketching as a visual chain of thought for multi- modal language models

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multi- modal language models. arXiv preprint arXiv:2406.09403,

    arXiv

  6. [14]

    Selective attention

    William A Johnston and Veronica J Dark. Selective attention. Annual review of psychology, 1986. 1

    1986

  7. [15]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 5648–5656, 2018. 1

    2018

  8. [16]

    Tablevqa- bench: A visual question answering benchmark on multiple table domains

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa- bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205, 2024. 1, 2, 3, 4, 5

    2024 arXiv

  9. [17]

    Semantic-sam: Segment and recognize anything at any granu- larity

    Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granu- larity. arXiv preprint arXiv:2307.04767, 2023. 3

    2023 arXiv

  10. [18]

    Matcha: Enhancing visual language pretraining with math reasoning and chart derender- ing

    Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Col- lier, and Julian Martin Eisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derender- ing. arXiv preprint arXiv:2212.09662, 20...

    2022 arXiv

  11. [19]

    Deplot: One- shot visual language reasoning by plot-to-table translation

    Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One- shot visual language reasoning by plot-to-table translation. In Findings of the 61st Annual Meeting of ...

    2023

  12. [20]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 6, 7

    2024

  13. [21]

    Llava-plus: Learning to use tools for creating multimodal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437, 2023. 1

    2023 arXiv

  14. [22]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun 9 Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3

    2023 arXiv

  15. [23]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 2

    2024

  16. [24]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 1, 3, 4, 5, 8

    2022 arXiv

  17. [25]

    Unichart: A universal vision- language pretrained model for chart comprehension and rea- soning

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Ena- mul Hoque, and Shafiq Joty. Unichart: A universal vision- language pretrained model for chart comprehension and rea- soning. arXiv preprint arXiv:2305.14761, 2023. 1, 2

    2023 arXiv

  18. [27]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 6

    2023

  19. [28]

    Compositional se- mantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional se- mantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015. 3

    2015 arXiv

  20. [29]

    Space and selective attention

    Giacomo Rizzolatti, Lucia Riggio, Boris M Sheliga, et al. Space and selective attention. Attention and performance XV, 15:231–265, 1994. 1

    1994

  21. [30]

    Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. In The Thirty-eight Conference on Neural Informat...

    2024

  22. [31]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11987–11997, 2023. 3

    2023

  23. [32]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. Proceed- ings of IEEE International Conference on Computer Vision (ICCV), 2023. 1, 2

    2023

  24. [33]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6

    2023 arXiv

  25. [34]

    Charxiv: Charting gaps in realis- tic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yi- tao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realis- tic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521, 2024. 1, 4

    2024 arXiv

  26. [35]

    Chain-of- thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837, 2022. 1

    2022

  27. [36]

    List items one by one: A new data source and learning paradigm for multimodal llms

    An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. List items one by one: A new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404.16375, 2024. 3

    2024 arXiv

  28. [37]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 2, 3, 7

    2023 arXiv

  29. [38]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 3

    2024

  30. [39]

    Multimodal chain-of-thought rea- soning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought rea- soning in language models. arXiv preprint arXiv:2302.00923,

    arXiv

  31. [40]

    id": "train-two_col_103562

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021. 1 Acknowledgments We would like to...

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.