Pith. sign in

REVIEW 4 major objections 6 minor 21 references

ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChartAgent claims that chart QA's weak spot — unannotated charts — is fixed by an agent that segments, measures, and visually verifies chart geometry, posting chart-benchmark scores up to 16.07 points above prior methods.

desk verdict The chart-specialized tool loop is a real and well-ablated contribution, but the abstract’s SOTA claim rests on an unreleased 125-pair dataset and a leakage hypothesis the paper itself contradicts in §7. read the letter →

arxiv 2510.04514 v3 pith:TLTFLRHL submitted 2025-10-06 cs.AI cs.CEcs.CLcs.CVstat.ME

Rachneet Kaur , Nishan Srishankar , Zhen Zeng , Sumitra Ganesh , Manuela Veloso This is my paper
classification cs.AIcs.CEcs.CLcs.CVstat.ME
keywords chartvisualquestionansweringmultimodalagentsvisuallygroundedreasoningunannotatedchartschart-specificvisiontoolsself-verificationBenchX
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the right way to answer hard questions about charts is to stop relying on a model's eyes alone and instead give it hands: an agent that iteratively segments pie slices, isolates bars, reads axes, and verifies its own intermediate drawings before answering. The target is unannotated charts — visualizations with no printed numbers to copy — where even the strongest multimodal LLMs collapse to roughly a quarter or a third accuracy. ChartAgent claims the top published scores on the ChartBench and ChartX benchmarks by a wide margin on exactly those charts, up to 16.07% absolute overall and 17.31% on unannotated, numerically intensive queries, and shows its tool layer lifts accuracy across every base model it wraps. If the mechanism is real, it maps a route to precision visual reasoning that scales with tools and agents rather than with model size or prompt engineering alone.

What carries the argument

The central mechanism is a perception–reasoning loop that gives the model hands in the image itself. An orchestrator extracts chart metadata and routes annotated charts — ones with readable numbers — straight to the base model, reserving the tool loop for unannotated charts and numeric queries. Each cycle has three parts: a Thought that picks the next subtask, a tool Action that manipulates the chart, and an Observation in which the model inspects what the tool produced. The tool library pairs universal tools (legend annotation, marker-color extraction, segmentation, axis localization, pixel-to-value interpolation) with chart-type tools (segment-area computation for pies and treemaps, bar de

What would settle it

Take a fresh set of unannotated charts generated after all model training cutoffs — charts that have never appeared on the web or in any public benchmark — and run ChartAgent against the newest frontier models plus the same base model with chain-of-thought prompting. If ChartAgent's margin over its own base model shrinks to well below the reported 12–28 points, or if the newest models keep their public-benchmark edge on the fresh set, the leakage explanation and the SOTA claim both fall. A cheaper probe: re-render ChartBench's charts with altered colors, legends, and axis scales and measure wh

Watch

Extended reading notes

Core claim

ChartAgent's central claim is that chart question answering fails not because vision-language models reason poorly but because they never actually measure — they guess bar heights and pie fractions from a compressed image, with no way to check. The discovery is that an iterative think–act–observe loop, in which the model calls chart-specific tools that annotate legends, crop regions, segment pie slices, isolate bars, and map pixels to axis values, then inspects the resulting visualizations to verify its work, produces large accuracy gains precisely where prompting fails: unannotated charts and numeric questions. ChartAgent reports 71.39% overall and 60.81% on unannotated ChartBench (17.31 po

Load-bearing premise

The load-bearing premise is that the newest frontier models' higher scores on the public benchmarks (GPT-o3 and GPT-o4-mini beat ChartAgent by 8–12 points on ChartBench) are artifacts of training-data leakage rather than real ability; the paper supports that with behavioral anecdotes, counters on a private 125-pair dataset with no specified sampling, and concedes in Section 7 that no strictly held-out evaluation was performed.

Editorial extensions

If this is right

  • If ChartAgent is right, the bottleneck in chart understanding is perception, not reasoning: grafting its tool loop onto a base model lifts accuracy on unannotated, numeric questions from 31.6% to 58.3% for GPT-4o and by 23.9–28.4 points for GPT-4o-mini and Claude 3 Haiku.
  • The agentic loop alone is not the win; the chart-specific tools are: in the paper's ablation, replacing them with generic crop/zoom tools drops overall accuracy from 71.4% to 41.4%, and a tool-free version lands at 38.8%.
  • Self-verification converts imperfect tools into a workable pipeline: the paper's trajectory analysis shows 50% of sampled runs needed recovery, and 70% of those recovered successfully, bounding tool-error leakage into final answers at 15%.
  • The framework is model-agnostic in principle: because the tools and loop wrap around any multimodal LLM, ChartAgent's accuracy should keep improving as base models improve — a cumulative-gain property the paper states as a design goal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper's leakage hypothesis is accepted, public chart benchmarks have a shelf life: the newest models it lists (GPT-o3 at 83.39, GPT-o4-mini at 80.18 on ChartBench) outscore ChartAgent's 71.39 on the public sets, and the claimed lead over them rests on a private 125-pair dataset (Appendix J.6.2) that is unreleased and whose sampling is unspecified — while Section 7 concedes no strictly held-
  • The recipe should transfer to any visual domain whose questions reduce to measurable primitives — maps, floor plans, circuit schematics, scientific figures. The paper's own failure taxonomy, which attributes most errors to perception rather than planning, suggests the agent's ceiling is set by tool robustness, so replacing its segmentation and OCR backends with sturdier ones is the cheapest predic
  • The per-query cost of the 15-step loop — roughly 125 seconds and $0.40 with the default base model — trades about an order of magnitude of latency for the accuracy gain. Whether that trade is worth it depends on the plug-and-play results with cheap base models (the paper quotes about $0.025 per sample with a smaller one) holding up on harder, unannotated-heavy benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ChartAgent, an agentic ReAct-style framework that augments a base multimodal LLM (GPT-4o) with a library of chart-specialized perception tools—legend annotation, segmentation, axis localization, bar-height estimation, pie-area computation, boxplot extraction, and so on—together with a visual self-verification loop. The system routes annotated, qualitative queries directly to the base MLLM and invokes the tool loop for unannotated charts and numeric questions. The central empirical claims are: (i) state-of-the-art accuracy on ChartBench and ChartX, with large absolute gains on unannotated and numeric chart QA; (ii) plug-and-play gains across multiple base MLLMs; and (iii) superiority over generic natural-image tool-use and no-tool ReAct baselines. The paper also reports analyses by chart type, visual/reasoning complexity, tool-usage patterns, recovery behavior, and failure modes.

Significance. If the central claims hold, the paper makes a meaningful contribution: it demonstrates that explicit, chart-specific visual manipulation—rather than larger models or prompt engineering alone—can substantially improve chart QA, and it provides a detailed, reusable tool taxonomy. The core ablation (Fig. 4c / App. Table 9) is a genuinely informative controlled comparison, and the evaluation on external benchmarks with fixed ground truth gives the results an objective anchor. The plug-and-play experiments across four base MLLMs are a useful generalization check. However, the headline SOTA claim is not currently load-bearing: on the public benchmarks, the paper's own Table 13 shows several concurrent models outperforming ChartAgent, and the only evidence for SOTA over those models rests on an unreleased 125-pair internal dataset and an unverified data-leakage hypothesis. The mechanistic contribution is credible, but the paper's strongest framing needs to be either verified with a released/audited evaluation or appropriately qualified.

major comments (4)
  1. [Abstract and §5.1, Table 13] The abstract and Section 5.1 claim state-of-the-art accuracy on ChartBench and ChartX. This is not supported by the paper's own public-benchmark comparison. Table 13 shows GPT-o3 (83.39 vs 71.39), GPT-o4-mini (80.18), GPT-4.1 (76.58), and Claude 3.7 Sonnet (72.18) all exceed ChartAgent on ChartBench, and GPT-o3/o4-mini/4.1/Claude 3.7 also exceed ChartAgent on ChartX (78.82/79.77/78.56/71.44 vs 59.69). Under these numbers, ChartAgent is SOTA only among non-concurrent baselines. The claim should be restricted to that comparison class, or supported by the released/audited internal evaluation.
  2. [App. J.6.2 and §7] The only evidence for SOTA over concurrent models is Table 14 in App. J.6.2: a private, self-curated 125-pair dataset. The sampling protocol is unspecified, the dataset is not released, and Section 7 explicitly states: 'we did not evaluate our method on a newly constructed or strictly held-out dataset to check for potential data leakage.' This directly contradicts the appendix's description of a curated held-out dataset. A 10.48-point gap over GPT-5 corresponds to roughly 13 questions, so small curation biases could change the ranking. Please release the internal set (or provide a detailed audit) and report confidence intervals, or remove the concurrent-model superiority claim from the headline.
  3. [App. J.6.1] The leakage hypothesis—that newer models memorized ChartBench/ChartX—is supported only by anecdotal observations of GPT-o3 and GPT-o4-mini behavior (correct answers despite wrong reasoning, decimal-level precision). This is plausible but does not constitute controlled evidence. Provide a systematic analysis, e.g., accuracy changes on re-rendered or perturbed charts, a known-provenance held-out sample, or public contamination checks, before treating the hypothesis as 'strong evidence.' Without such evidence, the SOTA claim is conditional on an unverified assumption.
  4. [Fig. 4c / App. Table 9] The ablation is the strongest evidence for the paper's mechanistic claim, but the figure and table report only single deterministic runs. Please add repeated-run statistics or at least state the stability of the reported differences across seeds and sampled trajectories; otherwise the large numeric gaps, while likely meaningful, lack variance estimates. This is especially relevant because the recovery analysis (Fig. 3b/Table 15) is based on only 30 trajectories.
minor comments (6)
  1. [App. L.3.1] The evaluation standardizer is GPT-4o, which is also ChartAgent's base model. All models are processed by the same standardizer, but a short agreement check against a deterministic normalization or a second judge would rule out systematic bias in the accuracy metric.
  2. [Table 14] The 125-pair internal dataset should be described in the appendix with a breakdown by chart type, annotation status, and question type. Without this, it is difficult to assess representativeness or the risk of selective curation.
  3. [Table 1] The table header in panel (b) uses 'Global Conception QA'; the text elsewhere uses 'Global Perception.' Please standardize the terminology.
  4. [Table 8] Minor typo: 'analyse_radial_geometry' in Table 8 should be 'analyze_radial_geometry' to match the tool name in App. F and L.1.2.
  5. [§5.1] The sentence 'ChartAgent consistently outperforms all competing methods' should be qualified as 'all non-concurrent baselines' to avoid contradicting Table 13.
  6. [App. J.9/J.10] The latency (~125s per query for ChartAgent vs ~12s for GPT-4o) and cost ($0.40 per sample) are important practical trade-offs. They are reported, but the abstract and Section 1 would benefit from a one-sentence acknowledgment that the gains come at substantial compute/cost overhead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons against fixed external benchmark ground truth, and no fitted parameter, uniqueness theorem, or load-bearing self-citation chain is present.

full rationale

ChartAgent's central claims are benchmark-accuracy measurements on ChartBench and ChartX (and an internal 125-pair set), not derivations from fitted quantities. There is no step where a parameter is fit to a subset and then reported as a prediction, no self-definitional relation between an input and an output, and no uniqueness theorem or ansatz imported through self-citation. The only self-citation (Verma et al. 2024) appears in a related-work list about GUI agents and is not load-bearing. The use of GPT-4o both as the base model and as the response standardizer is an evaluation-harness overlap, but it is not a definitional reduction: standardization is formatting-only and is followed by an arithmetic/string correctness check against fixed ground truth, so accuracy is not constructed from ChartAgent's own outputs. The internal-dataset and leakage concerns raised in §7 and Appendix J.6 are empirical validity issues—the internal benchmark is unreleased and its sampling protocol is unspecified, and Table 13 shows concurrent models outperforming ChartAgent on the public benchmarks—but those concerns do not make any prediction equivalent to an input by construction. Accordingly, this paper exhibits no meaningful circularity under the stated criteria.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

Everything the central claim rests on that the reader did not pay for upstream: public-benchmark gains depend on a stack of computer-vision assumptions (SAM segmentation, OCR reliability, RGB uniqueness, linear axes) plus hand-set tool thresholds with no sensitivity analysis; the claim of beating GPT-5/GPT-o3-class models depends on a private 125-pair dataset and an ad hoc leakage hypothesis about newer models memorizing public benchmarks. No physical or mathematical entities are invented; the only new artifacts are software tools and the unreleased evaluation set.

free parameters (6)
  • axis_threshold = 0.15–0.2 (fraction of image height/width scanned for tick labels)
    Hand-set hyperparameter in axis_localizer, compute_bar_height, get_boxplot, get_edgepoints; no sensitivity analysis provided. It determines which tick labels are detected and therefore every pixel-to-value interpolation.
  • segmentation cleanup thresholds = min_area=5000, iou_thresh_unique=0.9, iou_thresh_composite=0.98, white_ratio_thresh=0.95
    Hand-chosen filters in segment_and_mark that decide which SAM masks survive; they directly determine which bars, pie slices, and box segments are measurable.
  • ReAct iteration limit = 15
    Maximum loop steps per query; truncating or extending this changes how often visual self-verification can correct failed tool outputs (average trajectory ~125 s).
  • ICL examples per chart type = 1–2, retrieved by predicted chart type
    Few-shot trajectories appended to the prompt; the paper notes they are textual-only and that multimodal ICL was not explored, which is a stated limitation.
  • evaluation tolerance and unit-stripping rule = 5% relative error; 500M vs 500 counted as equal when only one value carries a unit
    The scoring rule is itself a free choice that shapes every reported accuracy number (App. L.3.1); the 500M vs 500 equivalence is a lenient comparison that can count wrong-by-10^6 answers as correct.
  • internal dataset size and construction = 125 chart–QA pairs
    Self-curated held-out set used for the concurrent-model comparison and the claim of beating GPT-5; sampling protocol and selection criteria are not described.
assumptions (7)
  • domain assumption SAM (ViT-H) segmentation reliably isolates chart elements (bars, slices, boxes) across 40+ chart types
    segment_and_mark and all chart-specific tools depend on it; the paper's own failure modes 1.5–1.6 (Sec. 5.4, App. K) show it fails on axis-overlapping and heavily overlapping elements.
  • domain assumption OCR (Tesseract/EasyOCR) reads chart text, including slanted or partial tick labels
    axis_localizer and annotate_legend rely on OCR; the fallback analysis (§5.3) admits OCR returning None for legends/axis labels as a common trigger.
  • standard math Linear interpolation between adjacent detected axis ticks yields correct pixel-to-value mapping
    interpolate_pixel_to_value implements linear interpolation; valid for linear axes, unstated and unverified for log/nonlinear axes found in some chart subtypes.
  • domain assumption RGB color uniquely identifies a category or series within a chart
    The get_marker_rgb → filter_rgb pipeline assumes legend marker colors match data element colors and are distinct; failure modes 1.2 (poor contrast) and 1.6 (overlap) show where this breaks.
  • domain assumption GPT-4o-orchestrator classifies annotated vs unannotated charts correctly
    The routing decision in §3/A.2 determines whether the tool loop runs at all; the claimed 100% accuracy is on a 'uniformly sampled subset' whose size is not reported.
  • ad hoc to paper Concurrent models (GPT-o3, GPT-o4-mini, GPT-4.1, Gemini 2.0, etc.) memorized or leaked ChartBench/ChartX into training
    §J.6.1 uses this hypothesis to discount concurrent models that score 6–20 points higher than ChartAgent on the public benchmarks; the evidence is observational (correct answers despite wrong tool traces, decimal-level precision) and not verified.
  • domain assumption Counting discrete dots or color-filtered pixels approximates pie/treemap segment area
    compute_segment_area relies on dot/pixel counts as an area proxy (Fig. 1: 350/874 dots → 40.05%); accuracy depends on chart rendering, antialiasing, and dithering and is not characterized.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering." pith.science (2026). https://pith.science/paper/TLTFLRHL

@misc{pith2026251004514,
  author       = {Pith},
  title        = {Pith review of: ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLTFLRHL}},
  note         = {Machine review of arXiv:2510.04514}
}
read the original abstract

Recent multimodal LLMs have shown promise in chart-based visual question answering, but their performance declines sharply on unannotated charts-those requiring precise visual interpretation rather than relying on textual shortcuts. To address this, we introduce ChartAgent, a novel agentic framework that explicitly performs visual reasoning directly within the chart's spatial domain. Unlike textual chain-of-thought reasoning, ChartAgent iteratively decomposes queries into visual subtasks and actively manipulates and interacts with chart images through specialized actions such as drawing annotations, cropping regions (e.g., segmenting pie slices, isolating bars), and localizing axes, using a library of chart-specific vision tools to fulfill each subtask. This iterative reasoning process closely mirrors human cognitive strategies for chart comprehension. ChartAgent achieves state-of-the-art accuracy on the ChartBench and ChartX benchmarks, surpassing prior methods by up to 16.07% absolute gain overall and 17.31% on unannotated, numerically intensive queries. Furthermore, our analyses show that ChartAgent is (a) effective across diverse chart types, (b) achieves the highest scores across varying visual and reasoning complexity levels, and (c) serves as a plug-and-play framework that boosts performance across diverse underlying LLMs. Our work is among the first to demonstrate visually grounded reasoning for chart understanding using tool-augmented multimodal agents.

Figures

Figures reproduced from arXiv: 2510.04514 by the authors.

Figure 1
Figure 1. Comparison of our work with the existing SoTA. (a) ChartAgent performs visually grounded reasoning in the chart domain. For this unannotated chart, GPT-4o fails to produce the correct answer, whereas ChartAgent succeeds. (b) ChartAgent performance on unannotated charts and numeric QA compared with the top-10 SoTA. knowledge, and complementary to existing chart VQA approaches that rely on prompting or fine￾tuning MLL… view at source ↗
Figure 2
Figure 2. ChartAgent. The (A) orchestrator extracts chart metadata and routes annotated charts with textual shortcuts and qualitative QA to the base MLLM, while unannotated charts and numeric queries trigger the ReAct￾style loop. The system includes (B) a library of universal and chart-specific tools, (C) metadata for parameterizing tool usage and retrieving chart-type-specific ICL examples, and (D) few-shot ICL retrieval. Us… view at source ↗
Figure 3
Figure 3. (a) Left: ChartAgent vs. concurrent works: overall accuracy (↑) and average absolute error (↓). (b) Right: Effectiveness of visual self-verification: en￾abled 70% successful recoveries when invoked. enforce thresholds or miss small deviations (see App. J.5). See App. L.3 for evaluation prompts. 5 Results and Analysis 5.1 Performance Comparison to State-of-the-art [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Analysis of ChartAgent Performance. (a) Left: Stratified by visual complexity of charts and reasoning complexity of chart–QA pairs on unannotated charts, compared with top-10 SoTA. (b) Middle: ChartAgent performance on unannotated+numeric chartQA when instantiated with…
Figure 5
Figure 5. Figure 5: Examples of annotated (top) vs. unannotated (bottom) charts. An annotated chart contains explicit textual annotations or shortcuts, whereas an unannotated chart lacks such explicit value indicators. For instance, in the first column (top), the bar chart includes printe…
Figure 6
Figure 6. Figure 6: Chart types in the ChartX dataset: 18 types organized into three subcategories—general, fine-grained, and domain-specific chart types, with the percentage of data in each subcategory indicated. Over 60% of the data is unannotated, making ChartX a robust testbed for vis…
Figure 7
Figure 7. Figure 7: Chart types in the ChartBench dataset: 9 major types with 38 subtypes (excluding 4 subtypes with corrupted or incorrect ground-truth labels). Annotated subtypes are marked in green, and unannotated subtypes are marked in red. Over 75% of the data is unannotated, making…
Figure 8
Figure 8. Figure 8: Illustrative examples of key intermediate and final output visualizations for universal tools in ChartAgent. These visualizations are critical to facilitating visual self-verification in ChartAgent. Such tool observations enable ChartAgent to perceptually assess the ou…
Figure 9
Figure 9. Figure 9: Illustrative examples of key intermediate and final output visualizations for chart-specific tools in ChartAgent. These visualizations enable visual self-verification in ChartAgent, allowing it to refine tool usage through perceptual assessment and iterative correction…
Figure 10
Figure 10. Figure 10: Complexity dimensions in chart–QA pairs. Representative examples are shown for (a) visual complexity of charts and (b) reasoning complexity of chart–QA pairs, each categorized into Easy, Medium, and Hard levels. (a) For visual complexity: Easy charts (e.g., single bar…
Figure 11
Figure 11. Figure 11: Tool-use statistics across benchmark datasets. Percentage of times ChartAgent employs a given tool when solving queries for each chart type. As expected, universal tools are used broadly across all chart types, whereas chart-specific tools are invoked selectively depe…
Figure 12
Figure 12. Figure 12: Failure Mode Analysis. Examples where ChartAgent fails to produce the correct response due to visual perception challenges or reasoning ambiguities. (A) Perception-based failures include OCR obstruction by overlays, poor color contrast, key chart element occlusions (e…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 linked inside Pith

  1. [1]

    • Normalize both answers carefully by checking both the <groundtruth answer> and <predicted answer>values in context

    If both answers arenumeric, first extract the numeric portion of each value. • Normalize both answers carefully by checking both the <groundtruth answer> and <predicted answer>values in context. • If both values include thesame unit(e.g., K, M, B, or their full-word equivalents likethousand, million, billion),do not scale. Simply strip the unit and compar...

  2. [2]

    ground_truth_filtered

    If the answers are not numeric, return the string response. Return a JSON object in the following format: { "ground_truth_filtered": <normalized ground truth value>, "response_filtered": <normalized predicted value> } Input Format: <question> {entry["query"]} <groundtruth answer> {entry["ground_truth"]} <response> {original_response} L.3.2 LLM-as-a-Judge ...

    2023

  3. [3]

    How much higher is X than Y?

    Evaluate the complexity of each step, considering: • The need for precise perception (e.g., color or shape differentiation, relative positioning) • Cross-referencing multiple regions, axes, or visual types • Complex chart features (e.g., stacked vs. overlayed areas, 3D perspective) • Occlusion or ambiguity in label visibility (e.g., overlapping text or hi...

  4. [4]

    Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools.Preprint, arXiv:2406.12793. Google. 2025. Gemini 2.0 flash model card. Accessed: 2025-07-09. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The ll...

    arXiv 2025

  5. [6]

    InInternational conference on ma- chine learning, pages 19730–19742

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. InInternational conference on ma- chine learning, pages 19730–19742. PMLR. Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. 2025. Draw-and- understand: Leveraging visual prompts...

    arXiv 2025

  6. [7]

    Visual chatgpt: Talking, drawing and edit- ing with visual foundation models.arXiv preprint arXiv:2303.04671. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024a. Autogen: Enabling next-gen LLM applications via mult...

    arXiv 2024

  7. [8]

    arXiv preprint arXiv:2404.07972

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2023. Chartbench: A bench- mark for complex visual reasoning in charts.arXiv preprint arXiv:2312.15915. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan...

    arXiv 2023

  8. [9]

    Preprint, arXiv:2408.04840

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. Preprint, arXiv:2408.04840. Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang

    arXiv

Show all 21 references
  1. [10]

    thinking with images

    Tinychart: Efficient chart understanding with program-of-thoughts learning and visual token merg- ing. InProceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pages 1882–1898. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. G...

    2024 arXiv

  2. [11]

    What is the value of India in 2021?

    integrates text-based language models with external APIs, demonstrating improved reason- ing through external knowledge retrieval. Simi- arly, Visual ChatGPT (Wu et al., 2023) and MM- ReAct (Yang et al., 2023b) enhance text-only Chat- GPT with vision expert tools for multimoda...

    2023

  3. [16]

    in thousands

    model as the default, which has a size of 2.56 GB and 641 million parameters. For OCR tasks, we utilize Tesseract OCR (Smith, 2007) and Easy- OCR (AI, 2020). This setup facilitates efficient execution of both model inference and vision tool pipelines. All ChartAgent prompts ar...

    2007

  4. [19]

    Identify the key visual elements referenced by the question

  5. [20]

    Determine the number of distinct reasoning steps needed to answer it

  6. [42]

    For segmentation tasks, we use the Segment Anything (ViT-H) (Kirillov et al.,

    All experiments are conducted on a Linux ma- chine using an AWS g4dn.xlarge instance, which is equipped with a single NVIDIA T4 GPU with 16 GB of memory. For segmentation tasks, we use the Segment Anything (ViT-H) (Kirillov et al.,

  7. [203]

    (b) Chart-specific Tools Localized the X-axis and applied pixel-to-value interpolation to compute the median from the left and right parts of the detected box of interest

    for the category of interest ‘Renewable Energy’. (b) Chart-specific Tools Localized the X-axis and applied pixel-to-value interpolation to compute the median from the left and right parts of the detected box of interest. 100 150 200 250 100 150 200 250 Left Right Localized the...

    2015

  8. [2020]

    The tools that utilize EasyOCR are the same as above

    is used as a backup forTesseract OCR which can fail in more complicated and noisy images. The tools that utilize EasyOCR are the same as above. EasyOCR uses a VGG16- based CRAFT (Character Region Awareness for Text Detection, 138M parameters) for text de- tection followed by a...

    2000

  9. [2022]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi

    Interactive visualization applications in popu- lation health and health services research: systematic scoping review.Journal of medical Internet research, 24(2):e27534. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale ...

  10. [2023]

    Instructblip: Towards general-purpose vision- language models with instruction tuning.Preprint, arXiv:2305.06500. Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Ro- jas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Su...

    arXiv

  11. [2024]

    https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

    GPT4o-mini. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ . Accessed: 2025-05-01. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jian- min Bao, Harkirat Behl,...

    2025 arXiv

  12. [2025]

    InICASSP 2025-2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5

    Interactive robot action replanning using multi- modal llm trained from human demonstration videos. InICASSP 2025-2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, B...

    2025 arXiv

  13. [2409]

    For reproducibility, the random seed is set to

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.