Pith. sign in

REVIEW 17 cited by

ChartBench: A Benchmark for Complex Visual Reasoning in Charts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.15915 v3 pith:6ZQBCOZ5 submitted 2023-12-26 cs.CV

Zhengzhuo Xu , Sinan Du , Yiyan Qi , Chengjin Xu , Chun Yuan , Jian Guo This is my paper
classification cs.CV
keywords chartmllmschartbenchchartscomprehensionbenchmarkcomplexdata
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have shown impressive capabilities in image understanding and generation. However, current benchmarks fail to accurately evaluate the chart comprehension of MLLMs due to limited chart types and inappropriate metrics. To address this, we propose ChartBench, a comprehensive benchmark designed to assess chart comprehension and data reliability through complex visual reasoning. ChartBench includes 42 categories, 66.6k charts, and 600k question-answer pairs. Notably, many charts lack data point annotations, which requires MLLMs to derive values similar to human understanding by leveraging inherent chart elements such as color, legends, and coordinate systems. We also design an enhanced evaluation metric, Acc+, to evaluate MLLMs without extensive manual or costly LLM-based evaluations. Furthermore, we propose two baselines based on the chain of thought and supervised fine-tuning to improve model performance on unannotated charts. Extensive experimental evaluations of 18 open-sourced and 3 proprietary MLLMs reveal their limitations in chart comprehension and offer valuable insights for further research. Code and dataset are publicly available at https://chartbench.github.io.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Attentive-CoT is an attention-guided fine-tuning objective that improves chain-of-thought performance in multimodal LLMs by delaying answer commitment and increasing sustained visual-token access during rationale generation.

  2. MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    MM-JudgeBias benchmark shows that many MLLM judges neglect modalities and produce unstable evaluations under small input changes, based on tests of 26 models with over 1,800 samples.

  3. PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading

    cs.AI 2026-01 conditional novelty 7.0 of 10

    PlotChain benchmark reports top MLLMs reaching ~80% field-level accuracy on engineering plot reading under human-like tolerances, but with persistent failures on frequency-domain tasks like bandpass and FFT spectra.

  4. ChartCap: Mitigating Hallucination of Dense Chart Captioning

    cs.CV 2025-08 conditional novelty 7.0 of 10

    A new 565K-pair chart-caption dataset with schema-based dense captions and a reference-free visual consistency metric improves VLM captioning and reduces hallucination.

  5. LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

    cs.CL 2026-08 reject novelty 6.0 of 10

    LongChart is a graph-consistent multi-chart VQA benchmark where 10 multimodal LLMs lose accuracy as question reasoning hops grow.

  6. MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Current multimodal LLMs can copy the look of multi-view dashboards but mostly fail to bind real data and implement cross-view interactions.

  7. Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

    cs.HC 2026-06 unverdicted novelty 6.0 of 10

    Introduces a benchmark for MLLM-based chart data extraction from unlabeled images and a human-centered training framework that reaches SOTA numerical accuracy with a 7B model.

  8. CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    CrystalXRD-Bench is a new 250-sample benchmark for VLMs on XRD peak indexing, where the best model (GPT-5.4) reaches Jaccard 0.5888 and 37.6% exact match while most stay below 0.50, showing the task remains unsolved.

  9. FinDocMRE: A Benchmark for Document-Level Financial Multimodal Reasoning Evaluation

    cs.CE 2026-05 unverdicted novelty 6.0 of 10

    FinDocMRE is a new multi-image document-level benchmark spanning 12 financial domains and 5 task types, showing that 11 tested LMMs all score below 65 overall with particular weaknesses in numerical estimation and cro...

  10. Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Chart-FR1 uses Focus-CoT for linking reasoning to visual cues and Focus-GRPO reinforcement learning with efficiency rewards to outperform prior MLLMs on dense chart reasoning tasks.

  11. CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    CharTool equips MLLMs with cropping and code tools plus agentic RL on DuoChart data to raise chart-reasoning accuracy by up to 9.78 percent on benchmarks.

  12. ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch

    cs.CV 2026-01 conditional novelty 6.0 of 10

    ChartVerse uses Rollout Posterior Entropy and truth-anchored inverse QA synthesis to produce 640K high-quality chart reasoning samples, training an 8B model that surpasses its 30B teacher.

  13. CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generation

    cs.CL 2025-12 unverdicted novelty 6.0 of 10

    CycleChart is a consistency-based framework that organizes chart generation, schema parsing, data parsing, and QA around single data instances to enforce bidirectional semantic alignment and improve cross-task generalization.

  14. Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.

  15. FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new benchmark of real-world financial charts shows current vision-language models lag badly on questions that require reading values from chart axes.

  16. In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    ChartScope, using a template-based synthetic data pipeline and dual-path reasoning training, outperforms prior chart-reading models on several advanced chart benchmarks.

  17. Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    The preprint's abstract claims a sparse softmax variant that masks non-competitive classes and accelerates training, but the provided body contains an unrelated chart-captioning paper and none of the claimed method.

Pith tools