REVIEW 8 cited by
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. ChartMimic includes 4,800 human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span 18 regular types and 4 advanced types, diversifying into 201 subcategories. Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of $3$ proprietary models and 14 open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of 82.2 and 61.6, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.
Forward citations
Cited by 8 Pith papers
-
DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable
A human-validated 39-criterion benchmark plus a parse-plan-reconstruct workflow for measuring and improving how multimodal agents convert raster images into editable vector artifacts.
-
CodePercept: Code-Grounded Visual STEM Perception for MLLMs
Perception, not reasoning, is the main bottleneck for MLLM STEM visual reasoning, and training on executable reconstruction code measurably fixes it.
-
Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation
A 7B VLM trained with a structured chart-specification reward beats larger and commercial models on chart-to-code benchmarks using only 3K-4K training samples.
-
Visual Programmability: A Guide for Code-as-Thought in Chart Understanding
A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.
-
Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.
-
In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
ChartScope, using a template-based synthetic data pipeline and dual-path reasoning training, outperforms prior chart-reading models on several advanced chart benchmarks.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
Merging a coding LLM into a vision-language model via task vectors yields an open-source multimodal coder that reaches near-GPT-4o performance on the authors' new benchmark.
Discussion (0). Continue with ORCID to comment.