REVIEW 13 cited by
Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The remarkable progress of Multi-modal Large Language Models (MLLMs) has attracted significant attention due to their superior performance in visual contexts. However, their capabilities in turning visual figure to executable code, have not been evaluated thoroughly. To address this, we introduce Plot2Code, a comprehensive visual coding benchmark designed for a fair and in-depth assessment of MLLMs. We carefully collect 132 manually selected high-quality matplotlib plots across six plot types from publicly available matplotlib galleries. For each plot, we carefully offer its source code, and an descriptive instruction summarized by GPT-4. This approach enables Plot2Code to extensively evaluate MLLMs' code capabilities across various input modalities. Furthermore, we propose three automatic evaluation metrics, including code pass rate, text-match ratio, and GPT-4V overall rating, for a fine-grained assessment of the output code and rendered images. Instead of simply judging pass or fail, we employ GPT-4V to make an overall judgement between the generated and reference images, which has been shown to be consistent with human evaluation. The evaluation results, which include analyses of 14 MLLMs such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini, highlight the substantial challenges presented by Plot2Code. With Plot2Code, we reveal that most existing MLLMs struggle with visual coding for text-dense plots, heavily relying on textual instruction. We hope that the evaluation results from Plot2Code on visual coding will guide the future development of MLLMs. All data involved with Plot2Code are available at https://huggingface.co/datasets/TencentARC/Plot2Code.
Forward citations
Cited by 13 Pith papers
-
Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?
A new benchmark called ToxiMol evaluates how well 43 multimodal LLMs can edit toxic molecules into structurally similar, non-toxic, drug-like candidates; the best model succeeds on 43.3% of tasks.
-
Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation
A 7B VLM trained with a structured chart-specification reward beats larger and commercial models on chart-to-code benchmarks using only 3K-4K training samples.
-
Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation
A fully automated pipeline generates a 222.5K-pair synthetic chart dataset with 27 chart types and 11 plotting libraries, and a GPT-4o-judged benchmark shows current open-weights VLMs still underperform on chart-to-co...
-
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
A 55-criteria guideline and audit of 274 code benchmarks finds that most benchmarks skip data quality checks, prompting calls for more rigorous, reproducible benchmark construction.
-
ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
ChartCoder, a 7B multimodal LLM with a code-LLM backbone trained on 160k synthetic chart-code pairs, surpasses previous open-source models at converting chart images into executable plotting code.
-
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
A new multimodal reasoning benchmark shows that state-of-the-art AI models lag human experts by more than 30 percentage points, with visual reasoning errors as the main bottleneck.
-
An Exploratory Study of ML Sketches and Visual Code Assistants
A 19-participant study finds vision LLMs convert ML whiteboard sketches into correct code outlines about 79% of the time, but only about 36% of implementation details are correct.
-
Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
PandasPlotBench is a 175-task benchmark showing that LLMs generate good Matplotlib and Seaborn plots from brief instructions but struggle with Plotly.
-
ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges
A new 305-question benchmark uses Scratch block programs to test multimodal AI visual programming reasoning, and all ten tested models fall below 70% accuracy.
-
DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation
A dual-phase diffusion planner with dynamics-consistency and LLM-written guidance achieves strong success on goal-adaptive dexterous manipulation in simulation.
-
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
Merging a coding LLM into a vision-language model via task vectors yields an open-source multimodal coder that reaches near-GPT-4o performance on the authors' new benchmark.
Discussion (0). Continue with ORCID to comment.