REVIEW 4 cited by
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models' capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.
Forward citations
Cited by 4 Pith papers
-
ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts
A large-scale benchmark shows that leading multimodal language models still underperform expert humans at verifying climate claims from scientific charts.
-
SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning
Integrating binary self-verification into multi-turn GRPO rollouts raises VLM multimodal reasoning accuracy over matched GRPO baselines while the model learns to need fewer rethinks.
-
Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation
A 7B VLM trained with a structured chart-specification reward beats larger and commercial models on chart-to-code benchmarks using only 3K-4K training samples.
-
ChartLens: Fine-grained Visual Attribution in Charts
ChartLens uses segmentation and set-of-marks prompting to attribute chart-based answers to specific visual elements, and the authors release a new benchmark for evaluating such attribution.
Discussion (0). Sign in to comment.