REVIEW 9 cited by
MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
read the original abstract
Scientific figure interpretation is a crucial capability for AI-driven scientific assistants built on advanced Large Vision Language Models. However, current datasets and benchmarks primarily focus on simple charts or other relatively straightforward figures from limited science domains. To address this gap, we present a comprehensive dataset compiled from peer-reviewed Nature Communications articles covering 72 scientific fields, encompassing complex visualizations such as schematic diagrams, microscopic images, and experimental data which require graduate-level expertise to interpret. We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation. Our analysis revealed significant task challenges and performance gaps among models. Beyond serving as a benchmark, this dataset serves as a valuable resource for large-scale training. Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations. Furthermore, continuous pre-training on our interleaved article and figure data substantially enhanced the model's downstream task performance in materials science. We have released our dataset to support further research.
Forward citations
Cited by 9 Pith papers
-
SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detection
The first benchmark for AI-generated scientific figure detection shows existing detectors fail in zero-shot transfer, overfit to specific generators, and break under common image corruptions.
-
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
MMMU-Pro is a stricter multimodal benchmark that removes text-only solvable questions, augments options, and requires reading text from images, yielding substantially lower model scores of 16.8-26.9%.
-
GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models
GENFIG1 is a new benchmark that tests whether vision-language models can create effective Figure 1 visuals capturing the central scientific idea from paper text.
-
SciGA: A Comprehensive Dataset for Designing Graphical Abstracts in Academic Papers
Introduces the SciGA-145k dataset with intra-paper and cross-paper graphical abstract recommendation tasks plus the CAR evaluation metric.
-
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.
-
SciForma: Structure-Faithful Generation of Scientific Diagrams
A 9B open-weights model trained with axis-decomposed conjunctive preference optimization (M-DPO) and a structural inventory beats GPT-Image-1.5 on scientific-diagram structural-fidelity benchmarks.
-
A Multimodal Reasoning Typology for Grounding Chart-Image Coherence in Science Communication
A five-level typology of reasoning gaps (R1–R5) classifies how chart-image pairs in scientific papers jointly convey claims, predicting where expert and non-expert interpretations diverge.
-
SD-GRPO: Verifiable Segment Decomposition for Long-Form Vision-Language Generation
SD-GRPO extends GRPO by computing per-segment advantages via z-normalization of verifiable segment rewards, yielding gains on long-form VL tasks with varying semantic independence across segments.
-
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
DeepSeek-VL2 is a series of MoE vision-language models using dynamic tiling and latent attention that reach competitive or state-of-the-art results on VQA, OCR, document understanding and grounding with 1.0B to 4.5B a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.