REVIEW 6 cited by
SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this work, we present SciGraphQA, a synthetic multi-turn question-answer dataset related to academic graphs. SciGraphQA is 13 times larger than ChartVQA, the previously largest chart-visual question-answering dataset. It is also the largest open-sourced chart VQA dataset with non-synthetic charts. To build our dataset, we selected 290,000 Computer Science or Machine Learning ArXiv papers published between 2010 and 2020, and then used Palm-2 to generate 295K samples of open-vocabulary multi-turn question-answering dialogues about the graphs. As context, we provided the text-only Palm-2 with paper title, abstract, paragraph mentioning the graph, and rich text contextual data from the graph itself, obtaining dialogues with an average 2.23 question-answer turns for each graph. We asked GPT-4 to assess the matching quality of our question-answer turns given the paper's context, obtaining an average rating of 8.7/10 on our 3K test set. We evaluated the 0-shot capability of the most popular MLLM models such as LLaVa, mPLUGowl, BLIP-2, and openFlamingo's on our dataset, finding LLaVA-13B being the most performant with a CIDEr score of 0.08. We further enriched the question prompts for LLAVA by including the serialized data tables extracted from the graphs using the DePlot model, boosting LLaVA's 0-shot CIDEr to 0.15. To verify the validity of our dataset, we also fine-tuned LLaVa using our dataset, reaching a substantially higher CIDEr score of 0.26. We anticipate further accuracy improvement by including segmentation mask tokens and leveraging larger LLM backbones coupled with emergent prompting techniques. Our code and data are open-sourced.
Forward citations
Cited by 6 Pith papers
-
SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence
A fine-tuned CLIP+SciBERT multimodal scorer with manuscript-bound inputs and within-paper ranking reaches MAE 0.35 and 82% pairwise accuracy on held-out conference figures, far above LLM judges.
-
FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models
A new benchmark of real-world financial charts shows current vision-language models lag badly on questions that require reading values from chart axes.
-
Heterogeneous Graph Prompt Learning via Adaptive Weight Pruning
GPAWP prunes low-importance graph prompts using gradient-based importance masks and retunes the remaining prompts, improving few-shot node classification over HGPrompt on ACM, DBLP, and Freebase.
-
Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
A new 1,430-item multimodal benchmark shows that leading multimodal LLMs rarely notice small visual traps needed for commonsense safety reasoning, with top scores near 0.46 on a scale whose maximum is about 0.97.
-
Coling-UniA at SciVQA 2025: Few-Shot Example Retrieval and Confidence-Informed Ensembling for Multimodal Large Language Models
An ensemble of InternVL3-78B and Pixtral-Large with retrievable few-shot examples and a confidence threshold beats individual models on scientific figure question answering.
-
Evaluating LLMs for Visualization Generation and Understanding
GPT-4o led on most chart generation and understanding tests in this sample, but all four models struggled with complex charts, dotted lines, and close bar lengths.
Discussion (0). Sign in to comment.