Chart Question Answering from Real-World Analytical Narratives

Aidan Slingsby; Jo Wood; Maeve Hutchinson; Pranava Madhyastha; Radu Jianu

REVIEW 3 major objections 4 minor 17 references

Chart Question Answering from Real-World Analytical Narratives

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a chart question answering dataset built from real literate-visualization notebooks, with questions generated from the students' analytical narratives rather than from chart images alone, and shows that…

desk verdict A small but genuinely new CQA dataset with an ecological-validity angle; the single-annotator validation is the main weakness, not a fatal one. read the letter →

arxiv 2507.01627 v1 pith:26X3L5VO submitted 2025-07-02 cs.CL

Maeve Hutchinson , Radu Jianu , Aidan Slingsby , Jo Wood , Pranava Madhyastha This is my paper

classification cs.CL

keywords chartquestionansweringvisualizationnotebooksmultimodallargelanguagemodelsbenchmarkdatasetanalyticalnarrativesinteractivevisualizationsmultiple-choiceevaluationecologicalvalidity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a chart question answering dataset built from real literate-visualization notebooks, with questions generated from the students' analytical narratives rather than from chart images alone. The resulting pairs involve multiple charts, multiple views, and some interactive states, and include a 'cannot be determined' option for genuinely unanswerable questions. The central claim is that this setup is more ecologically valid than existing CQA benchmarks, and that state-of-the-art multimodal language models still leave a clear performance gap: GPT-4.1 reaches 69.3% accuracy, with smaller models well below that. A sympathetic reading takes this as evidence that real-world chart reasoning is not yet solved by current MLLMs.

What carries the argument

The construction pipeline is the carrier: literate visualization notebooks provide narrative text plus custom charts; narrative is segmented, an LLM generates question-answer pairs grounded in quoted narrative and classified by the VLAT task taxonomy; a data-visualization expert manually validates each pair against actual screenshots, rejecting pairs that refer to unavailable visualizations or non-analytical narrative, and labels pairs as unanswerable when required information exists only in uncaptured interactive views. Interactive controls are captured statically by enumerating categorical options. This pipeline yields a benchmark whose questions cannot be answered from narrative alone, nor from a single static chart, and it is the mechanism that produces the claimed ecological validity.

What would settle it

If an independent second annotation of the 205 pairs found that a meaningful fraction, say more than 10%, of questions labeled answerable cannot actually be answered from the supplied screenshots alone, the dataset's core claim of visual answerability would be undercut. A simpler check: run GPT-4.1 again with the relevant narrative text appended to the question; if accuracy jumps dramatically, the questions are not actually being answered from the charts.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is a dataset—205 expert-validated multiple-choice QA pairs paired with 103 chart images from 22 student notebooks—that re-anchors chart question answering in authentic analytical workflows. It shows that when questions derive from written narrative and target multi-view or interactive charts, prior benchmarks' single-image, single-view format misses an important part of real practice. Benchmarking GPT-4.1 and Qwen2.5-VL at two scales gives 69.3%, 56.6%, and 31.7% accuracy respectively, a substantial gap that varies by task type, with interpretive tasks like Make Comparisons and Find Anomalies proving harder than literal retrieval.

Load-bearing premise

The benchmark's value rests on the assumption that one data-visualization expert correctly determined, from static screenshot captures, that each retained question is answerable from the provided visual inputs, and that no retained question implicitly depends on narrative-only information or on interactive dynamics that screenshots miss.

Editorial extensions

If this is right

Current MLLMs answer only about 69% of these real-world chart questions, leaving a roughly 30-point gap for deployment in visual-assistant settings where multi-chart, narrative-grounded reasoning is common.
The dataset includes a 'cannot be determined' answer option, so models that cannot find evidence are expected to say so, steering CQA evaluation toward calibration and honesty rather than guessing.
Performance varies by task type: literal retrieval tasks are easier, while finding anomalies and making comparisons are harder, suggesting the VLAT taxonomy is a useful diagnostic for where multimodal reasoning breaks.
Because the data comes from student notebooks with explicit consent, the dataset offers a copyright-clean and ethically approved alternative to scraped chart corpora for future benchmark construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The interactive-view capture method generalizes: any benchmark built from notebooks or dashboards with discrete controls could adopt the same enumeration trick, pushing CQA beyond static images without needing full interaction logs.
A natural next experiment is to condition the same questions on narrative text plus charts, separating how much of the gap is visual versus contextual; the paper's design, with narrative removed from the vision-only evaluation, already sets up this control.
The unanswerable-pair subset could be used to test model abstention behavior separately, since over-answering on 'cannot be determined' items would indicate a tendency to hallucinate from ambiguous visual evidence.
With only 205 pairs and a single expert validator, the 69.3% figure is a point estimate; re-annotation studies or larger notebooks would tell whether the performance gap is stable or partly a small-sample artifact.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper makes a real addition to the chart QA space. The dataset is small (205 QA pairs) but the design is new: questions generated from the narrative text of student-authored litvis notebooks, then validated against static captures that include multiple interactive views, plus a fifth 'cannot be determined' option. That combination is something CharXiv and MultiChartQA don't offer, and the ecological-validity argument is plausible. The method is clearly described, the data is public, and the benchmarking protocol is straightforward. Credit where it's due: the pipeline from narrative to QA via an LLM is adapted from Changpinyo et al., but applying it to visualization notebooks with a validation pass is a legitimate contribution.

The soft spots are in proportion to the paper's size. The stress-test concern about single-expert validation is correct. Section 3.3 relies on one 'data visualization expert' to decide whether each of 429 pairs is answerable from the supplied screenshots, and no inter-annotator reliability or audit trail is reported. With 205 final items, small per-task cells (e.g., Find Clusters n=2), and no confidence intervals or significance tests, the headline accuracy gap between GPT-4.1 and Qwen 32B (69% vs 57%) is not established as statistically meaningful. The 'cannot be determined' label is smart, but its correctness is exactly what isn't verified. I don't think this is fatal—the dataset doesn't need perfect labels to be useful as a pilot—but it does mean the strong claims in the abstract and conclusion outrun the evidence.

I'd push back gently on the reader's conditional verdict: I think the paper is slightly stronger than the reader's moderate confidence suggests, because the qualitative examples in Appendix C show the questions do require multi-view reasoning and the 'cannot be determined' option is genuinely used. But the underlying concern is valid, and a referee would rightly ask for a second annotation pass, some measure of agreement, and error bars.

Who is this for? People building or benchmarking CQA models will find it a fresh test bed, even if small. Visualization literacy researchers may also find the narrative-grounded generation method interesting. I'd bring it to a reading group, and I'd cite it if I were writing on CQA evaluation axes. It deserves a serious peer review, but the authors should be prepared to harden the validation and statistics.

Recommendation: send to a venue that takes dataset papers seriously; ask for major revisions on annotation reliability and statistical grounding. If they add a second annotator and report uncertainty, this becomes a solid contribution.

Referee Report

3 major / 4 minor

Summary. This paper introduces RealWorld-ChartQA, a chart question answering dataset built from 22 student-authored litvis notebooks that combine analytical narratives with custom visualizations. The construction pipeline segments the narrative, uses an LLM to generate 429 multiple-choice QA pairs from text alone, and then filters them through a single-expert human validation step, yielding 205 QA pairs across 103 chart images; 36.6% of questions reference multiple views and 16.1% are labeled 'cannot be determined'. The authors benchmark GPT-4.1 and Qwen2.5-VL at 7B and 32B scales, reporting overall accuracies of 69.27%, 56.59%, and 31.71%, respectively, with task-level breakdowns. The central claims are that the dataset reflects ecologically valid analytical workflows, unlike prior CQA benchmarks, and that state-of-the-art MLLMs still show a significant performance gap on this more realistic setting.

Significance. The dataset addresses a real gap in CQA evaluation: most existing benchmarks use single static charts with questions derived from the image or caption, whereas this resource derives questions from surrounding analytical narratives and includes multi-view and interactively captured visualizations. The explicit 'cannot be determined' option is a useful design choice for probing over-confidence in MLLMs, and the public release of the dataset with a clearly documented pipeline is a strength. If the validity of the 205 answer labels can be established, the benchmark would complement ChartQA and CharXiv and give the community a more realistic testbed; the reported GPT-4.1 accuracy of 69.27% is a credible existence proof of an unsolved difficulty gap. However, the manuscript currently does not quantify the reliability of the single annotator's answerability decisions, and the small per-task sample sizes do not support the comparative task-level conclusions drawn in Section 5.

major comments (3)

[Section 3.3] The benchmark's validity rests on a single 'data visualization expert' deciding that each retained QA pair is answerable from the available static screenshots, and this decision is not quantified. Because question generation in Section 3.2 explicitly operates on narrative text 'without parsing the chart images', the validator's answerability judgment is the only bridge between the LLM-generated questions and the visual evidence. The paper reports rejection criteria but gives no inter-annotator agreement, no second review pass, and no per-item audit trail. Since Table 2 contains task cells with as few as 2, 9, 12, and 15 items, even a 5-10% labeling error rate could change task-level rankings and the headline accuracy gap. Please add a second-annotator reliability study (e.g., Cohen's kappa on accept/reject decisions and on the final answer choice) or provide a complete item-level appendix documenting the specific visual evidence used for each retained and rejected pair.
[Section 5, Table 2] Accuracy is reported as point estimates without confidence intervals or significance tests on a 205-item set. The text claims that Qwen2.5-VL-32B outperforms GPT-4.1 on Make Comparisons (59.09% vs 50.00%, n=22) and Find Anomalies (55.56% vs 44.44%, n=9), but with these sample sizes the differences are within sampling error, and the Find Clusters row (n=2) cannot support any comparative conclusion. Please report binomial confidence intervals or exact significance tests and soften the task-level comparative claims accordingly; in particular, the conclusion that 'performance differences across task types are meaningful' is not supported by the reported statistics.
[Section 5, Appendix B] The evaluation protocol presents models only with the question and visualization image(s), not the analytical narrative from which the questions were derived. If the benchmark's contribution is that questions are grounded in analytical narratives, then the ecological-validity claim is stronger than what is measured: the reported performance gap reflects static visual QA, not narrative-grounded reasoning. Please either include a condition in which the narrative context is provided to the model, or explicitly scope the claims to 'questions derived from narratives' rather than to 'narrative-grounded reasoning workflows' as stated in the abstract and conclusion.

minor comments (4)

[Section 3.1] The description of interactive-view capture should quantify how many of the 103 images are interactive captures and how many interactive states could not be captured; Table 1's checkmark for 'Interactive' implies coverage that the dataset analysis does not currently substantiate.
[Appendix B] The model-evaluation prompt asks for only a letter, but the paper does not report how raw model outputs were post-processed (e.g., handling of refusals, extra text, or invalid letters) or what decoding parameters were used; this information is needed for reproducibility.
[Section 2] The description of CharXiv states that its questions 'still focus on one image'; since CharXiv images can contain multiple chart panels, the sentence should clarify that the restriction is to a single image rather than a single view.
[Limitations] The Limitations section candidly notes the small dataset size and task imbalance, but it does not mention the unquantified annotator reliability; adding that caveat alongside the proposed second-annotation study would make the limitations statement more complete.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's benchmark construction and model evaluations are self-contained, and the only author-overlapping citation (litvis) is descriptive rather than load-bearing.

full rationale

This paper does not present a theoretical derivation or a fitted model; it constructs a dataset and reports benchmark accuracies. The question-answer pairs are generated by an LLM from narrative text and then human-validated against the visualizations (Section 3.3), which is an external validity check rather than a circular step. The reported accuracy of GPT-4.1 at 69.3% is an empirical measurement on held-out items, not a quantity fitted to the data or derived from the same assumptions that define the benchmark. The only citation with author overlap is Wood et al. (2019) for litvis, used to describe the source notebook format; it is not invoked as evidence for the paper's central claims about ecological validity or model performance. The 'cannot be determined' option and the single-expert validation procedure raise annotation-reliability and validity concerns, but these are not circularity: the benchmark scores are not equal to the validation decisions by construction. No equation is fitted, no prediction reduces to an input, and no uniqueness claim is imported from the authors' prior work. The derivation chain, such as it is, is self-contained.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The benchmark relies on domain assumptions about task taxonomy, source representativeness, and annotation reliability.

assumptions (3)

domain assumption The VLAT task taxonomy (eight categories from Lee et al., 2017) is a complete and appropriate representation of real-world analytical visualization tasks.
Adopted in Section 3.2 without validation that this taxonomy covers the reasoning in the student notebooks.
domain assumption Student-authored litvis notebooks from a 10-week visualization module at one university are ecologically valid examples of real-world analytical workflows.
Stated in Section 3.1; the dataset's realism claim depends on this representativeness.
domain assumption A single expert's manual validation correctly classifies each QA pair as answerable or unanswerable from the captured static images.
Section 3.3 describes one 'data visualization expert' with no inter-annotator agreement or second pass.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Chart Question Answering from Real-World Analytical Narratives." pith.science (2026). https://pith.science/paper/26X3L5VO

@misc{pith2026250701627,
  author       = {Pith},
  title        = {Pith review of: Chart Question Answering from Real-World Analytical Narratives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26X3L5VO}},
  note         = {Machine review of arXiv:2507.01627}
}

read the original abstract

We present a new dataset for chart question answering (CQA) constructed from visualization notebooks. The dataset features real-world, multi-view charts paired with natural language questions grounded in analytical narratives. Unlike prior benchmarks, our data reflects ecologically valid reasoning workflows. Benchmarking state-of-the-art multimodal large language models reveals a significant performance gap, with GPT-4.1 achieving an accuracy of 69.3%, underscoring the challenges posed by this more authentic CQA setting.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages

[1]

R. Amar, J. Eagan, and J. Stasko. 2005. https://doi.org/10.1109/INFVIS.2005.1532136 Low-level components of analytic activity in information visualization . In IEEE Symposium on Information Visualization , 2005. INFOVIS 2005. , pages 111--117. ISSN: 1522-404X

arXiv 2005
[2]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://doi.org/10.48550/arXiv.2502.13923 Qwen2.5- VL Technical Report . arXiv preprint. ArXiv:2502.13923 [cs]

arXiv doi:10.48550/arxiv.2502.13923 2025
[3]

Alexander Bendeck and John Stasko. 2024. https://doi.org/10.1109/TVCG.2024.3456155 An Empirical Evaluation of the GPT -4 Multimodal Language Model on Visualization Literacy Tasks . IEEE Transactions on Visualization and Computer Graphics, pages 1--11. Conference Name: IEEE Transactions on Visualization and Computer Graphics

arXiv 2024
[4]

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022. https://doi.org/10.18653/v1/2022.naacl-main.142 All you may need for VQA are image captions . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1947--1963, Sea...

work page doi:10.18653/v1/2022.naacl-main.142 2022
[5]

Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. https://doi.org/10.1109/WACV45572.2020.9093269 LEAF - QA : Locate , Encode & Attend for Figure Question Answering . In 2020 IEEE Winter Conference on Applications of Computer Vision ( WACV ) , pages 3501--3510. ISSN: 2642-9381

arXiv 2020
[6]

Yang Chen, Jing Yang, and William Ribarsky. 2009. https://doi.org/10.1109/PACIFICVIS.2009.4906837 Toward effective insight management in visual analytics systems . In 2009 IEEE Pacific Visualization Symposium , pages 49--56. ISSN: 2165-8773

arXiv 2009
[7]

Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. https://doi.org/10.1109/CVPR.2018.00592 DVQA : Understanding Data Visualizations via Question Answering . In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 5648--5656. ISSN: 2575-7075

arXiv 2018
[8]

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. https://doi.org/10.48550/arXiv.1710.07300 FigureQA : An Annotated Figure Dataset for Visual Reasoning . In 6th International Conference on Learning Representations , ICLR 2018, Vancouver , BC , Canada , April 30 - May 3, 2018, Workshop Track Proce...

arXiv doi:10.48550/arxiv.1710.07300 2018

Show all 17 references

[9]

Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020. https://doi.org/10.1145/3313831.3376467 Answering Questions about Charts and Generating Visual Explanations . In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , CHI '20, pages 1--13, New Yor...

2020
[10]

Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2017. https://doi.org/10.1109/TVCG.2016.2598920 VLAT : Development of a Visualization Literacy Assessment Test . IEEE Transactions on Visualization and Computer Graphics, 23(1):551--560. Conference Name: IEEE Transactions on Visuali...

2017
[11]

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. https://doi.org/10.18653/v1/2022.findings-acl.177 C hart QA : A benchmark for question answering about charts with visual and logical reasoning . In Findings of the Association for Computational Ling...

2022 doi
[12]

OpenAI . 2025. https://openai.com/index/gpt-4-1/ GPT -4.1 ( April 14 version)

2025
[13]

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/cdf6f8e9fd9aeaf79b6024caec24f15b-Paper...

2024
[14]

Jo Wood, Alexander Kachkaev, and Jason Dykes. 2019. https://doi.org/10.1109/TVCG.2018.2864836 Design Exposition with Literate Visualization . IEEE Transactions on Visualization and Computer Graphics, 25(1):759--768

2019
[15]

Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, and Meng Jiang. 2025. https://aclanthology.org/2025.naacl-long.566/ M ulti C hart QA : Benchmarking vision-language models on multi-chart problems . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of...

2025
[16]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[17]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

R. Amar, J. Eagan, and J. Stasko. 2005. https://doi.org/10.1109/INFVIS.2005.1532136 Low-level components of analytic activity in information visualization . In IEEE Symposium on Information Visualization , 2005. INFOVIS 2005. , pages 111--117. ISSN: 1522-404X

arXiv 2005

[2] [2]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://doi.org/10.48550/arXiv.2502.13923 Qwen2.5- VL Technical Report . arXiv preprint. ArXiv:2502.13923 [cs]

arXiv doi:10.48550/arxiv.2502.13923 2025

[3] [3]

Alexander Bendeck and John Stasko. 2024. https://doi.org/10.1109/TVCG.2024.3456155 An Empirical Evaluation of the GPT -4 Multimodal Language Model on Visualization Literacy Tasks . IEEE Transactions on Visualization and Computer Graphics, pages 1--11. Conference Name: IEEE Transactions on Visualization and Computer Graphics

arXiv 2024

[4] [4]

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022. https://doi.org/10.18653/v1/2022.naacl-main.142 All you may need for VQA are image captions . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1947--1963, Sea...

work page doi:10.18653/v1/2022.naacl-main.142 2022

[5] [5]

Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. https://doi.org/10.1109/WACV45572.2020.9093269 LEAF - QA : Locate , Encode & Attend for Figure Question Answering . In 2020 IEEE Winter Conference on Applications of Computer Vision ( WACV ) , pages 3501--3510. ISSN: 2642-9381

arXiv 2020

[6] [6]

Yang Chen, Jing Yang, and William Ribarsky. 2009. https://doi.org/10.1109/PACIFICVIS.2009.4906837 Toward effective insight management in visual analytics systems . In 2009 IEEE Pacific Visualization Symposium , pages 49--56. ISSN: 2165-8773

arXiv 2009

[7] [7]

Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. https://doi.org/10.1109/CVPR.2018.00592 DVQA : Understanding Data Visualizations via Question Answering . In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 5648--5656. ISSN: 2575-7075

arXiv 2018

[8] [8]

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. https://doi.org/10.48550/arXiv.1710.07300 FigureQA : An Annotated Figure Dataset for Visual Reasoning . In 6th International Conference on Learning Representations , ICLR 2018, Vancouver , BC , Canada , April 30 - May 3, 2018, Workshop Track Proce...

arXiv doi:10.48550/arxiv.1710.07300 2018