ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts

Janet B. Pierrehumbert; Jiasheng Si; Ruiran Su; Zhijiang Guo

REVIEW 2 major objections 5 minor 54 references

ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Best AI models reach only 78% accuracy when fact-checking scientific charts, far below human performance, a new 49,862-claim benchmark shows.

desk verdict A genuinely useful chart fact-checking benchmark, but the headline 'models far below humans' claim rests on comparing models on the test set to humans on a different random sample. read the letter →

arxiv 2506.08700 v2 pith:WC35GHC7 submitted 2025-06-10 cs.CL cs.CV

Ruiran Su , Jiasheng Si , Zhijiang Guo , Janet B. Pierrehumbert This is my paper

classification cs.CLcs.CV

keywords scientificfact-checkingchartunderstandingmultimodallargelanguagemodelsstatisticalreasoningknowledgegraphbenchmarkclimatescienceclaimverification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ClimateViz is the first large-scale benchmark designed to test whether AI systems can verify scientific claims against real charts, not just text or tables. The paper argues that this task requires genuine statistical reasoning—reading trends, comparing values across time and space, and judging whether a chart contains enough information. Evaluating state-of-the-art multimodal models, the authors find that even the strongest systems (Gemini 2.5, InternVL 2.5) top out at 76.2–77.8% label accuracy, while human experts score 89.3–92.7%. If the benchmark is valid, chart-based fact-checking is currently an unsolved capability, and ClimateViz provides a yardstick for measuring future progress.

What carries the argument

The load-bearing instrument is the ClimateViz dataset itself: a three-label fact-checking task (support, refute, not enough information) built from real, expert-curated scientific charts, with each example carrying a canonicalized knowledge graph of (head, relation, tail) triplets that encode the chart's statistical content. These knowledge graphs serve both as structured explanations for models to use or generate and as a mechanism for making the verification process interpretable and multi-hop.

What would settle it

Running the same three state-of-the-art models (e.g., Gemini 2.5 and InternVL 2.5) on the exact 150 examples scored by humans and finding their accuracy equals or exceeds 89–93% would falsify the claim that models perform far below human experts on chart fact-checking.

Watch

Extended reading notes

Core claim

The paper's central claim is that statistical reasoning over scientific charts poses a capability gap for current multimodal large language models, and that ClimateViz quantifies this gap. The benchmark contains 49,862 claims paired with 2,896 expert-curated charts from reputable climate institutions, each labeled support, refute, or not enough information, plus structured knowledge-graph explanations capturing trends, comparisons, and causal relations. In controlled evaluations, the best proprietary and open-source models reach only 76.2–77.8% accuracy in label-only settings, substantially below human performance (89.3% and 92.7%); adding structured table inputs helps some open-source models, and explanation-augmented outputs help closed-source models like o3 and Gemini 2.5, but no configuration closes the gap.

Load-bearing premise

The headline human-versus-model gap assumes that the 150 randomly sampled examples scored by human experts have the same difficulty and label distribution as the full test set on which all models were evaluated.

Editorial extensions

If this is right

If ClimateViz is a valid benchmark, then chart-based statistical reasoning is an unresolved problem for state-of-the-art multimodal models, not a side effect of a specific model choice.
The benchmark provides a reusable testbed: any future multimodal model can be scored on the same 2,896 charts and 49,862 claims to measure progress toward human-level chart fact-checking.
Explanation-augmented outputs (structured triplets) improve accuracy for some closed-source models, suggesting that forcing intermediate structured reasoning helps ground predictions on complex charts.
Few-shot prompting alone gives limited or inconsistent gains in the chart+text setting, indicating that in-context learning is not sufficient to elicit statistical chart reasoning.
Adding a DePlot-extracted table alongside the chart image boosts open-source models under few-shot prompting, showing that structured tabular representations partially compensate for visual reasoning weaknesses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same construction pipeline—curating domain charts, generating refuted and NEI claims via targeted perturbation, and attaching canonicalized KG triplets—could transfer to other quantitative sciences such as epidemiology, economics, or materials science, which the paper leaves untested.
The gap between BERTScore and BLEU on generated triplets suggests models often produce semantically plausible explanations that fail canonicalization; a canonical-form matching metric or fine-grained human evaluation might reveal whether these explanations are genuinely faithful to the chart.
The paper does not test chain-of-thought or other reasoning-time techniques; if those techniques close much of the 76%-to-89% gap, the claim that chart-based statistical reasoning is inherently out of reach for current models would need qualification.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — ClimateViz is worth knowing about. It's the first large-scale chart-based scientific fact-checking benchmark that I've seen: 49,862 claims over 2,896 expert-curated climate charts, with Zooniverse annotation, expert review, and knowledge-graph explanations. That resource fills a real gap — ChartCheck is smaller and observationally shallow, and prior fact-checking benchmarks are text- or table-only. The dataset construction is careful: six annotators per chart, Randolph's kappa reported, explicit strategies for generating refuted and NEI claims, and the labels come from humans, not from the models being evaluated. If you work on multimodal fact-checking or chart reasoning, this is the benchmark you'll want to cite and use.

The soft spots are mostly in the evaluation section, and one is load-bearing. Human performance is measured on 150 randomly sampled examples per setting, drawn from the whole benchmark, while every model is scored on the held-out 20% test split. The paper never reports model accuracy on those same 150 examples or human accuracy on the full test set. Since the human sample is drawn from the full benchmark, roughly 80% of it can come from training or development splits, whose difficulty is unknown. So the 12–16 point gap in Table 5 is not actually a measured quantity; it's an apples-to-oranges comparison. With n=150 and accuracy near 90%, the standard error is about 2.5 points, so the gap could shrink by several points even in the best case — and the central conclusion that models are 'far below human performance' depends on it. This is fixable: rerun the models on the exact human subset, or get human labels on a sample of the test set, and report both.

The other concern is the knowledge-graph explanations. They're generated by GPT-4o using the chart, caption, and the supported claims themselves, and then the generated triplets are used to evaluate explanation-augmented reasoning. That's a circularity for the explanation component, though not for the label benchmark. The paper describes the pipeline in Appendix D but doesn't flag the circularity explicitly. Also, human performance has no error bars and the human sample appears to involve one expert per example, so we don't see annotator variance.

Proportionally, these don't sink the dataset. The label benchmark itself is human-grounded and the resource is reusable. But the headline empirical result needs to be re-established on a common evaluation set before it can be taken at face value.

Recommendation: send it to peer review. It deserves a serious referee, but the revision should require the matched human-model evaluation and should temper the claims until that's done.

Referee Report

2 major / 5 minor

Summary. ClimateViz is a new benchmark for fact-checking scientific claims against chart images, containing 49,862 claims paired with 2,896 expert-curated climate charts, labeled as support, refute, or not enough information (NEI), plus knowledge-graph triplet explanations generated with GPT-4o. The authors evaluate several open- and closed-source multimodal LLMs in zero-shot and few-shot settings, with label-only and explanation-augmented outputs, and compare them to a human baseline. They report that the best models reach only 76.2–77.8% label accuracy, far below human accuracies of 89.3–92.7%, and that explanation-augmented outputs improve some closed-source models. The paper also presents dataset statistics, inter-annotator agreement, a reasoning-taxonomy analysis, and an ablation of table-only inputs.

Significance. If the reported human-model gap is confirmed on a common evaluation set, ClimateViz would be a valuable and much-needed benchmark for chart-based statistical reasoning and scientific fact-checking. The dataset is substantially larger than prior chart fact-checking resources, uses real-world expert-curated charts, includes multi-annotator setup with inter-annotator agreement, and is released with code; these are concrete strengths. The central label benchmark is not circular: labels originate from citizen science and expert review, not from the evaluated models. However, the headline human-model comparison is currently not a head-to-head measurement, and the explanation gold-standard construction introduces a degree of circularity. Both issues are load-bearing and need to be resolved before the benchmark's central claims can be fully credited.

major comments (2)

[Section 4.2 and Table 5] The central result that models are 'far below human performance' is not supported by the reported evaluation because human scores and model scores are computed on different instances. Models are evaluated on the held-out 20% test set, while the human baseline uses 150 randomly sampled examples per setting drawn from the entire CLIMATEVIZ benchmark, so most human examples likely come from training/development splits whose difficulty is never measured. The paper does not report model accuracy on the exact human subset, nor human accuracy on the full test set. With n=150 and accuracy near 90%, the sampling standard error is roughly 2.5 percentage points, so the observed 12–16 point gap could plausibly shrink by several points if the human sample is easier than the test set; the reported gap is therefore an unverified quantity rather than a measured result. Please recompute both evaluations on a common set of examples (e.g., the test set), report confidence intervals or inter-annotator variability for the human scores, and either qualify the headline accordingly or remove it.
[Section 3.1.3 and Appendix D.2] The gold knowledge-graph explanations are generated by GPT-4o using as input the chart, the caption, and a 'chart summary' constructed by aggregating the supported claims. Because the supported claims are part of the input to the generator and are also the basis of the support labels, evaluating model-generated triplets against these gold KGs with BLEU, METEOR, ROUGE-L, and BERTScore largely rewards reproducing GPT-4o's surface canonicalization rather than independently verified factual content. The paper acknowledges in the Limitations section that automatic metrics may not capture factual soundness, but the dataset description in Appendix D.2 presents the KGs as structured gold explanations without any human validation of triplet factuality. Please either add a human-validated subset of triplets (or a factual-consistency metric against chart-derived values) and report results on it, or reframe the explanation evaluation as a 'match to GPT-4o canonicalization' study rather than as fact-verification explanation accuracy.

minor comments (5)

[Section 5] The claim that explanation-augmented outputs 'notably' improve closed-source models such as Gemini 2.5 is not fully supported by Table 5: in the CT zero-shot condition, Gemini 2.5's accuracy drops from 76.2 (Acc-L) to 73.2 (Acc-E), and GPT-4o's CTT accuracy drops from 64.3 to 60.2. Please specify the exact settings and models for which the improvement holds.
[Section 4.2] The phrase 'using the same input modalities as the corresponding model configuration' for human annotators is ambiguous; in the CT setting it is unclear whether humans saw the chart image, the caption, and the claim exactly as presented to models, or just the caption and claim. Please specify the exact interface.
[Appendix A.2.2] There is a typo: 'the intructions' should be 'the instructions'.
[Table 5] The bold formatting for 'best score per column' does not include the human performance row, which has the highest accuracy in both CT and CTT columns; please clarify whether humans are excluded from the bold marking or add bold to the human row.
[Section 3.2.2] The reasoning-taxonomy analysis of 300 claims is performed by a single author without reporting inter-annotator agreement; since this analysis supports the claim that multi-hop statistical reasoning is prevalent, a second annotator on a subset would strengthen it.

Circularity Check

1 steps flagged · score 4.0 of 10

The central label benchmark is not circular, but the explanation-quality evaluation partially reduces by construction: gold KGs are generated by GPT-4o from the chart, caption, and the supported claims that the benchmark is meant to verify.

self definitional [Section 3.1.3 and Appendix D.2; Table 6]
"We construct triplets automatically using GPT-4o (OpenAI, 2024), using the chart, caption, and the set of supported claims as the chart summary as inputs. ... Triplets are generated only for supported claims to ensure factual consistency with the chart evidence."

The gold knowledge graphs used as references to score explanation generation (Table 6) are produced by GPT-4o from the chart, caption, and the supported claims themselves. The supported claims are exactly the instances whose fact-checking and explanations the benchmark evaluates. Therefore the gold explanation for a claim is not an independent ground truth; it is a GPT-4o restatement of the claim plus caption. When models, including GPT-4o itself, generate triplets, the BLEU/METEOR/ROUGE scores compare against that self-derived reference. GPT-4o's high triplet scores therefore partly measure self-agreement with the reference-generation pipeline rather than the factual or statistical quality of the explanation.

full rationale

The central label benchmark is self-contained: the support/refute/NEI labels are produced by citizen-science annotation and two-expert review, not by the models being evaluated. The headline claim that models underperform humans is therefore not circular in its construction, although the human-baseline comparison is statistically weakened by evaluating models on the held-out test split while humans see a random sample of the whole benchmark; that is a validity concern, not a circularity. The one genuine circular element is localized to the explanation component: the gold KGs are generated by GPT-4o from the chart, caption, and the supported claims, and then used as references to score model-generated triplets, including GPT-4o's own triplets. This makes the triplet-quality numbers partially measures of self-agreement with the reference generator. No load-bearing self-citation, imported uniqueness theorem, ansatz-smuggling via citation, or renaming of a known result is present. Because the main label results do not depend on the circular KG scoring, the overall circularity is partial and localized rather than total.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The benchmark's label claims rest on expert-validated annotations and the assumption that curated charts are accurate. The explanation component rests on GPT-4o-generated triplets used as ground truth, which is the largest circularity risk. The human-subset comparison and DePlot tables introduce additional domain assumptions that are load-bearing for the reported performance gaps.

free parameters (2)

MNLI contradiction threshold = 0.8
Used in Section 3.1.2 and Appendix C.1 to filter GPT-4o-generated refuted claims; chosen by hand rather than estimated from data, and it shapes which refuted claims enter the dataset.
Minimum claim length = 10 words
Applied as a preprocessing filter in Sections 3.1.2 and A.3.1; a design choice that shapes the claim distribution but is not fitted to data.

assumptions (6)

domain assumption Expert-validated citizen-science annotations provide correct ground-truth labels for support, refute, and NEI.
The benchmark treats these labels as ground truth; no independent gold set or adjudication statistics beyond Randolph's Kappa are reported.
domain assumption The 2,896 selected charts from NOAA, Met Office, Copernicus, NASA, Climate.gov, and Climate Reanalyzer accurately represent the real climate data they visualize.
Charts are used as evidence without independent verification of their accuracy against raw data.
ad hoc to paper GPT-4o-generated knowledge graph triplets are a valid gold standard for explanation evaluation.
Appendix D.2 constructs KGs with GPT-4o from chart, caption, and supported claims, without human validation, yet these triplets are used as references in Table 6.
domain assumption DeBERTa-Large-MNLI contradiction scores with threshold 0.8 reliably identify semantically contradictory refuted claims.
Section 3.1.2 uses this automatic filter to keep only high-confidence contradictions; errors in the NLI model propagate into the dataset.
ad hoc to paper Human performance on 150 sampled examples per setting is representative of the full held-out test set.
Used to compare human accuracy against model accuracy on the full test set; the subset may not have the same difficulty distribution.
domain assumption DePlot-generated tables are faithful enough for CTT evaluation.
Appendix E reports 8 of 50 conversions with major issues, yet CTT results are presented as reliable model scores.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts." pith.science (2026). https://pith.science/paper/WC35GHC7

@misc{pith2026250608700,
  author       = {Pith},
  title        = {Pith review of: ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WC35GHC7}},
  note         = {Machine review of arXiv:2506.08700}
}

read the original abstract

Scientific fact-checking has mostly focused on text and tables, overlooking scientific charts, which are key for presenting quantitative evidence and statistical reasoning. We introduce ClimateViz, the first large-scale benchmark for scientific fact-checking using expert-curated scientific charts. ClimateViz contains 49,862 claims linked to 2,896 visualizations, each labeled as support, refute, or not enough information. To improve interpretability, each example includes structured knowledge graph explanations covering trends, comparisons, and causal relations. We evaluate state-of-the-art multimodal language models, including both proprietary and open-source systems, in zero-shot and few-shot settings. Results show that current models struggle with chart-based reasoning: even the best systems, such as Gemini 2.5 and InternVL 2.5, reach only 76.2 to 77.8 percent accuracy in label-only settings, far below human performance (89.3 and 92.7 percent). Explanation-augmented outputs improve performance in some models. We released our dataset and code alongside the paper.

Figures

Figures reproduced from arXiv: 2506.08700 by the authors.

**Figure 1.** (a) A sample from CLIMATEVIZ showing a scientific chart, caption, claim, and its label. (b) Knowledge graph creation pipeline: raw triplets are extracted by a multimodal LLM and canonicalized. (c) Subgraph of relevant facts representing structured reasoning from the chart to the claim. spatial dimensions (e.g., geographic insets), and uncertainty bands; (ii) nuanced statistical reasoning to interpret temporal trend… view at source ↗

**Figure 2.** Distribution of statistical reasoning complex [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Quality Control Process: before, during, and [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Distributions of source and type of the charts [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2023 a . https://arxiv.org/abs/2301.11843 Reading and reasoning over chart images for evidence-based automated fact-checking . Preprint, arXiv:2301.11843

work page Pith review arXiv 2023
[4]

Mubashara Akhtar, Michael Sejr Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023 b . https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.361 Multimodal automated fact-checking: A survey . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pages 5430--5448. Association ...

work page doi:10.18653/v1/2023.findings-emnlp.361 2023
[5]

Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, and Elena Simperl. 2024. https://arxiv.org/abs/2311.07453 Chartcheck: Explainable fact-checking over real-world chart images . Preprint, arXiv:2311.07453

arXiv 2024
[6]

Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. https://arxiv.org/abs/2106.05707 Feverous: Fact extraction and verification over unstructured and structured information . Preprint, arXiv:2106.05707

arXiv 2021
[7]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. https://arxiv.org/abs/25...

arXiv 2025
[8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65--72. Association for Computational Linguistics

work page 2005

Show all 54 references

[9]

Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang. 2024. https://arxiv.org/abs/2304.09048 Codekgc: Code language model for generative knowledge graph construction . Preprint, arXiv:2304.09048

2024 arXiv
[10]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

2020 arXiv
[11]

Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. https://arxiv.org/abs/1909.02164 Tabfact: A large-scale dataset for table-based fact verification . Preprint, arXiv:1909.02164

2020 arXiv
[12]

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi...

2025 arXiv
[13]

Google DeepMind. 2025. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/. Accessed: 2024-05-01

2025
[14]

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2021. https://arxiv.org/abs/2012.00614 Climate-fever: A dataset for verification of real-world climate claims . Preprint, arXiv:2012.00614

2021 arXiv
[15]

Jan Drchal, Herbert Ullrich, Tomáš Mlynář, and Václav Moravec. 2024. https://arxiv.org/abs/2312.10171 Pipeline and dataset generation for automated fact-checking in almost any language . Preprint, arXiv:2312.10171. Preprint

2024 arXiv
[16]

John W. Fertig. 1958. https://api.semanticscholar.org/CorpusID:70925311 Introduction to statistical reasoning . American Journal of Public Health, 48:533--533

1958
[17]

Lucy Fortson, Karen Masters, Robert Nichol, Kirk Borne, Edd Edmondson, Chris Lintott, Jordan Raddick, Kevin Schawinski, and John Wallin. 2011. https://arxiv.org/abs/1104.5513 Galaxy zoo: Morphological classification and citizen science . Preprint, arXiv:1104.5513

2011 arXiv
[18]

e , James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych

Max Glockner, Ieva Stali \= u nait \. e , James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2024. https://doi.org/10.1162/tacl_a_00629 A mbi FC : Fact-checking ambiguous claims with evidence . Transactions of the Association for Computational Linguistics, 12:1--18

2024 doi
[19]

Zhijiang Guo, Michael Sejr Schlichtkrull, and Andreas Vlachos. 2022. https://doi.org/10.1162/TACL\_A\_00454 A survey on automated fact-checking . Trans. Assoc. Comput. Linguistics, 10:178--206

2022 doi
[20]

Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. https://doi.org/10.18653/v1/2020.acl-main.398 Tapas: Weakly supervised table parsing via pre-training . In Proceedings of the 58th Annual Meeting of the Association for Com...

2020 doi
[21]

Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji

Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2024. https://arxiv.org/abs/2403.12027 From pixels to insights: A survey on automatic chart understanding in the era of large foundation models . Preprint, arXiv:2403.12027

2024 arXiv
[22]

Richard Landis and Gary G

J. Richard Landis and Gary G. Koch. 1977. https://api.semanticscholar.org/CorpusID:11077516 The measurement of observer agreement for categorical data . Biometrics, 33(1):159--174

1977
[23]

Moritz Laurer. 2022. Deberta-large-mnli model. https://huggingface.co/MoritzLaurer/DeBERTa-Large-MNLI. Accessed: 2025-05-12

2022
[24]

Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. https://arxiv.org/abs/2304.11633 Evaluating chatgpt's information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness . Preprint, arXiv:2304.11633

2023 arXiv
[25]

Chin-Yew Lin. 2004. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81. Association for Computational Linguistics

2004
[26]

Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.660 D e P lot: One-shot visual language reasoning by plot-to-table t...

2023 doi
[27]

Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023 b . https://arxiv.org/abs/2212.10505 Deplot: One-shot visual language reasoning by plot-to-table translation . ...

2023 arXiv
[28]

Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2023 c . https://arxiv.org/abs/2212.09662 Matcha: Enhancing visual language pretraining with math reasoning and chart derenderin...

2023 arXiv
[29]

Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. https://arxiv.org/abs/2305.13186 Scitab: A challenging benchmark for compositional reasoning and claim verification on scientific tables . Preprint, arXiv:2305.13186

2023 arXiv
[30]

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022 a . https://arxiv.org/abs/2203.10244 Chartqa: A benchmark for question answering about charts with visual and logical reasoning . Preprint, arXiv:2203.10244

2022 arXiv
[31]

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022 b . Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022

2022
[32]

MetaAI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/meta-llama-4-and-code-llama-3. Accessed: 2025-05-01

2025
[33]

Khapra, and Pratyush Kumar

Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020 a . https://arxiv.org/abs/1909.00997 Plotqa: Reasoning over scientific plots . Preprint, arXiv:1909.00997

2020 arXiv
[34]

Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020 b . Plotqa: Reasoning over scientific plots. In Proceedings of the 28th ACM International Conference on Multimedia

2020
[35]

Isabelle Mohr, Amelie Wührl, and Roman Klinger. 2022. https://arxiv.org/abs/2204.12164 Covert: A corpus of fact-checked biomedical covid-19 tweets . Preprint, arXiv:2204.12164

2022 arXiv
[36]

OpenAI. 2024. Gpt-4o: A new frontier in openai's multimodal models. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-05-01

2024
[37]

OpenAI. 2025. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: 2025-05-01

2025
[38]

Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. https://doi.org/10.18653/v1/2023.acl-long.386 Fact-checking complex claims with program-guided reasoning . In Proceedings of the 61st Annual Meeting of the Association...

2023 doi
[39]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU : a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--318. Association for Computational Linguistics

2002
[40]

Randolph

Justus J. Randolph. 2005. Free-marginal multirater kappa: An alternative to fleiss' fixed-marginal multirater kappa. https://eric.ed.gov/?id=ED490661. ERIC Document ED490661

2005
[41]

Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. https://arxiv.org/abs/2106.03794 Covid-fact: Fact extraction and verification of real-world claims on covid-19 pandemic . Preprint, arXiv:2106.03794

2021 arXiv
[42]

Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.297 Evidence-based fact-checking of health-related claims . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499--35...

2021 doi
[43]

Michael Sejr Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/cd86a30526cd1aff61d6f89f107634e4-Abstract-Datasets\_and\_Benchmarks.html Averitec: A dataset for real-world claim verification with evidence from the web . I...

2023
[44]

Page, and David De Roure

Robert Simpson, Kevin R. Page, and David De Roure. 2014. https://doi.org/10.1145/2567948.2579215 Zooniverse: observing the world's largest citizen science platform . In Proceedings of the 23rd International Conference on World Wide Web, WWW '14 Companion, page 1049–1054, New Y...

2014
[45]

Juraj Vladika and Florian Matthes. 2023. https://arxiv.org/abs/2305.16859 Scientific fact-checking: A survey of resources and approaches . Preprint, arXiv:2305.16859

2023 arXiv
[46]

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. https://arxiv.org/abs/2004.14974 Fact or fiction: Verifying scientific claims . Preprint, arXiv:2004.14974

2020 arXiv
[47]

David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. https://arxiv.org/abs/2210.13777 Scifact-open: Towards open-domain scientific claim verification . Preprint, arXiv:2210.13777

2022 arXiv
[48]

Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. 2025. https://arxiv.org/abs/2503.12605 Multimodal chain-of-thought reasoning: A comprehensive survey . Preprint, arXiv:2503.12605

2025 arXiv
[49]

Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024. https://arxiv.org/abs/2401.06805 Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on...

2024 arXiv
[50]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models

2022
[51]

Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2024. https://arxiv.org/abs/2312.15915 Chartbench: A benchmark for complex visual reasoning in charts . Preprint, arXiv:2312.15915

2024 arXiv
[52]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models

2023
[53]

Bowen Zhang and Harold Soh. 2024. https://arxiv.org/abs/2404.03868 Extract, define, canonicalize: An llm-based framework for knowledge graph construction . Preprint, arXiv:2404.03868

2024 arXiv
[54]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2023 a . https://arxiv.org/abs/2301.11843 Reading and reasoning over chart images for evidence-based automated fact-checking . Preprint, arXiv:2301.11843

work page Pith review arXiv 2023

[4] [4]

Mubashara Akhtar, Michael Sejr Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023 b . https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.361 Multimodal automated fact-checking: A survey . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pages 5430--5448. Association ...

work page doi:10.18653/v1/2023.findings-emnlp.361 2023

[5] [5]

Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, and Elena Simperl. 2024. https://arxiv.org/abs/2311.07453 Chartcheck: Explainable fact-checking over real-world chart images . Preprint, arXiv:2311.07453

arXiv 2024

[6] [6]

Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. https://arxiv.org/abs/2106.05707 Feverous: Fact extraction and verification over unstructured and structured information . Preprint, arXiv:2106.05707

arXiv 2021

[7] [7]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. https://arxiv.org/abs/25...

arXiv 2025

[8] [8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65--72. Association for Computational Linguistics

work page 2005