Pith. sign in

REVIEW 4 major objections 3 minor 20 references

Chartography: A Benchmark for Professional Chart Understanding

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces Chartography, a benchmark of 100 expert-authored professional chart tasks on which the best frontier-model configuration passes only 45.0% of trials, and argues the gap is visual perception rather than reasoning.

desk verdict A credible, well-built professional-chart benchmark with a 45% ceiling, but the headline difficulty depends on a single un-audited judge and an explicitly curated hard item set. read the letter →

arxiv 2608.10677 v1 pith:UBFGWY5U submitted 2026-08-11 cs.CV

Suhaas Garre , Chris Mutty , Sushant Mehta , Edwin Chen This is my paper · ORCID
classification cs.CV
keywords professionalchartunderstandingQAbenchmarkvisualperceptionmetricgroundingmultimodalLLMevaluationexpert-verifiedtasksdomainconventionspass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing chart benchmarks are saturated and miss what professionals actually do with charts, so it builds a harder test: Chartography, 100 tasks using domain-native charts such as Kaplan-Meier curves, contour maps, Sankeys, and three-dimensional surface plots, with questions written by working professionals and independently verified by three additional experts. The result is a benchmark that resists saturation: the best of 30 frontier-model configurations passes only 45.0% of trials, and most configurations land between 9.0% and 39.5%. The paper's failure analysis localizes the gap to visual perception—missed thin features, misread values on sparsely labeled axes, mishandled projected geometry, and overlooked domain conventions—rather than to reasoning depth. A sympathetic reader should take from this that current multimodal models are not yet reliable readers of professional charts, and that earlier benchmark scores overstated that readiness.

What carries the argument

The central object is the benchmark itself: 100 tasks pairing a professional-domain chart with a free-form question that has a prespecified golden answer and, for 51 of 100 tasks, an expert-set acceptable range. The machinery that carries the argument is the expert-calibrated grading protocol: an LLM judge that never sees the chart decides whether the model's final answer falls within the expert range or matches the golden set exactly, with multi-part answers graded all-or-nothing; scores are reported as mean pass@1 over 20 trials per task. This design ensures that grades reward reading the chart at the precision it supports, and that the judge adjudicates answer equivalence rather than re-solving the task.

What would settle it

A reader could settle the central claim by running a judge-accuracy audit: take a random sample of model responses, have domain experts grade them against the golden answers using the paper's protocol, and compare those human verdicts with the automated judge's verdicts. If agreement is materially below the level reported for prior judge-validity studies, or if the judge systematically favors certain answer phrasings, then the absolute scores (including the 45.0% top score) would need revision; a different judge or an ensemble of judges would show how much the leaderboard shifts.

Watch

Extended reading notes

Core claim

Chartography establishes that frontier multimodal models, which score 80–90% on existing chart benchmarks, fail a majority of professionally relevant chart-reading tasks: the strongest evaluated configuration passes only 45.0% of 2,000 graded trials, and the paper attributes the shortfall to a failure of metric grounding—anchoring a numeric or categorical answer to the correct mark, axis position, or plotted geometry—rather than to inadequate reasoning. The benchmark is deliberately constructed so that this gap is measurable: expert-authored questions with walkthroughs, triple independent verification, adversarial difficulty screening, expert-set acceptable ranges that reflect the precision a chart actually supports, and all-or-nothing grading of multi-part answers.

Load-bearing premise

Every reported score depends on one automated judge whose accuracy on this benchmark is assumed from prior work rather than measured here, and that judge is itself one of the scored models.

Editorial extensions

If this is right

  • The same models that score above 80% on existing chart benchmarks fall below 50% on Chartography, so high marks on those benchmarks do not establish readiness for clinical, engineering, or financial chart-reading deployment.
  • Because 11 of 12 model families improved with elevated reasoning effort but gains were uneven, more deliberation helps when the chart is read correctly to begin with and cannot repair a misread value.
  • The failure analysis implies that improving visual reading alone, while holding reasoning fixed, would raise scores substantially.
  • With 100 expert-verified tasks, expert-calibrated ranges, and released provenance metadata, the benchmark gives future work a stable and auditable target that is not yet near saturation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'semantic recognition versus metric grounding' split likely generalizes beyond charts: any task where a model must anchor an answer to a specific position in a visual, such as maps, technical drawings, or medical images, could show the same pattern of correct identification but wrong measurement.
  • The no-tools protocol means the 45.0% ceiling describes bare perception; allowing zooming, cropping, or code execution under the same grading rules is a natural test of how much of the gap is a perception limit versus an interface limit.
  • Because the judge is a single model whose validity is borrowed from prior work rather than measured on this benchmark, the numerical scores should be read as a snapshot; a multi-judge or human-audited grading pass would tell whether the reported rank order is robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces Chartography, a benchmark of 100 professional chart-understanding tasks that pair real-world domain-specific charts (e.g., Kaplan-Meier curves, contour maps, candlesticks, Bode plots) with questions authored by practicing professionals and verified by three additional experts. Each task includes a golden answer with expert-set acceptable ranges, and grading is all-or-nothing across answer parts. The authors evaluate 30 frontier-model configurations using 20 trials per task with no tools, reporting a best mean pass@1 of 45.0% (GPT-5.6 Sol at maximum reasoning) and scores between 9.0% and 39.5% for the rest. They also analyze failure modes, which they attribute primarily to visual perception errors, and release the dataset, images, provenance metadata, and evaluation code.

Significance. If the benchmarking methodology holds up, Chartography addresses a real gap: professional chart reading is under-measured by existing benchmarks that are nearing saturation on standard formats. The expert authoring and verification process, the chart-specific acceptable ranges, and the emphasis on domain conventions are valuable contributions. The transparent protocol, public release, and reproducible harness are notable strengths. However, the central empirical claims—the low aggregate scores and the localization of failures in visual perception—depend critically on the validity of a single LLM judge and on the task-selection procedure, both of which require additional scrutiny before the results can be fully trusted.

major comments (4)
  1. [§4.2 Grading] The reported scores for all 30 configurations rest on a single LLM judge, Gemini 3.5 Flash, which never sees the chart; the paper cites MT-Bench [19] for judge validity, but that evidence concerns text-only preference following rather than equivalence adjudication of numeric answers against expert-set ranges in a multimodal chart-QA setting. No Chartography-specific judge-accuracy audit is reported, so a systematic judge bias (e.g., against certain phrasings or roundings) would shift every score in Table 3 and the 45.0% headline. This is load-bearing for the paper's central claim; please add a validation study comparing judge verdicts with expert human judgements on a sample of trials, report agreement and error patterns, and ideally use multiple judges or deterministic checks for numeric range membership.
  2. [§3.1 Task construction] The adversarial difficulty screening keeps a task only if it elicits a meaningful failure in at least one frontier model. This guarantees difficulty by construction and means the benchmark deliberately excludes tasks that current models can solve, so the statement that Chartography 'resists saturation' is partly a consequence of the selection rule rather than an empirical discovery. The paper should report how many candidate tasks were dropped during screening, describe the screening protocol in detail, and discuss the implications for what population of professional chart-reading tasks the 100-task set represents.
  3. [§4.2 Grading, self-judging condition] Gemini 3.5 Flash is both the judge and a scored configuration, so its leaderboard row (35.9% in Table 3) is self-graded. Because the judge is also used for every other row, this conflates the measurement instrument with the object of measurement. At minimum the Gemini 3.5 Flash row should be graded by a different judge, or the self-judging row should be reported separately; the paper should also disclose any potential conflict in the leaderboard description.
  4. [§5.4 Failure modes] The abstract and §6 claim that failures 'concentrate in visual perception,' but §5.4 presents only qualitative categories and anecdotal examples; no quantitative distribution of failure categories across the 55% of failed trials is reported. Since this claim is central to the paper's interpretation, please add a systematic, independently coded classification of a random sample of failed trials with inter-rater agreement, or soften the claim to 'observed failures frequently fall into these categories.'
minor comments (3)
  1. [Figure 1 caption] The caption contains a duplicated phrase: 'More charts with their with their tasks are shown in §5.5.'
  2. [Table 1] The 'Top model (no tools)' row for prior benchmarks cites a launch-analysis blog rather than a peer-reviewed source; please specify the exact model versions and evaluation dates, or explicitly mark those numbers as informal comparisons.
  3. [§4.3 Metric] The paper does not describe how the 95% confidence intervals in Table 3 are computed; please state the formula (e.g., normal approximation, Wilson interval) and whether clustering by task is accounted for.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: benchmark items are externally grounded in professional charts and expert-verified answers; disclosed difficulty screening and self-judged Gemini row are validity caveats, not circular steps.

full rationale

Chartography's evaluation chain is not circular. Items are externally grounded: 81 charts sourced online and 19 expert-created, questions written by working professionals, and golden answers independently verified by three domain experts, with per-item acceptable ranges. The grading judge receives only the question, golden answer, and model response, never the chart, so it checks against a prespecified target rather than generating the target from the evaluated model. The disclosed adversarial screening in Section 3.1 ('a task was retained only if it elicited a meaningful failure in at least one frontier model') guarantees hard items but does not force the specific 45.0% score or the observed failure distribution; moreover, the paper transparently states this selection. The Section 4.2 note that Gemini 3.5 Flash's row is 'a self-judging condition' is a caveat about one configuration, and all configurations are graded by the same external judge, so the ranking is not defined by the judge's own predictions. Judge-validity rests on an external MT-Bench citation, not on the current authors' self-citation. Self-citations to GDP.pdf and Riemann-Bench appear only in related work and carry no load. The main concerns - unmeasured judge accuracy on Chartography and the difficulty-screening distribution - are external-validity and correctness risks, not circularity. Score 2 reflects these minor validity caveats, not a derivation that reduces to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark's headline numbers rest on hand-chosen grading ranges and a screening rule that keeps only tasks where frontier models fail. The dominant unstated premise is judge validity on this benchmark. No invented entities are introduced; the benchmark is an external artifact.

free parameters (2)
  • Per-item acceptable ranges = 51/100 answers with ranges, 25 exact numeric, 24 categorical, 21 with multiple ranges
    These expert-set tolerances define a pass. They are chosen by hand and directly determine pass@1; tasks graded with wider ranges would produce higher scores. They are not fitted to model outputs but are not derivable from the images alone.
  • Adversarial difficulty screening threshold = Unspecified: 'a meaningful failure in at least one frontier model'
    The curation rule keeps only tasks that elicited a failure in at least one frontier model (§3.1). The threshold is not quantified, so the composition of the 100 tasks and the resulting scores depend on this hand-chosen criterion.
assumptions (3)
  • domain assumption The Gemini 3.5 Flash judge correctly determines whether a free-form response matches the golden answer and range, without seeing the chart.
    All pass@1 numbers in Table 3 depend on this judge. The paper cites MT-Bench for general LLM-judge agreement but reports no Chartography-specific judge-accuracy study, and the judge never sees the chart so it cannot correct visually grounded misreadings (§4.2).
  • domain assumption The chart images contain enough visual information for a human expert to extract the golden answer at the precision implied by the acceptable range.
    Grading assumes the expert answer is recoverable from the image alone. Three experts verify this per task, but no inter-annotator agreement or resolution analysis is reported (§3.1).
  • ad hoc to paper A task that causes at least one frontier model to fail is a valid sample of professional chart understanding.
    The adversarial screening in §3.1 removes tasks that models can solve, making the benchmark difficult by construction; the paper then interprets failure rates and categories as evidence about professional chart reading in general (§5, §6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chartography: A Benchmark for Professional Chart Understanding." pith.science (2026). https://pith.science/paper/UBFGWY5U

@misc{pith2026260810677,
  author       = {Pith},
  title        = {Pith review of: Chartography: A Benchmark for Professional Chart Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBFGWY5U}},
  note         = {Machine review of arXiv:2608.10677}
}
read the original abstract

Professionals across medicine, engineering, finance, manufacturing, and the sciences often make consequential decisions from charts. Existing chart benchmarks do not sufficiently measure this ability: they are dominated by bar, line, and pie formats, rely on shorter reasoning chains, and are nearing saturation, with frontier models already scoring 80-90%. We introduce Chartography, a benchmark of 100 tasks that pair charts drawn from professional practice, in domain-specific formats that standard chart benchmarks rarely include, with questions written by professionals who read these charts for a living and independently verified by three additional experts. In an evaluation of 30 frontier-model configurations (20 scored trials per task), the best configuration reaches only 45.0% mean pass@1; the remainder span 9.0-39.5%. Failures concentrate in visual perception: models can miss nuanced features, misread values along sparsely labeled axes, mishandle projected 3D geometry, and violate domain conventions encoded in the chart. We release all tasks, images, provenance metadata, and evaluation code.

Figures

Figures reproduced from arXiv: 2608.10677 by the authors.

Figure 1
Figure 1. Three of the 100 charts in Chartography: a Moody diagram, a three-dimensional surface plot, and a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Mean pass@1 versus inference cost (left) and generated tokens (right); ringed markers lie on the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Engineering design chart (USDA-SCS). “For an outlet pipe with a diameter of 48 inches and a flow of 73 cubic feet per second, what is the minimum downstream width of the riprap apron?” Golden answer: 30 feet (accepted 29–31). The flow lies below the 48-inch curve’s plotted range, so professional practice reads the curve’s minimum (La=26 ft) and applies W = Do +La = 4+26 = 30 ft. A representative failure read an infl… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pediatric growth chart (CDC). “What height corresponds to the 50th percentile for a four-year-old girl? Round to the nearest 0.5 centimeters.” Golden answer: 101.0 cm (accepted 100–101). A representative failure followed the correct percentile curve but read the inters…
Figure 5
Figure 5. Figure 5: Wind rose (USDA-ARS). “In which three directions did winds above 8.5 m/s occur for the longest total duration?” Golden answer: S, SSW, W. A representative failure fixated on the lower-left of the rose and answered S, SSW, SW, yet SW has negligible time above threshold …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages

  1. [19]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-judge with MT-Bench and chatbot arena.Advances in Neural Information Processing Systems, 36,

    work page

  2. [1]

    Riemann-Bench: A benchmark for moonshot mathematics

    Suhaas Garre, Erik Knutsen, Sushant Mehta, and Edwin Chen. Riemann-Bench: A benchmark for moonshot mathematics. arXiv preprint arXiv:2604.06802, 2026

    arXiv 2026

  3. [2]

    GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents

    Suhaas Garre, Emily Ritchie, Sushant Mehta, and Edwin Chen. GDP.pdf: Benchmarking grounded multimodal reasoning over professional PDF documents. arXiv preprint arXiv:2607.11192, 2026

    work page Pith review arXiv 2026

  4. [3]

    DVQA: Understanding data visualizations via question answering

    Kushal Kafle et al. DVQA: Understanding data visualizations via question answering. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    work page 2018

  5. [4]

    FigureQA: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou et al. FigureQA: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2018

    arXiv 2018

  6. [5]

    DePlot: One-shot visual language reasoning by plot-to-table translation

    Fangyu Liu et al. DePlot: One-shot visual language reasoning by plot-to-table translation. InFindings of the Association for Computational Linguistics: ACL 2023, 2023

    work page 2023

  7. [6]

    MatCha: Enhancing visual language pretraining with math reasoning and chart derendering

    Fangyu Liu et al. MatCha: Enhancing visual language pretraining with math reasoning and chart derendering. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

    work page 2023

  8. [7]

    ChartOCR: Data extraction from charts images via a deep hybrid framework

    Junyu Luo et al. ChartOCR: Data extraction from charts images via a deep hybrid framework. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021

    work page 2021

Show all 20 references
  1. [8]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Associ- ation for Computational Linguistics: ACL 2022. Association for Computational Linguistic...

    2022 arXiv

  2. [9]

    ChartQAPro: A more diverse and challenging benchmark for chart question answering

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. ChartQAPro: A more diverse and challengin...

    2025 doi

  3. [10]

    PlotQA: Reasoning over scientific plots

    Nitesh Methani et al. PlotQA: Reasoning over scientific plots. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020

    2020

  4. [11]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  5. [12]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  6. [13]

    Chartography leaderboard

    Surge AI. Chartography leaderboard. Public leaderboard, snapshot of July 16, 2026, 2026. https: //surgehq.ai/benchmarks/chartography

    2026

  7. [14]

    ChartMuseum: Testing visual reasoning capabilities of large vision-language models

    Liyan Tang et al. ChartMuseum: Testing visual reasoning capabilities of large vision-language models. arXiv preprint arXiv:2505.13444, 2025

    2025

  8. [15]

    Inspect AI: Framework for large language model evaluations

    UK AI Security Institute. Inspect AI: Framework for large language model evaluations. https: //inspect.aisi.org.uk, 2024

    2024

  9. [16]

    CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.Advances in Neural Information Processing Systems, 37, 2024

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.Advances in Neural Informati...

    2024 arXiv

  10. [17]

    ChartBench: A benchmark for complex visual reasoning in charts.arXiv preprint arXiv:2312.15915, 2023

    Zhengzhuo Xu et al. ChartBench: A benchmark for complex visual reasoning in charts.arXiv preprint arXiv:2312.15915, 2023. Official project page:https://chartbench.github.io/

    2023 arXiv

  11. [18]

    Acosta, Preston Carlson, Sacha Bron, Pierre-Loïc Doulcet, Daniel B

    Boyang Zhang, Sebastián G. Acosta, Preston Carlson, Sacha Bron, Pierre-Loïc Doulcet, Daniel B. Ospina, and Simon Suo. ParseBench: A document parsing benchmark for AI agents. arXiv preprint arXiv:2604.08538, 2026

    2026 arXiv

  12. [2023]

    For an outlet pipe with a diameter of 48 inches and a flow of 73 cubic feet per second, what is the minimum downstream width of the riprap apron?

    arXiv:2306.05685. 12 Figure 3:Engineering design chart (USDA-SCS).“For an outlet pipe with a diameter of 48 inches and a flow of 73 cubic feet per second, what is the minimum downstream width of the riprap apron?”Golden answer:30 feet(accepted 29–31). The flow lies below the 4...

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.