Pith. sign in

REVIEW 6 major objections 6 minor 1 cited by

Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

T0 review · 6 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Real scientific charts, especially flowcharts, show current multimodal models are far from human-level chart understanding, with the best proprietary model scoring 72.60 to humans' 90 on flowcharts.

desk verdict A substantial new chart QA benchmark with a real flowchart contribution, whose objective results are solid but whose open-ended scoring needs independent validation before the numbers are taken at face value. read the letter →

arxiv 2412.12150 v1 pith:5QHHFX3Q submitted 2024-12-11 cs.CL cs.CVcs.LG

Lingdong Shen , Qigqi , Kun Ding , Gaofeng Meng , Shiming Xiang This is my paper · ORCID
classification cs.CLcs.CVcs.LG
keywords scientificchartQAflowchartunderstandingmultimodallargelanguagemodelsbenchmarkevaluationcontext-basedreasoningquestionansweringexam-inspiredhallucinationmeasurement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SCI-CQA is a new benchmark for evaluating multimodal models on charts drawn from real scientific papers, with a deliberate focus on flowcharts—diagrams of model architectures, algorithm pipelines, and training processes—that most prior benchmarks ignore. The paper argues that existing chart benchmarks overestimate model abilities because they rely on narrow chart types, simple template-based questions, and one-note evaluation, so high scores do not survive contact with authentic scientific literature. The benchmark is built from 202,760 image-caption-context triplets mined from top computer science conferences, filtered to 37,607 charts, and paired with 5,629 expert-reviewed questions in multiple-choice, true/false, and open-ended formats. On the flowcharts, the best proprietary model scores 72.60 while human evaluators score around 90, and open-source models lag further, which the paper presents as evidence that real chart understanding remains unsolved. It also shows that adding the paper's surrounding textual context lets models answer questions that the image alone cannot answer.

What carries the argument

The load-bearing machinery is the SCI-CQA dataset itself, organized as (chart, caption, context) triplets extracted from LaTeX sources of 15 computer-science conferences, filtered by model-based classification and manual review into 37,607 charts. On top of this sits a 5,629-question bank with a 5:4:1 ratio of multiple-choice, true/false, and open-ended questions, assembled into 100-point exam papers that mirror human testing, with open-ended answers scored by a rubric-based model grader. The design that carries the argument is the addition of a sixth question type, scientific reasoning, and the inclusion of questions that cannot be answered from the image alone: these force the model either to use the surrounding caption and context or to say 'unable to answer', turning hallucination into a measurable failure. An automated annotation pipeline, built by fine-tuning an open-source vision-language model under supervision from free proprietary models, is used to generate pseudo-labels at reduced cost.

What would settle it

Have independent computer-science graduate students re-grade the open-ended answers from the three exam papers with no access to the model-generated scores; if the human scores differ systematically—say by more than one point in the direction of favoring GPT-style phrasing—then the reported model-versus-human gap and the inflation claim would be weakened.

Watch

Extended reading notes

Core claim

The central discovery is that existing benchmarks often overestimate chart understanding: when models are tested on authentic scientific charts, including flowcharts, their performance drops markedly relative to human performance. On SCI-CQA's flowchart test, the strongest proprietary model reaches 72.60 composite points while human graduate students in computer science score about 90; on data charts the same models score in the 60s, and open-source models trail by a wider margin. The paper attributes the gap to three flaws in prior benchmarks—limited chart variety, template-based questions that can be answered without reading the chart, and evaluation methods that do not separate perception from reasoning—and argues that a human-exam-inspired mix of multiple-choice, true/false, and open-ended questions gives a more accurate picture. A key supporting finding is that about 47% of machine-generated question-answer pairs were wrong before manual review, which the paper uses to argue that datasets without human review can mislead performance estimates.

Load-bearing premise

The load-bearing premise is that the human-curated, GPT-4o-generated questions are unbiased ground truth and that the GPT-4o-based grader scores open-ended answers fairly, without systematically favoring answers written in a GPT-4o-like style.

Editorial extensions

If this is right

  • High scores on template/synthetic chart benchmarks should not be read as evidence of real chart understanding; SCI-CQA shows the same models score far lower on authentic scientific charts.
  • Flowcharts should be treated as a first-class evaluation category, not an afterthought, because they stress structured reasoning and currently separate proprietary from open-source models.
  • Evaluations that mix multiple-choice, true/false, and open-ended questions reveal differences that a single question type masks, such as the 10.73-point composite gap between two models that differ by only 1.96 points on multiple-choice.
  • Supplying contextual text from the paper alongside the chart substantially improves performance on questions that are unanswerable from the image alone, while captions alone help much less.
  • Perception and reasoning are partially independent: once perception crosses a threshold, a model's reasoning score is not determined by its perception score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inflation claim generalizes, then public leaderboards built on synthetic/template chart QA should be re-read as measuring benchmark-fit rather than chart competence, and new benchmarks should include adversarial 'unanswerable' questions to expose hallucination.
  • A testable extension: because the paper reports that only about 47% of unaudited generated QA pairs were correct, other large multimodal benchmarks that skip human review may carry systematic label noise; re-auditing a sample of those datasets would reveal whether the inflation problem is widespread.
  • Since context mattered far more than captions for unanswerable questions, a natural next step is to measure how much context length or retrieval precision changes model scores, which could turn this benchmark into a tool for studying grounded scientific reasoning.
  • The automated annotation pipeline's gains were larger for objective questions than for open-ended ones, suggesting that genuinely hard open-ended chart reasoning may still require human annotation for the foreseeable future.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces SCI-CQA, a chart question-answering benchmark built from figures, captions, and surrounding context extracted from 75,506 papers at 15 computer science conferences. After filtering, 37,607 chart-context-caption triplets remain, and 5,629 multiple-choice, true/false, and open-ended questions are curated, with about half of the test items targeting flowcharts. The benchmark uses an exam-inspired evaluation protocol (40 MC, 40 TF, 4 open-ended per paper) and reports results for 14 multimodal models. The authors also propose an automated annotation pipeline based on a fine-tuned LLaVA variant (Chart2A/ChartAS) and study context-based reasoning with previously unanswerable questions. The main empirical claims are that existing benchmarks overestimate chart understanding, that flowcharts remain a hard problem for all models, and that proprietary models still lag humans by a large margin on SCI-CQA.

Significance. If the benchmark and its scoring procedure are validated, SCI-CQA would be a useful community resource: it draws on authentic scientific literature, includes flowcharts as a distinct and previously underevaluated chart type, combines objective and open-ended evaluation formats, and provides a human baseline. The automated annotation pipeline is a practical contribution with clear cost-savings potential, and the paper is transparent about its data collection prompts and scoring rubrics. The context-based experiment showing the value of surrounding text for answering chart questions is a valuable direction. However, the load-bearing quantitative findings depend on the validity of the ground truth, the scoring of open-ended answers, and the comparability of the human-model evaluation; these are not yet established with sufficient evidence.

major comments (6)
  1. [Experiment Setting; Fig. 8] Open-ended answers are scored by GPT-4o, which also generated the reference answers and is one of the evaluated models. The paper provides no validation that GPT-4o's rubric scores match human judgments, no blind evaluation, and no analysis of self-preference. Because open-ended questions contribute 20 of the 100 composite points, a systematic bias of 1 point on the 0-5 scale changes the composite by 4 points per test paper, which is large enough to reorder models near the top of Table 2. Please add a human-scored subset with inter-rater agreement, compare GPT-4o scores against human scores, and analyze whether GPT-4o favors answers in its own style.
  2. [Related Work; SCI-CQA data processing pipeline] The paper reports that only about 47% of GPT-4o-generated QA pairs were correct before manual review, yet it does not provide inter-annotator agreement, annotation guidelines, or an independent audit of the final 5,629 questions. If the human curation process has systematic blind spots—for example, accepting GPT-4o's phrasing or missing subtle errors—the ground truth would inherit generator bias. The dataset is not yet released, so the curation quality cannot be independently assessed. Please release the QA bank and provide detailed annotation statistics, including agreement measures and examples of rejected and corrected items.
  3. [Human-Inspired evaluation method; Table 5] The human-model comparison in Table 5 only includes Chartllama, CogVLM, and Qwen-VL-chat, with the highest model score reported as 41/100. This is not the same set of models whose scores appear in Tables 2-3, where GPT-4o reaches 72.60 on flowcharts and 66.17 on data charts. The sentence in the text stating that the highest model score was 41 is therefore misleading in context, and the claimed human-model gap of roughly 50 points does not apply to the best models evaluated in the paper. Please report human performance on the same test papers taken by the models in Tables 2-3, or clearly restrict the human comparison to the three models actually tested.
  4. [Performance analysis of flowchart [1]] The claim that existing benchmarks 'often overestimate' chart understanding is supported only by Chartllama's score drop from 69.66 on ChartQA to 21.12 on SCI-CQA flowcharts. A lower score on a new, more difficult benchmark does not by itself demonstrate that the old benchmark's scores are inflated; it may simply reflect task difficulty, distribution shift, or the model's training data. To support the inflation claim, the paper should either compare multiple models on both benchmarks with a human baseline on both, or show that prior benchmarks contain questions answerable without consulting the chart. Otherwise, the conclusion should be weakened to 'SCI-CQA is more challenging.'
  5. [Human-Inspired evaluation method; Table 2] The composite score definition—scaling open-ended scores to 0-100 and averaging with Avg MC and Avg TF—does not reproduce the reported composite scores under the natural interpretation. For example, GPT-4o on flowcharts would score (41.48 + 78.75 + 4.45*20)/3 = 69.7, not the reported 72.60. The column layout in Table 2 is also ambiguous. Please provide the exact composite formula and the underlying per-paper raw scores so that the headline numbers are reproducible.
  6. [Analysis of the Comparison Across Datasets; Table 1] Table 1 reports the number of evaluation samples as 1.5k for FigureQA, 1.5k for ChartQA, and 2.3k for CharXiv, but these numbers are inconsistent with the published versions of those benchmarks (e.g., ChartQA contains 32k questions, and FigureQA contains well over a million QA pairs). This makes the statement that 'SCI-CQA stands out with the highest number of evaluation samples (5.6k)' incorrect and undermines the comprehensiveness comparison. Please correct the table and revise the associated claim.
minor comments (6)
  1. [Figure 2 caption] The caption contains a typo: 'whichs led' should be 'which led'.
  2. [Automated labeling tools] The phrase 'a lager training dataset' should be 'a larger training dataset'.
  3. [Tables 2 and 3] The model name 'Qwen-VL-Chart' in Table 2 appears to be 'Qwen-VL-Chat' elsewhere in the paper; please standardize the naming.
  4. [Human-Inspired evaluation method] The paper lists 'Unanswerable Questions' as a question type, but the dataset statistics and evaluation description only cover multiple-choice, true/false, and open-ended questions. Please clarify how unanswerable questions and the 'unable to answer' option are scored.
  5. [Appendix, Q&A Data generation] The appendix uses inconsistent casing for 'GPT-4O' and 'GPT-4o'; please choose one convention throughout.
  6. [General] The dataset and code are only promised for future release ('will be released'); for a benchmark paper, availability at submission time is important for verification and adoption.

Circularity Check

2 steps flagged · score 4.0 of 10

Open-ended scores are GPT-4o judging GPT-4o-written answers, and the ChartAS annotation loop is trained on and evaluated by the same GPT-4o score distribution; objective questions and human scores keep the core benchmark from being fully circular.

  1. self definitional [Experiment Setting (open-ended scoring); Appendix Q&A Data generation; Figure 8 scoring prompt]
    "For open-ended questions, we use GPT-4o-based scoring (see the appendix for details). ... We used GPT-4O as the model for data generation, specifically for creating initial question-and-answer pairs. ... score the answers provided by a vision language model based on the content of the charts or diagrams and the standard answer."

    GPT-4o both writes the standard answers and acts as the judge for open-ended responses, while GPT-4o itself is one of the evaluated models. The 0-5 open-ended score therefore measures GPT-4o's agreement with its own generated reference, not an independently validated quality signal. The paper reports no human rating, no inter-annotator agreement, and no self-preference analysis for this scorer. Because four open-ended questions carry 20 of the 100 composite points, the reported GPT-4o open-ended advantage (4.45 vs. 1.23 for DocOwl1.5 in Table 2) is partly an artifact of the self-referential judging protocol, and any composite rankings affected by those 20 points inherit that artifact.

  2. fitted input called prediction [Automated labeling tools, stages [1]-[2]]
    "The fine-tuning dataset included 45k GPT-4o-annotated QA pairs, which had not undergone manual verification. ... we utilized all GPT-4o-based scores collected during testing, totaling 30k scores. ... After the completion of the first stage, we obtained two key models: Chart2A, used for data annotation, and ChartAS, used for question-answer scoring. ... we employed the free proprietary model Gemini-Pro-Vision and the fine-tuned ChartAS as supervisors to label 37k filtered chart-caption-context datasets. This labeling process generated 350k pseudo-labeled data points."

    ChartAS is fitted to GPT-4o's own scores, then used as a supervisor to score pseudo-labels, and those pseudo-labels train Chart2A. When the paper reports that Chart2A-fine-tuned models improve on SCI-CQA (Table 4), the open-ended portion of that benchmark is scored by the same GPT-4o scoring distribution from which ChartAS's supervision was derived. The pipeline's effectiveness is therefore self-confirming: the student is trained and evaluated within a closed loop anchored on GPT-4o's judgments, so the reported gains on open-ended questions are not independent evidence of annotation quality.

full rationale

SCI-CQA's headline comparison is not wholly circular: the benchmark uses real figures from scientific papers, the multiple-choice and true/false items are human-reviewed, and the human-vs-model gap in Table 5 provides an external anchor that does not depend on GPT-4o's self-scoring. The two self-referential loops are real but partial. First, open-ended answers carry 20 of 100 composite points and are graded by GPT-4o against standard answers also written by GPT-4o; no human rating, inter-annotator agreement, or self-preference analysis is reported, so the open-ended sub-scores and any composite ranking influenced by them measure GPT-4o's self-consistency rather than an independent quality signal. Second, the automated annotation pipeline trains ChartAS on GPT-4o scores, uses ChartAS to score pseudo-labels, trains Chart2A on those pseudo-labels, and then measures improvement on a benchmark whose open-ended portion is scored by the same GPT-4o scoring distribution; this makes the reported annotation-pipeline gains self-confirming. Because the central benchmark retains independent objective and human components, the derivation does not reduce entirely to its inputs, but the cited open-ended results should not be treated as independent evidence.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No mathematical free parameters are fitted to data in this benchmark paper, and no new physical or conceptual entities are postulated. The load-bearing assumptions are domain assumptions about dataset representativeness, ground-truth validity, and scoring reliability. Chart2A and ChartAS are fine-tuned models, not invented entities.

assumptions (4)
  • domain assumption Charts from 15 top-tier CS conferences over the past decade represent real-world scientific charts.
    The benchmark and its generalization claims rely on this representativeness; the paper selects only these conferences and treats them as the source of authentic scientific charts (Data processing pipeline, data source).
  • domain assumption Human-reviewed GPT-4o-generated questions form a valid ground truth.
    The paper reports only 47% of GPT-4o-generated QA pairs were correct before manual filtering, and the 5,629 final questions depend on undocumented review criteria with no inter-annotator agreement (Related Work; Experiment Setting).
  • domain assumption GPT-4o can reliably score open-ended chart-answer quality.
    Open-ended answers are scored with a GPT-4o prompt (Figure 8) without validation against human judges, despite GPT-4o also being one of the evaluated models (Experiment Setting).
  • domain assumption Nine graduate students on three test papers provide a reliable human gold standard.
    The model-versus-human gap is anchored on this small human evaluation sample, with no reported variance or sampling details (Human-Inspired evaluation method).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature." pith.science (2026). https://pith.science/paper/5QHHFX3Q

@misc{pith2026241212150,
  author       = {Pith},
  title        = {Pith review of: Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QHHFX3Q}},
  note         = {Machine review of arXiv:2412.12150}
}
read the original abstract

Scientific Literature charts often contain complex visual elements, including multi-plot figures, flowcharts, structural diagrams and etc. Evaluating multimodal models using these authentic and intricate charts provides a more accurate assessment of their understanding abilities. However, existing benchmarks face limitations: a narrow range of chart types, overly simplistic template-based questions and visual elements, and inadequate evaluation methods. These shortcomings lead to inflated performance scores that fail to hold up when models encounter real-world scientific charts. To address these challenges, we introduce a new benchmark, Scientific Chart QA (SCI-CQA), which emphasizes flowcharts as a critical yet often overlooked category. To overcome the limitations of chart variety and simplistic visual elements, we curated a dataset of 202,760 image-text pairs from 15 top-tier computer science conferences papers over the past decade. After rigorous filtering, we refined this to 37,607 high-quality charts with contextual information. SCI-CQA also introduces a novel evaluation framework inspired by human exams, encompassing 5,629 carefully curated questions, both objective and open-ended. Additionally, we propose an efficient annotation pipeline that significantly reduces data annotation costs. Finally, we explore context-based chart understanding, highlighting the crucial role of contextual information in solving previously unanswerable questions.

Figures

Figures reproduced from arXiv: 2412.12150 by the authors.

Figure 1
Figure 1. Comparison of SCI-CQA with previous work. The black box represents prior work, the green box highlights the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SCI-CQA data processing pipeline, multiple proprietary models is employed to monitor and enhance data quality, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data statistics (left) and automated annotation tool pipeline (right). High-quality annotated data and scoring data from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Impact of refined context and captions on answering previously unanswerable questions. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Generate question-and-answer pairs based solely on the provided flowchart image. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Generate question-and-answer pairs based on the provided flowchart image and additional text information. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Generate question-and-answer pairs based solely on the provided data chart image. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Criteria for Evaluating Open-Ended Answers by GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The multiple-choice question regarding the data chart. The labels on the x-axis highlighted with a red box. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The multiple-choice question regarding the flowchart. The number of convolutional layers in the top right corner is [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The multiple-choice question regarding the data chart. The red box in the figure clearly shows that the distribution [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The True/False question regarding the data chart. The figure shows that the median error of CWE-BOW [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: The True/False question regarding the flowchart. The Depth Encoder is not directly connects to the Semantic Map [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: The True/False question regarding the flowchart. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: The Open-Ended question regarding the flowchart. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: The Open-Ended question regarding the flowchart. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

    cs.CL 2026-08 reject novelty 6.0 of 10

    LongChart is a graph-consistent multi-chart VQA benchmark where 10 multimodal LLMs lose accuracy as question reasoning hops grow.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Ahmed, S.; Jawade, B.; Pandey, S.; Setlur, S.; and Govindaraju, V. 2023. RealCQA: Scientific Chart Question Answering as a Test-Bed for First-Order Logic. In ICDAR 2023 , volume 14189, 66--83

    work page 2023

  3. [3]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023 b . Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  5. [5]

    Chang, S.; Palzer, D.; Li, J.; Fosler-Lussier, E.; and Xiao, N. 2022. MapQA: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545

    arXiv 2022

  6. [6]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24185--24198

    2024

  7. [7]

    Dong, X.; Zhang, P.; Zang, Y.; Cao, Y.; Wang, B.; Ouyang, L.; Wei, X.; Zhang, S.; Duan, H.; Cao, M.; et al. 2024. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420

    arXiv 2024

  8. [8]

    Han, Y.; Zhang, C.; Chen, X.; Yang, X.; Wang, Z.; Yu, G.; Fu, B.; and Zhang, H. 2023. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483

    arXiv 2023

Show all 39 references
  1. [9]

    Hong, W.; Wang, W.; Lv, Q.; Xu, J.; Yu, W.; Ji, J.; Wang, Y.; Wang, Z.; Dong, Y.; Ding, M.; et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14281--14290

    2024

  2. [10]

    Hu, A.; Xu, H.; Ye, J.; Yan, M.; Zhang, L.; Zhang, B.; Li, C.; Zhang, J.; Jin, Q.; Huang, F.; et al. 2024. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895

    2024 arXiv

  3. [11]

    P.; Fung, Y

    Huang, K.-H.; Chan, H. P.; Fung, Y. R.; Qiu, H.; Zhou, M.; Joty, S.; Chang, S.-F.; and Ji, H. 2024. From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models. arXiv preprint arXiv:2403.12027

    2024 arXiv

  4. [12]

    L.; Cohen, S.; and Kanan, C

    Kafle, K.; Price, B. L.; Cohen, S.; and Kanan, C. 2018. DVQA: Understanding Data Visualizations via Question Answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , 5648--5656. Computer Vision Founda...

    2018

  5. [13]

    E.; Michalski, V.; Atkinson, A.; K \' a d \' a r, \' A .; Trischler, A.; and Bengio, Y

    Kahou, S. E.; Michalski, V.; Atkinson, A.; K \' a d \' a r, \' A .; Trischler, A.; and Bengio, Y. 2018. FigureQA: An Annotated Figure Dataset for Visual Reasoning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, ...

    2018

  6. [14]

    Kantharaj, S.; Leong, R. T. K.; Lin, X.; Masry, A.; Thakkar, M.; Hoque, E.; and Joty, S. R. 2022. Chart-to-Text: A Large-Scale Benchmark for Chart Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2022

  7. [15]

    Li, L.; Wang, Y.; Xu, R.; Wang, P.; Feng, X.; Kong, L.; and Liu, Q. 2024 a . Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00231

    2024 arXiv

  8. [16]

    Li, S.; and Tajbakhsh, N. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349

    2023 arXiv

  9. [17]

    Li, Y.; Zhang, Y.; Wang, C.; Zhong, Z.; Chen, Y.; Chu, R.; Liu, S.; and Jia, J. 2024 b . Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814

    2024 arXiv

  10. [18]

    Lin, Z.; Liu, C.; Zhang, R.; Gao, P.; Qiu, L.; Xiao, H.; Qiu, H.; Lin, C.; Shao, W.; Chen, K.; et al. 2023. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575

    2023 arXiv

  11. [19]

    Liu, F.; Wang, X.; Yao, W.; Chen, J.; Song, K.; Cho, S.; Yacoob, Y.; and Yu, D. 2023 a . Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774

    2023 arXiv

  12. [20]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  13. [21]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 b . Visual Instruction Tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023

    2023

  14. [22]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 c . Visual Instruction Tuning. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023

    2023

  15. [23]

    Lu, P.; Bansal, H.; Xia, T.; Liu, J.; Li, C.; Hajishirzi, H.; Cheng, H.; Chang, K.; Galley, M.; and Gao, J. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In The Twelfth International Conference on Learning Representations, ICLR 202...

    2024

  16. [24]

    Lu, P.; Bansal, H.; Xia, T.; Liu, J.; Li, C.; Hajishirzi, H.; Cheng, H.; Chang, K.-W.; Galley, M.; and Gao, J. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    2023 arXiv

  17. [25]

    R.; Hoque, E.; and Joty, S

    Masry, A.; Shahmohammadi, M.; Parvez, M. R.; Hoque, E.; and Joty, S. 2024. ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning. arXiv preprint arXiv:2403.09028

    2024 arXiv

  18. [26]

    M.; and Kumar, P

    Methani, N.; Ganguly, P.; Khapra, M. M.; and Kumar, P. 2020. PlotQA: Reasoning over Scientific Plots. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020 , 1516--1525. IEEE

    2020

  19. [27]

    Shen, L.; Huo, C.; Xu, N.; Han, C.; and Wang, Z. 2024. Learn How to See: Collaborative Embodied Learning for Object Detection and Camera Adjusting. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Art...

    2024

  20. [28]

    Singh, S.; Chaurasia, P.; Varun, Y.; Pandya, P.; Gupta, V.; Gupta, V.; and Roth, D. 2024. Flowvqa: Mapping multimodal logic in visual question answering with flowcharts. arXiv preprint arXiv:2406.19237

    2024 arXiv

  21. [29]

    Sun, Y.; Wang, S.; Feng, S.; Ding, S.; Pang, C.; Shang, J.; Liu, J.; Chen, X.; Zhao, Y.; Lu, Y.; et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137

    2021 arXiv

  22. [30]

    J.; Boggust, A

    Tang, B. J.; Boggust, A. W.; and Satyanarayan, A. 2023. VisText: A Benchmark for Semantically Rich Chart Captioning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 ,...

    2023

  23. [31]

    M.; Hauth, A.; et al

    Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  24. [32]

    Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; et al. 2023. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

    2023 arXiv

  25. [33]

    Wang, Z.; Xia, M.; He, L.; Chen, H.; Liu, Y.; Zhu, R.; Liang, K.; Wu, X.; Liu, H.; Malladi, S.; et al. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521

    2024 arXiv

  26. [34]

    Xia, R.; Zhang, B.; Ye, H.; Yan, X.; Liu, Q.; Zhou, H.; Chen, Z.; Dou, M.; Shi, B.; Yan, J.; et al. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185

    2024 arXiv

  27. [35]

    Xu, Z.; Du, S.; Qi, Y.; Xu, C.; Yuan, C.; and Guo, J. 2023. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915

    2023 arXiv

  28. [36]

    Ye, Q.; Xu, H.; Ye, J.; Yan, M.; Hu, A.; Liu, H.; Qian, Q.; Zhang, J.; and Huang, F. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13040--13051

    2024

  29. [37]

    Zhang, W.; Cheng, Z.; He, Y.; Wang, M.; Shen, Y.; Tan, Z.; Hou, G.; He, M.; Ma, Y.; Lu, W.; et al. 2024. Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model. arXiv preprint arXiv:2407.07053

    2024 arXiv

  30. [38]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.