Pith. sign in

REVIEW 3 major objections 5 minor 58 references

CHAOS: Chart Analysis with Outlier Samples

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that chart-specialized multimodal language models are the least robust to realistic chart perturbations, and that even noise humans barely notice costs every tested model at least 4% accuracy.

desk verdict CHAOS is a useful chart-robustness benchmark with a broken headline metric; the raw per-level accuracy data are the real contribution. read the letter →

arxiv 2505.17235 v1 pith:U4U7YUVB submitted 2025-05-22 cs.CV cs.CL

Omar Moured , Yufan Chen , Ruiping Liu , Simon Rei{ss} , Philip Torr , Jiaming Zhang , Rainer Stiefelhagen This is my paper
classification cs.CVcs.CL
keywords chartanalysisrobustnessbenchmarkmultimodallargelanguagemodelsvisualperturbationstextualquestionansweringsummarizationhumanstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds CHAOS, a benchmark that perturbs charts the way the real world does—blur, motion, fading, speckle, occlusions, warping, and character-level typos in questions—and asks whether 13 multimodal large language models can still read them. It argues that current MLLMs are far more fragile than clean-dataset scores suggest: every tested model loses at least 4% accuracy under easy distortions that humans barely notice, and textual typos alone can cost up to 31%. The central claim is a robustness trade-off: chart-specialized models win on clean charts but are the least robust ($R_{\mathrm{Chart}} = 68.7$), while general-purpose models, despite the lowest clean scores, are the most robust ($R_{\mathrm{Gen}} = 80.68$), with document specialists in between. If true, this matters wherever chart-reading models meet imperfect inputs—scans, camera photos, speech-recognized questions—because it means fine-tuning on clean chart data has not produced models that survive everyday noise.

What carries the argument

The CHAOS testbed itself: ten visual perturbations (defocus, vibration, warping, omission, ink-bleeding, ink-holdout, obstacles, fading, speckle, texture) and five textual perturbations (character addition, deletion, replacement, swap, and word modification), each at three severity levels anchored to human thresholds from a 42-participant study (at least 90% correct answers defines easy, the modal level defines middle, and the level with any correct answer defines hard). The argument is carried by two measurement tools: relaxed accuracy on ChartQA and BLEU-4/Content Selection on Chart-to-Text, plus the proposed robustness score $R$, which combines absolute perturbed accuracy with relative drop from clean performance so that equal absolute drops receive lower scores for models with lower clean accuracy.

What would settle it

Re-run CHAOS with severity thresholds re-estimated from an independent, larger human study: if chart-specialized models no longer score below general models under the recalibrated levels, the robustness trade-off claim collapses. A narrower check is to measure easy-level losses directly: the at-least-4% claim fails if any model loses less than 4% on the same easy perturbations in an out-of-sample replication.

Watch

Extended reading notes

Core claim

The paper's central discovery is that robustness to realistic chart perturbations does not follow clean accuracy. Across 10 visual and 5 textual perturbation types at three severity levels calibrated by human readers, every model's accuracy falls by at least 4% even at the easiest, nearly imperceptible settings. Ranking the three model families by the proposed robustness score $R$, general-purpose models prove most robust ($R_{\mathrm{Gen}} = 80.68$), document-specialized models next ($R_{\mathrm{Doc}} = 77.03$), and chart-specialized models, despite leading on clean ChartQA accuracy, prove least robust ($R_{\mathrm{Chart}} = 68.7$), with average drops of 23.25% at easy and roughly 50% at mid and hard visual levels. Textual perturbations are nearly as damaging as visual ones, and robustness does not track input resolution or parameter count. The authors suggest the chart models' fragility may trace to fine-tuning on synthetic, clean chart data, which brings out-of-domain degradation when a domain shift such as scanned or photographed charts appears.

Load-bearing premise

The benchmark's severity levels rest on a single online study with 42 participants, so the model rankings at each level are only as trustworthy as those human thresholds, whereas the raw clean-to-perturbed accuracy drops are direct measurements independent of them.

Editorial extensions

If this is right

  • Deployments of chart-reading MLLMs on real-world inputs—scans, photographs, audio-transcribed questions—should expect at least a 4-percentage-point accuracy loss even under distortions humans find trivial.
  • Chart-specific fine-tuning as currently practiced buys clean accuracy at a robustness cost: chart-specialized models lose roughly half their accuracy at mid and hard visual severity levels, whereas general-purpose models degrade more gracefully.
  • Textual robustness deserves equal weight with visual robustness in evaluation, since character-level typos alone can drop accuracy by up to 31%.
  • Robustness cannot be predicted from input resolution (up to 768x768) or parameter count (up to 13B); training data and fine-tuning strategy are the controlling factors.
  • On chart-to-text summarization, perturbations produce severe output degradation—hallucinated numbers, repetitive phrases, and up to a tenfold increase in inference time.
  • Augmented template-based questions remain easier and more robust under perturbation than human-written reasoning questions, so robustness scores should be reported per question type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit yardstick: the human calibration turns CHAOS into a pass/fail test against human perception—a model is human-robust at a level if it still answers where at least 90% of people can—so the benchmark can define robustness relative to people, not just to clean accuracy.
  • A testable extension the paper does not run: training on mildly perturbed charts below the human-easy threshold should raise the robustness score $R$ without sacrificing clean performance.
  • The general models' advantage under speckle and ink-bleeding suggests robustness tracks interpretive flexibility—using undamaged chart elements to infer damaged ones—which chart-specialized models could be taught explicitly through data augmentation or reasoning strategies.
  • If the synthetic-data explanation is right, collecting real noisy chart corpora (photographed, scanned, faded documents) is the direct remedy, and the public benchmark makes that hypothesis testable by retraining on perturbed samples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CHAOS, a robustness benchmark for multimodal large language models (MLLMs) on chart understanding. It defines 10 visual and 5 textual perturbation types at three severity levels derived from a 42-participant human study, evaluates 13 MLLMs (general, document, and chart-specific) on ChartQA and Chart-to-Text, and proposes a novel robustness metric R that combines relative and absolute accuracy drops. The main findings are that MLLMs are sensitive to even easy perturbations, that general models are more robust than chart-specialized models, and that textual perturbations are as impactful as visual ones.

Significance. If the results are reproducible, the benchmark would be a useful public resource for evaluating chart-understanding robustness. Strengths include the public release of data and code, the human-perception-based severity levels, the breadth of perturbations, and the inclusion of 13 models across three groups. The raw per-level accuracy drops are direct measurements and provide valuable observations. However, the central quantitative claims rest on a robustness metric whose printed definition is unparseable, and one headline finding contradicts the paper's own table. These issues currently prevent verification of the main comparative conclusions.

major comments (3)
  1. [Section 3.3, Eq. (1)] Equation (1) as typeset is not a well-formed expression and cannot yield the stated range [0,1]. For example, substituting A_x = A_clean (perfect robustness) gives 1/A_clean - A_clean, which is not equal to 1 and can exceed 1 for typical accuracy values. Because the R scores in Table 2 and the group averages in Finding 2 are computed from this equation, those results cannot be recomputed or verified from the manuscript. Please provide a corrected mathematical definition, a worked example, and make the evaluation code available so that Table 2's R_VP and R_TP values can be reproduced. Until then, Finding 2 ('general models highest robustness, chart-specific lowest') is unsupported.
  2. [Section 4.2.1, Finding 1] The claim that 'even under easy perturbations... performance drops by at least 4%' is contradicted by Table 2, where several models show smaller easy-level VP drops: GPT-4o (-2.60), Qwen2.5-VL (-2.33), DocOwl2 (-2.91), DocOwl1.5 (-3.52), and LLaVA-OneVision (-3.90). Please either restate the finding to reflect the actual distribution (e.g., the mean easy-level drop is X%, with chart-specific models dropping more) or provide the subset or condition under which the 4% floor holds.
  3. [Section 3.2, severity levels] The definitions of Easy, Middle, and Hard rely on a single online study with 42 participants, and the paper itself acknowledges in Section 4.4 that a larger and more diverse user base is needed. The thresholds (90% correct, mode, at least one correct) are ad hoc, and the description 'Starting from Level 10, we incrementally increase the severity' is ambiguous since Level 10 is the maximum severity. Please clarify the procedure and provide a sensitivity analysis (e.g., bootstrap over participants) so that the severity-stratified comparisons are not dependent on the exact outcome of one small study.
minor comments (5)
  1. [Abstract and Section 4] The manuscript inconsistently states the number of evaluated models: the Contributions and Section 4 say '8 MLLMs,' while the abstract and Table 2 list 13 models. Please correct this discrepancy.
  2. [Throughout] The phrase 'textural perturbations' should be 'textual perturbations' in the Fig. 1 caption, Table 2 header, and Section 4.1.2.
  3. [Section 4.1.3] The general MLLM list includes 'Qwen-VL [3],' but Table 2 uses 'Qwen2.5-VL [4]'; please use consistent naming and citations.
  4. [Section 4.1.2] The text states that the robustness metric R is compiled for both tasks, but Table 3 reports only BLEU-4 and no R values; please either include R values for Chart-to-Text or clarify that R is reported only for ChartQA.
  5. [Section 3.2] The 'Middle Level' definition uses 'mode' and 'majority' interchangeably; the mode is not necessarily a majority, so please rephrase precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CHAOS is an empirical benchmark whose findings rest on direct model evaluations, not on fitted constants or self-citation.

full rationale

The central findings (Finding 1: sensitivity to minor perturbations; Finding 2: general models are most robust, chart-specific models least robust) are direct measurements of relaxed accuracy on clean and perturbed inputs reported in Table 2 and Tables 6-7. The robustness metric R in Eq. (1), although typeset ambiguously and not recomputable from the printed formula, is proposed as an aggregation of the measured quantities A_clean and A_x; it is not fitted to the model rankings it produces, so the ranking is not circular by construction. The easy/middle/hard severity levels are anchored to an external human study with 42 participants (Sec. 3.1-3.2), not to model outputs, so the severity labels do not encode the findings. The citation to RoDLA [8], which shares authors with this paper, is used for the perturbation taxonomy, but the ten visual and five textual perturbations are explicitly specified in Table 1 and Appendix C with independent mathematical definitions, and the model evaluations are self-contained. The metric's illegibility and the small human-study sample are legitimate correctness and reproducibility concerns, but they do not make the derivation circular. Overall, no load-bearing step reduces to its own inputs or to a self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on benchmark construction choices: perturbation implementations, dataset choices, a human severity calibration, and the RA metric. No model parameters are fitted to derive the findings, but the benchmark's difficulty labels and metric limits are choices made by the authors.

free parameters (1)
  • Perturbation severity parameters (Gaussian sigma for VP1/VP3/VP9, kernel length L for VP2, alpha/beta for VP8, blob… = Per-level values defined by human-study thresholds, exact values not listed in main text
    These parameters define the easy/mid/hard perturbations. They are chosen by the authors based on human evaluation rather than fitted to model performance, but they shape all benchmark results.
assumptions (4)
  • domain assumption Gaussian convolution, morphological dilation/erosion, spatial transforms, and noise addition faithfully approximate real-world chart degradations.
    Invoked in Appendix C.1 for all visual perturbations; if these simulations do not match real conditions, the benchmark's external validity is limited.
  • domain assumption ChartQA and Chart-to-Text are representative proxies for chart understanding robustness.
    Both tasks are standard in the field, but results on these datasets may not generalize to all chart types or question styles.
  • domain assumption Relaxed Accuracy with 5% tolerance is an appropriate correctness metric.
    The paper itself notes in Section 4.4 that the 5% tolerance is problematic for small numbers and year-based answers, weakening this premise.
  • ad hoc to paper Severity levels derived from 42 participants using the stated thresholds (90%, mode, at least one correct) are stable and meaningful.
    Section 3.2 introduces this calibration; the small, unrepresentative sample is a load-bearing assumption for all per-level comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CHAOS: Chart Analysis with Outlier Samples." pith.science (2026). https://pith.science/paper/U4U7YUVB

@misc{pith2026250517235,
  author       = {Pith},
  title        = {Pith review of: CHAOS: Chart Analysis with Outlier Samples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4U7YUVB}},
  note         = {Machine review of arXiv:2505.17235}
}
read the original abstract

Charts play a critical role in data analysis and visualization, yet real-world applications often present charts with challenging or noisy features. However, "outlier charts" pose a substantial challenge even for Multimodal Large Language Models (MLLMs), which can struggle to interpret perturbed charts. In this work, we introduce CHAOS (CHart Analysis with Outlier Samples), a robustness benchmark to systematically evaluate MLLMs against chart perturbations. CHAOS encompasses five types of textual and ten types of visual perturbations, each presented at three levels of severity (easy, mid, hard) inspired by the study result of human evaluation. The benchmark includes 13 state-of-the-art MLLMs divided into three groups (i.e., general-, document-, and chart-specific models) according to the training scope and data. Comprehensive analysis involves two downstream tasks (ChartQA and Chart-to-Text). Extensive experiments and case studies highlight critical insights into robustness of models across chart perturbations, aiming to guide future research in chart understanding domain. Data and code are publicly available at: http://huggingface.co/datasets/omoured/CHAOS.

Figures

Figures reproduced from arXiv: 2505.17235 by the authors.

Figure 1
Figure 1. (a) ¢ CHart Analysis with Outlier Samples (CHAOS) benchmark includes 5 types of textural perturbations (TP) and 10 types of visual perturbations (VP), where each has 3 levels (Easy, Mid, Hard). Results of general, document- and chart-specific MLLMs are compared on (b) textual perturbations and (c) visual perturbations with the relaxed accuracy (RA) scores. Abstract Charts play a critical role in data analysis and vi… view at source ↗
Figure 2
Figure 2. Visualization of CHAOS benchmark with 10 types of visual perturbations (VPs) and 5 types of textual perturbations (TPs). on fine-tuning and visual instruction tuning to optimize performance across diverse language and vision tasks. These models are crucial for deeper multimodal learning, as evidenced by UNITER [7] and BLIP [28], which refine how images and text interact. UNITER selects specific image areas while BLI… view at source ↗
Figure 3
Figure 3. Distribution of human study results across [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of the metric R across per￾turbed and clean accuracy. All models on the same ‘contour’ have the same R score. For the same absolute drop (clean→perturbed), the model with a lower clean accuracy has a lower robustness. E.g., Ra>Rb=Rc, when a=(0.7, 0.8), b=…
Figure 5
Figure 5. Figure 5: Robustness analysis. The clean accuracy is repre [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Study design. Participants start at (a) the highest perturbation level (Level 10) for each [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Distribution of human study results across perturbation types and levels. Each cell shows [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Chart with FD perturbation used in the user study. The zoomed-in bounding box highlights [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Case study of hallucinations across TP and VP. The samples are selected from cases where [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 40 canonical work pages

  1. [1]

    Realcqa: Scientific chart question answering as a test-bed for first-order logic

    Saleem Ahmed, Bhavin Jawade, Shubham Pandey, Srirangaraj Setlur, and Venu Govindaraju. Realcqa: Scientific chart question answering as a test-bed for first-order logic. In Gernot A. Fink, Rajiv Jain, Koichi Kise, and Richard Zanibbi, editors, Document Analysis and Recognition - ICDAR 2023 , pages 66–83, Cham, 2023. Springer Nature Switzerland. 2

    work page 2023

  2. [2]

    Ic- dar 2023 competition on robust layout segmentation in corporate documents

    Christoph Auer, Ahmed Nassar, Maksym Lysak, Michele Dolfi, Nikolaos Livathinos, and Peter Staar. Ic- dar 2023 competition on robust layout segmentation in corporate documents. InInternational Conference on Document Analysis and Recognition, pages 471–482. Springer, 2023. 3

    work page 2023

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 6, 8, 16, 17, 21, 22

    arXiv 2023

  4. [4]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv

  5. [5]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoff- man. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 15

    arXiv 2022

  6. [6]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025. 6, 7, 17, 21, 22

    arXiv 2025

  7. [7]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pages 104–120. Springer, 2020. 3

    work page 2020

  8. [8]

    Rodla: Benchmarking the robustness of document layout analysis models

    Yufan Chen, Jiaming Zhang, Kunyu Peng, Junwei Zheng, Ruiping Liu, Philip Torr, and Rainer Stiefelha- gen. Rodla: Benchmarking the robustness of document layout analysis models. In CVPR, 2024. 3

    work page 2024

Show all 58 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. 15

    2020

  2. [10]

    Deep unrestricted document image rectification

    Hao Feng, Shaokai Liu, Jiajun Deng, Wengang Zhou, and Houqiang Li. Deep unrestricted document image rectification. IEEE Transactions on Multimedia, 2023. 3

    2023

  3. [11]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020. 3

    2020

  4. [12]

    Evaluating adversarial robustness on document image classification

    Timoth ´ee Fronteau, Arnaud Paran, and Aymen Shabou. Evaluating adversarial robustness on document image classification. In International Conference on Document Analysis and Recognition, pages 290–304. Springer, 2023. 3

    2023

  5. [13]

    Chartllama: A multimodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483,

    arXiv

  6. [14]

    Evaluation of deep convolutional nets for document image classification and retrieval

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991–995. IEEE, 2015. 3

    2015

  7. [15]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019. 3, 5

    2019

  8. [16]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021. 3, 5

    2021

  9. [17]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895, 2024. 6, 7, 8, 16, 21, 22

    2024 arXiv

  10. [18]

    mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024. 6, 7, 16, 21, 22, 23 10

    2024 arXiv

  11. [19]

    Vprochart: Answering chart question through visual perception alignment agent and programmatic solution reasoning

    Muye Huang, Lingling Zhang, Lai Han, Wenjun Wu, Xinyu Zhang, and Jun Liu. Vprochart: Answering chart question through visual perception alignment agent and programmatic solution reasoning. arXiv preprint arXiv:2409.01667, 2024. 2

    2024 arXiv

  12. [20]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    arXiv

  13. [21]

    @Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology

    Xin Jiang, Junwei Zheng, Ruiping Liu, Jiahang Li, Jiaming Zhang, Sven Matthiesen, and Rainer Stiefel- hagen. @Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology. In WACV, 2025. 2

    2025

  14. [22]

    Figureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, ´Akos K´ad´ar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300,

    arXiv

  15. [23]

    OpenCQA: Open-ended question answering with charts

    Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. OpenCQA: Open-ended question answering with charts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natura...

    2022

  16. [24]

    Chart-to-text: A large-scale benchmark for chart summarization

    Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2022

  17. [25]

    Faithful chart summarization with ChaTS-pi

    Syrine Krichene, Francesco Piccinno, Fangyu Liu, and Julian Eisenschlos. Faithful chart summarization with ChaTS-pi. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024

  18. [26]

    Pix2struct: Screenshot parsing as pretraining for visual language understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Ur- vashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICML, pages 18893–18912...

    2023

  19. [27]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 ,

    arXiv

  20. [28]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900. PMLR, 2022. 3

    2022

  21. [29]

    Imagenet-e: Benchmark- ing neural network robustness via attribute editing

    Xiaodan Li, Yuefeng Chen, Yao Zhu, Shuhui Wang, Rong Zhang, and Hui Xue. Imagenet-e: Benchmark- ing neural network robustness via attribute editing. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20371–20381, 2023. 3, 5

    2023

  22. [30]

    Matcha: Enhancing visual language pretraining with math reasoning and chart derendering, 2023

    Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering, 2023. 3

    2023

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 8, 16

    2023 arXiv

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

    2024

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 8

    2024

  26. [34]

    Llava- next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava- next: Improved reasoning, ocr, and world knowledge, 2024. 8, 17

    2024

  27. [35]

    ChartThinker: A contextual chain- of-thought approach to optimized chart summarization

    Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, and Ying Shen. ChartThinker: A contextual chain- of-thought approach to optimized chart summarization. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedin...

    2024

  28. [36]

    Data extraction from charts via single deep neural net- work

    Xiaoyi Liu, Diego Klabjan, and Patrick NBless. Data extraction from charts via single deep neural net- work. arXiv preprint arXiv:1906.11906, 2019. 2

    1906 arXiv

  29. [37]

    Chartocr: Data extraction from charts images via a deep hybrid framework

    Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. Chartocr: Data extraction from charts images via a deep hybrid framework. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1916–1924, 2021. 2

    2021

  30. [38]

    Towards an efficient framework for data extraction from chart images

    Weihong Ma, Hesuo Zhang, Shuang Yan, Guangshun Yao, Yichao Huang, Hui Li, Yaqiang Wu, and Lianwen Jin. Towards an efficient framework for data extraction from chart images. In International Conference on Document Analysis and Recognition, pages 583–597. Springer, 2021. 2

    2021

  31. [39]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 2, 3, 6

    2022

  32. [40]

    Chartin- struct: Instruction tuning for chart comprehension and reasoning.arXiv preprint arXiv:2403.09028, 2024

    Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartin- struct: Instruction tuning for chart comprehension and reasoning.arXiv preprint arXiv:2403.09028, 2024. 6, 7, 16, 21, 22

    2024 arXiv

  33. [41]

    Char- tassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning

    Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Char- tassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.02384, 2024. 6, 7, 16, 21, 22

    2024 arXiv

  34. [42]

    Plotqa: Reasoning over sci- entific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over sci- entific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1527–1536, 2020. 2, 6

    2020

  35. [43]

    Accessible document layout: An interface for 2d tactile displays

    Omar Moured, Sara Alzalabny, Thorsten Schwarz, Bastian Rapp, and Rainer Stiefelhagen. Accessible document layout: An interface for 2d tactile displays. InProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, pages 265–271...

    2023

  36. [44]

    Line graphics digitization: A step towards full automation

    Omar Moured, Jiaming Zhang, Alina Roitberg, Thorsten Schwarz, and Rainer Stiefelhagen. Line graphics digitization: A step towards full automation. In ICDAR, pages 438–453. Springer, 2023. 2

    2023

  37. [45]

    Altchart: Enhancing vlm- based chart summarization through multi-pretext tasks

    Omar Moured, Jiaming Zhang, M Saquib Sarfraz, and Rainer Stiefelhagen. Altchart: Enhancing vlm- based chart summarization through multi-pretext tasks. In ICDAR, pages 349–366. Springer, 2024. 2

    2024

  38. [46]

    Charteye: A deep learning framework for chart information extraction

    Osama Mustafa, Muhammad Khizer Ali, Momina Moetesum, and Imran Siddiqi. Charteye: A deep learning framework for chart information extraction. In 2023 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 554–561. IEEE, 2023. 2

    2023

  39. [47]

    Zero-shot embeddings inform learning and forgetting with vision-language encoders

    Laura Niss, Kevin V ogt-Lowell, and Theodoros Tsiligkaridis. Zero-shot embeddings inform learning and forgetting with vision-language encoders. arXiv preprint arXiv:2407.15731, 2024. 8

    2024 arXiv

  40. [48]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 3

    2021

  41. [49]

    ChartSumm: A large scale benchmark for Chart to Text Summarization

    Raian Rahman, Rizvi Hasan, and Abdullah Al Farhad. ChartSumm: A large scale benchmark for Chart to Text Summarization. PhD thesis, Department of Computer Science and Engineering (CSE), Islamic University of . . . , 2022. 2

    2022

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  43. [51]

    A robust system for document layout analysis using multilevel homogeneity structure

    Tuan Anh Tran, Kanghan Oh, In-Seop Na, Guee-Sang Lee, Hyung-Jeong Yang, and Soo-Hyung Kim. A robust system for document layout analysis using multilevel homogeneity structure. Expert Systems with Applications, 85:99–113, 2017. 3

    2017

  44. [52]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024. 6, 7, 17, 21, 22

    2024 arXiv

  45. [53]

    Chartmoe: Mixture of expert connector for advanced chart understanding

    Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of expert connector for advanced chart understanding. 2025. 6, 7, 9, 16, 21, 22, 23

    2025

  46. [54]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In EMNLP, 2023. 6, 7, 8, 16, 21, 22 12

    2023

  47. [55]

    Sigmoid loss for language im- age pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language im- age pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 17

    2023

  48. [56]

    Tiny- chart: Efficient chart understanding with visual token merging and program-of-thoughts learning

    Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tiny- chart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv preprint arXiv:2404.16635, 2024. 2, 6, 7, 15, 16, 20, 21, 22

    2024 arXiv

  49. [57]

    w” in “How,

    Mingliang Zhang, Zhen Cao, Juntao Liu, Liqiang Niu, Fandong Meng, and Jie Zhou. Welayout: Wechat layout analysis system for the icdar 2023 competition on robust layout segmentation in corporate docu- ments. ArXiv, abs/2305.06553, 2023. 3 13 A Human Evaluation Details To conduc...

    2023 arXiv

  50. [2024]

    2, 6, 7, 16, 17, 21, 22

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.