Pith. sign in

REVIEW 5 major objections 5 minor 74 references

ChartCap: Mitigating Hallucination of Dense Chart Captioning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChartCap pairs 565K real-world charts with dense captions limited to what the image shows, scores captions by rebuilding the chart in code, and trains models that beat proprietary baselines and human-written captions.

desk verdict Substantial dataset and a clever metric, but the 'beats humans' claim rides on a circular reconstruction pipeline and a filtered test set; deserves peer review with major revision. read the letter →

arxiv 2508.03164 v1 pith:2QRMBJKM submitted 2025-08-05 cs.CV cs.AIcs.CL

Junyoung Lim , Jaewoo Ahn , Gunhee Kim This is my paper
classification cs.CVcs.AIcs.CL
keywords chartcaptioningdensehallucinationmitigationvision-languagemodelsdatasetconstructioncycleconsistencyVisualScorereference-freeevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chart captions scraped from real documents are a weak training signal: they were written against the surrounding article, so they assert facts the chart cannot show, and they omit structural details the author expected readers to infer. ChartCap addresses this with a 565K-pair dataset of real-world chart images and dense, type-specific captions that state only what is visible in the image, marking absent elements explicitly as 'not specified'. The paper also introduces the Visual Consistency Score, which evaluates a caption by asking a large language model to write code that redraws the chart, then measuring how similar the redrawn chart is to the original. Vision-language models fine-tuned on ChartCap are rated by human judges as more informative, more accurate, and less hallucinated than open-source, chart-expert, and proprietary baselines, including Claude 3.5 Sonnet, and are preferred over human-written reference captions on two established benchmarks. The underlying claim is that for charts, the image is the ground truth, and caption quality can be measured by how faithfully the caption can be turned back into the chart.

What carries the argument

The load-bearing mechanism is cycle consistency between caption and chart, made possible by an intermediate modality: code. Because charts are deterministic outputs of plotting code, a caption can be translated by an LLM (Claude 3.5 Sonnet) into Matplotlib code, executed to produce a reconstructed chart image, and compared with the original. This one mechanism does double duty. In dataset construction it accelerates human verification: instead of reading caption-image pairs, annotators compare two images, which the paper reports is about 24 times faster with an F1 of 95%. In evaluation it becomes the Visual Consistency Score (VCS), the average cosine similarity (via SigLIP2 vision encoders at 512 resolution) between reconstructed and original charts, with OCRScore measuring text-element preservation as a complement. The other central object is the type-specific caption schema, derived from the Visualization Literacy Assessment Test's cognitive tasks, which specifies which structural elements (title, axes, legends, labels) and which key insights (extremum, range, trend, clusters, anomalies) must appear for each of nine chart types. The schema turns caption writing from an open-ended generation problem into a structured extraction problem, which is what lets the pipeline produce dense captions and lets 'not specified' serve as an explicit negative signal.

What would settle it

Take charts with log-scale axes or inset plots, write human-verified accurate captions for them, and run the Visual Consistency Score pipeline: if those captions score materially lower than equally accurate captions for simple charts purely because the code-generated rebuild fails, then the metric measures reconstruction feasibility, and the reported hallucination reduction may reflect which charts were kept in the test set rather than how well the models actually describe charts.

Watch

Extended reading notes

Core claim

The paper's central claim is that hallucination in chart captioning is mostly a data problem, and that the same reconstruction principle that fixes the data also fixes the metric. Existing real-world chart-caption pairs fail twice: captions contain information that cannot be inferred from the chart image alone because they were written against the surrounding document, and they omit the structural elements and key insights a reader needs. ChartCap replaces them with 565K pairs produced by a four-stage pipeline that filters non-chart images, classifies chart type and title, extracts type-specific structural and insight information using a schema grounded in the Visualization Literacy Assessment Test, and finalizes dense sentence captions, with 'not specified' used whenever an element is absent as a negative instruction against hallucination. A cycle-consistency procedure verifies quality by having an LLM write Matplotlib code that rebuilds the chart from the caption and comparing the rebuild to the original; the paper reports this is about 24 times faster than direct image-caption inspection at 95% F1. The Visual Consistency Score applies the same idea as an evaluation metric: translate a candidate caption to code, execute it, and average the SigLIP2 cosine similarity between rebuilt and original charts, supplemented by an OCRScore over extracted text. Experiments show fine-tuned InternVL2.5-8B and Phi3.5-Vision-4B surpass all baselines on reference-based metrics, VCS, OCRScore, and head-to-head human judgment, and that the gains transfer zero-shot to VisText and Chart-to-Text, where the fine-tuned captions are preferred to human-authored references.

Load-bearing premise

The whole method depends on being able to rebuild a chart from its caption with a large language model that writes Python code, so charts that resist rebuilding (log scales, insets, unusual styles) are filtered out of the dataset, and the measured quality is partly how rebuildable a caption is rather than how accurate or informative it is.

Editorial extensions

If this is right

  • Fine-tuning on ChartCap-style data reliably reduces chart-caption hallucination across model families: both InternVL2.5-8B and Phi3.5-Vision-4B improve over their base versions on every reported metric.
  • VCS and OCRScore provide a reference-free evaluation axis: they agree with human judgments more often than BERTScore, so caption quality can be audited at scale without the cost and bias of reference captions.
  • The gains transfer zero-shot to other human-verified benchmarks, VisText and Chart-to-Text, where ChartCap-trained models beat both Claude 3.5 Sonnet and human-annotated ground truth in human preference.
  • Training on original document captions from ArxivCap, ChartSumm, and ChartCheck degrades a base model's captioning performance, implying that low-quality reference data actively teaches hallucination and that the data, not model scale, is the main lever.
  • Cycle-consistency verification makes large-scale quality control economical, suggesting that human effort can be concentrated on comparing images rather than reading captions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because VCS measures rebuildability, a concise but correct caption such as 'prices rose then fell' would likely score below an exhaustive caption that lists every data point, since the code-writing LLM must guess unspecified details; an experiment comparing deliberately minimal and exhaustive captions of the same charts would show whether the metric measures faithfulness or exhaustiveness.
  • The pipeline's own manual audit places its failures in logarithmic-scale charts, scatter plots without clear correlation, and inset plots, exactly the charts that resist code reconstruction. Running VCS and human evaluation on a stress set of those excluded charts would show how much of the reported gain over baselines is genuine understanding and how much is selection on rebuildability.
  • The caption-to-code cycle transfers naturally to any visual format that has an executable intermediate representation, such as network diagrams, floor plans, or state machines, giving those domains the same reference-free metric and the same fast human-verification loop.
  • The appendix finding that caption-to-code distortion rises for less informative captions suggests VCS silently rewards exhaustiveness: models prompted to spell out every value would inflate their score even when human readers prefer a distilled summary, so VCS should be paired with a readability or conciseness check in practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ChartCap, a 565K-pair dataset of real-world chart images with dense, type-specific captions generated by a four-stage pipeline using GPT-4o, Claude 3.5 Sonnet, and GPT-4o-mini. The pipeline is designed to exclude extraneous information and to follow a caption schema covering structural components and key insights for nine chart types. The authors also propose the Visual Consistency Score (VCS), which regenerates a chart from a caption via LLM-written Matplotlib code and compares it to the original using SigLIP image embeddings and an OCR-based text score. They fine-tune InternVL2.5-8B and Phi3.5-Vision-4B on ChartCap and report improvements over open-source, chart-expert, and proprietary baselines in reference-based metrics, human evaluation, and VCS on ChartCap, VisText, and Chart-to-Text, including claims that the fine-tuned models surpass human-authored captions.

Significance. If the results hold, ChartCap would be a substantial new resource: it is the largest real-world chart-caption dataset with type-specific schemas, and the cycle-consistency verification could accelerate human quality control in future dataset construction. The paper is also useful for introducing a reference-free reconstruction-based metric for chart captions and for documenting the prompt and pipeline design in detail. However, the evaluation has a potential circularity because VCS and the test-set construction share the same caption-to-code reconstruction step, and the human-evaluation evidence for reduced hallucination is weakened by low inter-annotator agreement. The resource is nonetheless valuable, and the experiments are extensive, with detailed appendices and reproducible prompts.

major comments (5)
  1. [§3.4, §3.3] The VCS metric and the cycle-consistency filter used to build the test set share the same caption-to-code-then-image-similarity procedure (Claude 3.5 Sonnet to Matplotlib, SigLIP/OCR comparison). The 56K test set is precisely the subset of 68K caption-image pairs that survive this reconstruction test. Thus VCS measures the same property used to select the test distribution, and model gains on VCS are partly self-fulfilling for models trained on captions that already pass this filter. To support the claim that ChartCap reduces hallucination in general, the authors should evaluate on an unfiltered chart set or with a metric that does not rely on the same reconstruction step, e.g., direct human fact-checking on a random sample outside the filtered test set.
  2. [§4.2, Appendix J] The claim that ChartCap-trained models surpass human-authored captions on VisText rests on a 100-sample human evaluation, and the inter-annotator agreement for the 'fewer hallucinations' criterion is AC1=0.22–0.27 in the key comparisons. This level of agreement is close to chance and makes the reported 58–63% preference for the model caption not robust evidence of reduced hallucination. Please report per-item adjudication or a confidence interval, and consider a larger, more carefully defined hallucination protocol.
  3. [§3.2, §3.3] The dataset pipeline is validated on only 100 manually inspected samples, with 94% accuracy for type-specific information extraction, and the test set is filtered by the reconstruction step. The authors themselves note errors on logarithmic-scale charts, inset plots, and scatter plots without clear correlation (§3.2). Consequently, ChartCap may be biased toward charts that are easily reconstructable from captions, weakening the claim of general real-world coverage. The distribution of excluded charts and a breakdown by chart type should be reported, and a small unfiltered evaluation should be added.
  4. [Appendix F] Appendix F shows that for less informative captions the LLM 'hallucinated placeholder or arbitrary data values,' lowering VCS. This is exactly the situation of concise human-authored captions in VisText and Chart-to-Text, which omit exact coordinates as a stylistic choice. Therefore the VCS gap between ChartCap models and human captions may measure informativeness/verbosity of data listing rather than caption accuracy. The paper should either control for information density or show that VCS correlates with human accuracy judgments on captions matched for length.
  5. [Table 3] The reference-based metrics on the ChartCap test set are computed against reference captions generated by the same pipeline as the training captions. The large BLEU increase for fine-tuned models (from 8.41 to 23.82 for Phi3.5) may reflect stylistic overlap with the training distribution. Reporting reference-based metrics on VisText and Chart-to-Text, whose human-authored captions are not produced by this pipeline, would break this circularity; currently only VCS/OCRScore are reported there.
minor comments (5)
  1. [Abstract, §3.3, Table 1] The abstract states 565K pairs while §3.3 and Table 1 indicate a 56K human-verified test set and a 509K training set; please make the split explicit in the main text.
  2. [Figure 5] The two panels both label the model as 'Phi3.5-Vision-4B', which is confusing; the top panel should indicate 'Phi3.5-Vision-4BCHARTCAP vs Claude 3.5 Sonnet' and the bottom '... vs Phi3.5-Vision-4B'.
  3. [§3.4] The definition of Sim(Ii, Îi) should specify any preprocessing of images (e.g., aspect ratio, resolution) and whether the mean cosine similarity is over patches or the global embedding.
  4. [§3.3, Appendix D] The term 'human-verified' is used for the ChartCap test set, but the verification itself is cycle-consistency-based with a reported F1 of 95% against direct comparison; consider calling it 'cycle-consistency verified' to avoid implying full manual checking of every pair.
  5. [Appendix D.2] The premise that human inspectors make no mistakes is unrealistic; this does not invalidate the method but should be phrased as an idealization or a caveat rather than a premise.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: VCS re-runs the same caption-to-code reconstruction filter used to build the 56K test set, inflating ChartCap's advantage over human captions; independent human evals and zero-shot results keep the central claim from collapsing.

  1. self definitional [Sections 3.3-3.4, Appendix F, Section 4.2]
    "We generate Python code using Claude 3.5 Sonnet to recreate chart images from captions and then compare the reconstructed chart images with the originals. Applying human verification to 68K samples, we finalize a 56K test set. ... Given a caption Ci, an LLM is used to generate Matplotlib code Gi for recreating the chart. ... The similarity between Ii and ˆIi is computed using a cosine similarity with a vision encoder. ..."

    The 56K test set is selected by a caption-to-code reconstruction probe (Claude 3.5 Sonnet -> Matplotlib -> rendered image -> compare with original). VCS is the same probe averaged over captions, replacing the human comparison with SigLIP similarity. On the ChartCap test set, VCS re-scores the filter that defined the benchmark, and ChartCap-trained models are trained on captions produced by that same pipeline. Appendix F shows the probe punishes concise captions by making the LLM invent arbitrary data, so the Section 4.2 claim that ChartCap models 'surpass even the human-annotated ground-truth captions' on VCS measures reconstructability/verbosity rather than independent accuracy.

full rationale

The central evaluation chain is not fully self-contained. The dataset test set (Sec 3.3) and the proposed VCS metric (Sec 3.4) are built from the same caption-to-Matplotlib-code reconstruction operator using Claude 3.5 Sonnet. Since the 56K test set is the set of pairs that survived this operator (plus human verification), VCS on this test set is largely a re-run of the inclusion filter. ChartCap-trained models are optimized to emit captions with the explicit data values this probe requires, while human captions from VisText/Chart-to-Text are concise and are penalized by the LLM hallucinating placeholder values (Appendix F). Thus the VCS-based claim that ChartCap models surpass human-annotated captions is partly self-fulfilling. That said, the paper also reports head-to-head human evaluations, zero-shot performance on VisText and Chart-to-Text, and reference-based metrics; these provide independent evidence and prevent the whole claim from reducing to the pipeline. The cycle-consistency self-citation ([45], same group) is not load-bearing because the paper validates the process with its own F1=94.7% experiment (Appendix D). Overall, partial circularity in the VCS/test-set relationship warrants a moderate score, not a maximal one.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The central claims rest on domain assumptions about the reconstructability of charts from text, the validity of the caption schema, and the reliability of proprietary LLMs and the SigLIP2 encoder. No free parameters are fitted in the paper; the dataset and VCS are new artifacts, with the dataset lacking independent external verification because it is not released.

assumptions (5)
  • domain assumption A chart image can be reconstructed with sufficient fidelity from a natural-language caption via generated code.
    Central to both the dataset verification (Section 3.3) and the VCS metric (Section 3.4). Many real-world charts (e.g., choropleths, inset plots) may not be fully reconstructable.
  • domain assumption The type-specific caption schema derived from VLAT captures the key insights for each chart type.
    The schema defines what 'informative' means; if the schema is wrong or incomplete, the dataset's informativeness claim is undermined. Section 3.1.
  • domain assumption GPT-4o and Claude 3.5 Sonnet can extract accurate information from chart images with the reported accuracy (99%, 94%).
    The pipeline relies on these proprietary models; their error rates are estimated on a 100-sample manual inspection. Section 3.2.
  • domain assumption SigLIP2 embeddings capture visual similarity that aligns with human judgment of chart accuracy.
    VCS uses SigLIP2 cosine similarity as the reconstruction score; its agreement with humans is validated on 100 samples per comparison. Section 3.4 and Appendix E.
  • domain assumption The cycle-consistency verification process has 95% F1 and a 24x speedup over direct comparison.
    This justifies the human verification approach, but the F1 is computed on only 100 samples. Section 3.3 and Appendix D.
invented entities (2)
  • ChartCap dataset
    purpose: Training and evaluation resource for chart captioning with dense, schema-based captions.
    The dataset is claimed but not released; no independent handle to verify its contents. The paper provides statistics but no accessible data link.
  • Visual Consistency Score (VCS) independent evidence
    purpose: Reference-free metric for chart caption quality via chart reconstruction.
    VCS makes predictions about human preference (Appendix E) and can be computed by others on their own captions, providing a falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChartCap: Mitigating Hallucination of Dense Chart Captioning." pith.science (2026). https://pith.science/paper/2QRMBJKM

@misc{pith2026250803164,
  author       = {Pith},
  title        = {Pith review of: ChartCap: Mitigating Hallucination of Dense Chart Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QRMBJKM}},
  note         = {Machine review of arXiv:2508.03164}
}
read the original abstract

Generating accurate, informative, and hallucination-free captions for charts remains challenging for vision language models, primarily due to the lack of large-scale, high-quality datasets of real-world charts. However, existing real-world chart datasets suffer from the inclusion of extraneous information that cannot be inferred from the chart and failure to sufficiently capture structural elements and key insights. Therefore, we introduce ChartCap, a large-scale dataset of 565K real-world chart images paired with type-specific, dense captions that exclude extraneous information and highlight both structural elements and key insights in detail. To build ChartCap, we design a four-stage pipeline that generates captions using only the discernible data from the chart and employ a cycle consistency-based human verification, which accelerates quality control without sacrificing accuracy. Additionally, we propose a novel metric, the Visual Consistency Score, which evaluates caption quality by measuring the similarity between the chart regenerated from a caption and the original chart, independent of reference captions. Extensive experiments confirms that models fine-tuned on ChartCap consistently generate more accurate and informative captions with reduced hallucinations, surpassing both open-source and proprietary models and even human-annotated captions.

Figures

Figures reproduced from arXiv: 2508.03164 by the authors.

Figure 1
Figure 1. Comparison of the original caption and our C [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of the four-stage pipeline for our C [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An illustration of the cycle consistency-based human [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Results of human evaluation results comparing Phi3.5- [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Qualitative examples from VisText, comparing (a) the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Accuracy of GPT-4o and Claude 3.5 Sonnet on coarse [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Examples of the four main scenarios that arise during the cycle consistency-based human verification process. In Scenario A, [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Human evaluation results comparing Phi3.5-Vision [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: User interface for human evaluation comparing captions from different models on informativeness, accuracy, and fewer hallu [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: User interface for human evaluation comparing datasets (C [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 3 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024. 2, 6

    arXiv 2024

  2. [2]

    Chartcheck: Explainable fact-checking over real-world chart images

    Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tah- masebi, Oana Cocarascu, and Elena Simperl. Chartcheck: Explainable fact-checking over real-world chart images. In Findings of the Association for Computational Linguistics ACL 2024, pages 13921–13937, 2024. 1, 3

    2024

  3. [3]

    Estimation of the timing of human visual perception from magnetoen- cephalography

    Kaoru Amano, Naokazu Goda, Shin’ya Nishida, Yoshimichi Ejima, Tsunehiro Takeda, and Yoshio Ohtani. Estimation of the timing of human visual perception from magnetoen- cephalography. Journal of Neuroscience, 26(15):3981–3991,

  4. [4]

    Introducing Claude 3.5 Sonnet, 2024

    Anthropic. Introducing Claude 3.5 Sonnet, 2024. 2, 6

    2024

  5. [5]

    METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments. InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, 2005. Association for Computational Linguistics. 3, 6

    2005

  6. [6]

    Hartmann

    Bernd Burghardt and Alexander K. Hartmann. Rna sec- ondary structure design. Physical Review E , 75(2), 2007. 2

    2007

  7. [7]

    Infor- mation graphics: an untapped resource for digital libraries

    Sandra Carberry, Stephanie Elzer, and Seniz Demir. Infor- mation graphics: an untapped resource for digital libraries. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information re- trieval, pages 581–588, 2006. 1

    2006

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. ArXiv preprint, abs/2412.05271, 2024. 2, 4, 6

    arXiv 2024

Show all 74 references
  1. [9]

    The elements of graphing data

    William S Cleveland. The elements of graphing data . Wadsworth Publ. Co., 1985. 1

    1985

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20- 25 June 2009, Miami, Florida, USA , pages 248–255. I...

    2009

  3. [11]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applica- tions of Artificial Intelligence, IAAI 2024,...

    2024

  4. [12]

    Chartllama: A mul- timodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A mul- timodal llm for chart understanding and generation. ArXiv preprint, abs/2311.16483, 2023. 2, 3

    2023 arXiv

  5. [13]

    CLIPScore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 7514–7528, Online and Punta Cana, Do...

    2021

  6. [14]

    SciCap: Generating captions for scientific figures

    Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 3258–3264, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. 1, 3

    2021

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Represen- tations, ICLR 2022, Virtual Event, April 25-29, 2022....

    2022

  8. [16]

    Ciem: Contrastive instruction evaluation method for better instruction tuning

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning. ArXiv preprint, abs/2309.02301, 2023. 3

    2023 arXiv

  9. [17]

    Do lvlms understand charts? analyzing and correcting factual errors in chart captioning

    Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do lvlms understand charts? analyzing and correcting factual errors in chart captioning. ArXiv preprint, abs/2312.10160, 2023. 1

    2023 arXiv

  10. [18]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  11. [19]

    Swe- 9 bench: Can language models resolve real-world github is- sues? ArXiv preprint, abs/2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe- 9 bench: Can language models resolve real-world github is- sues? ArXiv preprint, abs/2310.06770, 2023. 5

    2023 arXiv

  12. [20]

    Chart-to-text: A large-scale benchmark for chart summariza- tion

    Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summariza- tion. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1...

    2022

  13. [21]

    Answering questions about charts and generating visual ex- planations

    Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. Answering questions about charts and generating visual ex- planations. In CHI ’20: CHI Conference on Human Fac- tors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020, pages 1–13. ACM, 2020. 1

    2020

  14. [22]

    Evaluating the factual consistency of ab- stractive text summarization

    Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of ab- stractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 9332–9346, Online, 2020. Ass...

    2020

  15. [23]

    Why a diagram is (some- times) worth ten thousand words

    Jill H Larkin and Herbert A Simon. Why a diagram is (some- times) worth ten thousand words. Cognitive science, 11(1): 65–100, 1987. 1

    1987

  16. [24]

    Vlat: Development of a visualization literacy assessment test

    Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics , 23(1):551–560, 2016. 2, 4, 8

    2016

  17. [25]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. ArXiv preprint, abs/2403.00231, 2024. 1, 2, 3

    2024 arXiv

  18. [26]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 292–305, Singapore, 2023. Association for ...

    2023

  19. [27]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, 2004. Association for Computa- tional Linguistics. 3, 6

    2004

  20. [28]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  21. [29]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 3, 4

    2023

  22. [30]

    MMC: Advancing multimodal chart understanding with large-scale instruction tuning

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. MMC: Advancing multimodal chart understanding with large-scale instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Associa- ti...

    2024

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

    2024

  24. [32]

    Misinformed by visualization: What do we learn from misinformative visualizations? In Computer Graphics Forum , pages 515–525

    Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, En- rico Bertini, and Huamin Qu. Misinformed by visualization: What do we learn from misinformative visualizations? In Computer Graphics Forum , pages 515–525. Wiley Online Library, 2022. 1

    2022

  25. [33]

    Accessible visual- ization via natural language descriptions: A four-level model of semantic content

    Alan Lundgard and Arvind Satyanarayan. Accessible visual- ization via natural language descriptions: A four-level model of semantic content. IEEE transactions on visualization and computer graphics, 28(1):1073–1083, 2021. 1, 4, 7

    2021

  26. [34]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. A...

    2022

  27. [35]

    UniChart: A universal vision- language pretrained model for chart comprehension and rea- soning

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. UniChart: A universal vision- language pretrained model for chart comprehension and rea- soning. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 14662...

    2023

  28. [36]

    Chartinstruct: Instruction tuning for chart comprehension and reasoning

    Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning. ArXiv preprint, abs/2403.09028, 2024. 2, 3, 6

    2024 arXiv

  29. [37]

    Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild

    Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild. ArXiv preprint, abs/2407.04172, 2024. 2, 3, 6

    2024 arXiv

  30. [38]

    On faithfulness and factuality in abstractive sum- marization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive sum- marization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906– 1919, Online, 2020. Association for Computatio...

    1906

  31. [39]

    Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning.ArXiv preprint, abs/2401.02384, 2024

    Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning.ArXiv preprint, abs/2401.02384, 2024. 2, 3

    2024 arXiv

  32. [40]

    Visualization analysis and design

    Tamara Munzner. Visualization analysis and design . CRC press, 2014. 2, 4

    2014

  33. [41]

    Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms

    Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms. In European Conference on Computer Vision, pages 395–413. Springer, 2025. 3

    2025

  34. [42]

    PaddleOCR Documentation, 2024

    PaddleOCR. PaddleOCR Documentation, 2024. 5 10

    2024

  35. [43]

    The persuasive power of data visualization

    Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Mar- garet Satterthwaite, and Enrico Bertini. The persuasive power of data visualization. IEEE Transactions on Visual- ization and Computer Graphics, 20(12):2211–2220, 2014. 1

    2014

  36. [44]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311– 318, Philadelphia, Pennsylvania, USA, 2002. Asso...

    2002

  37. [45]

    Text2Chart31: Instruction tuning for chart generation with automatic feedback

    Fatemeh Pesaran Zadeh, Juyeon Kim, Jin-Hwa Kim, and Gunhee Kim. Text2Chart31: Instruction tuning for chart generation with automatic feedback. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing , pages 11459–11480, Miami, Florida, USA, ...

    2024

  38. [46]

    The communicative function of ambiguity in language

    Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cogni- tion, 122(3):280–291, 2012. 4

    2012

  39. [47]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision, ICCV ...

    2015

  40. [48]

    A call for clarity in reporting BLEU scores

    Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Transla- tion: Research Papers , pages 186–191, Brussels, Belgium,

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  42. [50]

    Chartsumm: A comprehensive bench- mark for automatic chart summarization of long and short summaries

    Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tah- mid Rahman Laskar, Md Hamjajul Ashmafee, and Abu Rai- han Mostofa Kamal. Chartsumm: A comprehensive bench- mark for automatic chart summarization of long and short summaries. ArXiv preprint, abs/2304.13620, 2023. 1, 3, 5, 6

    2023 arXiv

  43. [51]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing , pages 4035– 4045, Brussels, Belgium, 2018. Association f...

    2018

  44. [52]

    Striking a balance: Reader takeaways and preferences when integrating text and charts

    Chase Stokes, Vidya Setlur, Bridget Cogley, Arvind Satya- narayan, and Marti A Hearst. Striking a balance: Reader takeaways and preferences when integrating text and charts. IEEE Transactions on Visualization and Computer Graph- ics, 29(1):1233–1243, 2022. 1

    2022

  45. [53]

    Aligning large multi- modal models with factually augmented rlhf.ArXiv preprint, abs/2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf.ArXiv preprint, abs/2309.14525, 2023. 3

    2023 arXiv

  46. [54]

    Vis- Text: A benchmark for semantically rich chart captioning

    Benny Tang, Angie Boggust, and Arvind Satyanarayan. Vis- Text: A benchmark for semantically rich chart captioning. In Proceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7268–7298, Toronto, Canada, 2023. Asso...

    2023

  47. [55]

    Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut

    Ashish V . Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multi- modal evaluation dataset. InProceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emira...

    2022

  48. [56]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025 arXiv

  49. [57]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. ArXiv preprint, abs/2403.18715, 2024. 3

    2024 arXiv

  50. [58]

    Charxiv: Charting gaps in realis- tic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yi- tao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realis- tic chart understanding in multimodal llms. ArXiv preprint, abs/2406.18521, 2024. 6

    2024 arXiv

  51. [59]

    Chartbench: A benchmark for complex visual reasoning in charts

    Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning in charts. ArXiv preprint, abs/2312.15915,

    arXiv

  52. [60]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback. In Proceedings of the IEEE/CVF Conference on ...

    2024

  53. [61]

    Ad- vancing multimodal large language models in chart question answering with visualization-referenced instruction tuning

    Xingchen Zeng, Haichuan Lin, Yilin Ye, and Wei Zeng. Ad- vancing multimodal large language models in chart question answering with visualization-referenced instruction tuning. IEEE Transactions on Visualization and Computer Graph- ics, 2024. 2

    2024

  54. [62]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 2, 3, 6

    2020

  55. [63]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. ArXiv preprint, abs/2311.16839, 2023. 3

    2023 arXiv

  56. [64]

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle- 11 consistent adversarial networks. In IEEE International Con- ference on Computer Vision, ICCV 2017, Venice, Italy, Octo- ber 22-29, 2017, pages 2242–2251. IEEE Com...

    2017

  57. [65]

    Retrieve Value

    Jiawen Zhu, Jinye Ran, Roy Ka-Wei Lee, Zhi Li, and Kenny Choo. AutoChart: A dataset for chart-to-text generation task. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pages 1636–1644, Held Online, 2021. INCOMA Ltd...

    2021

  58. [68]

    If the caption is incorrect, it produces faulty code leading to a mismatched image, which is identified and removed

    Scenario A. If the caption is incorrect, it produces faulty code leading to a mismatched image, which is identified and removed

  59. [69]

    If the caption lacks sufficient detail, an over- simplified chart is generated and subsequently filtered out

    Scenario B. If the caption lacks sufficient detail, an over- simplified chart is generated and subsequently filtered out

  60. [70]

    Even if the caption is accurate, errors in code generation or execution can result in a failed recon- struction, leading the sample to be excluded

    Scenario C. Even if the caption is accurate, errors in code generation or execution can result in a failed recon- struction, leading the sample to be excluded

  61. [71]

    bar”, “line

    Scenario D. Only when the caption is both accurate and informative, and the chart regenerates without errors, does the sample pass verification. This process ensures that only captions containing both correct and adequately detailed information are retained. In summary, our pr...

  62. [72]

    Informativeness – Does the caption adequately describe the chart’s structure and key insights (highlighted in green and blue in Fig 1)?

  63. [73]

    Accuracy – How faithfully does the caption reflect the chart’s structure and key insights?

  64. [74]

    SIRLC" represented by ared lineand red shaded area,

    Fewer Hallucinations – Does the caption avoid infor- mation that cannot be inferred from the chart (high- lighted in red in Fig 1)? For the dataset-level study in § 3.5 (C HART CAP vs Chart- Summ), we added a fourth question–overall preference–to capture holistic quality while...

  65. [2018]

    Association for Computational Linguistics. 6

  66. [2649]

    IEEE Computer Society, 2015. 5

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.