Pith. sign in

REVIEW 5 major objections 4 minor 24 references

SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Pre-training figure-QA models on SBS Figures, a one-million-image synthetic chart dataset, improves accuracy on real-world ChartQA and enables efficient fine-tuning with limited real-world data.

desk verdict Genuine synthetic pre-training gains on ChartQA, but the unvalidated LLM-generated QA labels and an off-budget PlotQA comparison keep this from being a clean win. read the letter →

arxiv 2412.17606 v1 pith:UG2H5OBK submitted 2024-12-23 cs.CV

classification cs.CV
keywords syntheticdatafigurequestionansweringchartunderstandingpre-trainingdata-to-figuregenerationLLM-generatedQAvisualreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SBS Figures, a fully synthetic dataset of one million chart images paired with 4.2M dense question-answer annotations, and a stage-by-stage pipeline that produces it without any manual annotation. It claims that pre-training chart-question-answering models on this synthetic data improves their accuracy on real-world benchmarks such as ChartQA, and that a model can then be trained efficiently with only a limited amount of real-world chart data. The dataset and pipeline are released publicly, and the paper also identifies which design choices matter most for synthetic pre-training, including appearance randomization, LLM-generated QAs, and scale.

What carries the argument

The central object is the stage-by-stage generation engine. It separates figure creation into three stages: an LLM first writes a topic and a JSON data file containing title, axes, labels, data points and colors; pre-defined Python code, one script per chart type, renders the image with randomized fonts, title placement, legend position, markers, spines and number visibility; a second LLM pass turns the JSON into question-answer pairs without OCR. Because rendering code is fixed and data follows a fixed JSON schema, code errors are largely avoided, and each JSON file can be reused for many appearance variants. The complete JSON annotation is what makes dense QA generation cheap and makes answers traceable to ground-truth data.

What would settle it

Take a random sample of SBS Figures QA pairs, recompute the answers from the JSON data, and measure the mismatch rate; if a meaningful share of answers are wrong, the pre-training signal is partly noisy and the reported accuracy gains would need to be reassessed after filtering.

Watch

Extended reading notes

Core claim

Pre-training a figure QA model on SBS Figures improves performance on real-world chart QA compared with no pre-training and with pre-training on existing synthetic datasets. Using the Donut model, SBS Figures pre-training raises ChartQA average accuracy from 54.42 to 60.84, with the human-annotated split climbing from 31.28 to 39.44; the benefit also appears with Pix2Struct, from 60.92 to 64.52. The pre-training transfers to fine-tuning on PlotQA, FigureQA, and the UniChart reasoning task, and the improvement grows as the number of synthetic images increases from 50k to 1M. The paper also reports that QA-based pre-training outperforms JSON-parsing pre-training and that LLM-generated QAs outperform template-based QAs.

Load-bearing premise

The pipeline assumes that the question-answer pairs generated by the LLM from JSON data are accurate enough to train on, yet no human or automatic check of those answers is performed.

Editorial extensions

If this is right

  • Pre-training with SBS Figures improves ChartQA accuracy on both the human and augmented test splits for Donut and Pix2Struct, so the benefit is not tied to one architecture.
  • Fine-tuning a model pre-trained on SBS Figures reaches higher accuracy on PlotQA and FigureQA than training from scratch, so the synthetic pre-training transfers across chart QA tasks.
  • The pre-training benefit also appears when the fine-tuning stage is a figure-specific reasoning model trained on real-world charts, at every training step count tested.
  • Scaling from 50k to 1M synthetic images steadily improves ChartQA accuracy, so further scaling the dataset is a plausible route to further gains.
  • QA-form pre-training beats JSON-parsing pre-training, so the reasoning task itself, not just visual exposure, contributes to the transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper never audits the correctness of the 4.2M LLM-generated QA pairs, the reported gains may actually understate what a cleaned version of SBS Figures could achieve; filtering mismatched answers is a natural next experiment.
  • The design choices shown to matter, appearance diversity, LLM QA quality, and dataset scale, suggest a recipe for synthetic pre-training in other structured document domains, such as tables, diagrams, or maps.
  • If the LLM-generated QAs contain systematic calculation errors, then a hybrid pipeline that combines LLM-generated questions with programmatically computed answers might exceed both the template and pure-LLM conditions reported in F3.
  • The finding that no special pre-training prompt is needed means future synthetic datasets can be plugged into existing fine-tuning pipelines without redesigning instruction formats.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces SBS Figures, a fully synthetic dataset of 1M chart images generated in stages: LLM-produced data topics and JSON data, deterministic Python rendering with randomized appearance, and LLM-generated QA pairs from the JSON. The authors claim that pre-training a figure QA model on this dataset improves performance on real-world ChartQA, and they support this with experiments using Donut and Pix2Struct, ablations over five factors, and further fine-tuning on PlotQA and FigureQA. The pipeline is claimed to be error-free, diverse, and annotation-free, with the dataset and code to be released.

Significance. The paper makes a useful practical contribution: a controllable, large-scale synthetic figure QA resource with a released pipeline, plus a set of ablation studies (appearance, pre-training task, QA source, prompt format, dataset size) that are informative for the community. The core empirical result—that SBS Figures pre-training improves ChartQA performance over a Donut or Pix2Struct baseline—is supported by the reported numbers and is the main strength. However, the significance is conditional on validating the QA label quality and on cleaning up several experimental inconsistencies; if those are addressed, the dataset and pipeline could become a solid pre-training resource for figure understanding.

major comments (5)
  1. [§3.3] The claim that QA numbers and calculations are reliable because the LLM does not use OCR is not supported by any measurement. Design criterion (3) in §3 requires QA pairs to 'accurately reflect the information presented in the figures,' but no human verification or automatic consistency check is reported for any of the 4.2M QA pairs. Pre-training on incorrect labels could teach spurious associations, so this is a load-bearing gap; please add a validation study (e.g., a random sample of QA pairs checked against the rendered figures) and report the error rate, or otherwise show that label noise is negligible.
  2. [§3.2/§3.3] The color randomization in §3.2 appears inconsistent with QA generation in §3.3. §3.2 states that 'colors' are randomized in the figure generation code, while §3.1 places 'corresponding colors' in the JSON that is later used for QA generation. If the renderer overrides the JSON colors, then color-related questions (e.g., 'What does the purple color represent?') will have incorrect answers. Please clarify whether the randomized colors are written back into the JSON before QA generation; if not, correct the pipeline or demonstrate empirically that the overlap between randomized and JSON colors is sufficient for the color-based QA pairs.
  3. [§4.2, Table 1] The comparison with other synthetic pre-training datasets is not fair because PlotQA was pre-trained for only 1 epoch while all other datasets, including SBS Figures, were pre-trained for 3 epochs (stated in §4.2). The smaller number of optimizer steps for PlotQA could explain part of its lower performance, so the claim that 'only SBS Figures demonstrates the improvements' is not fully supported. Please re-run the comparison with matched training steps (or matched number of updates) and report results across multiple seeds with error bars; this is important because the superiority of SBS Figures over other synthetic datasets is a central claim.
  4. [Abstract/§4] The abstract and introduction claim that pre-training on SBS Figures enables 'efficient training with a limited amount of real-world chart data,' but no experiment in §4 varies the amount of real-world fine-tuning data; all ChartQA fine-tuning experiments use the full ChartQA training set. Table 7 varies the number of synthetic pre-training images, not the amount of real-world data. Please add an experiment that reduces the ChartQA (or another real-world dataset) fine-tuning set size to support the stated claim, or revise the claim to match the evidence.
  5. [Table 1 vs Table 2; Appendix A] There are numerical inconsistencies that need to be resolved. The SBS Figures row in Table 1 reports 39.44/82.24/60.84, while the Donut+SBS Figures row in Table 2 reports 39.20/81.20/60.84 for the same setting; the identical averages make it unlikely that these are different experimental conditions, so one of the tables is wrong. Additionally, the main text in §4.1 states a pre-training learning rate of 1e-4 and a fine-tuning learning rate of 5e-5, while Appendix A states the reverse (5e-5 for pre-training and 1e-4 for fine-tuning). These discrepancies make reproduction ambiguous and should be corrected.
minor comments (4)
  1. [Table 8] The column headers of Table 8 are ambiguous: 'PlotQA FigureQA Pre-train V1 V2 V1 V2' does not make clear which V1/V2 columns belong to PlotQA and which to FigureQA; please use separate headers and clarify that FigureQA likely has only one validation split.
  2. [Figure 5 caption] The caption says 'We randomly select 10 questions from each figure type and manually analyze the topic of the figure,' but the analysis is of figures, not questions; this should read '10 figures from each type.'
  3. [§2] The related work contains a typo: 'LEFQA / LEAFQA++' should be 'LEAF-QA / LEAFQA++'.
  4. [Throughout] The dataset name is written inconsistently as 'SBS Figures' and 'SBSFigures'; please choose one spelling and use it consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pre-training corpus is generated independently of the evaluation benchmarks, and the reported gains are measured on externally annotated ChartQA, PlotQA, and FigureQA splits.

full rationale

The paper's central claim is empirical and externally anchored: models pre-trained on SBS Figures are fine-tuned and evaluated on ChartQA, and additional experiments use PlotQA and FigureQA. None of these benchmarks is used to construct SBS Figures, and no parameter of the pre-training pipeline is fitted to the ChartQA test set and then reported as a prediction. The JSON data, rendered figures, and QA pairs are all generated from the same synthetic source, but that internal consistency is the intended dataset design rather than a circular derivation of the evaluation result. The claim that 'LLM can make QA pairs without the OCR process, making the QA numbers and calculations reliable' (Section 3.3) is an unvalidated data-quality assertion; if many QA labels are wrong, pre-training could be harmed, but that is a correctness risk, not circular reasoning, because the downstream evaluation uses independent human and augmented ChartQA labels. The F3 ablation compares LLM-generated and template-generated QA by downstream accuracy, which is a proxy for QA quality rather than a direct label-accuracy measurement, but it does not reduce the ChartQA result to the dataset construction choice by definition. No load-bearing self-citation, uniqueness theorem, or imported ansatz is used to justify the main result. The paper is therefore self-contained against external benchmarks, and no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the accuracy of the automatically generated QA labels and the transferability of synthetic chart appearance. Neither is independently verified beyond downstream performance.

assumptions (2)
  • domain assumption The LLM-generated QA pairs are accurate and reliable enough for use as training labels.
    Section 3.3 assumes reliability without verification; no human eval or automatic check is reported.
  • domain assumption The pre-defined rendering code and randomized appearance parameters produce chart images with sufficient diversity and realism to transfer to real-world charts.
    Section 3.2 and ablations (F1) support diversity benefit, but transferability to all real-world chart styles is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images." pith.science (2026). https://pith.science/paper/UG2H5OBK

@misc{pith2026241217606,
  author       = {Pith},
  title        = {Pith review of: SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UG2H5OBK}},
  note         = {Machine review of arXiv:2412.17606}
}
read the original abstract

Building a large-scale figure QA dataset requires a considerable amount of work, from gathering and selecting figures to extracting attributes like text, numbers, and colors, and generating QAs. Although recent developments in LLMs have led to efforts to synthesize figures, most of these focus primarily on QA generation. Additionally, creating figures directly using LLMs often encounters issues such as code errors, similar-looking figures, and repetitive content in figures. To address this issue, we present SBSFigures (Stage-by-Stage Synthetic Figures), a dataset for pre-training figure QA. Our proposed pipeline enables the creation of chart figures with complete annotations of the visualized data and dense QA annotations without any manual annotation process. Our stage-by-stage pipeline makes it possible to create diverse topic and appearance figures efficiently while minimizing code errors. Our SBSFigures demonstrate a strong pre-training effect, making it possible to achieve efficient training with a limited amount of real-world chart data starting from our pre-trained weights.

Figures

Figures reproduced from arXiv: 2412.17606 by the authors.

Figure 1
Figure 1. SBS Figures (Stage-by-Stage Synthetic Fig￾ures). We create SBS Figures, a dataset for pre-training fig￾ure QA. Our stage-by-stage synthetic dataset creation en￾ables a strong pre-training effect for real-world chart data. utilized template-based QA augmentation or LLMs (Kahou et al. 2018; Methani et al. 2020; Carbune et al. 2024). While this reduces the cost of QA generation, the number of figures remains limited. R… view at source ↗
Figure 2
Figure 2. Generation pipeline of SBS Figures. SBS Figures was created using a fully synthetic method. First, we generate the visualization data, represented in JSON format, containing complete numbers, text, and colors. Next, we produce figure images from this data using pre-defined, error-free Python scripts. Finally, we generate dense and accurate QA pairs from visualization data without the need for OCR. JSON-style data fo… view at source ↗
Figure 3
Figure 3. Prompt templates used in the generation pipeline of SBS Figures. We adopt few-shot prompting to ensure con￾sistent formatting for both JSON data and QA generation. To improve efficiency, our pipeline includes code that repeatedly adjusts the context and prompts during the generation process. data generation process ensures the diversity of topics in our dataset and consistency of the data formatting. 3.2 Figure gene… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Example of SBS Figures QA pairs. The figures show diverse visual variations, with each data content containing around 2,000 combinations of visual components. Additionally, our pipeline generates dense and precise QA pairs, requiring complex reasoning skills to address…
Figure 5
Figure 5. Figure 5: Theme distribution of SBS Figures. We ran￾domly select 10 questions from each figure type and man￾ually analyze the topic of the figure. we use exact match accuracy with numerical tolerance for 5%. Model Our experiments are mainly based on the Donut (Kim et al. 2022). …
Figure 6
Figure 6. Figure 6: Qualitative Comparison. Our pre-trained Donut model on SBS Figures demonstrates its ability to answer complex reasoning questions. Incorrect answers are high￾lighted in red, while correct answers are highlighted in green. sity within our dataset, contributing to more r…
Figure 7
Figure 7. Figure 7: QA distribution of SBS Figures. We randomly selected 100 QAs and manually analyzed their QA types [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Examples of SBS Figures figure images and QA pairs. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

  1. [1]

    Carbune, V.; Mansoor, H.; Liu, F.; Aralikatte, R.; Baechler, G.; Chen, J.; and Sharma, A. 2024. Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs. arXiv:2403.12596

  2. [2]

    Chaudhry, R.; Shekhar, S.; Gupta, U.; Maneriker, P.; Bansal, P.; and Joshi, A. 2020. LEAF-QA: Locate, Encode & Attend for Figure Question Answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

  3. [3]

    Cho, J.; Lei, J.; Tan, H.; and Bansal, M. 2021. Unifying Vision-and-Language Tasks via Text Generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 1931--1942. PMLR

  4. [4]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR

  5. [5]

    Han, Y.; Zhang, C.; Chen, X.; Yang, X.; Wang, Z.; Yu, G.; Fu, B.; and Zhang, H. 2023. ChartLlama: A Multimodal LLM for Chart Understanding and Generation. arXiv:2311.16483

  6. [6]

    Hoque, E.; Kavehzadeh, P.; and Masry, A. 2022. Chart question answering: State of the art and future directions. In Computer Graphics Forum, volume 41, 555--572. Wiley Online Library

  7. [7]

    Kafle, K.; Price, B.; Cohen, S.; and Kanan, C. 2018. DVQA: Understanding Data Visualizations via Question Answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5648--5656

  8. [8]

    E.; Michalski, V.; Atkinson, A.; Kadar, A.; Trischler, A.; and Bengio, Y

    Kahou, S. E.; Michalski, V.; Atkinson, A.; Kadar, A.; Trischler, A.; and Bengio, Y. 2018. FigureQA: An Annotated Figure Dataset for Visual Reasoning. arXiv:1710.07300

Show all 24 references
  1. [9]

    Kim, G.; Hong, T.; Yim, M.; Nam, J.; Park, J.; Yim, J.; Hwang, W.; Yun, S.; Han, D.; and Park, S. 2022. OCR-Free Document Understanding Transformer. In European Conference on Computer Vision (ECCV)

  2. [10]

    Lee, K.; Joshi, M.; Turc, I.; Hu, H.; Liu, F.; Eisenschlos, J.; Khandelwal, U.; Shaw, P.; Chang, M.-W.; and Toutanova, K. 2023. Pix2Struct: screenshot parsing as pretraining for visual language understanding. In Proceedings of International Conference on Machine Learning (ICML)

  3. [11]

    Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020. BART : Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the Association for Computatio...

  4. [12]

    Li, Z.; Jasani, B.; Tang, P.; and Ghadar, S. 2024. Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13613--13623

  5. [13]

    Liu, F.; Eisenschlos, J.; Piccinno, F.; Krichene, S.; Pang, C.; Lee, K.; Joshi, M.; Chen, W.; Collier, N.; and Altun, Y. 2023 a . D e P lot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics (ACL), 103...

  6. [14]

    Liu, F.; Piccinno, F.; Krichene, S.; Pang, C.; Lee, K.; Joshi, M.; Altun, Y.; Collier, N.; and Eisenschlos, J. 2023 b . M at C ha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering. In Proceedings of the 61st Annual Meeting of the Association for ...

  7. [15]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

  8. [16]

    L.; Hoque, E.; and Joty, S

    Masry, A.; Kavehzadeh, P.; Do, X. L.; Hoque, E.; and Joty, S. 2023. U ni C hart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 14662--14684. Singapor...

  9. [17]

    Q.; Joty, S.; and Hoque, E

    Masry, A.; Long, D.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics (ACL), 2263--2279. Dublin, Ireland

  10. [18]

    M.; and Kumar, P

    Methani, N.; Ganguly, P.; Khapra, M. M.; and Kumar, P. 2020. PlotQA: Reasoning over Scientific Plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV)

  11. [19]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140): 1--67

  12. [20]

    K.; and Farhadi, A

    Siegel, N.; Horvitz, Z.; Levin, R.; Divvala, S. K.; and Farhadi, A. 2016. FigureSeer: Parsing Result-Figures in Research Papers. In European Conference on Computer Vision

  13. [21]

    Singh, H.; and Shekhar, S. 2020. STL-CQA : Structure-based Transformers with Localization and Encoding for Chart Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3275--3284. Online: Association for Computati...

  14. [22]

    S.; Love, J.; Tafti, P.; Hussenot, L.; Sessa, P

    Team, G.; Mesnard, T.; Hardin, C.; Dadashi, R.; Bhupatiraju, S.; Pathak, S.; Sifre, L.; Rivière, M.; Kale, M. S.; Love, J.; Tafti, P.; Hussenot, L.; Sessa, P. G.; Chowdhery, A.; Roberts, A.; Barua, A.; Botev, A.; Castro-Ros, A.; Slone, A.; Héliou, A.; Tacchetti, A.; Bulanova, ...

  15. [23]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.