REVIEW 5 major objections 4 minor 24 references
SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images
T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Pre-training figure-QA models on SBS Figures, a one-million-image synthetic chart dataset, improves accuracy on real-world ChartQA and enables efficient fine-tuning with limited real-world data.
desk verdict Genuine synthetic pre-training gains on ChartQA, but the unvalidated LLM-generated QA labels and an off-budget PlotQA comparison keep this from being a clean win. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the stage-by-stage generation engine. It separates figure creation into three stages: an LLM first writes a topic and a JSON data file containing title, axes, labels, data points and colors; pre-defined Python code, one script per chart type, renders the image with randomized fonts, title placement, legend position, markers, spines and number visibility; a second LLM pass turns the JSON into question-answer pairs without OCR. Because rendering code is fixed and data follows a fixed JSON schema, code errors are largely avoided, and each JSON file can be reused for many appearance variants. The complete JSON annotation is what makes dense QA generation cheap and makes answers traceable to ground-truth data.
What would settle it
Take a random sample of SBS Figures QA pairs, recompute the answers from the JSON data, and measure the mismatch rate; if a meaningful share of answers are wrong, the pre-training signal is partly noisy and the reported accuracy gains would need to be reassessed after filtering.
Extended reading notes
Core claim
Pre-training a figure QA model on SBS Figures improves performance on real-world chart QA compared with no pre-training and with pre-training on existing synthetic datasets. Using the Donut model, SBS Figures pre-training raises ChartQA average accuracy from 54.42 to 60.84, with the human-annotated split climbing from 31.28 to 39.44; the benefit also appears with Pix2Struct, from 60.92 to 64.52. The pre-training transfers to fine-tuning on PlotQA, FigureQA, and the UniChart reasoning task, and the improvement grows as the number of synthetic images increases from 50k to 1M. The paper also reports that QA-based pre-training outperforms JSON-parsing pre-training and that LLM-generated QAs outperform template-based QAs.
Load-bearing premise
The pipeline assumes that the question-answer pairs generated by the LLM from JSON data are accurate enough to train on, yet no human or automatic check of those answers is performed.
Editorial extensions
If this is right
- Pre-training with SBS Figures improves ChartQA accuracy on both the human and augmented test splits for Donut and Pix2Struct, so the benefit is not tied to one architecture.
- Fine-tuning a model pre-trained on SBS Figures reaches higher accuracy on PlotQA and FigureQA than training from scratch, so the synthetic pre-training transfers across chart QA tasks.
- The pre-training benefit also appears when the fine-tuning stage is a figure-specific reasoning model trained on real-world charts, at every training step count tested.
- Scaling from 50k to 1M synthetic images steadily improves ChartQA accuracy, so further scaling the dataset is a plausible route to further gains.
- QA-form pre-training beats JSON-parsing pre-training, so the reasoning task itself, not just visual exposure, contributes to the transfer.
Reading between the lines
- Because the paper never audits the correctness of the 4.2M LLM-generated QA pairs, the reported gains may actually understate what a cleaned version of SBS Figures could achieve; filtering mismatched answers is a natural next experiment.
- The design choices shown to matter, appearance diversity, LLM QA quality, and dataset scale, suggest a recipe for synthetic pre-training in other structured document domains, such as tables, diagrams, or maps.
- If the LLM-generated QAs contain systematic calculation errors, then a hybrid pipeline that combines LLM-generated questions with programmatically computed answers might exceed both the template and pure-LLM conditions reported in F3.
- The finding that no special pre-training prompt is needed means future synthetic datasets can be plugged into existing fine-tuning pipelines without redesigning instruction formats.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SBS Figures, a fully synthetic dataset of 1M chart images generated in stages: LLM-produced data topics and JSON data, deterministic Python rendering with randomized appearance, and LLM-generated QA pairs from the JSON. The authors claim that pre-training a figure QA model on this dataset improves performance on real-world ChartQA, and they support this with experiments using Donut and Pix2Struct, ablations over five factors, and further fine-tuning on PlotQA and FigureQA. The pipeline is claimed to be error-free, diverse, and annotation-free, with the dataset and code to be released.
Significance. The paper makes a useful practical contribution: a controllable, large-scale synthetic figure QA resource with a released pipeline, plus a set of ablation studies (appearance, pre-training task, QA source, prompt format, dataset size) that are informative for the community. The core empirical result—that SBS Figures pre-training improves ChartQA performance over a Donut or Pix2Struct baseline—is supported by the reported numbers and is the main strength. However, the significance is conditional on validating the QA label quality and on cleaning up several experimental inconsistencies; if those are addressed, the dataset and pipeline could become a solid pre-training resource for figure understanding.
major comments (5)
- [§3.3] The claim that QA numbers and calculations are reliable because the LLM does not use OCR is not supported by any measurement. Design criterion (3) in §3 requires QA pairs to 'accurately reflect the information presented in the figures,' but no human verification or automatic consistency check is reported for any of the 4.2M QA pairs. Pre-training on incorrect labels could teach spurious associations, so this is a load-bearing gap; please add a validation study (e.g., a random sample of QA pairs checked against the rendered figures) and report the error rate, or otherwise show that label noise is negligible.
- [§3.2/§3.3] The color randomization in §3.2 appears inconsistent with QA generation in §3.3. §3.2 states that 'colors' are randomized in the figure generation code, while §3.1 places 'corresponding colors' in the JSON that is later used for QA generation. If the renderer overrides the JSON colors, then color-related questions (e.g., 'What does the purple color represent?') will have incorrect answers. Please clarify whether the randomized colors are written back into the JSON before QA generation; if not, correct the pipeline or demonstrate empirically that the overlap between randomized and JSON colors is sufficient for the color-based QA pairs.
- [§4.2, Table 1] The comparison with other synthetic pre-training datasets is not fair because PlotQA was pre-trained for only 1 epoch while all other datasets, including SBS Figures, were pre-trained for 3 epochs (stated in §4.2). The smaller number of optimizer steps for PlotQA could explain part of its lower performance, so the claim that 'only SBS Figures demonstrates the improvements' is not fully supported. Please re-run the comparison with matched training steps (or matched number of updates) and report results across multiple seeds with error bars; this is important because the superiority of SBS Figures over other synthetic datasets is a central claim.
- [Abstract/§4] The abstract and introduction claim that pre-training on SBS Figures enables 'efficient training with a limited amount of real-world chart data,' but no experiment in §4 varies the amount of real-world fine-tuning data; all ChartQA fine-tuning experiments use the full ChartQA training set. Table 7 varies the number of synthetic pre-training images, not the amount of real-world data. Please add an experiment that reduces the ChartQA (or another real-world dataset) fine-tuning set size to support the stated claim, or revise the claim to match the evidence.
- [Table 1 vs Table 2; Appendix A] There are numerical inconsistencies that need to be resolved. The SBS Figures row in Table 1 reports 39.44/82.24/60.84, while the Donut+SBS Figures row in Table 2 reports 39.20/81.20/60.84 for the same setting; the identical averages make it unlikely that these are different experimental conditions, so one of the tables is wrong. Additionally, the main text in §4.1 states a pre-training learning rate of 1e-4 and a fine-tuning learning rate of 5e-5, while Appendix A states the reverse (5e-5 for pre-training and 1e-4 for fine-tuning). These discrepancies make reproduction ambiguous and should be corrected.
minor comments (4)
- [Table 8] The column headers of Table 8 are ambiguous: 'PlotQA FigureQA Pre-train V1 V2 V1 V2' does not make clear which V1/V2 columns belong to PlotQA and which to FigureQA; please use separate headers and clarify that FigureQA likely has only one validation split.
- [Figure 5 caption] The caption says 'We randomly select 10 questions from each figure type and manually analyze the topic of the figure,' but the analysis is of figures, not questions; this should read '10 figures from each type.'
- [§2] The related work contains a typo: 'LEFQA / LEAFQA++' should be 'LEAF-QA / LEAFQA++'.
- [Throughout] The dataset name is written inconsistently as 'SBS Figures' and 'SBSFigures'; please choose one spelling and use it consistently.
Circularity Check
No circularity: the pre-training corpus is generated independently of the evaluation benchmarks, and the reported gains are measured on externally annotated ChartQA, PlotQA, and FigureQA splits.
full rationale
The paper's central claim is empirical and externally anchored: models pre-trained on SBS Figures are fine-tuned and evaluated on ChartQA, and additional experiments use PlotQA and FigureQA. None of these benchmarks is used to construct SBS Figures, and no parameter of the pre-training pipeline is fitted to the ChartQA test set and then reported as a prediction. The JSON data, rendered figures, and QA pairs are all generated from the same synthetic source, but that internal consistency is the intended dataset design rather than a circular derivation of the evaluation result. The claim that 'LLM can make QA pairs without the OCR process, making the QA numbers and calculations reliable' (Section 3.3) is an unvalidated data-quality assertion; if many QA labels are wrong, pre-training could be harmed, but that is a correctness risk, not circular reasoning, because the downstream evaluation uses independent human and augmented ChartQA labels. The F3 ablation compares LLM-generated and template-generated QA by downstream accuracy, which is a proxy for QA quality rather than a direct label-accuracy measurement, but it does not reduce the ChartQA result to the dataset construction choice by definition. No load-bearing self-citation, uniqueness theorem, or imported ansatz is used to justify the main result. The paper is therefore self-contained against external benchmarks, and no circular step can be exhibited.
Assumptions & free parameters
assumptions (2)
- domain assumption The LLM-generated QA pairs are accurate and reliable enough for use as training labels.
- domain assumption The pre-defined rendering code and randomized appearance parameters produce chart images with sufficient diversity and realism to transfer to real-world charts.
Cite this review
Pith. "Pith review of SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images." pith.science (2026). https://pith.science/paper/UG2H5OBK
@misc{pith2026241217606,
author = {Pith},
title = {Pith review of: SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/UG2H5OBK}},
note = {Machine review of arXiv:2412.17606}
}
read the original abstract
Building a large-scale figure QA dataset requires a considerable amount of work, from gathering and selecting figures to extracting attributes like text, numbers, and colors, and generating QAs. Although recent developments in LLMs have led to efforts to synthesize figures, most of these focus primarily on QA generation. Additionally, creating figures directly using LLMs often encounters issues such as code errors, similar-looking figures, and repetitive content in figures. To address this issue, we present SBSFigures (Stage-by-Stage Synthetic Figures), a dataset for pre-training figure QA. Our proposed pipeline enables the creation of chart figures with complete annotations of the visualized data and dense QA annotations without any manual annotation process. Our stage-by-stage pipeline makes it possible to create diverse topic and appearance figures efficiently while minimizing code errors. Our SBSFigures demonstrate a strong pre-training effect, making it possible to achieve efficient training with a limited amount of real-world chart data starting from our pre-trained weights.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Carbune, V.; Mansoor, H.; Liu, F.; Aralikatte, R.; Baechler, G.; Chen, J.; and Sharma, A. 2024. Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs. arXiv:2403.12596
arXiv 2024
-
[2]
Chaudhry, R.; Shekhar, S.; Gupta, U.; Maneriker, P.; Bansal, P.; and Joshi, A. 2020. LEAF-QA: Locate, Encode & Attend for Figure Question Answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
work page 2020
-
[3]
Cho, J.; Lei, J.; Tan, H.; and Bansal, M. 2021. Unifying Vision-and-Language Tasks via Text Generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 1931--1942. PMLR
work page 2021
-
[4]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR
2021
-
[5]
Han, Y.; Zhang, C.; Chen, X.; Yang, X.; Wang, Z.; Yu, G.; Fu, B.; and Zhang, H. 2023. ChartLlama: A Multimodal LLM for Chart Understanding and Generation. arXiv:2311.16483
arXiv 2023
-
[6]
Hoque, E.; Kavehzadeh, P.; and Masry, A. 2022. Chart question answering: State of the art and future directions. In Computer Graphics Forum, volume 41, 555--572. Wiley Online Library
work page 2022
-
[7]
Kafle, K.; Price, B.; Cohen, S.; and Kanan, C. 2018. DVQA: Understanding Data Visualizations via Question Answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5648--5656
work page 2018
-
[8]
E.; Michalski, V.; Atkinson, A.; Kadar, A.; Trischler, A.; and Bengio, Y
Kahou, S. E.; Michalski, V.; Atkinson, A.; Kadar, A.; Trischler, A.; and Bengio, Y. 2018. FigureQA: An Annotated Figure Dataset for Visual Reasoning. arXiv:1710.07300
arXiv 2018
Show all 24 references
-
[9]
Kim, G.; Hong, T.; Yim, M.; Nam, J.; Park, J.; Yim, J.; Hwang, W.; Yun, S.; Han, D.; and Park, S. 2022. OCR-Free Document Understanding Transformer. In European Conference on Computer Vision (ECCV)
2022
-
[10]
Lee, K.; Joshi, M.; Turc, I.; Hu, H.; Liu, F.; Eisenschlos, J.; Khandelwal, U.; Shaw, P.; Chang, M.-W.; and Toutanova, K. 2023. Pix2Struct: screenshot parsing as pretraining for visual language understanding. In Proceedings of International Conference on Machine Learning (ICML)
2023
-
[11]
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020. BART : Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the Association for Computatio...
2020
-
[12]
Li, Z.; Jasani, B.; Tang, P.; and Ghadar, S. 2024. Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13613--13623
2024
-
[13]
Liu, F.; Eisenschlos, J.; Piccinno, F.; Krichene, S.; Pang, C.; Lee, K.; Joshi, M.; Chen, W.; Collier, N.; and Altun, Y. 2023 a . D e P lot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics (ACL), 103...
2023
-
[14]
Liu, F.; Piccinno, F.; Krichene, S.; Pang, C.; Lee, K.; Joshi, M.; Altun, Y.; Collier, N.; and Eisenschlos, J. 2023 b . M at C ha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering. In Proceedings of the 61st Annual Meeting of the Association for ...
2023
-
[15]
Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
2021
-
[16]
L.; Hoque, E.; and Joty, S
Masry, A.; Kavehzadeh, P.; Do, X. L.; Hoque, E.; and Joty, S. 2023. U ni C hart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 14662--14684. Singapor...
2023
-
[17]
Q.; Joty, S.; and Hoque, E
Masry, A.; Long, D.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics (ACL), 2263--2279. Dublin, Ireland
2022
-
[18]
M.; and Kumar, P
Methani, N.; Ganguly, P.; Khapra, M. M.; and Kumar, P. 2020. PlotQA: Reasoning over Scientific Plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV)
2020
-
[19]
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140): 1--67
2020
-
[20]
K.; and Farhadi, A
Siegel, N.; Horvitz, Z.; Levin, R.; Divvala, S. K.; and Farhadi, A. 2016. FigureSeer: Parsing Result-Figures in Research Papers. In European Conference on Computer Vision
2016
-
[21]
Singh, H.; and Shekhar, S. 2020. STL-CQA : Structure-based Transformers with Localization and Encoding for Chart Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3275--3284. Online: Association for Computati...
2020
-
[22]
S.; Love, J.; Tafti, P.; Hussenot, L.; Sessa, P
Team, G.; Mesnard, T.; Hardin, C.; Dadashi, R.; Bhupatiraju, S.; Pathak, S.; Sifre, L.; Rivière, M.; Kale, M. S.; Love, J.; Tafti, P.; Hussenot, L.; Sessa, P. G.; Chowdhery, A.; Roberts, A.; Barua, A.; Botev, A.; Castro-Ros, A.; Slone, A.; Héliou, A.; Tacchetti, A.; Bulanova, ...
2024 arXiv
-
[23]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[24]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.