{"id":"f1e3b7fb-3720-4601-8d02-641cb88914eb","arxiv_id":"2412.17606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A synthetic figure generation pipeline and a one-million-image dataset that improve pre-training for figure question answering.","lead":"The authors build a stage-by-stage pipeline that generates one million synthetic chart images with question-answer pairs, and show that pre-training on this data improves chart question-answering models. A generalist might care because synthetic data can reduce the need for expensive human annotations when teaching computers to read charts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's QA labels are asserted accurate in §3.3 but never validated against the rendered figures; errors in 4.2M GPT-generated pairs could silently undermine the pre-training benefit.","rationale":"The paper has real strengths: the stage-by-stage pipeline is concrete, the code and dataset are promised to be released, and the pre-training gains are consistent across two architectures and several ablations. The investigation of appearance, task type, QA source, prompt, and image count is useful and goes beyond many synthetic-data papers. I do not see a reason to reject the paper outright. The load-bearing concern is the accuracy of the 4.2M LLM-generated QA pairs, which the authors claim as a design goal but never measure directly. A downstream accuracy gain on ChartQA is indirect evidence: fine-tuning on human-annotated labels can overwrite incorrect pre-training associations, and Table 5 compares QA generators only by end-task performance, not by answer correctness. If a meaningful fraction of answers are wrong or refer to attributes not visible in the randomized figure, the central pre-training signal is corrupted. An independent sample-based validation of QA labels against the rendered figures would settle this. Since the reader already returned a conditional verdict, my read does not move that verdict; the condition should explicitly include QA-label validation before the dataset is fully endorsed.","tokens_in":13311,"tokens_out":6525,"duration_ms":65270,"concrete_test":"Sample 500 QA pairs stratified by chart type and question category from the released dataset. Re-render each figure from its JSON with the released code, then have two annotators independently verify each answer against both the JSON and the rendered image, checking value, arithmetic, label, color, and whether the queried attribute is actually visible. Compute per-category error rates and inter-annotator agreement. If the overall error rate exceeds about 5%, or color/title questions show systematic mismatches, retrain the Donut pre-training on a corrected or validated subset and compare ChartQA human split; if the roughly +8 point gain in Table 1 shrinks materially, the concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 asserts that 'LLM can make QA pairs without the OCR process, making the QA numbers and calculations reliable,' but no accuracy measurement is reported for any of the 4.2M QA pairs. Reliability does not follow from skipping OCR: GPT-3.5-turbo can misread the JSON, miscalculate aggregates, or ask about attributes that are randomized out of the rendered image. Section 3.2 randomizes title position, legend presence, number visibility, and possibly colors, yet the QA pairs are generated from the JSON alone, not from the rendered figure. If a substantial fraction of answers are wrong, pre-training on SBS Figures teaches incorrect reasoning and color/label associations rather than chart comprehension. The downstream gains on ChartQA do not rule this out: fine-tuning on real labels can correct some errors, and Table 5 only compares QA sources by downstream accuracy, not by label correctness. Since the stated design criterion (3) requires QA pairs to 'accurately reflect the information presented in the figures,' the unmeasured label error rate is the load-bearing risk for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SBS Figures, a fully synthetic dataset of 1M chart images generated in stages: LLM-produced data topics and JSON data, deterministic Python rendering with randomized appearance, and LLM-generated QA pairs from the JSON. The authors claim that pre-training a figure QA model on this dataset improves performance on real-world ChartQA, and they support this with experiments using Donut and Pix2Struct, ablations over five factors, and further fine-tuning on PlotQA and FigureQA. The pipeline is claimed to be error-free, diverse, and annotation-free, with the dataset and code to be released.","tokens_in":13523,"tokens_out":7644,"duration_ms":63245,"significance":"The paper makes a useful practical contribution: a controllable, large-scale synthetic figure QA resource with a released pipeline, plus a set of ablation studies (appearance, pre-training task, QA source, prompt format, dataset size) that are informative for the community. The core empirical result—that SBS Figures pre-training improves ChartQA performance over a Donut or Pix2Struct baseline—is supported by the reported numbers and is the main strength. However, the significance is conditional on validating the QA label quality and on cleaning up several experimental inconsistencies; if those are addressed, the dataset and pipeline could become a solid pre-training resource for figure understanding.","major_comments":[{"comment":"The claim that QA numbers and calculations are reliable because the LLM does not use OCR is not supported by any measurement. Design criterion (3) in §3 requires QA pairs to 'accurately reflect the information presented in the figures,' but no human verification or automatic consistency check is reported for any of the 4.2M QA pairs. Pre-training on incorrect labels could teach spurious associations, so this is a load-bearing gap; please add a validation study (e.g., a random sample of QA pairs checked against the rendered figures) and report the error rate, or otherwise show that label noise is negligible.","section":"§3.3"},{"comment":"The color randomization in §3.2 appears inconsistent with QA generation in §3.3. §3.2 states that 'colors' are randomized in the figure generation code, while §3.1 places 'corresponding colors' in the JSON that is later used for QA generation. If the renderer overrides the JSON colors, then color-related questions (e.g., 'What does the purple color represent?') will have incorrect answers. Please clarify whether the randomized colors are written back into the JSON before QA generation; if not, correct the pipeline or demonstrate empirically that the overlap between randomized and JSON colors is sufficient for the color-based QA pairs.","section":"§3.2/§3.3"},{"comment":"The comparison with other synthetic pre-training datasets is not fair because PlotQA was pre-trained for only 1 epoch while all other datasets, including SBS Figures, were pre-trained for 3 epochs (stated in §4.2). The smaller number of optimizer steps for PlotQA could explain part of its lower performance, so the claim that 'only SBS Figures demonstrates the improvements' is not fully supported. Please re-run the comparison with matched training steps (or matched number of updates) and report results across multiple seeds with error bars; this is important because the superiority of SBS Figures over other synthetic datasets is a central claim.","section":"§4.2, Table 1"},{"comment":"The abstract and introduction claim that pre-training on SBS Figures enables 'efficient training with a limited amount of real-world chart data,' but no experiment in §4 varies the amount of real-world fine-tuning data; all ChartQA fine-tuning experiments use the full ChartQA training set. Table 7 varies the number of synthetic pre-training images, not the amount of real-world data. Please add an experiment that reduces the ChartQA (or another real-world dataset) fine-tuning set size to support the stated claim, or revise the claim to match the evidence.","section":"Abstract/§4"},{"comment":"There are numerical inconsistencies that need to be resolved. The SBS Figures row in Table 1 reports 39.44/82.24/60.84, while the Donut+SBS Figures row in Table 2 reports 39.20/81.20/60.84 for the same setting; the identical averages make it unlikely that these are different experimental conditions, so one of the tables is wrong. Additionally, the main text in §4.1 states a pre-training learning rate of 1e-4 and a fine-tuning learning rate of 5e-5, while Appendix A states the reverse (5e-5 for pre-training and 1e-4 for fine-tuning). These discrepancies make reproduction ambiguous and should be corrected.","section":"Table 1 vs Table 2; Appendix A"}],"minor_comments":[{"comment":"The column headers of Table 8 are ambiguous: 'PlotQA FigureQA Pre-train V1 V2 V1 V2' does not make clear which V1/V2 columns belong to PlotQA and which to FigureQA; please use separate headers and clarify that FigureQA likely has only one validation split.","section":"Table 8"},{"comment":"The caption says 'We randomly select 10 questions from each figure type and manually analyze the topic of the figure,' but the analysis is of figures, not questions; this should read '10 figures from each type.'","section":"Figure 5 caption"},{"comment":"The related work contains a typo: 'LEFQA / LEAFQA++' should be 'LEAF-QA / LEAFQA++'.","section":"§2"},{"comment":"The dataset name is written inconsistently as 'SBS Figures' and 'SBSFigures'; please choose one spelling and use it consistently.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core result—synthetic SBS pre-training improving ChartQA—appears real and the resource itself is valuable. However, the lack of QA validation, the unfair PlotQA comparison, the color mismatch concern, and the overclaim about limited real-world data are all fixable with additional experiments or clarification. The numerical inconsistencies between Table 1 and Table 2 are concerning and must be resolved before the paper is reliable. I see no sign of circularity; the pre-training data is independent of the evaluation benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading: the main result is real, and the main comparison is not as clean as it looks. SBS Figures is a fully synthetic 1M-image chart dataset with 4.2M QA pairs, built by a stage-by-stage pipeline that separates data-topic generation, data-content JSON generation, error-free rendering code, and LLM-based QA generation. The paper shows consistent pre-training gains on ChartQA over scratch (about +8 points on the human split for Donut, +6 for Pix2Struct), and similar gains when fine-tuning on PlotQA and FigureQA. That is a useful, credible result. The pipeline itself is a solid engineering contribution, and the authors release code, data, and prompts.\n\nThe soft spots are two, and they are of different sizes. The smaller one is the baseline comparison in Table 1. PlotQA is pre-trained for 1 epoch while SBS Figures gets 3 epochs; the authors note this is due to QA count differences, but it still makes the comparison unfair. No error bars or multiple seeds are reported, so the 60.84 vs 52.28 gap cannot be taken at face value as a superiority claim over PlotQA. This is fixable with matched budgets and variance reporting.\n\nThe larger one is the QA label validity. Section 3.3 asserts that generating QA from the JSON directly, without OCR, makes the numbers and calculations reliable. That does not follow. GPT-3.5-turbo can miscalculate, misread the JSON, or ask about attributes that the rendering step randomized out of the figure (e.g., a question about a color when the legend is absent). The paper's own criterion (3) requires QA pairs to accurately reflect the figures, yet no accuracy measurement is reported for any of the 4.2M pairs. The ablation in F3 compares QA sources by downstream ChartQA accuracy, not by label correctness, so it does not resolve the concern. I think this is a genuine risk, but not a fatal one: the downstream fine-tuning on real labels can absorb some label errors, and the gains are consistent across models and datasets. Still, for a dataset paper, a label-quality audit on a random sample is essential.\n\nThe paper is otherwise careful. The factor analyses (appearance, task, prompt, dataset size) are useful and the findings are plausible; the Pix2Struct generalization strengthens the contribution. The limitations are understated (only compute constraints), but that is minor.\n\nWho should read this: anyone working on chart understanding, synthetic data for VLMs, or pre-training strategies. I would bring it to a reading group and I would cite it if I were in the area. It deserves a serious referee. My recommendation: send to peer review, and require the authors to (1) validate a random sample of QA pairs against the rendered figures, and (2) rerun the cross-dataset pre-training comparison with matched epochs and multiple seeds.","headline":"Genuine synthetic pre-training gains on ChartQA, but the unvalidated LLM-generated QA labels and an off-budget PlotQA comparison keep this from being a clean win.","tokens_in":13998,"tokens_out":2555,"would_cite":true,"duration_ms":23211,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training figure-QA models on SBS Figures, a one-million-image synthetic chart dataset, improves accuracy on real-world ChartQA and enables efficient fine-tuning with limited real-world data.","keywords":["synthetic data","figure question answering","chart understanding","pre-training","data-to-figure generation","LLM-generated QA","visual reasoning","ChartQA"],"falsifier":"Take a random sample of SBS Figures QA pairs, recompute the answers from the JSON data, and measure the mismatch rate; if a meaningful share of answers are wrong, the pre-training signal is partly noisy and the reported accuracy gains would need to be reassessed after filtering.","tokens_in":13168,"feed_emoji":"📊","tokens_out":7593,"duration_ms":60485,"temperature":0.7,"pith_summary":"The paper introduces SBS Figures, a fully synthetic dataset of one million chart images paired with 4.2M dense question-answer annotations, and a stage-by-stage pipeline that produces it without any manual annotation. It claims that pre-training chart-question-answering models on this synthetic data improves their accuracy on real-world benchmarks such as ChartQA, and that a model can then be trained efficiently with only a limited amount of real-world chart data. The dataset and pipeline are released publicly, and the paper also identifies which design choices matter most for synthetic pre-training, including appearance randomization, LLM-generated QAs, and scale.","feed_headline":"Pre-training on 1M synthetic charts lifts ChartQA accuracy","feed_subtitle":"Synthetic charts with dense QA pairs let models learn without costly manual annotation.","key_machinery":"The central object is the stage-by-stage generation engine. It separates figure creation into three stages: an LLM first writes a topic and a JSON data file containing title, axes, labels, data points and colors; pre-defined Python code, one script per chart type, renders the image with randomized fonts, title placement, legend position, markers, spines and number visibility; a second LLM pass turns the JSON into question-answer pairs without OCR. Because rendering code is fixed and data follows a fixed JSON schema, code errors are largely avoided, and each JSON file can be reused for many appearance variants. The complete JSON annotation is what makes dense QA generation cheap and makes answers traceable to ground-truth data.","core_discovery":"Pre-training a figure QA model on SBS Figures improves performance on real-world chart QA compared with no pre-training and with pre-training on existing synthetic datasets. Using the Donut model, SBS Figures pre-training raises ChartQA average accuracy from 54.42 to 60.84, with the human-annotated split climbing from 31.28 to 39.44; the benefit also appears with Pix2Struct, from 60.92 to 64.52. The pre-training transfers to fine-tuning on PlotQA, FigureQA, and the UniChart reasoning task, and the improvement grows as the number of synthetic images increases from 50k to 1M. The paper also reports that QA-based pre-training outperforms JSON-parsing pre-training and that LLM-generated QAs outperform template-based QAs.","pith_inferences":["Because the paper never audits the correctness of the 4.2M LLM-generated QA pairs, the reported gains may actually understate what a cleaned version of SBS Figures could achieve; filtering mismatched answers is a natural next experiment.","The design choices shown to matter, appearance diversity, LLM QA quality, and dataset scale, suggest a recipe for synthetic pre-training in other structured document domains, such as tables, diagrams, or maps.","If the LLM-generated QAs contain systematic calculation errors, then a hybrid pipeline that combines LLM-generated questions with programmatically computed answers might exceed both the template and pure-LLM conditions reported in F3.","The finding that no special pre-training prompt is needed means future synthetic datasets can be plugged into existing fine-tuning pipelines without redesigning instruction formats."],"forward_implications":["Pre-training with SBS Figures improves ChartQA accuracy on both the human and augmented test splits for Donut and Pix2Struct, so the benefit is not tied to one architecture.","Fine-tuning a model pre-trained on SBS Figures reaches higher accuracy on PlotQA and FigureQA than training from scratch, so the synthetic pre-training transfers across chart QA tasks.","The pre-training benefit also appears when the fine-tuning stage is a figure-specific reasoning model trained on real-world charts, at every training step count tested.","Scaling from 50k to 1M synthetic images steadily improves ChartQA accuracy, so further scaling the dataset is a plausible route to further gains.","QA-form pre-training beats JSON-parsing pre-training, so the reasoning task itself, not just visual exposure, contributes to the transfer."],"supporting_citations":[{"why":"ChartQA is the primary real-world benchmark; the paper pre-trains and fine-tunes on its human and augmented splits.","marker":"Masry et al. 2022"},{"why":"Donut is the base model for most experiments, including the main dataset comparison.","marker":"Kim et al. 2022"},{"why":"Pix2Struct is the second architecture used to show the pre-training effect generalizes.","marker":"Lee et al. 2023"},{"why":"PlotQA serves as a comparison dataset for pre-training and as a fine-tuning target for transfer experiments.","marker":"Methani et al. 2020"},{"why":"UniChart provides hyperparameters and reasoning training used in the explorative study.","marker":"Masry et al. 2023"},{"why":"DePlot's JSON-style data representation motivates the data format used by SBS Figures.","marker":"Liu et al. 2023a"},{"why":"ChartLlama is the concurrent LLM-based figure synthesis approach the paper contrasts with its own stage-by-stage pipeline.","marker":"Han et al. 2023"},{"why":"FigureQA is an earlier template-based synthetic dataset used as a pre-training comparison and fine-tuning target.","marker":"Kahou et al. 2018"}],"fun_headline_variants":["SBSFigures: synthetic chart pre-training boosts ChartQA","Stage-by-stage synthetic figures lift ChartQA by 6+ points","1M synthetic charts pre-train for real-world chart QA","Synthetic figure QA pre-training beats manual annotation cost","SBSFigures: dense QA on synthetic charts without hand labeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the question-answer pairs generated by the LLM from JSON data are accurate enough to train on, yet no human or automatic check of those answers is performed.","fun_headline_variants_meta":{"raw":{"variants":["SBSFigures: synthetic chart pre-training boosts ChartQA","Stage-by-stage synthetic figures lift ChartQA by 6+ points","1M synthetic charts pre-train for real-world chart QA","Synthetic figure QA pre-training beats manual annotation cost","SBSFigures: dense QA on synthetic charts without hand labeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2806,"prompt_tokens":866,"completion_tokens":1940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1856}},"tokens_in":482,"tokens_out":1940,"duration_ms":13740,"temperature":1.0,"reasoning_tokens":1856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:20:31.930772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of SBS Figures QA pairs, recompute the answers from the JSON data, and measure the mismatch rate; if a meaningful share of answers are wrong, the pre-training signal is partly noisy and the reported accuracy gains would need to be reassessed after filtering.","supporting_citations":[{"cited_title":"Q.; Joty, S.; and Hoque, E","cited_arxiv_id":null,"evidence_quote":"ChartQA is the primary real-world benchmark; the paper pre-trains and fine-tunes on its human and augmented splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Donut is the base model for most experiments, including the main dataset comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pix2Struct is the second architecture used to show the pre-training effect generalizes."},{"cited_title":"M.; and Kumar, P","cited_arxiv_id":null,"evidence_quote":"PlotQA serves as a comparison dataset for pre-training and as a fine-tuning target for transfer experiments."},{"cited_title":"L.; Hoque, E.; and Joty, S","cited_arxiv_id":null,"evidence_quote":"UniChart provides hyperparameters and reasoning training used in the explorative study."}],"review_version":1}