{"id":"86e51a6f-5728-465c-bee1-3ed8a1c7abca","arxiv_id":"2501.04675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning DePlot on synthetic financial bar charts improves chart-to-table extraction, and supplying the extracted table to a vision-language model reduces numeric QA error far below image-only prompting.","lead":"This paper fine-tunes the chart-to-table model DePlot on 50,000 synthetic financial bar charts and shows that the fine-tuned model extracts tables much more accurately than the base model. It then shows that giving those tables to a vision-language model alongside the chart image sharply improves numeric question answering, and that smaller models with tables can beat GPT-4o with images only.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic same-generator evaluation makes the headline gain a distribution-fit result; external chart benchmarks are needed before the VQA claim generalizes.","rationale":"The paper's internal comparisons support the mechanism it claims: within the synthetic domain, fine-tuning DEPLOT improves table extraction (Tables II and III), and for all four LLMs the fine-tuned-table configuration beats the image-only configuration on MAPE (Table IV). Those are controlled comparisons and are not invalidated by the external-validity concern. What must be true for the headline to generalize to real financial VQA is that the measured gains are not an artifact of training and test sharing the same generator and label vocabulary. The paper provides no such evidence and explicitly names this as a limitation. The reader's weakest assumption identifies this same load-bearing condition, and the RNSS discrepancy in Table III is a separate reporting flaw that further supports a conditional rather than unconditional accept. I therefore see no reason to change the reader's verdict, but I would emphasize the external benchmark as the decisive next step.","tokens_in":17429,"tokens_out":9041,"duration_ms":92224,"concrete_test":"Run the fine-tuned DEPLOT and the Ft table-augmented QA protocol on the bar-chart subset of ChartQA's test split (human-annotated real charts), reporting RMS F1 and MAPE/RMSE for the same four LLMs. If the fine-tuned RMS F1 drops by more than 20 points relative to the synthetic test set, or the MAPE gap between Image-Only and Ft shrinks by more than half, the reported gains are specific to the synthetic generator distribution rather than general chart-reading.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Training and test bar charts are generated by the same randomized Matplotlib/Seaborn pipeline (Section 5.3) with a fixed vocabulary of financial titles, labels, and categories (Section 5.1), and the 100 QA charts are also selected from this same synthetic distribution with QA pairs generated from the ground-truth tables. Consequently, the fine-tuned DEPLOT sees test images whose fonts, colors, layout, label set, and value ranges closely match training; the large RMS F1 increase (50.93% to 91.06%) and QA MAPE reduction (32.63% to 2.98% for Qwen2-VL-7B) may substantially reflect learning the generator's distribution rather than general chart-reading ability. The paper's own limitations paragraph concedes that the model has not been evaluated on more complex visualizations and that domain-specific fine-tuning may limit generalization to unfamiliar chart types. Because the central claim is about financial VQA broadly, this external-validity gap is load-bearing: without out-of-distribution evaluation, the claim is only established for same-distribution synthetic charts. Secondary issues, including the base RNSS inconsistency (89.67% in Section 6.2 vs. 86.67% in Table III), single-run evaluation, and absence of released artifacts, reinforce the need for a conditional verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning DePlot on 50,000 synthetic financial bar charts (simple, stacked, and grouped) and using the resulting tables as intermediate structured representations for question answering with vision-language models and LLMs. It reports that fine-tuned DePlot improves RNSS from 89.67% to 98.07% and RMS F1 from 50.93% to 91.06% on a held-out set from the same generator, and that providing the fine-tuned table to GPT-4o, Qwen2-VL-2B/7B, and LLaMa-11B reduces MAPE and RMSE relative to image-only prompting. An appendix provides qualitative examples comparing base and fine-tuned tables, along with zero-shot VLM inference on three charts.","tokens_in":17667,"tokens_out":5040,"duration_ms":43014,"significance":"The central idea—domain-adaptive fine-tuning of a plot-to-table module to improve downstream numerical reasoning—is sensible, and the internal experiments are self-consistent, providing a clear proof-of-concept on the synthetic distribution. The paper describes the dataset generation and metrics in enough detail to be reproduced in principle, and it explicitly defines the RNSS and RMS scoring rules. However, the central claim is about financial VQA broadly, and the evidence is entirely from a single synthetic generator with a fixed vocabulary and layout distribution. The paper's own limitations paragraph concedes the absence of evaluation on complex or out-of-distribution charts. Consequently, the headline numbers should be read as distribution-specific results, not as established generalization. The approach merits publication only after external validation on a public chart benchmark or real financial documents.","major_comments":[{"comment":"The training, test, and QA charts are all produced by the same Matplotlib/Seaborn generator with a fixed financial vocabulary (Section 5.1), and the QA pairs are generated from the same ground-truth tables. Because the fine-tuned DePlot is tested on images that share fonts, colors, label sets, and value ranges with training, the large improvements (RMS F1 50.93% to 91.06%; Qwen-7b MAPE 32.63% to 2.98%) may substantially reflect fitting the generator's distribution rather than general chart-reading ability. The limitations paragraph in Section 7 explicitly acknowledges that the model has not been evaluated on more complex visualizations and that domain-specific fine-tuning may limit generalization. To support the paper's general claim, an out-of-distribution evaluation on a public chart benchmark (e.g., ChartQA, ChartOCR, or a collection of real financial charts) is required; without it, the external-validity gap is load-bearing.","section":"§5.3, §8, Table IV"},{"comment":"The base DePlot average RNSS is reported as 89.67% in the text of Section 6.2, but Table III lists it as 86.67%. This discrepancy is not explained, and it directly affects the reported RNSS improvement of \"approximately 8.4 percentage points\" in Section 8.1. The authors should correct the inconsistent number and recompute the gain consistently, or justify why the two values differ.","section":"§6.2 vs. Table III"},{"comment":"The QA results are point estimates from a single run over 100 charts, with no confidence intervals, error bars, or significance tests. Moreover, the base-table configuration is not consistently better than image-only: for GPT-4o, RMSE increases from 60.51 to 65.89, and for Qwen-2b, from 80.84 to 90.52. The statement that \"transitioning from the image-only configuration to the fine-tuned table configuration consistently and significantly reduces both MAPE and RMSE\" is supported for the fine-tuned table, but the base-table comparison should be reported with uncertainty and the negative cases should be discussed rather than swept into an average.","section":"Table IV and §9.2"},{"comment":"The fine-tuning section specifies 10 epochs, AdamW, and an H100 GPU, but omits learning rate, batch size, gradient accumulation, warmup, the exact DePlot checkpoint used for initialization, and the input resolution or patch configuration (beyond the 1,024-patch limit). Without these details—and without code or dataset release—the fine-tuning results are difficult to reproduce independently. The authors should provide the full hyperparameter configuration and, ideally, release the dataset and trained checkpoints.","section":"§7.3 and §7.2"}],"minor_comments":[{"comment":"The conclusion is numbered \"7\" while Section 7 is \"Fine-Tuning DEPLOT on the Custom Bar Chart Dataset\"; subsequent sections continue to 9, so the numbering is inconsistent and should be renumbered.","section":"Section numbering"},{"comment":"The sentence \"the fine-tuned model achieves its highest RNSS score of 98.07% and an RMS F1 score of 91.51% at epoch 6 and epoch 2, respectively\" matches Table II, but Table III compares the fine-tuned model at epoch 6 for both metrics. Clarify whether the comparison model uses epoch-6 weights for RMS or the epoch-2 checkpoint, and why.","section":"§8.1 and Table II"},{"comment":"The matching matrix X in the RNSS and RMS equations is not fully specified; please state that it is obtained via a minimal-cost assignment, and give the value of the threshold parameter τ used for the Levenshtein-distance partial credit in RMS.","section":"§6.1 equations"},{"comment":"Reference [14] appears to have incomplete metadata (only a date is given); please provide a full citation. Also, reference [19] is repeated as reference [16] in spirit; check for duplicate or missing entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an industry technical report with a promising but narrowly validated idea. The main concern is the discrepancy between the broad claim ('financial VQA') and the evidence that is confined to one synthetic generator. I recommend requesting an out-of-distribution evaluation and artifact release before resubmission. There are also several internal inconsistencies (RNSS values, section numbering, caption errors) that suggest a careful editorial pass is needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper shows a clean internal result — fine-tuning DePlot on 50k synthetic financial bar charts lifts RNSS/RMS on the held-out synthetic test set, and routing the extracted table into an LLM drops MAPE from 32.63% to 2.98% for Qwen2-VL-7B. The table-vs-image comparison in Table IV is genuinely informative: even the base DePlot table helps most models, so the effect of structured intermediates isn't an artifact of fine-tuning. That part holds up.\n\nWhat's new is modest: the domain-specific synthetic dataset, the fine-tuned checkpoint, and the three-way QA study. The dataset generation is described in enough detail to replicate, but nothing is released — no code, no data, no model weights. That's a big strike for a paper whose main product is a dataset and a fine-tuned model.\n\nThe soft spots in proportion: the evaluation is entirely within the distribution of the same synthetic generator. Training and test charts share fonts, label vocabularies, color palettes, value ranges, and layout rules. So the large gains are, at least in part, the model learning the generator's distribution. The paper's own limitations paragraph concedes it hasn't been tested on more complex visualizations and that fine-tuning may limit generalization to unfamiliar chart types. That concession is accurate, and it makes the broad 'financial VQA' claim in the title and abstract unsupported as written. Also: the base RNSS number is inconsistent (89.67% in Section 6.2 vs 86.67% in Table III); the experiments appear to be single-run with no variance information; and the 100-chart QA set is small, with questions generated from the same ground-truth tables, so the setup is maximally favorable to the table route.\n\nNone of this kills the narrow claim. If you read it as 'on this class of synthetic charts, table intermediates help,' the evidence is coherent. It just doesn't license the generalization to real financial documents.\n\nWho this is for: someone working on chart-to-table models or LLM reasoning over structured data, who wants a quick sanity check that the DePlot recipe transfers to a narrow domain. It deserves a serious referee, not a desk reject, because the internal comparison is clean and the synthesis pipeline is reproducible in principle. But I'd send it back with a firm request to release artifacts and add at least one out-of-distribution test — e.g., a small set of real financial charts or a public benchmark like ChartQA — before it can be considered a general result.","headline":"Clean internal result on same-distribution synthetic charts, but the financial-VQA generalization claim needs out-of-distribution evidence and released artifacts before it persuades.","tokens_in":18225,"tokens_out":2988,"would_cite":false,"duration_ms":29554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured tables extracted from chart images, rather than the images themselves, sharply improve vision-language model reasoning on financial bar-chart questions, and fine-tuning DePlot on synthetic bar charts is enough to realize the…","keywords":["chart understanding","visual question answering","DePlot","fine-tuning","bar charts","structured representations","financial documents","vision language models"],"falsifier":"Take a set of real financial charts from filings, earnings reports, or financial news with manually verified ground-truth values, run the fine-tuned table extractor and the same table-plus-image LLM setup on them, and compare MAPE and RMS F1 against the synthetic-test numbers; if the advantage over image-only queries shrinks or disappears, the claim that the gains carry outside the synthetic distribution is refuted.","tokens_in":17219,"feed_emoji":"📊","tokens_out":7421,"duration_ms":66508,"temperature":0.7,"pith_summary":"The paper argues that feeding a vision-language model a structured table extracted from a chart image, rather than the raw image, produces far more accurate numerical answers to financial chart questions. To get good tables, the authors fine-tune DePlot, a plot-to-table translation model, on 50,000 synthetic bar charts spanning simple, stacked, and grouped layouts. On a held-out test set the fine-tuned table extractor raises categorical mapping F1 from about 51% to 91%, and for one small vision-language model it cuts mean absolute percentage error on chart-based questions from 32.6% to 3.0%. The intended significance is that a relatively cheap intermediate representation, a linearized table, can make even small models competitive with much larger ones that try to read charts directly.","feed_headline":"Fine-tuned table extraction cuts chart-question error by 90%","feed_subtitle":"Turning charts into accurate tables lets small models out-answer large image-only systems on financial Q&A.","key_machinery":"The load-bearing object is DePlot, a modality-conversion module that turns a chart image into a linearized table string, using '|' to separate columns and '<0x0A>' to mark row ends. The authors fine-tune DePlot on a custom 50,000-image dataset of financial bar charts, then feed the resulting tables to downstream LLMs alongside the image. The evaluation uses Relative Mapping Similarity (RMS), which measures how well predicted row-and-column-to-value mappings align with ground truth, and Relative Number Set Similarity (RNSS), which checks numerical sets; these metrics make the improvement in table fidelity legible and connect the table-extraction step to the downstream QA gains.","core_discovery":"The central claim is that accurate intermediate structured representations are the bottleneck for chart question answering: once a chart is correctly rendered as a table, LLM reasoning over that table is dramatically better than over the image, and fine-tuning the table extractor on domain-specific synthetic charts is enough to make this work. The authors demonstrate that base DePlot produces tables with poor categorical alignment, reaching an RMS F1 of 50.93%, while the fine-tuned model reaches 91.06% on the same 1,000-image test set. Downstream, adding the fine-tuned table lowers Qwen2-VL-7B's MAPE from 32.63% to 2.98% and its RMSE from 90.99% to 8.10%, and it lets smaller models such as Qwen2-VL-7B and Llama-3.2-11B outperform GPT-4o when GPT-4o is given only the raw image. The paper presents this as evidence that the quality of the table, not the scale of the reasoning model, drives numerical question-answering performance.","pith_inferences":["If table quality is truly the bottleneck, then any reliable chart-to-table system, not just DePlot, should reproduce the same downstream gains, making the claim testable in a model-agnostic way.","The large improvement likely reflects the LLM's difficulty in reading exact numeric values from images; once the table removes perceptual noise, residual errors should track table-extraction accuracy rather than image understanding.","Because training and test charts come from the same randomized generator, the measured gains probably overestimate performance on real-world charts with noise, OCR errors, and complex encodings; testing on real financial report pages would clarify this.","The same table-first recipe could be applied to other structured visual inputs, such as line charts, scatter plots, or diagrams, wherever a textual derendering of the visual exists."],"forward_implications":["Fine-tuning a chart-to-table model on domain-specific synthetic charts can produce large gains in table fidelity, with RMS F1 rising by roughly 40 percentage points, without changing the downstream LLM.","Providing a high-quality table alongside the image reduces MAPE for all tested LLMs, and the relative reduction is largest for the smaller models.","With structured tables, smaller vision-language models can beat a much larger model, GPT-4o, that reads the image directly.","Because a correct table is the key input, the quality of the intermediate representation, rather than model scale, dominates numerical QA performance in this setting.","The same dataset-generation recipe, with domain-specific labels and values, is claimed to extend the approach to other domains such as healthcare or marketing charts."],"supporting_citations":[{"why":"Supplies the base plot-to-table conversion module, DePlot, that the paper fine-tunes and then uses to generate intermediate tables.","marker":"[1]"},{"why":"Establishes the few-shot reasoning capability of language models that the downstream LLM step relies on when given the structured table.","marker":"[2]"},{"why":"Provides the ChartQA benchmark and the RNSS metric used to evaluate numerical interpretation accuracy of the extracted tables.","marker":"[3]"},{"why":"Offers a hybrid chart-extraction baseline that motivates the need for learned structured table extraction and is cited for the RNSS metric.","marker":"[4]"},{"why":"Supplies the Qwen2-VL-7B vision-language model that shows the largest relative improvement when given the fine-tuned table.","marker":"[8]"},{"why":"Provides the chart-generation library used to create the synthetic bar-chart dataset that is central to fine-tuning and evaluation.","marker":"[11]"}],"fun_headline_variants":["Fine-tuned chart-to-table model boosts financial QA accuracy","Structured tables beat raw images for chart question answering","Small models with DePlot tables outperform GPT-4o on chart QA","Table accuracy drives LLM chart reasoning, not model size","Fine-tuning DePlot on 50K charts slashes financial QA error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the synthetic bar charts used for training and testing behave like real-world financial charts; if real charts differ in layout, noise, or encoding, the measured accuracy gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned chart-to-table model boosts financial QA accuracy","Structured tables beat raw images for chart question answering","Small models with DePlot tables outperform GPT-4o on chart QA","Table accuracy drives LLM chart reasoning, not model size","Fine-tuning DePlot on 50K charts slashes financial QA error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1289,"prompt_tokens":937,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":266}},"tokens_in":553,"tokens_out":352,"duration_ms":3827,"temperature":1.0,"reasoning_tokens":266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:26:36.225755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real financial charts from filings, earnings reports, or financial news with manually verified ground-truth values, run the fine-tuned table extractor and the same table-plus-image LLM setup on them, and compare MAPE and RMS F1 against the synthetic-test numbers; if the advantage over image-only queries shrinks or disappears, the claim that the gains carry outside the synthetic distribution is refuted.","supporting_citations":[{"cited_title":"ChartQA: A benchmark for question answering about charts with visual and logical reasoning,","cited_arxiv_id":null,"evidence_quote":"Provides the ChartQA benchmark and the RNSS metric used to evaluate numerical interpretation accuracy of the extracted tables."},{"cited_title":"Chartocr: Data extraction from charts images via a deep hybrid framework,","cited_arxiv_id":null,"evidence_quote":"Offers a hybrid chart-extraction baseline that motivates the need for learned structured table extraction and is cited for the RNSS metric."}],"review_version":1}