{"id":"54402737-7ece-4ee8-a5f9-4b65a5fbad1f","arxiv_id":"2506.21319","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new chart dataset with simplified vector encodings and chain-of-thought answers improves fine-tuned MLLM performance on data-centric chart QA, at least for models with strong spatial perception.","lead":"This paper introduces SimVecVis, a dataset of nearly 3,000 synthetic charts paired with a simplified vector format and chain-of-thought question-answer annotations, and shows that fine-tuning multimodal language models on it improves their ability to answer data questions about charts. It matters because chart understanding is a weak spot for current multimodal models, and the paper offers a concrete training resource plus evidence about what helps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gain in Table 2 rests on an undocumented evaluation protocol and an impossible accuracy entry; the 53.84%-vs-11.92% result may reflect in-distribution template learning rather than generalizable chart understanding.","rationale":"The reader's verdict is CONDITIONAL, and this stress-test pass identifies a concern that sharpens the same condition rather than moving the verdict. The reader's weakest assumption focused on the CoT annotations being built from ground-truth axis metadata and the question of transfer to real charts; the concern raised here is closely related but more directly about the evaluation protocol: without a documented held-out split and without any external validation, the headline accuracy gain could be explained by the model learning template-specific shortcuts in synthetic charts. The impossible <20% entry in Table 2 is additional, concrete evidence that the quantitative support for the central claim is not currently reliable. This concern is load-bearing because the paper's central claim is an empirical one, and its strongest quantitative statement is the 53.84% versus 11.92% comparison; if the evaluation overlaps training or the numbers are mismeasured, that statement does not support the claimed improvement. The proposed test would settle the issue by checking the released data for split integrity and by measuring held-out and external performance. The paper still contributes a dataset and a reconstruction demonstration, and the code is promised, so the appropriate outcome remains conditional acceptance pending the requested evaluation details and table correction.","tokens_in":9877,"tokens_out":6196,"duration_ms":73283,"concrete_test":"Inspect the released repository to verify whether the 260 evaluation instances in Tables 2 and 3 are disjoint from the 2,999 training charts; if no explicit split is defined, re-run the MiniCPM (SimVec + QA w/CoT) fine-tuning on a randomly held-out 260-chart test set and also evaluate the same checkpoint on the ChartQA test split. If held-out SimVecVis accuracy drops materially below 53.84% or ChartQA accuracy shows no improvement over the zero-shot baseline, the reported gain is an in-distribution artifact rather than evidence of generalizable chart understanding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends entirely on Tables 2 and 3, yet the manuscript never states whether the 260 evaluation instances are held out from the 2,999 training charts, how they were selected, or whether any external benchmark was used. Because SimVecVis is generated from a small set of predefined templates with randomized colors and axis ranges, a model trained on a large fraction of the same template distribution can achieve high accuracy by exploiting layout regularities (e.g., fixed canvas size, axis pixel span) without learning to decode arbitrary charts. The CoT annotations in Section 4.1 amplify this risk: the reasoning steps quote exact ground-truth pixel coordinates, such as \"the Y-axis maps from 50 pixels to 450 pixels,\" which are derived from template metadata rather than from visual perception, so the model can mimic the CoT without acquiring a general pixel-to-data mapping. The reliability of the quantitative evidence is further undermined by an internally inconsistent row in Table 2: MiniCPM (QA w/o CoT) is reported as 26.92% (<5%), 41.92% (<10%), and then 25.38% (<20%); with cumulative thresholds, the <20% accuracy cannot be lower than the <10% accuracy. No error bars or repeated runs are reported. Until the train/test split and evaluation protocol are documented and the table corrected, the 53.84%-versus-11.92% comparison does not establish a generalizable improvement in visualization understanding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SimVec, a simplified vector representation for chart elements, and SimVecVis, a dataset of 2,999 generated bar, line, and area charts, each with a bitmap image, a SimVec representation, and data-centric QA pairs with chain-of-thought annotations. The authors fine-tune MiniCPM and Qwen-VL under several ablations and report large gains, e.g., MiniCPM with SimVec + QA w/ CoT reaches 53.84% within-5% accuracy versus 11.92% zero-shot (Table 2). They also report a reconstruction experiment in which a fine-tuned model generates SimVec from chart images (Section 5.3). The paper concludes that SimVecVis and CoT supervision substantially improve data-centric visualization understanding.","tokens_in":10140,"tokens_out":4528,"duration_ms":49040,"significance":"If validated, the dataset is a useful resource: it pairs raster charts with an intermediate structured representation and CoT reasoning, and the compact SimVec format is a sensible target for reconstruction. The paper also reports a concrete training recipe with modest compute. However, the current evidence does not yet establish generalization, because the evaluation protocol is undocumented and one table row is internally inconsistent; the result is plausible but under-evidenced. The central contribution (dataset plus representation) is worth pursuing, and most concerns are addressable with additional documentation and experiments.","major_comments":[{"comment":"The MiniCPM (QA w/o CoT) row reports 26.92% for <5%, 41.92% for <10%, and 25.38% for <20%; since the <20% criterion is a superset of the <10% criterion, this ordering is impossible. This entry must be corrected and the whole table rechecked, because the headline comparison (SimVec + QA w/ CoT 53.84% vs. zero-shot 11.92%) depends on this table.","section":"Section 5, Table 2"},{"comment":"The manuscript never defines the train/test split, the test-set size, or the selection procedure for evaluation instances. Several percentages are consistent with n=260, but this is not stated. Because all charts are generated from a small set of predefined templates with randomized colors and axis ranges (Section 4.1), high in-distribution accuracy could arise from template regularity rather than chart understanding; without a held-out split or an external benchmark, the reported gains do not establish generalization.","section":"Section 5, Tables 2 and 3"},{"comment":"The CoT annotations are synthesized from ground-truth axis metadata, e.g., 'the Y-axis ... maps from 50 pixels to 450 pixels, corresponding to ... 0% to 100%'. This assumes a known, linear pixel-to-data mapping during training. Since real charts must be decoded from noisy images, the paper should test whether the learned reasoning transfers, for example by evaluating on an external chart-QA benchmark or on charts whose pixel-to-axis mapping must be visually estimated.","section":"Section 4.1"},{"comment":"The reconstruction experiment uses MiniCPM (SimVec + QA w/ CoT), a model fine-tuned on SimVecVis, to generate SimVec from chart images. Interpreting high reconstruction quality as evidence of SimVec's expressiveness is partly circular because the model was trained to emit this exact format; report held-out or few-shot performance with a model that has not seen SimVec training data, or reframe H3 as a test of learnability rather than expressiveness.","section":"Section 5.3"}],"minor_comments":[{"comment":"The note that 'all coordinates and size are described using a uniform value where the size is set to 1000' is ambiguous; please state explicitly that all coordinates are relative to a normalized 1000-unit canvas.","section":"Section 3, Table 1"},{"comment":"The relationship among 2,999 visualizations, 2,999 identification tasks, 5,642 extreme-value tasks, and the number of QA pairs used for training and evaluation should be clarified.","section":"Section 4"},{"comment":"No hyperparameters, number of epochs, learning rate, or data split are reported; one paragraph on the training setup would improve reproducibility.","section":"Section 5.2"},{"comment":"The term 'mocked historical visualizations' is informal; 'simulated' or 'synthesized' would be clearer.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to target a visualization or HCI venue. Given that the contribution is a dataset, I would ask the editor to require the authors to provide the exact data split and corrected tables during revision; the current evaluation cannot be checked without them. The internal inconsistency in Table 2 and the missing evaluation protocol are substantive but fixable issues, so I do not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the resource is real: SimVecVis pairs 2,999 synthetic bar/line/area charts with a simplified vector format (SimVec), bitmap images, and data-centric QA pairs with chain-of-thought annotations. That combination is new and plausibly useful for training chart-understanding models. Second, the main quantitative claim, that fine-tuning MiniCPM on SimVec+CoT jumps accuracy from 11.92% to 53.84% within 5% of ground truth, is currently not something you can rely on. There is no stated train/test split, the test set size is never given, and no external benchmark is used. The reported gain may largely reflect in-distribution template learning rather than generalizable chart understanding, because the charts are generated from a small set of templates with randomized colors and axis ranges.\n\nWhat the paper does well: SimVec is a sensible simplification of SVG, cutting token count dramatically while retaining the geometry and color needed for reconstruction. The CoT annotations are grounded in explicit axis-pixel mappings, which gives the model a clear reasoning trace for value extraction. The reconstruction experiment, while secondary, shows the fine-tuned model can emit SimVec structure with decent fidelity; those numbers are at least internally coherent.\n\nThe soft spots are real but not all equal. The most serious is the evaluation transparency problem: no split, no seeds, no error bars, and no test on a held-out distribution or existing benchmark such as ChartQA. That makes the 53.84% figure hard to interpret. There is also a clear error in Table 2: MiniCPM (QA w/o CoT) reports 41.92% within 10% but only 25.38% within 20%, which is impossible for cumulative thresholds. That kind of mistake makes you worry the other numbers were not carefully checked. A separate concern is that the CoT traces quote ground-truth axis metadata (e.g., \"the Y-axis maps from 50 pixels to 450 pixels\"), so the model may learn to mimic a template-specific mapping rather than develop a general pixel-to-data decoder. Transfer to noisy real-world charts is untested. The H3 reconstruction experiment is mildly circular because the model was trained on SimVec, but that is a minor issue for a secondary claim.\n\nWho is this for? Researchers working on chart QA or fine-tuning MLLMs for structured output from images will find the dataset worth examining, but they should treat the current accuracy numbers as provisional. The paper deserves a serious referee, and I would send it out, but it needs major revision: document the evaluation protocol, correct the table, add variance estimates, and show at least one external benchmark. If the authors address those, the dataset is a legitimate contribution.","headline":"A useful synthetic chart dataset and a plausible fine-tuning recipe, but the headline accuracy numbers are not yet reliable due to an undocumented evaluation protocol and an internally inconsistent table row.","tokens_in":787,"tokens_out":962,"would_cite":false,"duration_ms":32557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on SimVecVis lifts MLLM chart-data accuracy from 12% to 54%.","keywords":["visualization understanding","multimodal large language models","chart question answering","chain-of-thought reasoning","vector representation","fine-tuning","dataset construction","chart reconstruction"],"falsifier":"Run the fine-tuned MiniCPM model from Table 2 on a held-out set of real-world charts with non-linear axes, such as logarithmic scales, or with heavily distorted hand-drawn axes, and measure accuracy within 5% of ground truth; if it collapses toward the 11.92% zero-shot baseline, the learned pixel-to-value reasoning is an artifact of the linear template rather than a general chart-reading skill.","tokens_in":9667,"feed_emoji":"📊","tokens_out":5815,"duration_ms":62523,"temperature":0.7,"pith_summary":"The paper claims that a new training dataset, SimVecVis, can substantially improve how multimodal large language models read data from charts. The dataset pairs 2,999 bitmap chart images with two extra supervision signals: a compact structured 'SimVec' description of the chart's marks and coordinates, and question-answer pairs whose answers are justified by explicit chain-of-thought reasoning anchored to axis scales. The headline result is that fine-tuning MiniCPM on SimVec plus QA-with-CoT raises data-question accuracy within 5% of ground truth from 11.92% zero-shot to 53.84%, outperforming zero-shot GPT-4o's 16.54%. The paper further shows that a model trained with SimVec can reconstruct charts from pixels at high fidelity, and that the recipe only helps models with strong spatial localization. If true, this gives a route to practical chart QA for scanned documents and historical visualizations, where data is not available as metadata.","feed_headline":"New training data lifts chart reading from 12% to 54%","feed_subtitle":"SimVecVis pairs charts with simplified vector code and step-by-step reasoning, and fine-tuning on it transforms data-centric chart QA.","key_machinery":"SimVec is a simplified vector format that flattens a chart into an ordered list of text, rectangle, line, and polygon elements, each with normalized coordinates in a 0–1000 space and HSL colors quantized to a 0–20 range. It is compact, reducing token count by about 90% compared to the SVG version of the same chart, and it is machine-readable enough to serve both as training supervision and as a reconstruction target. The other load-bearing component is the chain-of-thought QA annotation, which anchors each reasoning step to ground-truth axis metadata, for example: the Y-axis maps 50 pixels to 450 pixels corresponding to 0% to 100%, the bar height is 140 pixels, so the value is (140/(450−50))×100 = 35%. Together, these two components train the model to decode visual marks into data values rather than merely matching image patterns.","core_discovery":"The paper's central claim is that fine-tuning a multimodal large language model on SimVecVis—where each chart image is paired with a compact SimVec encoding and chain-of-thought QA traces—substantially improves data-centric chart question answering. The strongest demonstration is with MiniCPM: with SimVec plus QA-with-CoT supervision, 53.84% of predictions fall within 5% of the ground-truth value, compared with 11.92% for the zero-shot model and 16.54% for zero-shot GPT-4o. The same training does not help Qwen-VL, which the authors attribute to its weaker chart-element localization. The paper also shows that a model trained to output SimVec can reconstruct bar, line, and area charts from pixels, with text hit rates near 99% and average element position errors around 1% of image size on the evaluated cases, while noting that errors in early reasoning steps such as axis decoding still propagate to final answers.","pith_inferences":["Beyond the stated results, the linear axis-mapping assumption in the chain-of-thought annotations implies that the gains may not transfer to charts with logarithmic scales, broken axes, or other non-linear pixel-to-value mappings; that is a concrete stress test for the method.","If transfer to uncontrolled charts does hold, model-generated SimVec reconstructions could serve as pseudo-labels for unlabeled chart corpora, turning the explicit vector representation into a self-training signal.","The high reconstruction fidelity suggests SimVec could also function as an intermediate target for chart-to-code or chart-to-data-table pipelines, not just for question answering.","Since the recipe helped only a model with strong localization, pairing SimVec supervision with a localization-focused auxiliary objective might extend the benefit to weaker visual-language models."],"forward_implications":["Fine-tuning a spatially capable MLLM on SimVecVis with SimVec and chain-of-thought supervision yields large accuracy gains over zero-shot models on data-centric chart questions.","Chain-of-thought supervision alone improves accuracy over direct-answer training, and adding SimVec predictions produces a further large jump for models that can localize chart elements.","The benefit is not model-agnostic: a model with weaker chart-element localization showed no improvement from SimVec, so the recipe depends on the base model's spatial perception.","A model trained with SimVec can reconstruct chart structure from pixels with high fidelity, suggesting that SimVec is an expressive intermediate representation for chart understanding.","The main remaining failure mode is error accumulation in multi-step reasoning, such as an incorrect axis reading propagating to the final numerical answer."],"supporting_citations":[{"why":"Supplies MiniCPM, the primary spatially capable MLLM whose zero-shot baseline and fine-tuning are measured.","marker":"[14]"},{"why":"Supplies Qwen-VL, the comparison model that does not benefit from SimVec.","marker":"[2]"},{"why":"Supplies DeepSeek-VL as another zero-shot baseline in the comparison.","marker":"[23]"},{"why":"Defines the low-level analytic tasks, specifically retrieving values and finding extremes, that the dataset's QA pairs target.","marker":"[1]"},{"why":"Introduces chain-of-thought reasoning, the supervision format used in the dataset's QA annotations.","marker":"[31]"},{"why":"Provides the graph-comprehension rationale that axis-scale interpretation precedes mapping visual elements to values.","marker":"[28]"},{"why":"Represents prior chart-QA benchmarks that lack explicit reasoning traces, the gap SimVecVis fills.","marker":"[32]"},{"why":"Supplies the normalized Levenshtein distance used to evaluate text reconstruction quality.","marker":"[36]"}],"fun_headline_variants":["Fine-tuning on SimVecVis lifts chart QA accuracy from 12% to 54%","SimVecVis dataset quadruples chart-reading accuracy in MLLMs","Vector-format training lifts MLLM chart QA to 54% accuracy","From 12% to 54%: SimVecVis fine-tuning fixes chart reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The chain-of-thought annotations are generated from ground-truth axis metadata that assumes a known, linear pixel-to-value mapping for each chart, and the training recipe depends on that mapping being present and learnable; real charts without clean, linear axes may not transfer the learned reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning on SimVecVis lifts chart QA accuracy from 12% to 54%","SimVecVis dataset quadruples chart-reading accuracy in MLLMs","Vector-format training lifts MLLM chart QA to 54% accuracy","From 12% to 54%: SimVecVis fine-tuning fixes chart reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2759,"prompt_tokens":961,"completion_tokens":1798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1712}},"tokens_in":577,"tokens_out":1798,"duration_ms":15310,"temperature":1.0,"reasoning_tokens":1712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:27:04.506072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fine-tuned MiniCPM model from Table 2 on a held-out set of real-world charts with non-linear axes, such as logarithmic scales, or with heavily distorted hand-drawn axes, and measure accuracy within 5% of ground truth; if it collapses toward the 11.92% zero-shot baseline, the learned pixel-to-value reasoning is an artifact of the linear template rather than a general chart-reading skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the graph-comprehension rationale that axis-scale interpretation precedes mapping visual elements to values."},{"cited_title":"mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning","cited_arxiv_id":"2404.01548","evidence_quote":"Represents prior chart-QA benchmarks that lack explicit reasoning traces, the gap SimVecVis fills."},{"cited_title":"Yujian and L","cited_arxiv_id":null,"evidence_quote":"Supplies the normalized Levenshtein distance used to evaluate text reconstruction quality."}],"review_version":1}