REVIEW 4 major objections 4 minor 39 references
SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fine-tuning on SimVecVis lifts MLLM chart-data accuracy from 12% to 54%.
desk verdict A useful synthetic chart dataset and a plausible fine-tuning recipe, but the headline accuracy numbers are not yet reliable due to an undocumented evaluation protocol and an internally inconsistent table row. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
SimVec is a simplified vector format that flattens a chart into an ordered list of text, rectangle, line, and polygon elements, each with normalized coordinates in a 0–1000 space and HSL colors quantized to a 0–20 range. It is compact, reducing token count by about 90% compared to the SVG version of the same chart, and it is machine-readable enough to serve both as training supervision and as a reconstruction target. The other load-bearing component is the chain-of-thought QA annotation, which anchors each reasoning step to ground-truth axis metadata, for example: the Y-axis maps 50 pixels to 450 pixels corresponding to 0% to 100%, the bar height is 140 pixels, so the value is (140/(450−50))×100 = 35%. Together, these two components train the model to decode visual marks into data values rather than merely matching image patterns.
What would settle it
Run the fine-tuned MiniCPM model from Table 2 on a held-out set of real-world charts with non-linear axes, such as logarithmic scales, or with heavily distorted hand-drawn axes, and measure accuracy within 5% of ground truth; if it collapses toward the 11.92% zero-shot baseline, the learned pixel-to-value reasoning is an artifact of the linear template rather than a general chart-reading skill.
Extended reading notes
Core claim
The paper's central claim is that fine-tuning a multimodal large language model on SimVecVis—where each chart image is paired with a compact SimVec encoding and chain-of-thought QA traces—substantially improves data-centric chart question answering. The strongest demonstration is with MiniCPM: with SimVec plus QA-with-CoT supervision, 53.84% of predictions fall within 5% of the ground-truth value, compared with 11.92% for the zero-shot model and 16.54% for zero-shot GPT-4o. The same training does not help Qwen-VL, which the authors attribute to its weaker chart-element localization. The paper also shows that a model trained to output SimVec can reconstruct bar, line, and area charts from pixels, with text hit rates near 99% and average element position errors around 1% of image size on the evaluated cases, while noting that errors in early reasoning steps such as axis decoding still propagate to final answers.
Load-bearing premise
The chain-of-thought annotations are generated from ground-truth axis metadata that assumes a known, linear pixel-to-value mapping for each chart, and the training recipe depends on that mapping being present and learnable; real charts without clean, linear axes may not transfer the learned reasoning.
Editorial extensions
If this is right
- Fine-tuning a spatially capable MLLM on SimVecVis with SimVec and chain-of-thought supervision yields large accuracy gains over zero-shot models on data-centric chart questions.
- Chain-of-thought supervision alone improves accuracy over direct-answer training, and adding SimVec predictions produces a further large jump for models that can localize chart elements.
- The benefit is not model-agnostic: a model with weaker chart-element localization showed no improvement from SimVec, so the recipe depends on the base model's spatial perception.
- A model trained with SimVec can reconstruct chart structure from pixels with high fidelity, suggesting that SimVec is an expressive intermediate representation for chart understanding.
- The main remaining failure mode is error accumulation in multi-step reasoning, such as an incorrect axis reading propagating to the final numerical answer.
Reading between the lines
- Beyond the stated results, the linear axis-mapping assumption in the chain-of-thought annotations implies that the gains may not transfer to charts with logarithmic scales, broken axes, or other non-linear pixel-to-value mappings; that is a concrete stress test for the method.
- If transfer to uncontrolled charts does hold, model-generated SimVec reconstructions could serve as pseudo-labels for unlabeled chart corpora, turning the explicit vector representation into a self-training signal.
- The high reconstruction fidelity suggests SimVec could also function as an intermediate target for chart-to-code or chart-to-data-table pipelines, not just for question answering.
- Since the recipe helped only a model with strong localization, pairing SimVec supervision with a localization-focused auxiliary objective might extend the benefit to weaker visual-language models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SimVec, a simplified vector representation for chart elements, and SimVecVis, a dataset of 2,999 generated bar, line, and area charts, each with a bitmap image, a SimVec representation, and data-centric QA pairs with chain-of-thought annotations. The authors fine-tune MiniCPM and Qwen-VL under several ablations and report large gains, e.g., MiniCPM with SimVec + QA w/ CoT reaches 53.84% within-5% accuracy versus 11.92% zero-shot (Table 2). They also report a reconstruction experiment in which a fine-tuned model generates SimVec from chart images (Section 5.3). The paper concludes that SimVecVis and CoT supervision substantially improve data-centric visualization understanding.
Significance. If validated, the dataset is a useful resource: it pairs raster charts with an intermediate structured representation and CoT reasoning, and the compact SimVec format is a sensible target for reconstruction. The paper also reports a concrete training recipe with modest compute. However, the current evidence does not yet establish generalization, because the evaluation protocol is undocumented and one table row is internally inconsistent; the result is plausible but under-evidenced. The central contribution (dataset plus representation) is worth pursuing, and most concerns are addressable with additional documentation and experiments.
major comments (4)
- [Section 5, Table 2] The MiniCPM (QA w/o CoT) row reports 26.92% for <5%, 41.92% for <10%, and 25.38% for <20%; since the <20% criterion is a superset of the <10% criterion, this ordering is impossible. This entry must be corrected and the whole table rechecked, because the headline comparison (SimVec + QA w/ CoT 53.84% vs. zero-shot 11.92%) depends on this table.
- [Section 5, Tables 2 and 3] The manuscript never defines the train/test split, the test-set size, or the selection procedure for evaluation instances. Several percentages are consistent with n=260, but this is not stated. Because all charts are generated from a small set of predefined templates with randomized colors and axis ranges (Section 4.1), high in-distribution accuracy could arise from template regularity rather than chart understanding; without a held-out split or an external benchmark, the reported gains do not establish generalization.
- [Section 4.1] The CoT annotations are synthesized from ground-truth axis metadata, e.g., 'the Y-axis ... maps from 50 pixels to 450 pixels, corresponding to ... 0% to 100%'. This assumes a known, linear pixel-to-data mapping during training. Since real charts must be decoded from noisy images, the paper should test whether the learned reasoning transfers, for example by evaluating on an external chart-QA benchmark or on charts whose pixel-to-axis mapping must be visually estimated.
- [Section 5.3] The reconstruction experiment uses MiniCPM (SimVec + QA w/ CoT), a model fine-tuned on SimVecVis, to generate SimVec from chart images. Interpreting high reconstruction quality as evidence of SimVec's expressiveness is partly circular because the model was trained to emit this exact format; report held-out or few-shot performance with a model that has not seen SimVec training data, or reframe H3 as a test of learnability rather than expressiveness.
minor comments (4)
- [Section 3, Table 1] The note that 'all coordinates and size are described using a uniform value where the size is set to 1000' is ambiguous; please state explicitly that all coordinates are relative to a normalized 1000-unit canvas.
- [Section 4] The relationship among 2,999 visualizations, 2,999 identification tasks, 5,642 extreme-value tasks, and the number of QA pairs used for training and evaluation should be clarified.
- [Section 5.2] No hyperparameters, number of epochs, learning rate, or data split are reported; one paragraph on the training setup would improve reproducibility.
- [Section 4.2] The term 'mocked historical visualizations' is informal; 'simulated' or 'synthesized' would be clearer.
Circularity Check
Minor circularity: the H3 reconstruction experiment tests a model fine-tuned to emit SimVec, so it partly assumes SimVec's sufficiency; the central QA improvement remains an independent empirical comparison.
-
fitted input called prediction
[Section 5.3 (Hypothesis H3), supported by the training definition in Section 5.2]
"To evaluate hypothesis H3, we assess the reconstruction capabilities using the SimVec format. We use MiniCPM (SimVec+CoT) to take an image as input and generate the corresponding SimVec as output. ... These results validate H3, confirming that the SimVec format effectively supports high-fidelity reconstruction of both textual and graphical elements."
The evaluated model, MiniCPM (SimVec+CoT), was fine-tuned with the additional objective to 'predict the SimVec representation of the chart' (Section 5.2). The reconstruction test then asks that same model to output exactly the quantity it was supervised to produce—the image-to-SimVec mapping. High reconstruction fidelity is therefore primarily a check that the model learned its training target, not an independent demonstration that SimVec is an expressive or sufficient chart encoding. No non-SimVec-trained baseline is reported, and the 100 evaluation images are not stated to be held out from training, so the H3 validation partly assumes what it claims to show.
full rationale
The principal claim—that fine-tuning MLLMs on SimVecVis improves data-centric chart QA—is supported by a direct empirical comparison across model variants (Table 2, e.g., MiniCPM SimVec+QA w/ CoT 53.84% vs. 11.92% zero-shot at <5% tolerance). This comparison is not circular: the QA supervision and the evaluation metric are distinct, and the result is a measured outcome rather than an identity. The only load-bearing reduction I can exhibit is in the secondary H3 reconstruction experiment: the evaluated model was fine-tuned to output SimVec from chart images, and the reconstruction test then measures that same output; calling this a validation of SimVec's expressiveness partly assumes the representation's sufficiency rather than establishing it independently. No uniqueness theorem or load-bearing self-citation is invoked, and the CoT annotations, while derived from ground-truth axis metadata, are training data rather than a circular evaluation input. Concerns about undocumented train/test splits or the inconsistent Table 2 row are correctness/validity issues, not circularity, so they do not raise the circularity score.
Assumptions & free parameters
assumptions (3)
- domain assumption SimVec conversion removes enough SVG detail that no data-relevant information is lost.
- domain assumption The axis metadata used to build CoT traces gives a correct linear pixel-to-data mapping.
- domain assumption Synthetic charts generated from GPT-4o topics and predefined templates are representative of the visualizations for which MLLM understanding should improve.
Cite this review
Pith. "Pith review of SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding." pith.science (2026). https://pith.science/paper/QOVXOC5X
@misc{pith2026250621319,
author = {Pith},
title = {Pith review of: SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/QOVXOC5X}},
note = {Machine review of arXiv:2506.21319}
}
read the original abstract
Current multimodal large language models (MLLMs), while effective in natural image understanding, struggle with visualization understanding due to their inability to decode the data-to-visual mapping and extract structured information. To address these challenges, we propose SimVec, a novel simplified vector format that encodes chart elements such as mark type, position, and size. The effectiveness of SimVec is demonstrated by using MLLMs to reconstruct chart information from SimVec formats. Then, we build a new visualization dataset, SimVecVis, to enhance the performance of MLLMs in visualization understanding, which consists of three key dimensions: bitmap images of charts, their SimVec representations, and corresponding data-centric question-answering (QA) pairs with explanatory chain-of-thought (CoT) descriptions. We finetune state-of-the-art MLLMs (e.g., MiniCPM and Qwen-VL), using SimVecVis with different dataset dimensions. The experimental results show that it leads to substantial performance improvements of MLLMs with good spatial perception capabilities (e.g., MiniCPM) in data-centric QA tasks. Our dataset and source code are available at: https://github.com/VIDA-Lab/SimVecVis.
Figures
Reference graph
Works this paper leans on
-
[1]
R. Amar, J. Eagan, and J. Stasko. Low-level components of analytic activity in information visualization. In Proc. IEEE InfoVis. Symp., pp. 111–117, 2005. doi: 10.1109/INFVIS.2005.1532136
arXiv 2005
-
[2]
J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. 2023. doi: 10. 48550/ARXIV.2308.12966
- [3]
-
[4]
M. A. Borkin, A. A. V o, Z. Bylinskii, P. Isola, S. Sunkavalli, A. Oliva, and H. Pfister. What makes a visualization memorable? IEEE Trans. Vis. Comput. Graph., 19(12):2306–2315, 2013. doi: 10.1109/TVCG. 2013.234
doi:10.1109/tvcg 2013
-
[5]
M. Bostock, V . Ogievetsky, and J. Heer. D3: Data-Driven Documents. IEEE Trans. Vis. Comput. Graph., 17(12):2301–2309, 2011. doi: 10. 1109/TVCG.2011.185
work page 2011
-
[6]
C. Chen, R. Zhang, E. Koh, S. Kim, S. Cohen, and R. Rossi. Figure captioning with relation maps for reasoning. InProc. IEEE Winter Conf. Appl. Comput. Vis., pp. 1537–1545, 2020. doi: 10.1109/W ACV45572. 2020.9093592
arXiv 2020
-
[7]
K. Cox, R. E. Grinter, S. L. Hibino, L. J. Jagadeesan, and D. Mantilla. A multi-modal natural language interface to an information visualization environment. Int. J. Speech Technol., 4(3-4):297–314, 2001. doi: 10. 1023/A:1011368926479
work page 2001
-
[8]
D. Deng, Y . Wu, X. Shu, J. Wu, S. Fu, W. Cui, and Y . Wu. VisImages: A fine-grained expert-annotated visualization dataset. IEEE Trans. Vis. Comput. Graph., 29(7):3298–3311, 2023. doi: 10.1109/TVCG.2022. 3155440
Show all 39 references
-
[9]
E. Fast, B. Chen, J. Mendelsohn, J. Bassen, and M. S. Bernstein. Iris: A conversational agent for complex tasks. In Proc. ACM Conf. Hum. Factors Comput. Syst., 2018. doi: 10.1145/3173574.3174047
2018
-
[10]
X. Fu, Y . Wang, H. Dong, W. Cui, and H. Zhang. Visualization assessment: A machine learning approach. In Proc. IEEE Vis. Conf., pp. 126–130, 2019. doi: 10.1109/VISUAL.2019.8933570
2019
-
[11]
Hoque and M
E. Hoque and M. Agrawala. Searching the visual style and structure of D3 visualizations. IEEE Trans. Vis. Comput. Graph., 26(1):1236–1245,
-
[12]
K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo. VizML: A machine learning approach to visualization recommendation. In Proc. ACM Conf. Hum. Factors Comput. Syst. , pp. 1–12, 2019. doi: 10. 1145/3290605.3300358
2019
-
[13]
K. Hu, S. Gaikwad, M. Hulsebos, M. A. Bakker, E. Zgraggen, C. Hi- dalgo, T. Kraska, G. Li, A. Satyanarayan, and C ¸. Demiralp. VizNet: Towards a large-scale visualization learning and benchmarking reposi- tory. In Proc. ACM Conf. Hum. Factors Comput. Syst., pp. 1–12, 2019. doi...
2019
- [14]
-
[15]
Kafle, B
K. Kafle, B. L. Price, S. Cohen, and C. Kanan. DVQA: understanding data visualizations via question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5648–5656, 2018
2018
-
[16]
S. E. Kahou, V . Michalski, A. Atkinson, ´A. K´ad´ar, A. Trischler, and Y . Bengio. FigureQA: An annotated figure dataset for visual reasoning. In Workshop Proceedings of International Conference on Learning Representations, 2018
2018
- [17]
-
[19]
C. Lai, Z. Lin, R. Jiang, Y . Han, C. Liu, and X. Yuan. Automatic annotation synchronizing with textual description for visualization. In Proc. ACM Conf. Hum. Factors Comput. Syst., 2020. doi: 10.1145/ 3313831.3376443
2020
-
[20]
C. Liu, Y . Han, R. Jiang, and X. Yuan. ADVISor: Automatic visual- ization answer for natural-language question on tabular data. In Proc. IEEE Pac. Vis. Symp., pp. 6–15, 2021. doi: 10.1109/PacificVis52677. 2021.00010
2021
-
[21]
C. Liu, R. Jiang, S. Tan, J. Yu, C. Yang, H. Shao, and X. Yuan. Datasets of visualization for machine learning, 2024. doi: 10.48550/arXiv.2407. 16351
2024 doi
-
[22]
C. Liu, L. Xie, Y . Han, X. Yuan, et al. AutoCaption: An approach to generate natural language description from visualization automatically. In Proc. IEEE Pac. Vis. Symp. , pp. 191–195, 2020. doi: 10.1109/ PacificVis48177.2020.1043
2020
- [23]
-
[24]
Y . Luo, X. Qin, N. Tang, and G. Li. DeepEye: Towards automatic data visualization. In Proc. IEEE Int. Conf. Data Eng., pp. 101–112, 2018. doi: 10.1109/ICDE.2018.00019
2018
-
[25]
Mackinlay, P
J. Mackinlay, P. Hanrahan, and C. Stolte. Show me: Automatic presen- tation for visual analysis. IEEE Trans. Vis. Comp. Graph., 13(6):1137– 1144, 2007. doi: 10.1109/TVCG.2007.70594
2007
-
[26]
X. Mei, Y . Zhang, C. Yang, R. Shi, and X. Yuan. ZuantuSet: A collection of historical chinese visualizations and illustrations. In Proc. ACM Conf. Hum. Factors Comput. Syst., 2025
2025
-
[27]
Moritz, C
D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer. Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco. IEEE Trans. Vis. Comp. Graph., 25(1):438–448, 2018
2018
-
[28]
S. Pinker. A theory of graph comprehension. Artificial Intelligence and the Future of Testing, pp. 73–126, 1990
1990
-
[29]
Poco and J
J. Poco and J. Heer. Reverse-engineering visualizations: Recover- ing visual encodings from chart images. Comput. Graph. Forum., 36(3):353–363, 2017. doi: 10.1111/cgf.13193
2017 doi
-
[30]
Y . Sun, J. Leigh, A. Johnson, and S. Lee. Articulate: A semi-automated model for translating natural language queries into meaningful visual- izations. In Proc. Int. Symp. Smart Graph., pp. 184–195, 2010. doi: 10. 1007/978-3-642-13544-6 18
2010
-
[31]
J. Wei, X. Wang, et al. Chain-of-thought prompting elicits reasoning in large language models. Proc. NeurIPS, 35:24824–24837, 2022. doi: 10.5555/3600270.3602070
2022
- [32]
-
[33]
Wongsuphasawat, D
K. Wongsuphasawat, D. Moritz, A. Anand, J. D. Mackinlay, B. Howe, and J. Heer. V oyager: Exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Vis. Comp. Graph., 22(1):649–658, 2016. doi: 10.1109/TVCG.2015.2467191
2016
- [34]
-
[35]
Yu and C
B. Yu and C. T. Silva. FlowSense: A natural language interface for visual data exploration within a dataflow system.IEEE Trans. Vis. Com- put. Graph., 26(1):1–11, 2020. doi: 10.1109/TVCG.2019.2934668
2020
-
[36]
Yujian and L
L. Yujian and L. Bo. A normalized levenshtein distance metric. IEEE Trans. Pattern Anal. Mach. Intell., 29(6):1091–1095, 2007
2007
- [37]
-
[38]
Zhang, B
Y . Zhang, B. Coecke, and M. Chen. MI3: Machine-initiated intelligent interaction for interactive classification and data reconstruction. ACM Transactions on Interactive Intelligent Systems, 11(3–4), Aug. 2021
2021
-
[39]
Zhang, R
Y . Zhang, R. Jiang, L. Xie, Y . Zhao, C. Liu, T. Ding, S. Chen, and X. Yuan. OldVisOnline: Curating a dataset of historical visualizations. IEEE Trans. Vis. Comput. Graph., 30(1):551–561, 2024. doi: 10.1109/ TVCG.2023.3326908
2024
-
[2019]
doi: 10.1109/TVCG.2019.2934431
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.