pith. sign in

arxiv: 2604.21134 · v1 · submitted 2026-04-22 · 💻 cs.CL

Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents

Yiyang Lu , Woong Shin , Ahmad Maroof Karimi , Feiyi Wang , Jie Ren , Evgenia Smirni This is my paper

Pith reviewed 2026-05-09 23:54 UTC · model grok-4.3

classification 💻 cs.CL
keywords visualization agentsvision-language modelschart understandingintrospective groundinginteractive groundingPlotlydata reconstructionquestion answering
0
0 comments X

The pith

Visualization agents overcome the pixel-only bottleneck by querying chart specifications and manipulating views to resolve ambiguities.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models misread charts because they interpret interactive figures as static images and lose access to exact values encoded in the underlying specification. The paper introduces a framework that adds spec-grounded introspection, which retrieves deterministic evidence from the chart spec, together with view-grounded interaction that changes the displayed view to clear visual overlaps or ambiguities. These two capabilities are evaluated on a new benchmark of 500 interactive Plotly figures containing over six thousand binary questions with known ground truth. Experiments show that introspection alone improves data reconstruction while the full combination reaches 0.81 QA accuracy and a 6.7 percent gain on overlapping geometries. The approach is further shown working in autonomous data-exploration agents and real-time human collaboration settings.

Core claim

The central claim is that combining spec-grounded introspection with view-grounded interaction lets visualization agents move beyond pixel-only interpretation, producing higher-fidelity data reconstruction and question-answering accuracy of 0.81, including a 6.7 percent improvement on charts with overlapping geometries.

What carries the argument

Introspective and Interactive Visual Grounding (IVG), a framework that pairs queries to the chart's underlying specification for exact values with controlled manipulations of the interactive view to resolve visual ambiguity.

Load-bearing premise

The underlying chart specification is always accessible to the agent and that view manipulations can be performed without introducing new visual or interaction errors the model cannot handle.

What would settle it

An experiment on the same benchmark in which the specification is withheld from the agent or view changes create additional ambiguities that the model fails to resolve, producing no accuracy gain over pixel-only reading.

Figures

Figures reproduced from arXiv: 2604.21134 by Ahmad Maroof Karimi, Evgenia Smirni, Feiyi Wang, Jie Ren, Woong Shin, Yiyang Lu.

Figure 1
Figure 1. Figure 1: Overview of IVG. Rather than reasoning by interpreting rendered pixels with a VLM (left), the agent [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: IVG workflow on a concrete example. Given a question, the agent recreates the chart (Chart Creation), [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Real-time collaboration. The user points at a region of interest and asks a question; IVG captures this as [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Autonomous exploration: IVG enables evidence-grounded analysis. The agent [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ML solution search. (a) The agent surveys candidate solutions across a search space, but (b) overlapping [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Real-time collaboration interface. Left: chat panel where users interact with the agent. Right: interactive [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Autonomous exploration interface. Left: agent-generated analysis report with findings linked to supporting [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: ML solution search interface. The tree visualization shows candidate solutions as nodes, with metrics [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
read the original abstract

Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that VLMs suffer from a 'Pixel-Only Bottleneck' when interpreting interactive charts and introduces Introspective and Interactive Visual Grounding (IVG), which augments agents with (1) spec-grounded introspection to query underlying chart specifications for deterministic evidence and (2) view-grounded interaction to manipulate views and resolve ambiguities. It presents the iPlotBench benchmark of 500 Plotly figures with 6,706 binary questions and ground-truth specs, reporting that introspection improves data reconstruction while the full IVG combination reaches 0.81 QA accuracy (+6.7% on overlapping geometries) and enables autonomous exploration and real-time human collaboration.

Significance. If the empirical gains prove robust, the work offers a concrete path to improve VLM reliability on structured visual data by bridging pixel interpretation with programmatic access and interaction. The controlled benchmark with explicit ground-truth specifications is a useful contribution for reproducible evaluation of grounding techniques in visualization agents.

major comments (2)
  1. [Abstract] Abstract, results paragraph: the headline numbers (0.81 QA accuracy, +6.7% gain on overlapping geometries) are reported without error bars, confidence intervals, statistical significance tests, or details on question generation, introspection implementation, or VLM prompting. This is load-bearing for the central empirical claim and prevents assessment of whether the improvements are reliable or reproducible.
  2. [Evaluation / iPlotBench] iPlotBench and Experiments: all reported gains are obtained inside a deterministic Plotly environment where ground-truth specifications are directly queryable and view manipulations (zoom, filter, etc.) introduce no new rendering artifacts or parsing errors. No ablation injects realistic spec noise, missing attributes, or post-interaction visual degradation, so the fidelity and accuracy lifts are not shown to survive the conditions the skeptic note identifies as critical for generalizability.
minor comments (2)
  1. The abstract mentions 'deployed agents that explore data autonomously and collaborate with human users in real time' but supplies no quantitative metrics, failure cases, or interface details for these demonstrations; moving this material to an appendix or adding a short table of observed behaviors would improve clarity.
  2. Notation for the two grounding components (spec-grounded introspection vs. view-grounded interaction) is introduced clearly in the abstract but could be reinforced with a small diagram or pseudocode snippet early in the methods to help readers track which module contributes to each reported gain.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below with clarifications and indicate where we will revise the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract, results paragraph: the headline numbers (0.81 QA accuracy, +6.7% gain on overlapping geometries) are reported without error bars, confidence intervals, statistical significance tests, or details on question generation, introspection implementation, or VLM prompting. This is load-bearing for the central empirical claim and prevents assessment of whether the improvements are reliable or reproducible.

    Authors: We agree the abstract is concise and omits supporting details. The full manuscript describes question generation (Section 3.2), introspection (Section 3.1), and prompting (Section 4.1). We will revise the abstract to reference the controlled benchmark setup and direct readers to the experiments section for methodology. Our results use a fixed deterministic environment, so variance arises mainly from VLM stochasticity; we will add error bars from repeated prompt runs and note statistical significance in the revised experimental tables. revision: partial

  2. Referee: [Evaluation / iPlotBench] iPlotBench and Experiments: all reported gains are obtained inside a deterministic Plotly environment where ground-truth specifications are directly queryable and view manipulations (zoom, filter, etc.) introduce no new rendering artifacts or parsing errors. No ablation injects realistic spec noise, missing attributes, or post-interaction visual degradation, so the fidelity and accuracy lifts are not shown to survive the conditions the skeptic note identifies as critical for generalizability.

    Authors: The deterministic Plotly setting with explicit ground-truth specs was deliberately chosen to isolate the Pixel-Only Bottleneck and measure exact contributions of introspection and interaction without confounding rendering or parsing noise. This enables reproducible evaluation on the 6,706 questions. We acknowledge that real deployments may encounter spec noise or visual degradation. In revision we will add an explicit limitations paragraph discussing this gap and outlining planned extensions for noisy conditions, but we do not introduce new ablations here as they require substantial additional experiments. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical results on introduced benchmark

full rationale

The paper introduces the IVG framework and iPlotBench benchmark, then reports direct experimental measurements of QA accuracy (0.81) and fidelity gains on 500 figures. No equations, derivations, or predictions are presented that reduce by construction to fitted parameters or self-citations. The central claims rest on external evaluation against ground-truth specifications within the new benchmark, satisfying the default expectation of a self-contained empirical study with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review reveals no explicit free parameters, axioms, or invented entities; the approach assumes access to chart specifications and interaction APIs that are treated as given.

pith-pipeline@v0.9.0 · 5490 in / 1153 out tokens · 40074 ms · 2026-05-09T23:54:05.821497+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    InFindings of the Association for Computa- tional Linguistics: ACL 2025, pages 19796–19821, Vienna, Austria

    Data interpreter: An LLM agent for data sci- ence. InFindings of the Association for Computa- tional Linguistics: ACL 2025, pages 19796–19821, Vienna, Austria. Association for Computational Lin- guistics. Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented...

    work page 2025

  2. [2]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 27228–27238

    Throne: An object-based hallucination bench- mark for the free-form generations of large vision- language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 27228–27238. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing

    work page

  3. [3]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882

    Mitigating object hallucinations in large vision- language models through visual contrastive decod- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882. Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, and Nanyun Peng. 2025. Metal: A multi-agent framework for chart generation with test-time scali...

    work page arXiv 2025

  4. [4]

    Chartcoder: Advancing multimodal large language model for chart-to-code generation, 2025

    V oyager: Exploratory analysis via faceted browsing of visualization recommendations.IEEE Transactions on Visualization and Computer Graph- ics, 22(1):649–658. Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2017. V oy- ager 2: Augmenting visual analysis with partial view...

    work page arXiv 2017

  5. [5]

    Load any data files you find (csv, json, parquet, etc.)

    Look– List files in the directory. Load any data files you find (csv, json, parquet, etc.)

    work page

  6. [6]

    Ask– What is this dataset about? What questions are worth investigating?

    work page

  7. [7]

    Sup- port each finding with evidence (numbers or plotly visualizations)

    Investigate– Answer your questions. Sup- port each finding with evidence (numbers or plotly visualizations). 4.Synthesize– What did you learn? Improvement Prompt.The agent iteratively re- fines its analysis: You have {time_remaining} seconds remaining. Reflect: • Does your evidence support your findings? • Dig deeper. Check specific values. Use get_plot_i...

    work page

  8. [8]

    Findings– List each finding with evidence (numbers, tables, or plot references) To finish:

    work page

  9. [9]

    Redraw any unclear plots (fix overlapping, ensure readable)

    work page

  10. [10]

    Decide which plots are evidence and their display order (e.g., plots 4, 7, 2)

    work page

  11. [11]

    analysis.md

    Write “analysis.md” referencing plots as Plot 1, Plot 2, Plot 3... (matching the order you chose)

    work page

  12. [12]

    Manhattan Pickup Dominance

    Call submit_summary(evidence_plots=[4, 7, 2]) with original IDs in that order. Report Interface.Figure 7 shows the final report interface with a dual-panel layout: a left panel con- taining the agent-generated report and a right panel hosting interactive Plotly figures in tabbed views. Each finding in the report links to a supporting plot, and the report ...

    work page 2015