{"id":"364c0c9e-86f7-40d6-b9b3-f5e76a08bf53","arxiv_id":"2607.27938","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VizPilot uses a two-stage multimodal-LLM pipeline to decompose SVG-based composite visualizations into explained, highlighted components and generate interactive scrollytelling and free-exploration onboarding automatically.","lead":"VizPilot automatically creates interactive, step-by-step guides for complex multi-chart web visualizations by analyzing their SVG code and screenshots with a multimodal AI model. It could save developers the manual work of writing onboarding tours and help newcomers read unfamiliar composite charts faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no manual authoring' claim is never tested against the developer-provided description: the pipeline is never run without it, so 'directly from raw artifacts' may be overstated.","rationale":"The reader's CONDITIONAL verdict is appropriate, and this stress-test reinforces the conditionality by identifying a more targeted gap: the system's flagship 'no manual authoring' / 'directly from raw artifacts' claim is not supported by any condition that removes the developer description. The paper is otherwise strong: it has a concrete two-stage pipeline, a programmatic validation step for selectors, a documented user study with objective completion-time evidence, and a public gallery. The description ablation is the single most load-bearing missing experiment because it directly tests whether the reverse-engineering is actually automatic or whether it depends on a human-provided semantic summary. The existing modality comparison is informative but not sufficient, since the description is held constant across all conditions. The user study does provide independent evidence that the final onboarding experience is helpful, but it does not test the automation claim; a human-authored description plus formatted output could produce the same user experience. Therefore the paper should remain CONDITIONAL pending this ablation and the release of code/data for independent verification. I do not recommend REJECT: the claimed UX benefit is supported, and the missing ablation is a testable, addressable gap rather than a known failure.","tokens_in":27272,"tokens_out":4630,"duration_ms":44825,"concrete_test":"Re-run the gallery evaluation from Section 7.3 on all 18 visualizations with the developer description field removed (empty string), keeping SVG code, bitmap image, and optional interaction source code unchanged, and compare decomposition F1, semantic mapping exact accuracy, and end-to-end success rate against Table 3. If the end-to-end success rate drops materially (e.g., more than 0.1, or below roughly 0.7), the 'directly from raw artifacts' claim is dependent on the description; if performance is statistically unchanged, the concern is resolved. Also report the fraction of explanation steps whose text overlaps verbatim with the removed description, to quantify description-driven content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VizPilot reverse-engineers composite visualizations 'directly from raw visualization artifacts' with 'no manual authoring effort.' The load-bearing condition is that the developer-supplied 'brief description' is not doing the semantic work. The paper never ablates this input. Section 1 states that VizPilot 'requires no manual authoring effort,' while the Abstract and Section 3 require 'only a brief visualization description and optional interaction source code.' The Semantic Inference prompt in Appendix A.1 lists the 'user-authored description' as a primary input, and Section 7.2 shows Alex providing a description plus interaction source code. The modality analysis in Section 7.1 and Appendix A.8 varies SVG, image, and interaction source code, but the description is present in all conditions, so it cannot separate description-driven semantics from artifact-driven reverse-engineering. If the description already contains component names, their roles, relationships, and reading order, the claimed autonomy from raw artifacts is materially weakened: the system may be formatting an authored summary rather than reconstructing the visualization's semantics. This concern is distinct from the DOM-scope limitation flagged by the reader; even on well-grouped, DOM-accessible SVGs, the description may be load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VizPilot is a browser extension that generates interactive onboarding experiences for SVG-based composite visualizations. It takes as input the extracted SVG code, a rendered bitmap image, a developer-provided brief description, and optionally the interaction source code. A two-stage Composite Visualization Analyzer first performs Semantic Inference to decompose the visualization into visual components and produce atomic explanation units organized by a five-category knowledge taxonomy, then Semantic Mapping anchors these units to specific SVG elements via hierarchical selector inference with programmatic validation. The Onboarding Interface provides narrative scrollytelling and free exploration. The paper evaluates the approach with a comparative input-modality analysis, a gallery of 18 visualizations scored by LLM-as-judge, a usage scenario with expert feedback, and a 16-participant user study contrasting VizPilot against a text baseline.","tokens_in":27627,"tokens_out":5510,"duration_ms":48457,"significance":"If the central claims hold, VizPilot addresses a real gap: prior visualization onboarding approaches are platform-dependent, require substantial manual authoring, and mostly target simple charts or dashboards, whereas VizPilot targets customized composite SVG visualizations in a browser context. The two-stage decomposition with hierarchical selector inference and programmatic DOM validation is a sensible design that combines semantic reasoning with deterministic grounding. The paper contributes a working implementation, a public gallery, a multimodal ablation of SVG/image/interaction-code inputs, and a user study with both quantitative and qualitative data. These are concrete strengths. However, the significance is tempered by three issues: the developer-provided description is never ablated, so the claim of reverse-engineering 'directly from raw visualization artifacts' is not fully established; the gallery evaluation depends on author-defined ground truth and LLM-as-judge with manual escalation; and the user study has a ceiling effect on accuracy and a single marginal objective performance result.","major_comments":[{"comment":"The central claim of automated reverse-engineering 'directly from raw visualization artifacts' is not adequately supported because the developer-provided description is a primary input to Semantic Inference and is never ablated. The Semantic Inference prompt in Appendix A.1 lists the 'user-authored description' as one of the four inputs, and the modality analysis in §7.1 varies SVG, image, and interaction source code but keeps the description fixed in all conditions. Consequently, component names, roles, relationships, and reading order may be largely supplied by the description rather than reconstructed from the SVG. I request an ablation that runs the pipeline without the description and reports decomposition, mapping, and explanation-quality metrics, and a revised claim that reflects the actual input requirements.","section":"§1, §3, Appendix A.1"},{"comment":"The gallery evaluation relies on author-defined ground-truth component lists and an LLM-as-judge protocol with a confidence threshold of 0.7; cases below the threshold (5 of 18) are escalated to manual review by the authors. Because the ground truth and rubric are derived from the same component definition that the pipeline implements, the reported E2E success rate of 0.898 may be optimistic. Please report inter-judge agreement among the three judge models, results computed over the full set excluding manually reviewed cases, and a pre-registered or independently verified random sample of the manual-review decisions.","section":"§7.3, Table 3"},{"comment":"The user study's only objective performance benefit is task completion time (p=0.039) from a 16-participant, two-visualization design, and task accuracy shows a ceiling effect (baseline 3.62/4 vs. VizPilot 3.75/4). Although the NASA-TLX reductions are large and consistent, the paper should report effect sizes and confidence intervals for the time and workload results, and should discuss whether the p=0.039 task-time result would remain significant after correcting for the number of Wilcoxon tests performed. As written, the statement that 'VizPilot accelerates insight extraction' rests on a single marginal p-value.","section":"§7.4.1"},{"comment":"The claimed reduction in developer authoring effort is supported only by a single usage scenario and an interview with two experts. No quantitative comparison against a manual authoring baseline (e.g., authoring time, number of editing operations, or interaction-code adjustments) is provided. Without such a comparison, the 'no manual authoring effort' claim remains a design assertion rather than an evaluated outcome. Adding a small comparative study or at least a structured time measurement for the authoring workflow would materially strengthen the paper.","section":"§7.2, Appendix A.7"}],"minor_comments":[{"comment":"The abstract and §1 say VizPilot 'requires no manual authoring effort,' while the abstract also says it 'requires only a brief visualization description and optional interaction source code.' These statements are in tension; please rephrase to avoid an overclaim, e.g., 'requires a brief description but no step-by-step authoring.'","section":"Abstract and §1"},{"comment":"The limitations regarding canvas-rendered visualizations and heavily flattened SVGs are stated only in the Discussion. Since they directly qualify the platform-agnostic framing, they should be mentioned prominently in the Introduction or the system overview.","section":"§8.1"},{"comment":"The procedure section says 'three phrases' where it means 'three phases'; please fix this typo.","section":"§7.4"},{"comment":"The row labeled 'Multiple Time SeriesMultiple Time Series' contains a duplicated phrase; the label should be 'Multiple Time Series.'","section":"Table 5"},{"comment":"The phrase 'maintaining user anatomy' appears to be a typo; it should likely be 'user agency' or 'user autonomy.'","section":"§8.2"},{"comment":"The figures mark several comparisons with '*** (p<0.001)' but the exact p-values are not reported in the text for each item; please list them in the figure captions or in the results section.","section":"Figures 6 and 7"},{"comment":"The implementation uses GPT-5 for inference while the gallery judges include GPT-5.5, Gemini 3.1 Pro Preview, and Claude 3.5 Sonnet; please clarify whether the inference model is the same across all evaluations and whether the judge models are distinct from the generation model, as this affects reproducibility and the interpretation of the LLM-as-judge results.","section":"§6 and §7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of TVCG and the core approach is interesting. The main risk is that the developer-provided description may be carrying much of the semantic content, so the requested description-ablation study is essential before acceptance. The evaluation would also benefit from stronger evidence on authoring effort and from a more conservative treatment of the marginal task-time result. I recommend major revision rather than reject because the central design is defensible and the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VizPilot is a genuinely useful system: a two-stage MLLM pipeline that decomposes an SVG composite visualization into components, extracts structured explanations, and maps them to DOM selectors with programmatic validation. That last part is the real contribution—hierarchical selector inference plus a validator that rejects ill-formed selectors makes the output actionable, not just plausible text. The modality analysis (SVG only, image only, SVG+image, plus interaction source code) is a nice piece of evidence that each input earns its place. The user study is small but sensible: 16 participants, within-subjects, and the task-time improvement (p=0.039) and TLX reductions are consistent with the qualitative comments about split-attention. The gallery of 18 visualizations across four composite patterns is a good breadth check, and putting it online is a plus.\n\nThe soft spots are real but not fatal. The biggest one is that the developer-provided description is never ablated. The abstract says the system needs only 'a brief visualization description,' but the intro claims 'no manual authoring effort,' and the Semantic Inference prompt treats the user-authored description as a primary input. The modality analysis keeps the description fixed across all conditions, so it cannot rule out that the description is doing much of the semantic work. On the gallery, the component names and relationships may already be in the description, which would make the 'reverse-engineering directly from raw artifacts' claim materially weaker. This needs a fix: either an ablation without the description, or a careful reframing of claims.\n\nThe gallery evaluation relies on LLM-as-judge with author-defined ground truth and manual escalation for 5 of 18 cases. That is not independent human grounding, but the rubric is detailed and the error analysis is honest. The user study has ceiling-effect accuracy—meaning the task was too easy—but the time and workload differences are still interpretable. Code and data are not shipped, which limits reproducibility; the online gallery helps but is not the same as an executable pipeline. The DOM accessibility assumption is stated plainly in Section 8.1, so that is a scope limit rather than an oversight.\n\nOverall, the central argument holds: for well-grouped, DOM-accessible SVGs, VizPilot can produce a working onboarding tour with minimal human input beyond a high-level description. The citations to prior work are relevant and not self-promotional. This deserves serious peer review. I would ask the authors to ablate the description, release the pipeline and data, and temper the 'no manual authoring' phrasing.","headline":"VizPilot is a solid systems contribution with a clever mapping validator and a fair evaluation, but the 'no manual authoring' claim is not supported because the developer description is never ablated.","tokens_in":28014,"tokens_out":2793,"would_cite":true,"duration_ms":23763,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VizPilot claims it can automatically generate interactive guided tours for complex multi-chart visualizations straight from the SVG code and a screenshot, with no manual authoring.","keywords":["composite visualizations","visualization onboarding","multimodal large language models","SVG semantic mapping","hierarchical selector inference","scrollytelling","cognitive load","browser extension"],"falsifier":"Run VizPilot on a composite visualization whose SVG has been flattened into a single path with no grouped containers: if the system cannot produce any valid selectors or onboarding steps, that confirms the central raw-artifact claim is limited to structurally grouped SVG inputs. A complementary check is to apply it to a canvas-rendered visualization; the paper's own scope statement predicts component extraction accuracy drops.","tokens_in":27036,"feed_emoji":"🧭","tokens_out":5251,"duration_ms":45055,"temperature":0.7,"pith_summary":"VizPilot claims that interactive onboarding for complex composite visualizations can be generated automatically, without manual authoring, from the raw artifacts a developer already has: the SVG code, a rendered bitmap screenshot, a brief description, and optionally the interaction source code. The system decomposes the visualization into visual components, extracts structured knowledge about each one, and maps every explanation to the exact SVG elements that should be highlighted. If this works, developers no longer need to hand-write tour scripts or bind events by hand, and first-time viewers of unfamiliar multi-chart displays can learn them through guided scrolling and free-form exploration instead of staring at static text. The paper supports the claim with a gallery of 18 visualizations spanning all four composite design patterns and a user study in which 16 participants answered questions faster and with lower reported mental demand using VizPilot than with text-only descriptions.","feed_headline":"Browser extension auto-generates onboarding tours for complex charts","feed_subtitle":"It reads the raw SVG and a screenshot, maps each explanation to the right visual element, and cuts user task time by 68 seconds.","key_machinery":"The load-bearing object is the Composite Visualization Analyzer, a two-stage MLLM pipeline. Semantic Inference decomposes the composite visualization into visual components and emits atomic explanation units — one sentence, one component, one knowledge category — using a five-category taxonomy drawn from onboarding literature. Semantic Mapping then performs Hierarchical Selector Inference, prompting the model to write CSS-like selector paths from parent containers down to target marks rather than raw IDs, and passes every candidate selector through a three-check programmatic validation: DOM validity, component containment, and layout consistency. This combination converts probabilistic model output into deterministic, verifiable highlighting targets.","core_discovery":"The paper's central discovery is that a multimodal LLM, constrained by a structured two-stage pipeline, can reverse-engineer the semantic structure of a composite SVG visualization and turn it into a mapped, interactive onboarding experience. The first stage, Semantic Inference, decomposes the chart into visual components and generates atomic explanation units under a five-category knowledge taxonomy (structure, data, encoding, analytical, interaction). The second stage, Semantic Mapping, anchors each unit to DOM nodes via hierarchical selector inference, then verifies the selectors programmatically. The claimed result is end-to-end onboarding generation with a mean end-to-end success rate of 0.898 across 18 gallery visualizations, and a user study showing significantly faster task completion (194 seconds vs. 263 seconds), lower NASA-TLX mental demand, effort, and frustration, and higher PSSUQ ratings, with task accuracy at ceiling in both conditions.","pith_inferences":["The same mapped semantic representation could power accessibility features beyond onboarding, such as screen-reader narration, keyboard-navigable tours, and automated alternative text, since each explanation is already anchored to precise DOM elements.","If the mapping mechanism generalizes, it could turn the pipeline into a general evaluation harness for whether multimodal models truly understand chart structure, by scoring decomposition and grounding quality on any SVG-based visualization.","A testable extension is to apply the decomposition to dashboards composed of multiple SVG containers; the paper currently assumes a single SVG container, so multi-view dashboards would be the natural next boundary.","The flattened-SVG limitation suggests a targeted stress test: feed the system an SVG where all shapes are merged into one path, and measure how much decomposition recall and mapping accuracy drop; the paper's own scope statement predicts a sharp drop."],"forward_implications":["Visualization developers can publish onboarding for a composite chart by providing a brief description and optional interaction code; the rest of the tour is generated and exported as a JSON file plus a lightweight runtime script.","End users of unfamiliar composite visualizations get two complementary paths — guided narrative scrollytelling with synchronized highlighting, and free exploration with hover inspectors and a context-constrained assistant — which the study ties to reduced split-attention effort.","The approach generalizes across the four composite design patterns (juxtaposition, overloading, superimposition, nesting), with exact mapping accuracy above 0.89 and end-to-end success above 0.80 in the gallery evaluation.","Because all mappings are expressed as validated SVG selectors, the generated onboarding can be deployed without modifying the visualization's underlying source code.","Interaction knowledge, when interaction source code is available, shifts from guesswork to verified action-consequence descriptions of hover, click, brush, and linked updates."],"supporting_citations":[{"why":"Supplies the visualization-onboarding knowledge taxonomy and the definition of onboarding that VizPilot operationalizes.","marker":"[41]"},{"why":"Defines the composite visualization design space (juxtaposition, overloading, superimposition, nesting) used to structure the gallery evaluation.","marker":"[22]"},{"why":"Prior SVG reverse-engineering method that VizPilot builds on for recovering chart structure from SVG primitives.","marker":"[7]"},{"why":"Shows that multimodal LLMs can derive semantic representations from SVG visualizations, the capability VizPilot extends to onboarding with deterministic grounding.","marker":"[51]"},{"why":"D3.js, one of the main SVG-based libraries whose output the pipeline targets.","marker":"[5]"},{"why":"Vega-Lite, another SVG-producing grammar the pipeline is designed to be compatible with.","marker":"[36]"},{"why":"Supplies the narrative visualization and scrollytelling principles guiding the onboarding interface design.","marker":"[37]"},{"why":"NASA-TLX instrument used to measure cognitive load in the user study.","marker":"[18]"},{"why":"PSSUQ instrument used to measure system usefulness and information quality.","marker":"[27]"}],"fun_headline_variants":["Multimodal LLM auto-generates interactive tours for composite charts","AI turns SVG artifacts into guided onboarding experiences","Browser extension creates interactive onboarding from raw SVG","VizPilot: 0.898 success in auto-onboarding composite charts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline works only when the target visualization exposes a hierarchical, DOM-queryable SVG structure; canvas-rendered charts and heavily flattened SVGs are outside its scope and would break the semantic mapping stage.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal LLM auto-generates interactive tours for composite charts","AI turns SVG artifacts into guided onboarding experiences","Browser extension creates interactive onboarding from raw SVG","VizPilot: 0.898 success in auto-onboarding composite charts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3255,"prompt_tokens":982,"completion_tokens":2273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":598,"tokens_out":2273,"duration_ms":15026,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:22:04.914219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VizPilot on a composite visualization whose SVG has been flattened into a single path with no grouped containers: if the system cannot produce any valid selectors or onboarding steps, that confirms the central raw-artifact claim is limited to structurally grouped SVG inputs. A complementary check is to apply it to a canvas-rendered visualization; the paper's own scope statement predicts component extraction accuracy drops.","supporting_citations":[{"cited_title":"Stoiber, F","cited_arxiv_id":null,"evidence_quote":"Supplies the visualization-onboarding knowledge taxonomy and the definition of onboarding that VizPilot operationalizes."},{"cited_title":"Javed and N","cited_arxiv_id":null,"evidence_quote":"Defines the composite visualization design space (juxtaposition, overloading, superimposition, nesting) used to structure the gallery evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that multimodal LLMs can derive semantic representations from SVG visualizations, the capability VizPilot extends to onboarding with deterministic grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PSSUQ instrument used to measure system usefulness and information quality."}],"review_version":2}