{"id":"eeb57d8a-b64e-45e9-98fd-ceb3dfc386fb","arxiv_id":"2607.04727","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Proactive explore-then-code reconstruction of interactive Plotly+Dash dashboards is hard: Gemini 3 Pro scores 79.4 overall but only 64.2 on Level-3 callback complexity, with a large open/closed gap.","lead":"The paper defines Dashboard2Code: multimodal models must actively explore a live interactive dashboard, use their own click/filter feedback, and emit Plotly+Dash code that rebuilds both looks and callbacks. It ships a 180-dashboard benchmark and an automated visual-plus-behavior scorer, then shows even top closed models drop sharply on complex interaction logic.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified that would overturn the central empirical claim.","rationale":"The paper's central claim is empirical hardness of proactive Dashboard2Code plus a large open/closed gap, not that the weighted metric is a perfect universal proxy. That claim is multiply supported by execution rates, FigSim, exploration statistics, complexity breakdowns, and text-anonymization drops that do not all depend on Gemini-as-judge. The metric-validation caveats the reader flags are correctly noted as limitations (single framework, synthetic majority, LLM judges) but are secondary; they do not invert the ranking or the L3 difficulty signal. No internal inconsistency or untested assumption undercuts the reported numbers. Therefore the ACCEPT / HIGH-confidence verdict stands without adjustment.","tokens_in":27926,"tokens_out":457,"duration_ms":4619,"concrete_test":"Re-score the 90 human-rated dashboards (or a stratified 30-sample L1/L2/L3 subset) with an independent non-Gemini judge (e.g., Claude or GPT-5.1 only) and recompute Pearson vs. human scores plus model ranking on Total/L3; if r stays ≥0.70 and Gemini vs. open-source / L3 gaps remain within ~5 points of Table 1, the proxy concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (metric fidelity under Gemini-family judges + synthetic majority) is a real caveat but not load-bearing against the strongest claim. Tables 1–4 and Figures 3–5 show large, consistent gaps: Gemini 3 Pro 79.4 overall / 64.2 on L3, open-source collapse, DOM removal and text anonymization sharply cut LLM-sem and exploration quality. These patterns hold under code-exec rates, component coverage, FigSim (LLM-judge-free), and ablations; human correlation r=0.78 on 90 samples and GPT-5.1 cross-judge robustness (A.4.3) further stabilize the ranking. Single-framework and synthetic-data limits are acknowledged and do not reverse the hardness or gap findings.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces Dashboard2Code: a multimodal agent must proactively explore a live interactive dashboard (click, filter, drag, etc.), integrate its own interaction feedback, and emit Plotly+Dash code that reproduces both visual appearance and callback logic. To support the task it releases DashboardMimic (180 manually verified dashboard–code pairs: 58 real + 122 LLM-augmented), spanning three callback-complexity levels and eight interaction patterns, plus an automated evaluation stack that mixes static code-semantic metrics with dynamic task-based testing (Figure Similarity, LLM-visual, LLM-behavior). Experiments on closed- and open-source VLMs show that even Gemini 3 Pro reaches only 79.4 overall and 64.2 on Complexity L3, that open-source models lag sharply, and that removing DOM or anonymizing control text substantially degrades semantic/callback fidelity.","tokens_in":28210,"tokens_out":1302,"duration_ms":20648,"significance":"If the empirical picture holds, the work supplies a missing intermediate benchmark between static chart-to-code and open-ended GUI agents, with a concrete, reproducible Python target (Plotly+Dash under MIT). Strengths that should be credited include: explicit filtering principles and dual human recheck; multi-generator synthesis plus a rewrite-and-refactor contamination pipeline with n-gram/embedding checks; a judge-free Figure Similarity metric; human correlation of the aggregate score (Pearson r=0.78 on 90 dashboards); and a useful suite of ablations (DOM on/off, text anonymization, role separation, error typology). These make DashboardMimic a practical resource for measuring active exploration, long-horizon visual reasoning, and interactive code generation.","major_comments":[{"comment":"§4.4 and A.4.2: the final Score weights (0.1 KCC + 0.3 LLM-sem + 0.2 FigSim + 0.2 LLM-vis + 0.2 LLM-beh) are obtained by optimizing correlation against human ratings on the same 90-dashboard set used for validation. This is a mild circularity risk for the meta-metric. Please report (i) leave-one-out or held-out weight stability, (ii) unweighted / equal-weight aggregates, and (iii) primary model rankings under the judge-free components alone (Code Exec., Comp. Cov., FigSim). The central hardness claim is already visible in those components, so this is a transparency fix rather than a redesign.","section":"§4.4, A.4.2"},{"comment":"Table 1 / §5.2: open-source models show very low Code Exec. rates (e.g., InternVL3.5-8B 5.6%, Qwen3-VL-8B 2.8%). The reported open/closed gap therefore conflates general executable-code generation skill with the Dashboard2Code-specific abilities (active exploration, callback recovery). A controlled follow-up—e.g., providing a layout skeleton or measuring only among successfully executing generations—would better isolate the contribution of interactive exploration. Without it, the claim that open-source models specifically fail at Dashboard2Code remains only partially supported.","section":"Table 1, §5.2"},{"comment":"§3.4 and A.3.5: 122/180 dashboards are LLM-synthesized (including Gemini/GPT/Claude families that are also evaluated). Although multi-generator synthesis and human filtering reduce single-model bias, the paper should report a real-seed (n=58) vs. synthetic (n=122) performance split for at least the top closed-source models. If rankings or absolute scores differ materially on the real subset, that should be stated; if they do not, the split still strengthens the contamination/favoritism defense.","section":"§3.4, A.3.5"}],"minor_comments":[{"comment":"A.4.3 cross-judge table is helpful but only covers Gemini 3 Pro and GPT-5.1 generators. Extending the same GPT-5.1-as-judge check to Claude Sonnet 4.5 (or reporting Spearman rank agreement across judges) would further reassure readers about self-preference.","section":"A.4.3"},{"comment":"Figure 5 error typology is informative; adding absolute counts (not only percentages) per complexity level would make the L3 shift toward State Misinterpretation easier to interpret.","section":"Figure 5, §5.3.4"},{"comment":"Eq. (5) and Eq. (6) use free sub-weights (FigureSimilarity: 0.3 Data / 0.2 Style / 0.2 Type / 0.3 Text). Briefly justify these choices or show sensitivity; they are currently listed without derivation.","section":"§4.4, A.4.5"},{"comment":"Limitations correctly note Plotly+Dash-only scope; a short forward-looking paragraph on what would change under React/Tableau targets would help readers gauge generalizability without overclaiming.","section":"Limitations"},{"comment":"Minor typography: “end -to-end”, “front -end”, “multi -modal” spacing inconsistencies appear in the introduction and related-work sections; unify hyphenation.","section":"§1, A.1"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical systems paper; the central hardness and open/closed gap findings are robust enough that I would not block on the metric-weight or synthetic-data points if the authors supply the requested splits and unweighted rankings. Fit for a top-tier SE/ML venue that values new interactive benchmarks. No integrity or novelty concerns beyond the usual concurrent-work note (WebVIA is already discussed)."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this paper defines Dashboard2Code properly as an active problem—model must explore a live dashboard, integrate its own click/filter feedback, then emit full Plotly+Dash code with callbacks—and shows even Gemini 3 Pro only hits 79.4 overall and 64.2 on L3, with open-source models collapsing and DOM/text cues mattering a lot.\n\nWhat is actually new is the combination of proactive exploration (not human trajectories as in IWR-Bench/Interaction2Code), dashboard-specific callback topologies (eight patterns, three complexity levels), and a 180-pair DashboardMimic set with verified interaction tasks. Construction looks careful: real seed filtering, multi-generator synthesis, dual human recheck, rewrite/refactor against contamination with n-gram and embedding checks. The auto-eval mixes static component/semantic scores with dynamic task execution, figure JSON similarity (judge-free on data/style/type/text), and LLM visual/behavior scores; they report Pearson 0.78 vs humans on 90 samples and a GPT-5.1 cross-judge check. Ablations on text anonymization, role separation, exploration completeness, and error types (especially transformation hallucination and missing components on L3) are useful, not decorative.\n\nSoft spots are real but proportionate. Scope is deliberately Plotly+Dash only; majority of pairs are synthetic; final metric weights were optimized against the human set and several judges are Gemini-family. Those are standard systems-benchmark caveats, not load-bearing against the ranking or the L3 hardness claim—code-exec rates, component coverage, and FigSim move in the same direction. Free parameters (weights, 25-step limit, 1920×1080 viewport) are documented.\n\nThis is for people working on multimodal GUI agents, chart/dashboard code generation, and SE evaluation of interactive UIs. It does not invent a new technology class, but it is a clean, reproducible evaluation contribution with honest limitations. I would send it to peer review; the central empirical story holds up under the stress test. Worth a reading-group slot if your group cares about agents that must reverse-engineer interactive analytics software.","headline":"Useful new proactive-exploration-to-code task and Plotly+Dash benchmark; the hardness and open/closed gap claims are well supported, with narrow but honest scope limits.","tokens_in":28831,"tokens_out":562,"would_cite":true,"duration_ms":10521,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Even the strongest multimodal models struggle to rebuild interactive dashboards by actively exploring them and writing faithful code.","keywords":["Dashboard2Code","interactive dashboards","multimodal code generation","GUI agents","Plotly Dash","benchmark","active exploration","callback logic"],"falsifier":"A new model that scores high on the automated metric yet is consistently rated by human judges as mismatched in interaction logic or visual detail on the same human-rated sample set would falsify both the metric's validity and the reported performance gaps.","tokens_in":28853,"feed_emoji":"📊","tokens_out":840,"duration_ms":17117,"temperature":0.7,"pith_summary":"This paper argues that reconstructing interactive dashboards is a harder end-to-end challenge for multimodal models than static chart-to-code tasks. It defines Dashboard2Code: the model must proactively click, filter, and navigate a live dashboard, integrate the visual and structural feedback it receives, and emit code that reproduces both appearance and interaction logic. To measure this, the authors release DashboardMimic, 180 verified Plotly+Dash dashboard-code pairs spanning three complexity levels and eight common callback patterns, plus an automated evaluator that mixes static code analysis with live interaction tests and tracks human judgments closely. Experiments show that top closed-source systems reach only partial fidelity on the hardest cases, open-source models lag far behind, and removing DOM structure or anonymizing control labels sharply hurts recovery of the underlying logic. The work therefore claims that current multimodal agents still lack robust active exploration, long-context integration, and callback reasoning needed for real data apps.","feed_headline":"Top models still fail to rebuild interactive dashboards","feed_subtitle":"Active exploration plus code generation falls short on complex real-world callback logic","key_machinery":"The Dashboard2Code active-exploration loop (model navigates the live dashboard, collects its own interaction feedback, then emits reproducing Plotly+Dash code) together with the DashboardMimic benchmark and the hybrid automated evaluator that combines code-semantics scores with dynamic figure-similarity and behavior-consistency tests.","core_discovery":"Dashboard2Code is a genuine open challenge: even the strongest evaluated multimodal systems achieve only middling overall fidelity and drop further on high-complexity multi-callback dashboards, while open-source models remain far behind, especially when forced to rely on screenshots alone or when textual labels are anonymized.","pith_inferences":["Improvements in exploration completeness may yield larger gains than pure code-generation skill for reverse-engineering interactive apps.","The same active-reconstruction protocol could diagnose similar gaps in other stateful interfaces such as spreadsheets or multi-step forms.","Synthetic dashboards generated under controlled taxonomies may systematically under- or over-estimate difficulty relative to messy production systems.","Text anonymization results suggest many current systems still shortcut via label heuristics rather than learning callback graphs from trajectories."],"forward_implications":["Static chart-to-code benchmarks understate the difficulty of programming real interactive data interfaces.","GUI agents will need stronger long-horizon exploration and causal-graph recovery before they can reliably reverse-engineer dashboards.","Removing structured UI trees or control labels exposes heavy reliance on surface text cues rather than true interaction semantics.","Hybrid static-plus-dynamic evaluation will become necessary for any claim of faithful interactive UI reconstruction.","The open-versus-closed performance gap on this task points to missing capabilities in open multimodal models beyond simple visual recognition."],"fun_headline_variants":["Even top models struggle to rebuild interactive dashboards","Multimodal models fall short reconstructing complex dashboards","Dashboard2Code exposes gaps in interactive dashboard coding","Strongest systems only middling on multi-callback dashboards","Open-source models lag far behind on dashboard reconstruction"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The weighted automated metric, tuned and validated on a human-rated subset of ninety dashboards, is assumed to remain a faithful proxy for true visual-and-interaction fidelity across the full benchmark and future models.","fun_headline_variants_meta":{"raw":{"variants":["Even top models struggle to rebuild interactive dashboards","Multimodal models fall short reconstructing complex dashboards","Dashboard2Code exposes gaps in interactive dashboard coding","Strongest systems only middling on multi-callback dashboards","Open-source models lag far behind on dashboard reconstruction"]},"model":"grok-4.5","effort":"low","cost_usd":0.00313,"raw_usage":{"total_tokens":1035,"prompt_tokens":722,"num_sources_used":0,"completion_tokens":77,"cost_in_usd_ticks":31300000,"prompt_tokens_details":{"text_tokens":722,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":236,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":722,"tokens_out":77,"duration_ms":2735,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T14:29:32.276091+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A new model that scores high on the automated metric yet is consistently rated by human judges as mismatched in interaction logic or visual detail on the same human-rated sample set would falsify both the metric's validity and the reported performance gaps.","supporting_citations":[],"review_version":1}