{"id":"10cc04a8-a823-4045-b40e-52819255b2ce","arxiv_id":"2607.19910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current multimodal LLMs can copy the look of multi-view dashboards but mostly fail to bind real data and implement cross-view interactions.","lead":"MV-Bench introduces over 1,000 coordinated multi-view dashboard interfaces with executable code, datasets, and interaction checks, built by translating Tableau workbooks. Five leading multimodal AI models reproduced dashboard appearance well (up to 75%) but scored poorly on data binding (21.7%) and interaction logic (11.7%).","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference ground truth is only verified at the source/build/render level, never for data-binding or interaction semantics, so Data and Int. scores may measure agreement with an LLM-generated artifact rather than functional correctness.","rationale":"The paper's central qualitative finding—visual fidelity far exceeds functional correctness—depends entirely on the functional correctness of the benchmark ground truth. The verification pipeline explicitly validates only source consistency with the CTS, build success, and non-empty rendering; it never validates data-binding semantics or interaction behavior against the original Tableau workbook. Because the CTS is itself produced from Tableau workbooks through a pipeline that includes LLM-based generation, a wrong CTS field mapping, filter scope, or interaction definition will be faithfully implemented by the reference code and pass all three checks. The Data and Int. metrics then measure agreement with a possibly flawed reference, not correctness. This directly threatens construct validity, which is more fundamental than the separate reporting inconsistency in the 10% subset (72 base vs. 31 derived labeled 'Full Benchmark'); that inconsistency affects the quantitative headline but not the qualitative claim. The reader's weakest_assumption identified the same ground-truth concern, and I agree. A targeted audit is necessary and sufficient to resolve it. Since the concern is real but testable, and the reader already conditioned on it, the verdict remains CONDITIONAL (no change).","tokens_in":19132,"tokens_out":5250,"duration_ms":59333,"concrete_test":"Audit a random sample of the 92 base interfaces (e.g., 20) by independently reconstructing expected data bindings and interaction episodes from the original Tableau .twb files—either manually or using Tableau's own SDK. For each reference implementation: (1) extract rendered data values per view and compare against the Tableau workbook's data extract and aggregations; (2) replay each interaction episode and compare post-conditions against the workbook's defined dashboard actions. If any reference contains data-binding or interaction errors, repair or replace those references, then recompute Data and Int. scores for all five models. If the Static−Data/Int. gap shrinks substantially (e.g., Data scores rise by >10 points), the central claim is unsupported; if the gap persists, the claim survives this audit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MV-Bench's central claim—that current MLLMs reproduce visual appearance but fail at coordinated data binding and interaction logic—rests on comparing model outputs against reference implementations that are themselves generated by GLM 4.7 and accepted after three checks (Sec. 3.4.1, 3.4.2, 3.6.1). Those checks are: source (fields specified in CTS are referenced, data file loads), build (install/lint/type/build pass), and render (screenshots show expected views non-empty). None verifies that the rendered data values are semantically correct relative to the original Tableau workbook, nor that interaction actions produce the correct observable cross-view effects. Since the CTS is the grounding artifact, any parsing/translation error in TSS/CTS (e.g., an incorrectly inferred filter scope, a mis-mapped field, or an omitted interaction hook) is inherited by the reference interface and passes verification. The Data metric (Eq. 3) compares rendered values against those of the reference U; a model that reproduces the same wrong binding scores high. Similarly, Int. (Eq. 4) rewards models that implement the same missing or miswired interactions. The only fidelity check against the original Tableau dashboards is visual (SSIM 0.7042, Sec. 3.5.3), which does not test data or interaction correctness. Thus the observed gap could be inflated if the references themselves contain the data/coordination failures attributed to the models. The paper's Limitations section does not flag this ground-truth verification gap, despite asserting that instances are 'verified.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MV-Bench, a benchmark for evaluating multimodal large language models (MLLMs) on generating coordinated multi-view interfaces from screenshots, data, and interaction specifications. Tableau workbooks from the DMiner corpus are converted via two intermediate representations (TSS, CTS) into executable React/TypeScript/D3 code. The benchmark claims 92 base interfaces and 956 derived samples (1,048 total). Five state-of-the-art MLLMs are evaluated under single-pass and repair conditions with metrics for static visual fidelity, data binding correctness, and interaction completeness. The headline finding is a consistent gap: models achieve 67.9–76.4% static fidelity but only 0.9–21.7% data binding and 0–11.7% interaction completeness. The paper argues this gap reflects a fundamental capability limitation rather than an artifact of the single-pass setting, since three repair rounds raise executability but not the functional scores.","tokens_in":19501,"tokens_out":7283,"duration_ms":71591,"significance":"If the benchmark and its ground truth are valid, MV-Bench fills a real gap: existing image-to-code evaluations largely ignore data semantics and cross-view coordination. The metrics go beyond visual similarity by extracting rendered data values and replaying interaction episodes, and the pipeline from Tableau workbooks to executable code is a useful contribution. The consistent visual-vs-functional gap across five models and both benchmark partitions is an important empirical observation for the visualization and MLLM communities. However, the significance is conditional on two issues that are load-bearing for the quantitative claims: the reported results appear to come from a 103-instance subset rather than the claimed 1,048 instances, and the reference ground truth is validated only at the source/build/render level, not for data-binding or interaction semantics.","major_comments":[{"comment":"The paper states the benchmark comprises 92 base interfaces and 956 derived samples (1,048 total, Sec. 3.5.3), but Table 1 reports evaluation on '72 Base Interfaces, 31 Derived Samples' and labels the combined 103 instances as 'Full Benchmark.' The headline numbers (e.g., Kimi 75.45% Static, 21.71% Data, 11.68% Int.) therefore rest on a subset containing 78% of base interfaces but only 3.2% of derived samples, not on the full 1,048 instances. No selection protocol for this subset is described. The paper must either evaluate on the full benchmark or clearly rename Table 1 as a subset and revise the abstract, conclusion, and all 'full benchmark' claims accordingly.","section":"Sec. 5 / Table 1; Sec. 3.5.3"},{"comment":"Reference implementations are generated by GLM 4.7 and accepted after three checks: source (CTS fields referenced, data file loads), build (lint/type/build pass), and render (views present and non-empty). None of these verifies that rendered data values are semantically correct relative to the original Tableau workbook, nor that interaction actions produce correct cross-view effects. The only fidelity check against the original Tableau dashboards is visual (SSIM 0.7042, Sec. 3.5.3). Consequently, Sdata and Sint measure agreement with an LLM-generated reference artifact, not functional correctness. This circularity risk is not acknowledged in the Limitations section. The authors should add independent validation of reference data bindings and interaction behavior (e.g., human audit or automated extraction from .twb) and report error rates.","section":"Sec. 3.4.2, 3.6.1, Eqs. (3)-(4)"},{"comment":"The 'representative 10% subset' is unbalanced: 72/92 base (78%) and 31/956 derived (3.2%). The aggregate Full Benchmark scores average these 72+31 instances per interface, giving base interfaces about 70% weight, whereas the full benchmark has only ~8.8% base. Because Data and Int. scores differ substantially between partitions (e.g., Kimi Data: 20.04% base vs. 36.05% derived), the subset composition can materially change aggregate results. The selection protocol (random seed, stratification criteria) must be specified, and results should be reported with per-partition weighting or on the full 1,048 instances.","section":"Sec. 5 intro; Sec. 5.6"}],"minor_comments":[{"comment":"The Data metric should specify how DOM-extracted values are matched (tolerances for numerical values, handling of axis scales or coordinate transforms). Without this detail, the metric is not fully reproducible.","section":"Sec. 4.2, Eq. (3)"},{"comment":"Consider reporting confidence intervals or bootstrap estimates for the reported scores, especially given the small subset and the acknowledged API nondeterminism (Sec. 6.2).","section":"Sec. 5 / Table 1"},{"comment":"The average SSIM of 0.7042 between reference implementations and original Tableau dashboards indicates nontrivial visual divergence. The paper should discuss whether this limits the use of these references as ground truth for Static scores.","section":"Sec. 3.5.3"},{"comment":"The abstract and conclusion repeat the 1,048-instance claim without noting that the evaluation reported in Table 1 was performed on a 103-instance subset. These statements need to be reconciled with the revised Table 1 presentation.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The benchmark idea and the three-way evaluation metrics are valuable, and the paper is generally well-written. However, the two central issues — the mismatch between the claimed 1,048-instance benchmark and the 103-instance evaluation, and the lack of semantic validation of the LLM-generated ground truth — are load-bearing for the quantitative claims. Both are fixable in revision, but they require substantial additional validation work rather than minor edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuinely useful benchmark, but the headline numbers are not what they appear to be. The core contribution—a pipeline that turns Tableau workbooks into executable multi-view web interfaces with screenshots, data, and interaction annotations—is real, and the qualitative finding that models reproduce visual layouts far better than data bindings or interaction logic is probably real too. The TSS/CTS intermediate representation, the recomposition strategy, and the three-way metric design are all sensible, and shipping the code and data is the right move. The repair experiments are a nice check: repair raises executability but not data/interaction fidelity, which strengthens the capability-limitation story.\n\nThat said, two issues need to be fixed before I'd trust the quantitative claims. First, the reporting is internally inconsistent. Section 5 says the results are on a \"representative 10% subset\" with 72 Base Interfaces and 31 Derived Samples. But the full benchmark is 92 base and 956 derived. A true 10% subset would be roughly 9 base and 96 derived. So the \"Full Benchmark\" columns in Table 1 are not the full benchmark—they are the combined 103-instance subset. The abstract's 75.45%/21.71%/11.68% numbers come from this mislabeled subset, not the full 1,048 instances. That's a factual error in reporting, not a nit.\n\nSecond, the ground-truth verification gap is real. The reference implementations are generated by GLM 4.7 and accepted after source/build/render checks. None of those checks verifies that rendered data values match the original Tableau workbook, and none verifies that interactions produce the correct cross-view effects. The Data and Int. metrics compare against these references, so a model that reproduces a wrong binding can score high, while a model that implements correct behavior missing from the reference scores low. The stress-test note is on target. This is a moderate-to-serious flaw: it doesn't sink the benchmark, but it means the magnitude of the visual-vs-functional gap is uncertain. The Limitations section does not flag this, even though it asserts instances are \"verified.\"\n\nThe same-family circularity (reference generated by GLM 4.7, evaluated model GLM 4.6V) is less worrying than the missing semantic verification. The absence of error bars is minor; the authors note temperature=0 and API nondeterminism.\n\nWho should read this: visualization researchers and people building MLLM code-generation benchmarks. It's a useful measurement instrument, not yet an authoritative ranking. I would send it to peer review and ask for major revision: correct the subset/full-benchmark reporting, add semantic verification against the Tableau workbooks or soften the \"verified\" language, and report variance. If those are addressed, I'd cite and use it.","headline":"Useful new benchmark for coordinated multi-view interface generation, but the headline numbers come from a mislabeled subset and the reference ground truth is only verified visually, not semantically.","tokens_in":19968,"tokens_out":4862,"would_cite":true,"duration_ms":54698,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current AI models can copy how a dashboard looks, but they cannot wire up the data and cross-view interactions that make it work.","keywords":["multimodal large language models","coordinated multiple views","benchmark","image-to-code generation","data binding","interaction completeness","Tableau","visualization code generation"],"falsifier":"Take a random subset of the reference interfaces, have a human expert verify the data bindings and interaction behavior against the original Tableau workbooks, and re-run the interaction episodes; if a substantial fraction of reference interfaces fail their own interaction tests, the scores would no longer support the capability-limit claim. Conversely, if a model that is explicitly given the CTS specification rather than only the screenshot achieves high Data and Int. scores, the gap may be a prompting artifact rather than a capability limit.","tokens_in":19032,"feed_emoji":"📊","tokens_out":4625,"duration_ms":49228,"temperature":0.7,"pith_summary":"This paper claims that today's multimodal large language models, when asked to turn a screenshot of a coordinated multi-view dashboard into working code, can reproduce how the dashboard looks but largely fail at the two things that make it functional: binding the right data values to each view, and implementing the cross-view interactions (filtering, highlighting, brushing) that link views together. To test this, the authors built MV-Bench, a benchmark of 1,048 verified dashboard instances generated from Tableau workbook files through a pipeline that converts structured specifications into executable web code with data and interaction annotations. On the strongest model, visual fidelity reached 75.45%, while data binding scored 21.71% and interaction completeness 11.68%; the gap persisted even after up to three rounds of error-driven repair. The paper argues this is a capability limit, not a single-attempt artifact, and that visual appearance is an unreliable proxy for functional correctness in MLLM-generated interfaces.","feed_headline":"75% looks, 12% behavior: AI dashboards don't work yet","feed_subtitle":"MV-Bench's 1,048 verified dashboards show the visual gap is real: even repair rounds don't close data-binding and interaction scores.","key_machinery":"The benchmark's engine is a multi-stage pipeline that turns Tableau workbook XML into executable web applications via two structured intermediate representations: the Tableau Structured Specification (TSS), which normalizes authoring intent, and the Code Translation Specification (CTS), which resolves rendering commitments into execution directives. Reference code is produced by an LLM agent and accepted only after source, build, and render validation; the 92 accepted base interfaces are then recombined across chart types, datasets, and interaction patterns under interaction-preservation rules to yield 1,048 verified instances. Evaluation decomposes into three automatic metrics—Static (visua","core_discovery":"Using Tableau workbook files as ground truth because they explicitly encode data bindings and interactions, MV-Bench provides an image-to-code benchmark for coordinated multi-view interface construction. The central finding is a consistent, large separation between static visual fidelity and functional correctness across all five evaluated models: every model scores much higher on reproducing the appearance of the reference image than on matching its data values or replaying its interaction episodes, and this separation survives iterative repair. The paper attributes the gap to representation asymmetry—layout is visible in the input image while data semantics and interaction logic are latent","pith_inferences":["If the benchmark's ground truth is itself LLM-generated and only checked for source, build, and render validity, then the Data and Int. scores could partially measure agreement with an LLM's own errors rather than functional correctness; a hand-verified subset would settle this.","The same pipeline could be turned into a training signal: because data-binding and interaction failures are localized in the code, a future system could use MV-Bench's DOM-level checks as reward signals for reinforcement learning on cross-view reasoning.","The paper's framing suggests a concrete testable extension: evaluate MLLMs on declarative targets such as Vega-Lite where views, selections, and links are compositional—if the gap persists there, it points to a reasoning deficit rather than an imperative-coding artifact."],"forward_implications":["Any tool that evaluates or ranks MLLM code generation for dashboards using only screenshots will systematically overstate capability; correctness must be verified by executing the generated code and checking data and behavior.","The visual-versus-functional gap, persistent across models and repair rounds, indicates that simply scaling current models or giving them more attempts will not close the interaction and data-binding hole; new training objectives or architectures that model shared state across views are needed.","MV-Bench's replayable interaction episodes and sandboxed execution can serve as a feedback signal for iterative generation, letting a repair loop target data-binding and interaction failures rather than just compile errors.","Because the benchmark is built from Tableau dashboards with a fixed React/TypeScript/D3 stack, its results describe this stack; retargeting the pipeline to declarative grammars is left as future work but would test whether the gap is stack-specific."],"fun_headline_variants":["AI nails dashboard looks, flops on data logic","MV-Bench: Vision models see, but don't connect","Dashboards that look right but act wrong","Why AI dashboards look perfect but break","Tableau-based benchmark exposes AI's dashboard blind spot"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's ground-truth interfaces are generated by an LLM and validated only for whether the code builds and renders, not for whether the data bindings and interaction behavior are actually correct, so the Data and Int. scores rest on the assumption that this reference generation preserves the original Tableau semantics.","fun_headline_variants_meta":{"raw":{"variants":["AI nails dashboard looks, flops on data logic","MV-Bench: Vision models see, but don't connect","Dashboards that look right but act wrong","Why AI dashboards look perfect but break","Tableau-based benchmark exposes AI's dashboard blind spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2479,"prompt_tokens":810,"completion_tokens":1669,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1609}},"tokens_in":554,"tokens_out":1669,"duration_ms":12562,"temperature":1.0,"reasoning_tokens":1609,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:19:11.803465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of the reference interfaces, have a human expert verify the data bindings and interaction behavior against the original Tableau workbooks, and re-run the interaction episodes; if a substantial fraction of reference interfaces fail their own interaction tests, the scores would no longer support the capability-limit claim. Conversely, if a model that is explicitly given the CTS specification rather than only the screenshot achieves high Data and Int. scores, the gap may be a prompting artifact rather than a capability limit.","supporting_citations":[],"review_version":1}