{"id":"e95293c5-aada-48b2-ac6e-d46b94770883","arxiv_id":"2608.10567","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DashArena evaluates LLM-generated interactive dashboards by replaying each model's own interaction walkthrough in a browser and judging the evidence with a distilled VLM judge, showing that current frontier models frequently generate non-functional or analytically weak dashboards.","lead":"The authors introduce DashArena, a benchmark that tests whether AI models can build interactive data dashboards from a data file and a written goal, and also asks each model to provide a recorded walkthrough of how its dashboard should be used.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human calibration shares the trajectory evidence with the judge, so high agreement may measure trajectory persuasiveness rather than dashboard utility; a free-exploration study is needed to validate the protocol.","rationale":"The paper is well-engineered and provides genuine evidence: a deterministic browser executor, ablations showing interaction evidence matters, a 100-case audit with step-level attribution, aggregation robustness checks, and a released open-weight judge. My concern is not that the protocol is fake, but that its validity argument has a circularity at the calibration step. Human annotators receive the same trajectory and execution report as the judge, so the 79.8% agreement demonstrates inter-rater reliability about the self-description, not independent verification of the artifact. The audit partially mitigates this by probing unauthored controls, but it measures control mechanics, not analytical completeness; a trajectory can be technically accurate yet omit the most important analysis. This is load-bearing because the benchmark's novelty—and its claimed superiority over static or execution-only checks—rests on the assumption that trajectories are trustworthy windows into a dashboard's intended workflow. If that assumption fails, the leaderboard partly measures persuasive writing. I do not think this warrants rejection: the audit and ablations reduce the likelihood of gross manipulation, and the proposed free-exploration study, while labor-intensive, is feasible with the released artifacts. The reader's CONDITIONAL verdict is therefore appropriate; no verdict change is needed.","tokens_in":12498,"tokens_out":5701,"duration_ms":61344,"concrete_test":"Recruit a fresh panel of dashboard experts (distinct from the six annotators) and have each freely explore 40–60 renderable model-generated dashboards from the held-out leaderboard tasks in a browser, without seeing any model-authored trajectory, then collect pairwise preferences on the same task-pairs used in calibration. Compare these free-exploration majority preferences with DashJudge-8B's labels. If agreement falls materially below the reported 79.8% (the trajectory-present calibration) or the rank correlation drops well below the reported tau=0.857 with Gemini, the current human calibration is confounded by shared trajectory evidence and cannot validate the protocol as a measure of dashboard utility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DashArena yields a valid measurement of interactive dashboard generation. Validity rests on the premise that a model-authored interaction trajectory faithfully represents the dashboard's actual analytical support. The paper's strongest evidence for this premise is the 100-pair human calibration: annotators inspect the task, field schema, screenshots, and interaction trajectory evidence—exactly the inputs the judge receives. High agreement (79.8%, kappa 0.600) therefore shows that DashJudge-8B matches humans in judging the persuasiveness of the submitted trajectory, not necessarily the utility of the underlying dashboard. The 100-case audit partially addresses concealment by probing unauthored controls, but it checks mechanical control functionality and page coverage, not whether the walkthrough's analytical narrative is complete or faithful. Omitted analytical paths that are never mentioned in the trajectory are invisible to both humans and the judge in the calibration protocol. Consequently, the benchmark's central measurement may conflate writing a plausible walkthrough with building a useful dashboard. This risk is not contradicted by face validity: the 73.3% best-model replay rate shows the protocol is not trivially gameable, but the specific failure mode of selective omission remains untested. The human baseline's Claude-authored trajectory is an additional weakness, but the trajectory-fidelity issue is the load-bearing one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DashArena is a benchmark for open-ended, task-grounded generation of interactive analytic dashboards. The authors construct 234 tasks from human-authored Tableau Public dashboards, ask each evaluated model to produce a single-file HTML dashboard plus a structured interaction trajectory, replay the trajectory in a Playwright executor to obtain deterministic execution evidence, and use a VLM judge (distilled into the open-weight DashJudge-8B from a Claude Opus 4.6 teacher) to make pairwise preference judgments aggregated with a Bradley–Terry model. The paper reports a human calibration study (99 evaluable pairs; 79.8% agreement, Cohen's κ=0.600), an ablation showing that interaction evidence improves judge–human agreement by 8.1 points, deterministic-rule comparisons (42.4% agreement), a 100-candidate trajectory-coverage audit, an 841-step failure audit, counterfactual control probes, and leaderboard robustness checks under alternative aggregation rules and task resampling. On 120 held-out tasks, GPT-5.5 ranks first, followed by the human baseline and GLM-5.2, with Grok 4.3 last; the best render and replay rates are 85.8% and 73.3%, respectively.","tokens_in":12665,"tokens_out":10219,"duration_ms":95897,"significance":"The contribution is substantial if the central validity claim holds. DashArena appears to be the first benchmark that evaluates open-ended interactive dashboard generation with reproducible runtime evidence, and the release of DashJudge-8B, tasks, and an anonymized human baseline makes the protocol usable by others. The paper's empirical effort is a strength: the human calibration is larger than typical, the trajectory ablation isolates the effect of interaction evidence, the 100-candidate audit and step-level inspection directly address the possibility of selective trajectory authorship, and the leaderboard is shown to be robust across aggregation rules and task resampling. The deterministic renderable/replayable metrics and the qualitative failure analysis also provide a useful bridge between preference judgments and concrete runtime defects. The main risk is external validity: because the human calibration shares the trajectory evidence with the judge, the reported agreement may reflect agreement about the persuasiveness of a walkthrough rather than about the full analytical utility of the dashboard.","major_comments":[{"comment":"The human calibration protocol gives annotators exactly the trajectory and execution report that DashJudge-8B consumes, so the 79.8% agreement and κ=0.600 in Table 2 validate agreement about the submitted walkthrough, not necessarily about the full analytical support of the dashboard. The audit in Table 4 shows that authored trajectories cover 80.8% of controls and that unauthored controls respond comparably when probed, but it does not test whether omitted analytical paths would change a preference; a cherry-picked trajectory can still omit the most demanding comparisons while satisfying the coverage metrics. I request either a free-exploration sub-study in which annotators interact with the dashboards directly, or a sensitivity check in which a stratified subset is re-judged using a second, independently authored trajectory. Without such a check, the load-bearing claim that DashArena measures dashboard utility rather than trajectory persuasiveness remains under-supported.","section":"Do Model-Authored Trajectories Test Their Dashboards?"},{"comment":"Claude Opus 4.6 is used to author task seeds, to generate the human-baseline trajectory, to teach DashJudge-8B, and as an evaluated candidate. The human baseline is therefore not a matched-protocol human artifact, and its trajectory may encode the same model's preferences used to train the judge. Because the baseline ranks second with a confidence interval overlapping GPT-5.5 (Figure 2a), this is not purely cosmetic. Please either add a truly human-authored baseline trajectory for a subset of tasks or demonstrate that the leaderboard is unchanged when baseline trajectories are produced by a different authoring model.","section":"Human baseline / Open-weight Judge Distillation"}],"minor_comments":[{"comment":"The sentence 'One author manually verify that every generated trajectory...' contains a typo; it should read 'verifies'. The same paragraph should state explicitly that the human baseline's execution report is simulated rather than produced by replaying the Tableau workbook.","section":"Human baseline"},{"comment":"Please clarify whether the 20 judge-test tasks are included in the 120-task leaderboard and whether the 45 judge-test pairs are part of the 3,325 aggregated comparisons; the phrase 'rather than an additional split' is ambiguous.","section":"Open-weight Judge Distillation"},{"comment":"The cell symbols are garbled in the submitted text (e.g., 'Code-only! % % % %'); please render check/cross marks unambiguously and add a legend.","section":"Table 1"},{"comment":"State the maximum trajectory length and the fixed delay per action, since the audit attributes coverage gaps to the single-trajectory length limit.","section":"Interaction Trajectory and Execution"},{"comment":"Report the kappa value for the Gemini/DashJudge-8B agreement (currently only the percentage is given), because agreement percentages are sensitive to base rates.","section":"Is DashJudge-8B Reliable?"},{"comment":"The phrase 'task-clustered Bradley–Terry aggregation' is used in the Introduction, but the experiments describe confidence intervals that resample tasks; clarify whether clustering is used in model fitting or only in resampling.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical paper with unusually thorough validation for a benchmark paper. My main concern is that the current human calibration validates agreement about trajectory evidence, not about independent dashboard exploration; I would like to see a free-exploration or independent-trajectory check before acceptance. The self-referential role of Claude Opus 4.6 (seed author, teacher, baseline author, candidate) is a secondary concern that should be disclosed more prominently and, if possible, tested with an alternative baseline-authoring model. I would not reject on the current evidence, but I would not accept without addressing the trajectory-fidelity concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first benchmark I've seen that evaluates open-ended interactive dashboard generation with a replayable trajectory, a browser executor, and a VLM judge. That protocol is the real contribution, and it's a good one. The paper is unusually careful empirically - 99-pair human calibration with kappa 0.600, an 8.1-point ablation for trajectory evidence, a deterministic-rule comparison at 42.4%, a 100-candidate coverage audit, counterfactual control probes, and robustness checks across aggregation methods and task subsamples. The citation pattern is appropriate, and the Bradley-Terry aggregation with task-level bootstrap CIs is standard. This deserves to become a reference evaluation for dashboard generation.\n\nThe soft spots are real but not fatal. First, the human calibration set is small and deliberately skewed toward replayable pairs, so judge reliability on the full distribution - including non-replayable or partially broken candidates - is unproven. Second, the human baseline's trajectory is written by Claude, not by a human, which makes the 'human' reference partially model-generated; the authors acknowledge this but it still weakens the anchor. Third, and most load-bearing, is the stress-test concern: both the human annotators and the judge see the same trajectory evidence, so the 79.8% agreement may partly measure agreement about the persuasiveness of the walkthrough rather than the utility of the underlying dashboard. The coverage audit (80.8% per-page control usage, 90.1% versus 86.0% downstream response for authored versus unauthored controls, and the judge catching 53 of 55 confirmed error steps) does mitigate the worry about deliberate concealment. But the audit is mechanical and post hoc; it cannot rule out omitted analytical paths that are never mentioned in the trajectory. That is a genuine validity gap, not a manufactured one.\n\nWho is this for? Anyone working on LLM-generated interactive artifacts, visualization evaluation, or benchmark methodology. The paper will be useful even if the trajectory-fidelity question remains open. My recommendation: send it to peer review. A serious referee should engage, and the expected revision should include either a free-exploration validation study or a broader calibration set that includes non-replayable and partially broken candidates. With that, the benchmark would be a solid foundation.","headline":"A serious, well-engineered benchmark for open-ended dashboard generation; the trajectory-evidence protocol is genuinely novel, though the human calibration shares the trajectory with the judge, so one more validation study is needed before the validity claim is fully trusted.","tokens_in":722,"tokens_out":1309,"would_cite":true,"duration_ms":25578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that open-ended dashboard generation can be judged by replaying each model's self-declared interaction walkthrough, and presents DashArena, a 234-task benchmark, as evidence the protocol works.","keywords":["dashboard generation","interactive dashboards","LLM evaluation","open-ended benchmark","replayable interaction trajectory","vision-language judge","Bradley-Terry aggregation","DashJudge-8B"],"falsifier":"Give the same models dashboards containing deliberately planted broken controls in unseen tasks, have them author trajectories, and measure whether broken controls are underrepresented in those trajectories; if the concealment rate is substantially higher than the audit's 90.1%-versus-86.0% authored/unauthored response gap, then trajectory fidelity—and the benchmark's validity—is not assured.","tokens_in":12213,"feed_emoji":"📊","tokens_out":8025,"duration_ms":76853,"temperature":0.7,"pith_summary":"DashArena sets out to measure something earlier benchmarks do not: whether an LLM-generated dashboard genuinely supports the analysis it was asked to support, interactively, not just whether it renders or matches a template. Its answer is to make each model submit both a dashboard and a replayable interaction trajectory—a structured walkthrough of the analytical steps the model intends its dashboard to support. A browser executor replays that trajectory and records which actions succeed and which charts, controls, and texts actually change, turning a model's self-description into checkable evidence. A vision-language judge then compares two anonymous candidates on the same task using this evidence, and Bradley–Terry aggregation turns the pairwise preferences into a leaderboard. Human calibration on 99 pairs shows the open-weight judge DashJudge-8B reproduces human majority preferences at 79.8% agreement ($\\kappa=0.600$), while deterministic execution rules reach only 42.4%; the best model still replays only 73.3% of its trajectories, so the paper concludes the task is far from solved.","feed_headline":"New benchmark: best AI dashboard replays only 73% of its actions","feed_subtitle":"DashArena scores dashboards by replaying each model's own interaction walkthrough, catching failures static checks miss.","key_machinery":"The load-bearing mechanism is the replayable interaction trajectory: an ordered JSON list of steps, each specifying an action type, a target identified by a stable data-test-id, a natural-language analytical intent, and an expected visual change. A browser-based executor replays these steps, validates the schema, captures page screenshots, and summarizes which charts, controls, and text components changed after each action; only those screenshots and execution summaries are shown to the judge. Pairwise comparison by a vision-language judge—first a proprietary teacher, then the distilled open-weight DashJudge-8B—converts this evidence into preferences, and Bradley–Terry aggregation converts the preferences into the leaderboard.","core_discovery":"The paper's central claim is that interaction quality in open-ended dashboard generation can be evaluated without an exploratory agent: the model itself declares what should be attempted and what change it expects, an independent browser executor verifies what actually happens, and a vision-language judge determines whether the demonstrated behavior serves the task. This replay-before-judging protocol is validated three ways. Interaction evidence improves judge agreement by 8.1 points (from 71.7% to 79.8%) and by 0.159 in Cohen's $\\kappa$; the distilled DashJudge-8B matches a proprietary teacher's human alignment while running on open weights; and a 100-candidate audit finds that authored trajectories reach 80.8% per-page control coverage and that authors' chosen controls respond downstream 90.1% of the time versus 86.0% for unauthored controls, giving no evidence of systematic concealment. On the 120-task held-out leaderboard, no model exceeds an 85.8% render rate or a 73.3% replay rate, and manual inspection finds data-binding, analytical, and readability defects even in execution-clean candidates.","pith_inferences":["The replayable-trajectory protocol could generalize beyond dashboards to other open-ended interactive artifacts such as web apps, data-analysis notebooks, and design tools, where a model's declared usage plan can be replayed and judged.","The paper's own numbers suggest that most remaining judge disagreement lives in genuinely ambiguous pairs: agreement is 92.3% on unanimous pairs but 66.0% on two-to-one majorities, so a follow-up could model preference distributions instead of a single majority label.","Because the audit's no-concealment result is a post hoc sample of 100 candidates, a straightforward stress test would be to incentivize models to hide known failures and check whether per-page control coverage remains near 80%.","The small 90.1%-versus-86.0% downstream-response gap between authored and unauthored controls implies that omitted controls are almost as functional as demonstrated ones, so a cheaper evaluation might combine execution reports with lightweight heuristics, though the 42.4% deterministic-rule agreement warns against going fully automatic."],"forward_implications":["Future dashboard-generation benchmarks should require replayable trajectories, because static screenshots miss the 8.1-point drop in judge agreement that interaction evidence provides.","Replayability is a distinct bottleneck from rendering: the best model renders 85.8% of dashboards but replays only 73.3% of its trajectories, so interaction failures are not captured by page-load success alone.","Because deterministic execution rules reach only 42.4% human agreement, raw action success is not a substitute for semantic judgment; both execution evidence and multimodal reasoning are needed.","The leaderboard ordering is stable across Bradley–Terry, Thurstone–Mosteller, and average-win-rate aggregation, so the reported capability tiers do not depend on the choice of aggregation model.","Since DashJudge-8B is open-weight and trained on 255 pairs, the evaluation is reproducible on local hardware and can be rerun as new models appear."],"supporting_citations":[{"why":"Supplies the replay-before-judging principle that motivates generator-authored, replayable trajectories.","marker":"Luo et al. 2026"},{"why":"Provides the paired-comparison model used to aggregate pairwise judge preferences into the leaderboard.","marker":"Bradley and Terry 1952"},{"why":"Documents multimodal judge behavior and motivates the pairwise judging and distillation approach.","marker":"Chen et al. 2024"},{"why":"Provides the prior single-chart VLM-judge baseline that DashArena extends to interactive dashboards.","marker":"Chen et al. 2025"},{"why":"Gives the nvBench NL2VIS benchmark that DashArena positions itself against as an open-ended, multi-view alternative.","marker":"Luo, Tang, and Li 2021"}],"fun_headline_variants":["DashArena: AI dashboards fail replay test, max 73% success","Replay your dashboard: new benchmark exposes AI interaction flaws","AI dashboards look good but fail when replayed, DashArena shows","Replay test: frontier LLMs still fail interactive dashboards","DashArena replays AI interactions to catch dashboard failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The protocol's validity rests on the assumption that a model's self-written interaction walkthrough is a faithful demonstration of what the dashboard can actually do, not a cherry-picked tour that avoids the dashboard's weak points.","fun_headline_variants_meta":{"raw":{"variants":["DashArena: AI dashboards fail replay test, max 73% success","Replay your dashboard: new benchmark exposes AI interaction flaws","AI dashboards look good but fail when replayed, DashArena shows","Replay test: frontier LLMs still fail interactive dashboards","DashArena replays AI interactions to catch dashboard failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3576,"prompt_tokens":976,"completion_tokens":2600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2509}},"tokens_in":592,"tokens_out":2600,"duration_ms":17717,"temperature":1.0,"reasoning_tokens":2509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:39:29.806667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the same models dashboards containing deliberately planted broken controls in unseen tasks, have them author trajectories, and measure whether broken controls are underrepresented in those trajectories; if the concealment rate is substantially higher than the audit's 90.1%-versus-86.0% authored/unauthored response gap, then trajectory fidelity—and the benchmark's validity—is not assured.","supporting_citations":[],"review_version":1}