{"id":"5b89b4fe-10c1-4910-95ee-59e51c37618f","arxiv_id":"2608.03689","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-agent, adaptive benchmark framework for evaluating LLM-generated frontend projects, with a 100-query benchmark, an 11-model leaderboard, and 86% agreement with human UI ratings.","lead":"LiveEvalBench is a new automated system that grades AI-generated websites by having three AI agents act as build engineer, code reviewer, and browser-based user tester. It aims to replace static screenshot benchmarks with a more interactive, adaptive evaluation that matches human expert judgment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive UI check decomposition biases interaction scores against feature-rich implementations, threatening the central 'interaction is biggest weakness' finding.","rationale":"The central empirical claim has two parts: (1) LiveEvalBench provides a human-aligned measure of web generation ability, and (2) current frontier models lose the most points on interactive runtime behavior. Both depend on the adaptive protocol's implementation-grounded checks being fair across diverse artifacts. The protocol is not a small implementation detail: Table 6's 'one check per interactive element, capped at five' and A.3's strict 'any single task fails, the entire criterion is considered failed' create a structural coupling between implementation complexity and failure probability. A feature-rich implementation can fail Interaction Experience because one of five checks fails, while a sparse implementation gets one check and passes. This concern is reinforced by the paper's own failure case (Figure 11), where the UI Tester reached an incorrect verdict even on a single-button hover check, showing that check-level noise is real. The human-agreement study validates the UI Tester against human judgment on the same adaptive items, but it does not validate that the number of adaptive items is comparable across models. A fixed-checklist re-scoring or a check-count-controlled regression would settle whether the 'interaction is the biggest weakness' finding is robust. This concern does not invalidate the framework's design or the benchmark's utility; it means the headline empirical claim is conditional on check-count fairness, which is exactly the reader's CONDITIONAL verdict.","tokens_in":39190,"tokens_out":5378,"duration_ms":58335,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that implementation-grounded check counts are a fair basis for cross-model comparison. The UI Tester's decomposition policy is 'one check per interactive element, capped at five' (Table 6), and scoring is strict: 'if any single task fails, the entire criterion is considered failed' (A.3). A project with five interactive elements therefore receives five independent failure opportunities on Interaction Experience, while a project with one element receives one; any one failure forfeits the full criterion. This systematically penalizes feature-rich implementations relative to minimal ones on the exact dimension where the paper reports the largest losses (Fig. 5). The leaderboard gap on interaction (e.g., Claude Opus 4.7 at 32.74 vs Qwen3.5-27B at 19.79) could partly reflect how many checks the adaptive planner generated, not solely interaction quality. Human-agreement validation (Table 2) rates the same criteria after the same check decomposition, so it does not independently validate fairness across implementations with different element counts. Without controlling for check count, the central finding that 'current frontier models fail most often on interactive runtime behavior' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LiveEvalBench, a multi-agent evaluation framework for LLM-generated frontend projects. It deploys three agents — Build Engineer, Code Engineer, and UI Tester — that examine deployment, source code, and runtime interaction, using an adaptive protocol that combines query-level shared criteria with implementation-grounded checks per artifact. The framework is demonstrated on 100 curated queries across 11 frontier models, producing a 90-point weighted score (Build 15, Code 30, UI 45). The authors report that current models lose the most points on interactive runtime behavior, that the framework aligns closely with human majority judgments on UI criteria (85.7% agreement, AC1=0.793), and that role-separated evaluation yields a graded score distribution whereas a single-evaluator ablation collapses to a bimodal 0/90 distribution.","tokens_in":39507,"tokens_out":4553,"duration_ms":45913,"significance":"If validated, LiveEvalBench addresses a real gap: existing web-generation benchmarks rely on static or template-bound evaluation, while frontend artifacts are interactive, open-ended, and evolve rapidly. The extensible evaluator-configuration schema and the adaptive check-decomposition protocol are useful and novel contributions, and the benchmark of 100 queries with 11 models is a substantial resource. The paper ships its code, prompts, and configuration schemas, which supports reproducibility and community reuse. However, two load-bearing issues must be resolved: the fairness of the adaptive check-count policy for cross-model comparison, and the scope of the human-alignment claim. The loss-attribution finding also depends on the hand-chosen weighting scheme. These are addressable through additional analyses and qualifications, but they currently prevent the central empirical claims from being accepted at face value.","major_comments":[{"comment":"Adaptive check counts confound interaction scores. The UI Tester's check_decomposition_policy is 'one check per interactive element, capped at five,' and scoring uses strict aggregation: 'if any single task fails, the entire criterion is considered failed.' A project with five interactive elements therefore receives five independent failure opportunities on Interaction Experience, while a project with one element receives one; any single failure forfeits the full 45-point criterion. This systematically penalizes feature-rich implementations on the exact dimension where the paper reports the largest losses. Human-agreement validation (Table 2) rates the same check decomposition and thus does not control for check-count variation. Please report per-check pass rates, normalize by element count, or include check count as a covariate, and show that the cross-model interaction ranking survives","section":"§A.3, Table 6, Fig. 5"},{"comment":"The human-alignment claim is overgeneralized. Human judgments are collected only for the UI Tester's criteria; the Build and Code dimensions are never validated against human raters. No human-human agreement is reported as an upper bound, so AC1=0.793 is difficult to interpret. The abstract/conclusion state that LiveEvalBench 'aligns closely with human expert judgment' without qualifying that this holds for the UI dimension only. Either collect human labels for Build and Code dimensions, or explicitly restrict the claim to UI, and report human-human agreement to contextualize the chance-corrected coefficient.","section":"Table 2, Abstract, Conclusion"},{"comment":"Loss attribution is not normalized by dimension weight. Fig. 5 reports raw lost points on a 90-point scale where Build has a maximum of 15 points, Code 30, and UI 45. UI losses therefore have up to three times the dynamic range of Build losses. The conclusion that 'UI behavior matters most' is partly a consequence of the chosen weights (UI = 45 of 90 points) rather than an independent empirical finding. Report per-dimension loss as a fraction of that dimension's maximum (e.g., 40% of UI points lost vs. 20% of Build points lost), or show the ordering of relative loss rates; otherwise the headline finding is not robust to the weighting scheme.","section":"Fig. 5, §Main Benchmark Evaluation"}],"minor_comments":[{"comment":"The text says 'The remaining 59 queries follow the same structure' but the benchmark contains 100 queries; this should be 'the remaining 99' or restated.","section":"§B.1"},{"comment":"The OCR/rendering of the figure text contains garbled strings such as 'modile user' (typo for 'mobile user') and long runs of 'c'/'w' characters. Figures need to be regenerated with clean text.","section":"Fig. 2"},{"comment":"'Design Arena 2025' is cited in the text and in the table but does not appear in the reference list; please add a proper reference.","section":"References"},{"comment":"The evaluator-model agreement uses 'Qwen3.7-Plus', which is not in the 11-model list evaluated in the main benchmark. Please clarify the model's provenance and whether it is a separate evaluator model.","section":"§Reliability Evaluation"},{"comment":"Tables 3 and 8 report the same evaluator-role ablation with the same numbers; consider merging to avoid redundancy.","section":"Tables 3 and 8"}],"recommendation":"major_revision","confidential_remarks":"The framework and benchmark are potentially valuable contributions, and the code release is a strong plus. The central empirical claims are currently vulnerable to the check-count confound and the weighting artifact, and the human-alignment scope is narrower than claimed. The requested analyses (per-check rates, normalized loss, human labels for Build/Code) should be feasible within the manuscript's scope and would substantially strengthen the paper. If the authors decline to provide them, the claims should be substantially diluted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LiveEvalBench is a solid, well-documented framework for evaluating web generation, and the role-separated jury is a genuine improvement over single-judge pipelines. The paper's empirical finding—that models lose the most points on interactive runtime behavior—is plausible, but I would not yet take the leaderboard at face value. The adaptive check policy can penalize feature-rich implementations, and that directly affects the interaction-loss measurement.\n\nWhat is actually new: the combination of a Build Engineer, Code Engineer, and UI Tester, each with distinct tools and evidence channels; implementation-grounded checks that adapt to the artifact; and an evaluator schema where adding a role is just a JSON config. The benchmark itself has 100 real-world queries across categories and difficulty levels, which is a useful resource. The human-agreement numbers for the UI Tester are decent (85.7% agreement, AC1=0.793), and the reliability analysis shows repeated evaluation reduces variance.\n\nThe soft spots are proportionate to their size. First, the abstract claims alignment with 'human expert judgment,' but only the UI Tester was validated against humans. Build and Code dimensions have no human ground truth, so the claim is broader than the evidence. Second, the check-decomposition policy is the real issue. The UI Tester uses 'one check per interactive element, capped at five,' and if any check fails the whole criterion fails. A project with five interactive elements has five chances to fail; a minimal project with one has one. This systematically disadvantages richer implementations on the exact dimension where the paper reports the largest losses. The human-agreement study uses the same decomposition, so it does not validate fairness across implementations with different element counts. Without controlling for check count, the 'interaction is the biggest weakness' finding could be partly an artifact of the policy. Third, the leaderboard has no uncertainty intervals, and the code/data are not yet available despite the abstract claiming a GitHub link—the supplementary says release upon publication.\n\nNone of this is fatal. The framework is a real contribution and the benchmark is useful. The authors need to add a robustness check (e.g., normalize by check count or show the result holds for a fixed number of checks) and validate more dimensions against humans. I would send this to peer review.","headline":"LiveEvalBench is a solid, reusable evaluation framework for web generation; the interaction-loss finding is plausible but the adaptive check policy needs a fairness check.","tokens_in":39965,"tokens_out":2878,"would_cite":true,"duration_ms":30159,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LiveEvalBench claims that LLM-generated websites must be judged by live, agentic review, and that on its 100-query benchmark, frontier models lose the most points on interactive UI behavior.","keywords":["web generation evaluation","LLM-as-judge","multi-agent evaluation","frontend code generation","adaptive evaluation protocol","interactive UI testing","benchmark construction","browser automation"],"falsifier":"Count the rate at which the UI Tester performs a state-changing action (like hover or click) without first capturing a baseline observation — the paper's own failure case documents one such false negative — then re-run the 100-query evaluation with per-criterion check counts normalized across projects. If interaction-experience losses shrink or the leaderboard reorders when feature-rich pages are no longer given more chances to fail than minimal ones, the central 'interaction is the biggest weakness' result is an artifact of check decomposition; if the gap persists, the finding is robust.","tokens_in":39145,"feed_emoji":"🖱️","tokens_out":7028,"duration_ms":64574,"temperature":0.7,"pith_summary":"LiveEvalBench argues that LLM-generated websites should be evaluated the way they are actually used — deployed, read as code, and clicked through a live browser — rather than scored in one static pass. To do this it assembles a three-agent jury (Build Engineer, Code Engineer, UI Tester) whose shared rubrics stay fixed across models while concrete checks adapt to what each artifact actually contains, and it packages each judge as a persona/criteria/tools configuration so new perspectives can be added without pipeline redesign. On 100 real-world queries across 11 frontier models the framework reports 85.7 percent agreement with human expert judgment, and its headline finding is that current models lose the most points on interactive runtime behavior — the exact dimension static benchmarks cannot see. If correct, the framework both provides a human-aligned measurement and identifies live UI interaction, not building or coding, as the current bottleneck in web generation.","feed_headline":"Frontier models lose most points on live UI interaction","feed_subtitle":"A 90-point benchmark scores generated sites by build, code, and browser use — interaction decides the ranking.","key_machinery":"The machinery is the adaptive evaluation protocol running inside a three-agent jury. The protocol decouples what to check from how to check: criteria (build smoothness, implementation quality, instruction following, visual quality, interaction experience, plus per-query functional requirements) stay fixed and shared across all models, while the concrete checks are synthesized at runtime from each artifact — the UI Tester, for instance, inspects the rendered DOM and accessibility tree and writes one check per interactive element, capped at five, then executes hover, click, and type actions in a real browser. Aggregation is strict: any single failing check fails its whole criterion. Around thi","core_discovery":"LiveEvalBench's central claim is that web generation cannot be judged by static inspection: a frontend's quality lives in how it behaves when someone deploys it, reads its code, and clicks through the running page. On a benchmark of 100 real-world queries and 11 frontier models, the paper reports that the resulting 90-point score aligns with human expert judgment (85.7 percent overall agreement, Gwet's AC1 = 0.793), and that the decisive failure pattern is runtime interaction: UI sub-scores span the widest range (19.8 to 32.7 of 45), the bottom tier of models loses the most points on interaction, and interaction experience is the dimension with the largest per-model loss. The paper interpret","pith_inferences":["Editorial: the check-decomposition policy — one check per interactive element capped at five, with any single failure failing the whole criterion — plausibly disadvantages feature-rich implementations, so re-running with normalized check counts would test whether the interaction-failure ranking survives as a real property of models.","Editorial: the 45/90 weight on UI behavior encodes the paper's own claim that interaction is the highest bar; given that Build Smoothness correlates with Visual Quality and Interaction Experience at r about 0.74 each, the dimension weights themselves partly determine the leaderboard order.","Editorial: the persona/criteria/tools triple could generalize beyond frontends to other interactive artifact classes — chat widgets, dashboards, agent UIs — where the binding failure mode is also runtime behavior rather than static output.","Editorial: the documented hover-feedback false negative (the judge ruled 'no hover effect' without ever capturing a non-hovered baseline) implies the current framework may undercount passing interaction checks; a baseline-forcing rule would likely raise reported UI scores across all models."],"forward_implications":["The main finding — interactive runtime behavior, not build success or code quality, is where frontier models shed the most points — directs improvement effort at the live UI layer, which carries half the score.","Shared rubrics plus implementation-grounded checks let one benchmark cover open-ended queries with many valid layouts, so new real-world requests can be added without rewriting judgment logic.","New evaluation perspectives (touch-only mobile user, color-blind user, accessibility-tree user) reduce to filling in a persona/criteria/tools configuration, letting the benchmark track evolving frontend capabilities.","Repeated three-run evaluation with majority voting cuts per-query score variability (mean SD 23.81 to 17.62), and two different evaluator models agree 90.1 percent (Cohen's kappa = 0.641), supporting the framework's reliability claims.","The single-evaluator ablation collapses to binary 0/90 scores while the three-role design spreads scores across the scale, indicating that role separation is what produces graded diagnostic signal."],"supporting_citations":[{"why":"ArtifactsBench: supplies the six task categories the benchmark adapts and the scripted-interaction checklist approach whose rigidity LiveEvalBench argues against; also a seed source for queries.","marker":"Zhang et al. 2025"},{"why":"Design2Code: the representative visual-reproduction benchmark that treats web generation as matching a target design, the contrast case for adaptive scoring.","marker":"Si et al. 2025"},{"why":"WebCompass: the closest prior agent-as-judge browser evaluation with synthesized test cases, which the paper extends with multi-role separation and extensibility.","marker":"Lei et al. 2026"},{"why":"Agent-as-a-Judge: supplies the tool-using evaluator mechanism that LiveEvalBench generalizes into three specialized agents.","marker":"Zhuge et al. 2024"},{"why":"Replacing Judges with Juries: motivates replacing one large judge with a panel of heterogeneous judges.","marker":"Verga et al. 2024"},{"why":"Web-Bench: example of sandboxed predefined-script functional evaluation whose fixed tests bind judgments to task templates.","marker":"Xu et al. 2025"},{"why":"FrontendBench: representative frontend benchmark with predefined scripts and visual checks, used as a comparison point in the limitations argument.","marker":"Zhu et al. 2025"}],"fun_headline_variants":["Interaction, not static code, decides web model quality","LiveEvalBench: UI interaction is the great separator","Runtime interaction exposes biggest gaps in web models","Static checks can't catch the web UI failures that matter","Where frontier models lose: live browser interaction"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The framework's scores assume that turning a criterion into concrete checks — one check per interactive element, capped at five, with any single failing check failing the whole criterion — treats minimal and feature-rich implementations equally, so the finding that interaction is models' biggest weakness could partly be an artifact of how checks are counted rather than a real property of the models.","fun_headline_variants_meta":{"raw":{"variants":["Interaction, not static code, decides web model quality","LiveEvalBench: UI interaction is the great separator","Runtime interaction exposes biggest gaps in web models","Static checks can't catch the web UI failures that matter","Where frontier models lose: live browser interaction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":2943,"prompt_tokens":752,"completion_tokens":2191,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2127}},"tokens_in":496,"tokens_out":2191,"duration_ms":16116,"temperature":1.0,"reasoning_tokens":2127,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:43:19.023257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the rate at which the UI Tester performs a state-changing action (like hover or click) without first capturing a baseline observation — the paper's own failure case documents one such false negative — then re-run the 100-query evaluation with per-criterion check counts normalized across projects. If interaction-experience losses shrink or the leaderboard reorders when feature-rich pages are no longer given more chances to fail than minimal ones, the central 'interaction is the biggest weakness' result is an artifact of check decomposition; if the gap persists, the finding is robust.","supporting_citations":[],"review_version":1}