{"id":"bac2135a-5ae3-459d-98a0-0a7e61b4e3f8","arxiv_id":"2605.30000","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Cookie-Bench is a reference-free 1,000-query web development benchmark paired with Cookie-Frame, a metacognition-inspired three-stage framework (static perception, agent interaction, dynamic scoring) that aligns with human ratings on 13 frontier LLMs.","lead":"The paper introduces Cookie-Bench, a 1,000-query benchmark across 11 domains for testing LLM-generated interactive web apps, and Cookie-Frame, a three-stage AI evaluator that observes, interacts with, and scores the apps autonomously. A smart generalist might read it because scalable, human-like evaluation of front-end code is becoming a bottleneck as LLMs are used to build real web products.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The alignment claim requires that Cookie-Frame's autonomous three-stage process produces verdicts equivalent to human live-session synthesis, yet the description provides no quantitative validation of that equivalence.","rationale":"The reader's weakest_assumption directly targets the condition required for the strongest_claim to hold. The full manuscript text does not alter this because the concern is structural (absence of reference or test-suite grounding for holistic scoring) rather than a detail that additional prose would automatically resolve. No other internal inconsistency or parameter-free derivation was identified that would supersede this point.","tokens_in":1763,"tokens_out":323,"duration_ms":22288,"concrete_test":"From the human alignment study section, extract the exact number of queries rated by experts, the reported agreement metric (e.g., Cohen's kappa or Pearson r), and whether the study was blinded to model identity; if N<100 or r<0.7, the central alignment claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim rests on Cookie-Frame replicating human reasoned judgment via Static Perception, Agent-Driven Interaction (continuous screen capture), and Dynamic Scoring. The paper states the regime is reference-free and holistic, but the load-bearing step is the untested mapping from captured evidence chain to structured failure attribution that matches expert raters. Without reported correlation coefficients, inter-rater reliability, or ablation results showing that the staged separation improves fidelity over direct LLM scoring, the \"aligns closely\" assertion cannot be evaluated. The 13-LLM headroom result inherits the same risk.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces Cookie-Bench, an 11-domain, 54-leaf, 1,000-query benchmark for LLM-generated web applications spanning static and interactive tasks across difficulty tiers and languages, with prompts rewritten to avoid recall. It also presents Cookie-Frame, a reference-free three-stage framework (Static Perception, Agent-Driven Interaction via continuous screen capture, and Dynamic Scoring) grounded in Flavell's metacognitive monitoring, claiming close alignment with expert human ratings and substantial headroom across 13 frontier LLMs on interactive web generation.","tokens_in":1889,"tokens_out":473,"duration_ms":16151,"significance":"If the alignment claim holds with quantitative support, the work could supply a scalable, autonomous alternative to human-judged leaderboards for evaluating complex interactive front-end code, addressing the scalability limits of existing reference- or test-suite-based proxies while enabling holistic, reasoned verdicts.","major_comments":[{"comment":"Abstract: the central claim that Cookie-Frame 'aligns closely with expert human ratings' is asserted without any reported metrics (e.g., correlation coefficients, agreement statistics, sample sizes, or inter-rater reliability). This is load-bearing for the primary contribution, as the evaluation regime's three-stage separation and mapping from captured evidence to structured verdicts remains unvalidated against human live-session synthesis.","section":"Abstract"},{"comment":"Abstract and evaluation regime description: the assertion of 'substantial headroom across 13 frontier LLMs' inherits the same validation gap; without ablation results comparing staged scoring to direct LLM scoring or details on how the autonomous agent replicates reasoned human judgment, the headroom result cannot be assessed for robustness.","section":"Abstract"}],"minor_comments":[{"comment":"The benchmark construction mentions 'briefs rewritten to resist recall from circulated prompts,' but provides no concrete details on the rewriting process or verification method.","section":null},{"comment":"Notation for the two artifacts uses placeholder macros (\\dataname, \\framename) in the abstract; consistent naming (Cookie-Bench, Cookie-Frame) should be used throughout.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful review and for identifying the need to make quantitative support for the alignment and headroom claims explicit in the abstract. We address each point below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the abstract should report the supporting metrics rather than asserting alignment without them. The full manuscript contains a human study with Pearson correlation, sample size, and inter-rater reliability figures validating the three-stage regime against expert ratings on live sessions. We will revise the abstract to include these quantitative results so that the validation of the staged evidence-to-verdict mapping is visible at the abstract level.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim that Cookie-Frame 'aligns closely with expert human ratings' is asserted without any reported metrics (e.g., correlation coefficients, agreement statistics, sample sizes, or inter-rater reliability). This is load-bearing for the primary contribution, as the evaluation regime's three-stage separation and mapping from captured evidence to structured verdicts remains unvalidated against human live-session synthesis."},{"response":"The headroom result is obtained by running the complete Cookie-Frame pipeline (including agent-driven continuous interaction and post-evidence dynamic scoring) on the 13 models; the primary validation remains the correlation with human ratings rather than an internal ablation against direct LLM scoring. We will add a concise clarification in the abstract and evaluation section describing how the metacognition-inspired separation of perception, interaction, and scoring is intended to approximate human live-session synthesis. An explicit ablation against direct LLM scoring is not present in the current manuscript and would require additional experiments; we therefore treat this as a partial revision focused on textual clarification.","revision_made":"partial","referee_comment":"[Abstract] Abstract and evaluation regime description: the assertion of 'substantial headroom across 13 frontier LLMs' inherits the same validation gap; without ablation results comparing staged scoring to direct LLM scoring or details on how the autonomous agent replicates reasoned human judgment, the headroom result cannot be assessed for robustness."}],"tokens_in":1410,"tokens_out":451,"duration_ms":20158,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The colleague should know two things up front. First, the paper builds Cookie-Bench, a 1,000-query set spanning 11 domains and three difficulty tiers for both static and interactive web tasks, with prompts rewritten to limit leakage. Second, it pairs the benchmark with Cookie-Frame, a three-stage procedure that does static perception, then lets an agent drive continuous screen recording and interaction, and only afterward produces holistic functionality and aesthetics scores with failure attribution.\n\nWhat is actually new is the explicit separation of evidence gathering from judgment, drawing on Flavell's metacognitive monitoring to justify the staged process. The benchmark construction itself looks careful on paper: balanced across languages and difficulties, and aimed at tasks where reference implementations are impractical. That addresses a real scaling problem with human leaderboards like Arena.\n\nThe soft spot is the central claim. The abstract states that Cookie-Frame aligns closely with expert human ratings and shows headroom on 13 LLMs, yet supplies no correlation numbers, agreement statistics, sample sizes, or ablation results comparing the staged method to direct scoring. The stress-test note is right that the load-bearing step—mapping the captured evidence chain to verdicts that match live human synthesis—remains unquantified in the visible description. Without those data, the equivalence cannot be assessed, and the headroom result inherits the same uncertainty.\n\nThis paper is for researchers working on automated evaluation of LLM-generated interactive applications. A reader focused on benchmark design or agent-based testing could extract useful ideas from the framework even if the validation is incomplete. It deserves a serious referee because the problem is concrete and the approach is distinct from checklist or reference-based methods, even though the current evidence for the main result is thin. I would send it to peer review so the authors can supply the missing metrics and any internal checks.","headline":"Cookie-Bench and Cookie-Frame introduce a reference-free three-stage autonomous eval for LLM web apps, but the alignment-with-humans claim has no supporting metrics visible.","tokens_in":2393,"tokens_out":445,"would_cite":false,"duration_ms":21761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Cookie-Frame matches expert human ratings on interactive web generation without references or test suites.","keywords":["web generation","LLM evaluation","interactive benchmarks","agent-based evaluation","front-end development","reference-free evaluation","continuous screen interaction","metacognitive monitoring"],"falsifier":"A side-by-side study in which multiple expert human raters independently score the same set of generated web applications and the resulting scores diverge substantially from Cookie-Frame verdicts on a non-negligible fraction of cases.","tokens_in":2662,"feed_emoji":"🤖","tokens_out":686,"duration_ms":23847,"temperature":0.7,"pith_summary":"The paper presents Cookie-Bench, a reference-free benchmark of one thousand web-development queries across eleven domains and three difficulty levels, to test both static pages and interactive applications. It introduces Cookie-Frame, a three-stage evaluator that first forms a static impression, then lets an autonomous agent explore the live interface while recording continuous screen video and per-step screenshots, and finally issues holistic functionality and aesthetics scores with failure attribution. This regime is designed to replicate the reasoned synthesis a human reviewer performs during a live session. On the benchmark the method aligns closely with expert human ratings and identifies substantial performance gaps among thirteen frontier LLMs. The approach therefore supports scalable, autonomous evaluation of web-generation models without requiring reference implementations.","feed_headline":"Agent evaluator matches humans on 1000 web queries","feed_subtitle":"Cookie-Frame scores interactive web apps autonomously after full screen interaction and aligns with expert ratings without test suites.","key_machinery":"Cookie-Frame, the three-stage process of static perception, agent-driven interaction with continuous screen-video capture, and post-evidence dynamic scoring with structured failure attribution.","core_discovery":"Cookie-Bench supplies an 11-domain, 54-leaf, 1000-query WebDev benchmark balanced across static-presentation and interactive-application tasks; Cookie-Frame implements a metacognition-inspired regime that separates evidence accumulation (static perception plus agent-driven continuous screen interaction) from holistic judgment (dynamic scoring), achieving close alignment with expert human ratings while exposing headroom across frontier LLMs on interactive web generation.","pith_inferences":["The continuous screen-capture record could serve as training data for reward models that learn to predict human preference directly from interaction traces.","The separation of evidence accumulation from judgment may generalize to other GUI domains such as mobile or desktop application generation.","Because the benchmark resists recall of circulated prompts, repeated use of the same queries is less likely to inflate reported performance over time."],"forward_implications":["Evaluation of LLM-generated interactive web applications can proceed at scale without human judges or reference code at each iteration.","Current frontier models exhibit measurable shortfalls on both functionality and aesthetics when judged under continuous-interaction conditions.","The same reference-free regime applies equally to static presentation tasks and to dynamic application tasks.","Structured failure attribution produced after full evidence collection supplies actionable diagnostic signals for model improvement."],"fun_headline_variants":["Human-aligned agent scores 1000 web queries","Autonomous screen review for interactive LLM web code","Reference-free benchmark matches expert web ratings","Agent-driven evidence accumulation for web evaluation"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"An autonomous agent performing continuous screen interaction and later holistic scoring can replicate the reasoned synthesis a human reviewer performs over a live session without any reference implementation or test suite.","fun_headline_variants_meta":{"raw":{"variants":["Human-aligned agent scores 1000 web queries","Autonomous screen review for interactive LLM web code","Reference-free benchmark matches expert web ratings","Agent-driven evidence accumulation for web evaluation"]},"model":"grok-4.3","cost_usd":0.00718,"raw_usage":{"total_tokens":3331,"prompt_tokens":703,"num_sources_used":0,"completion_tokens":53,"cost_in_usd_ticks":71799500,"prompt_tokens_details":{"text_tokens":703,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2575,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":703,"tokens_out":53,"duration_ms":31261,"temperature":1.0,"reasoning_tokens":2575,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:21:41.504842+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A side-by-side study in which multiple expert human raters independently score the same set of generated web applications and the resulting scores diverge substantially from Cookie-Frame verdicts on a non-negligible fraction of cases.","supporting_citations":[],"review_version":1}