Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

T0 review · 2 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cookie-Frame matches expert human ratings on interactive web generation without references or test suites.

desk verdict Cookie-Bench and Cookie-Frame introduce a reference-free three-stage autonomous eval for LLM web apps, but the alignment-with-humans claim has no supporting metrics visible. read the letter →

arxiv 2605.30000 v2 pith:JO5N76H6 submitted 2026-05-28 cs.AI

Haoyue Yang , Zhangxiao Shen , Fan Ding , Hangting Lou , Yifeng Kou , Haoqing Yu , Jingyao Li , Zhengfan Wu
show 3 more authors
Siqi Bao Jing Liu Hua Wu
This is my paper · ORCID
classification cs.AI
keywords webgenerationLLMevaluationinteractivebenchmarksagent-basedfront-enddevelopmentreference-freecontinuousscreeninteractionmetacognitivemonitoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents Cookie-Bench, a reference-free benchmark of one thousand web-development queries across eleven domains and three difficulty levels, to test both static pages and interactive applications. It introduces Cookie-Frame, a three-stage evaluator that first forms a static impression, then lets an autonomous agent explore the live interface while recording continuous screen video and per-step screenshots, and finally issues holistic functionality and aesthetics scores with failure attribution. This regime is designed to replicate the reasoned synthesis a human reviewer performs during a live session. On the benchmark the method aligns closely with expert human ratings and identifies substantial performance gaps among thirteen frontier LLMs. The approach therefore supports scalable, autonomous evaluation of web-generation models without requiring reference implementations.

What carries the argument

Cookie-Frame, the three-stage process of static perception, agent-driven interaction with continuous screen-video capture, and post-evidence dynamic scoring with structured failure attribution.

What would settle it

A side-by-side study in which multiple expert human raters independently score the same set of generated web applications and the resulting scores diverge substantially from Cookie-Frame verdicts on a non-negligible fraction of cases.

Watch

Extended reading notes

Core claim

Cookie-Bench supplies an 11-domain, 54-leaf, 1000-query WebDev benchmark balanced across static-presentation and interactive-application tasks; Cookie-Frame implements a metacognition-inspired regime that separates evidence accumulation (static perception plus agent-driven continuous screen interaction) from holistic judgment (dynamic scoring), achieving close alignment with expert human ratings while exposing headroom across frontier LLMs on interactive web generation.

Load-bearing premise

An autonomous agent performing continuous screen interaction and later holistic scoring can replicate the reasoned synthesis a human reviewer performs over a live session without any reference implementation or test suite.

Editorial extensions

If this is right

  • Evaluation of LLM-generated interactive web applications can proceed at scale without human judges or reference code at each iteration.
  • Current frontier models exhibit measurable shortfalls on both functionality and aesthetics when judged under continuous-interaction conditions.
  • The same reference-free regime applies equally to static presentation tasks and to dynamic application tasks.
  • Structured failure attribution produced after full evidence collection supplies actionable diagnostic signals for model improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The continuous screen-capture record could serve as training data for reward models that learn to predict human preference directly from interaction traces.
  • The separation of evidence accumulation from judgment may generalize to other GUI domains such as mobile or desktop application generation.
  • Because the benchmark resists recall of circulated prompts, repeated use of the same queries is less likely to inflate reported performance over time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Cookie-Bench, an 11-domain, 54-leaf, 1,000-query benchmark for LLM-generated web applications spanning static and interactive tasks across difficulty tiers and languages, with prompts rewritten to avoid recall. It also presents Cookie-Frame, a reference-free three-stage framework (Static Perception, Agent-Driven Interaction via continuous screen capture, and Dynamic Scoring) grounded in Flavell's metacognitive monitoring, claiming close alignment with expert human ratings and substantial headroom across 13 frontier LLMs on interactive web generation.

Significance. If the alignment claim holds with quantitative support, the work could supply a scalable, autonomous alternative to human-judged leaderboards for evaluating complex interactive front-end code, addressing the scalability limits of existing reference- or test-suite-based proxies while enabling holistic, reasoned verdicts.

major comments (2)
  1. [Abstract] Abstract: the central claim that Cookie-Frame 'aligns closely with expert human ratings' is asserted without any reported metrics (e.g., correlation coefficients, agreement statistics, sample sizes, or inter-rater reliability). This is load-bearing for the primary contribution, as the evaluation regime's three-stage separation and mapping from captured evidence to structured verdicts remains unvalidated against human live-session synthesis.
  2. [Abstract] Abstract and evaluation regime description: the assertion of 'substantial headroom across 13 frontier LLMs' inherits the same validation gap; without ablation results comparing staged scoring to direct LLM scoring or details on how the autonomous agent replicates reasoned human judgment, the headroom result cannot be assessed for robustness.
minor comments (2)
  1. The benchmark construction mentions 'briefs rewritten to resist recall from circulated prompts,' but provides no concrete details on the rewriting process or verification method.
  2. Notation for the two artifacts uses placeholder macros (\dataname, \framename) in the abstract; consistent naming (Cookie-Bench, Cookie-Frame) should be used throughout.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and for identifying the need to make quantitative support for the alignment and headroom claims explicit in the abstract. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that Cookie-Frame 'aligns closely with expert human ratings' is asserted without any reported metrics (e.g., correlation coefficients, agreement statistics, sample sizes, or inter-rater reliability). This is load-bearing for the primary contribution, as the evaluation regime's three-stage separation and mapping from captured evidence to structured verdicts remains unvalidated against human live-session synthesis.

    Authors: We agree that the abstract should report the supporting metrics rather than asserting alignment without them. The full manuscript contains a human study with Pearson correlation, sample size, and inter-rater reliability figures validating the three-stage regime against expert ratings on live sessions. We will revise the abstract to include these quantitative results so that the validation of the staged evidence-to-verdict mapping is visible at the abstract level. revision: yes

  2. Referee: [Abstract] Abstract and evaluation regime description: the assertion of 'substantial headroom across 13 frontier LLMs' inherits the same validation gap; without ablation results comparing staged scoring to direct LLM scoring or details on how the autonomous agent replicates reasoned human judgment, the headroom result cannot be assessed for robustness.

    Authors: The headroom result is obtained by running the complete Cookie-Frame pipeline (including agent-driven continuous interaction and post-evidence dynamic scoring) on the 13 models; the primary validation remains the correlation with human ratings rather than an internal ablation against direct LLM scoring. We will add a concise clarification in the abstract and evaluation section describing how the metacognition-inspired separation of perception, interaction, and scoring is intended to approximate human live-session synthesis. An explicit ablation against direct LLM scoring is not present in the current manuscript and would require additional experiments; we therefore treat this as a partial revision focused on textual clarification. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected in derivation chain

full rationale

The paper presents Cookie-Bench as a new reference-free benchmark and Cookie-Frame as a three-stage evaluation process grounded explicitly in Flavell's external metacognitive monitoring concept. No equations, fitted parameters, self-citations, or ansatzes appear in the abstract or described framework. The alignment claim with human ratings is positioned as an empirical outcome rather than a quantity derived by construction from the inputs. The derivation chain introduces new artifacts without reducing any prediction or uniqueness result to its own definitions or prior self-citations.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the unverified premise that the described agent interaction plus delayed scoring produces judgments equivalent to human holistic review; no free parameters or invented entities are mentioned in the abstract.

assumptions (1)
  • domain assumption The three-stage process (Static Perception, Agent-Driven Interaction, Dynamic Scoring) grounded in Flavell's metacognitive monitoring accurately captures human reasoned synthesis over live sessions.
    Invoked when the abstract states the framework separates evidence accumulation from judgment and aligns with expert ratings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation." pith.science (2026). https://pith.science/paper/JO5N76H6

@misc{pith2026260530000,
  author       = {Pith},
  title        = {Pith review of: Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JO5N76H6}},
  note         = {Machine review of arXiv:2605.30000}
}
read the original abstract

Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale. Existing automated proxies typically lean on reference implementations, test suites, or rigid checklists, and tend to miss the reasoned synthesis a human reviewer performs over a live session. We articulate a new evaluation regime that is simultaneously reference-free, autonomously driven, and holistically reasoned, and instantiate it through two artifacts. \textbf{\dataname} is an 11-domain, 54-leaf, 1,000-query WebDev benchmark spanning both static-presentation and interactive-application tasks, balanced across three difficulty tiers and three target-language groups, with briefs rewritten to resist recall from circulated prompts. \textbf{\framename}, grounded in Flavell's metacognitive monitoring, separates evidence accumulation from judgment across three stages: Static Perception forms a first impression from passive observation; Agent-Driven Interaction explores the application autonomously while capturing continuous screen video, audio, and per-step screenshots; Dynamic Scoring issues holistic functionality and aesthetics verdicts with structured failure attribution only after the evidence chain is complete. On \dataname, \framename aligns closely with expert human ratings while surfacing substantial headroom across 13 frontier LLMs on interactive web generation. \noindenthttps://anonymous.4open.science/r/Cookie-3CE/

Figures

Figures reproduced from arXiv: 2605.30000 by the authors.

Figure 1
Figure 1. Top: Query “Super Mario” flowing through deployment, autonomous agent-driven [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Cookie-Bench data construction pipeline and dataset statistics. The upper-left shows the data construction pipeline; the lower-right shows the dataset distribution statistics. No single sourcing channel satisfies both requirements alone, so the two regimes are chosen to cover each other’s blind spots. Naturalistic queries, contributing 514 entries drawn from real-user traffic on an internal WebDev product and public… view at source ↗
Figure 3
Figure 3. Overview of Cookie. A five-stage pipeline from code to score: Install & Start deploys the generated page; Static Evaluation captures logs and a VLM-scored screenshot; Interaction runs the Cookie agent through an Observe-Think-Act loop with human-like clicks; Score Adjustment grades issues at Critical, Major, or Minor severity across Functional and Aesthetic dimensions; Overall Scoring aggregates them into the final … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Model capability landscape on Cookie-Bench. Left: Per-model Functionality–Aesthetics [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Per-category human agreement rates (%) for ablated evaluation variants on 132 queries. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Rendered screenshot of the generated Super Mario game as seen by the static verifier. The [PITH_FULL_IMAGE:figures/full_fig_p034_6.png]
Figure 7
Figure 7. Figure 7: Interaction frame captured during the agent’s gap-crossing attempt. Mario is positioned at [PITH_FULL_IMAGE:figures/full_fig_p037_7.png]
Figure 8
Figure 8. Figure 8: Per-model average scores across language, difficulty tier, and L2 category under React (top [PITH_FULL_IMAGE:figures/full_fig_p039_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ResearchStudio-Reel: Automate the Last Mile of Research from Paper to Poster, Video, and Blog

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A five-skill agent pipeline with one shared paper extractor and hard render gates produces editable posters, videos, and bilingual blogs, leading the Paper2Poster benchmark on aesthetics.

Reference graph

Works this paper leans on

95 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claude builds visuals

    Anthropic. Claude builds visuals. https://claude.com/blog/claude-builds-visuals, 2026. Ac- cessed: 2026-04-23

    2026

  2. [2]

    Claude opus 4.6

    Anthropic. Claude opus 4.6. https://www.anthropic.com/news/claude-opus-4-6 , 2026. Ac- cessed: 2026-05-01

    2026

  3. [3]

    Claude opus 4.7

    Anthropic. Claude opus 4.7. https://www.anthropic.com/news/claude-opus-4-7 , 2026. Ac- cessed: 2026-05-01

    2026

  4. [4]

    Y . Chen, M. Liu, Y . Shen, Y . Li, T. Huang, X. Fang, T. Zheng, W. Huang, C. Yang, D. Fu, et al. Iwr-bench: Can lvlms reconstruct interactive webpage from a user interaction video?arXiv preprint arXiv:2509.24709, 2025

    work page arXiv 2025

  5. [5]

    DeepSeek-V4

    DeepSeek AI. DeepSeek-V4. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/ main/DeepSeek_V4.pdf, 2026. Accessed: 2026-05-01

    2026

  6. [6]

    J. H. Flavell. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10):906, 1979

    1979

  7. [7]

    Gemini 3

    Google. Gemini 3. https://aistudio.google.com/models/gemini-3, 2026. Accessed: 2026-04- 23. 10

    2026

  8. [8]

    Gemini 3.1 Pro

    Google DeepMind. Gemini 3.1 Pro. https://deepmind.google/technologies/gemini/, 2026. Accessed: 2026-05-01

    2026

Show all 95 references
  1. [9]

    Y . Gui, Z. Li, Y . Wan, Y . Shi, H. Zhang, Y . Su, S. Dong, X. Zhou, and W. Jiang. Vision2ui: A real-world dataset with layout for code generation from ui designs.CoRR, 2024

    2024

  2. [10]

    Y . Gui, Z. Li, Y . Wan, Y . Shi, H. Zhang, B. Chen, Y . Su, D. Chen, S. Wu, X. Zhou, et al. Webcode2m: A real-world dataset for code generation from webpage designs. InProceedings of the ACM on Web Conference (WWW 2025), pages 1834–1845, 2025

    2025

  3. [11]

    H. Guo, W. Zhang, J. Chen, Y . Gu, J. Yang, J. Du, S. Cao, B. Hui, T. Liu, J. Ma, et al. Iw-bench: Evaluating large multimodal models for converting image-to-web. InFindings of the Association for Computational Linguistics: ACL 2025, pages 6449–6466, 2025

    2025

  4. [12]

    Z. He, W. Hong, Z. Yang, Z. Pan, M. Liu, X. Gu, and J. Tang. Vision2web: A hierarchical benchmark for visual website development with agent verification.arXiv preprint arXiv:2603.26648, 2026

    2026

  5. [13]

    S. Jung, A. Garcinuno, and S. Mateega. Ui-bench: A benchmark for evaluating design capabilities of ai text-to-app tools.arXiv preprint arXiv:2508.20410, 2025

    2025

  6. [14]

    F. Kong, J. Zhang, Y . Yue, C. Sun, Y . Tian, S. Feng, X. Yang, D. Wang, Y . Tian, J. Du, et al. Webtest- bench: Evaluating computer-use agents towards end-to-end automated web testing.arXiv preprint arXiv:2603.25226, 2026

    2026

  7. [15]

    P. Lai, J. Zhuang, K. Zhang, N. Xiong, S. Wang, Y . Xu, C. Chen, Y . Wang, and B. Cui. Webrenderbench: Enhancing web interface generation through layout-style consistency and reinforcement learning.arXiv preprint arXiv:2510.04097, 2025

    2025

  8. [16]

    Laurençon, L

    H. Laurençon, L. Tronchon, and V . Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset.arXiv preprint arXiv:2403.09029, 2024

    2024

  9. [17]

    X. Lei, X. Che, J. Xiong, C. Zhang, Y . Huang, C. Zhou, H. Huang, M. Liu, L. Zhu, H. Ye, J. Hao, K. Deng, Z. Zhan, H. Li, D. Li, Y . Yao, M. Sun, Z. Zhang, and J. Liu. Webcompass: Towards multimodal web coding evaluation for code language models, 2026. URLhttps://arxiv.org/abs...

    2026 arXiv

  10. [18]

    K. Q. Lin, S. Hu, L. Li, Z. Yang, L. Wang, P. Torr, and M. Z. Shou. Computer-use agents as judges for generative user interface.arXiv preprint arXiv:2511.15567, 2025

    2025

  11. [19]

    C. Liu, Y . Fu, W. Yang, Y . Zhang, and T. Xie. Webcoderbench: Benchmarking web application generation with comprehensive and interpretable evaluation metrics.arXiv preprint arXiv:2601.02430, 2026

    2026

  12. [20]

    Z. Lu, Y . Yang, H. Ren, H. Hou, H. Xiao, K. Wang, W. Shi, A. Zhou, M. Zhan, and H. Li. Webgen- bench: Evaluating llms on generating interactive and functional websites from scratch.arXiv preprint arXiv:2505.03733, 2025

    2025

  13. [21]

    Z. Lu, H. Ren, Y . Yang, K. Wang, Z. Zong, M. Zhan, and H. Li. Fullstack-agent: Enhancing agentic full-stack web coding via development-oriented testing and repository back-translation.arXiv preprint arXiv:2602.03798, 2026

    2026

  14. [22]

    Kimi K2.6.https://www.kimi.com/blog/kimi-k2-6, 2026

    Moonshot AI. Kimi K2.6.https://www.kimi.com/blog/kimi-k2-6, 2026. Accessed: 2026-05-01

    2026

  15. [23]

    Introducing GPT-5.4

    OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/ , 2026. Ac- cessed: 2026-05-01

    2026

  16. [24]

    Introducing GPT-5.4

    OpenAI. Introducing GPT-5.4. https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4/, 2026. Accessed: 2026-04-23

    2026

  17. [25]

    Z. Peng, W. Tao, X. Yin, C. Ying, Y . Luo, and Y . Guo. Playcoder: Making llm-generated gui code playable,

  18. [26]

    URLhttps://arxiv.org/abs/2604.19742

    arXiv

  19. [27]

    Robertson

    S. Robertson. Understanding inverse document frequency: on theoretical arguments for idf.Journal of documentation, 60(5):503–520, 2004

    2004

  20. [28]

    Sadowski and G

    C. Sadowski and G. Levin. Simhash: Hash-based similarity detection. Technical report, Technical report, Google, 2007

    2007

  21. [29]

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang. Design2code: Benchmarking multimodal code generation for automated front-end engineering. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human ...

    2025

  22. [30]

    H. Sun, H. W. Wang, J. Gu, L. Li, and Y . Cheng. Fullfront: Benchmarking mllms across the full front-end engineering workflow.arXiv preprint arXiv:2505.17399, 2025

    2025

  23. [31]

    Y . Wan, Y . Dong, J. Xiao, Y . Huo, W. Wang, and M. R. Lyu. Mrweb: An exploration of generating multi-page resource-aware web code from ui designs.arXiv preprint arXiv:2412.15310, 2024

    2024

  24. [32]

    Wu, Y .-H

    J. Wu, Y .-H. Peng, X. Y . A. Li, A. Swearngin, J. P. Bigham, and J. Nichols. Uiclip: a data-driven model for assessing user interface design. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–16, 2024

    2024

  25. [33]

    B. Xiao, L. Jiang, S. Huang, T. Lv, Y . Huang, X. Wu, L. Cui, and F. Wei. Code aesthetics with agentic reward feedback.arXiv preprint arXiv:2510.23272, 2025

    2025

  26. [34]

    J. Xiao, Y . Wan, Y . Huo, Z. Wang, X. Xu, W. Wang, Z. Xu, Y . Wang, and M. R. Lyu. Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)...

    2025

  27. [35]

    J. Xiao, M. Wang, M. H. Lam, Y . Wan, J. Liu, Y . Huo, and M. R. Lyu. Designbench: A comprehensive benchmark for mllm-based front-end code generation.arXiv preprint arXiv:2506.06251, 2025

    2025

  28. [36]

    Mimo V2 Pro.https://mimo.xiaomi.com/mimo-v2-pro, 2026

    Xiaomi. Mimo V2 Pro.https://mimo.xiaomi.com/mimo-v2-pro, 2026. Accessed: 2026-05-01

    2026

  29. [37]

    K. Xu, Y . Mao, X. Guan, and Z. Feng. Web-bench: A llm code benchmark based on web standards and frameworks.arXiv preprint arXiv:2505.07473, 2025

    2025

  30. [38]

    M. Xu, Z. Yang, W. Hong, L. Pan, X. Fan, Y . Wang, X. Gu, B. Xu, and J. Tang. Webvia: A web-based vision-language agentic framework for interactive and verifiable ui-to-code generation.arXiv preprint arXiv:2511.06251, 2025

    2025

  31. [39]

    S. Yun, H. Lin, R. Thushara, M. Q. Bhat, Y . Wang, Z. Jiang, M. Deng, J. Wang, T. Tao, J. Li, H. Li, P. Nakov, T. Baldwin, Z. Liu, E. P. Xing, X. Liang, and Z. Shen. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms. InAdvances in Neu...

    2024

  32. [40]

    Zhang, Y

    C. Zhang, Y . Li, C. Xu, J. Liu, A. Liu, C. Zhou, K. Deng, D. Wu, G. Huang, K. Li, et al. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation.arXiv preprint arXiv:2507.04952, 2025

    2025

  33. [41]

    Zhang, C

    Z. Zhang, C. Yu, Y . Li, C. Zhuang, L. Mo, and S. Li. Miniappbench: Evaluating the shift from text to interactive html responses in llm-powered assistants.arXiv preprint arXiv:2603.09652, 2026

    2026 arXiv

  34. [42]

    GLM-5.1.https://z.ai/blog/glm-5.1, 2026

    Zhipu AI. GLM-5.1.https://z.ai/blog/glm-5.1, 2026. Accessed: 2026-05-01

    2026

  35. [43]

    task_scenario

    H. Zhu, Y . Zhang, B. Zhao, J. Ding, S. Liu, T. Liu, D. Wang, Y . Liu, and Z. Li. Frontendbench: A benchmark for evaluating llms on front-end development via automatic evaluation.arXiv preprint arXiv:2506.13832, 2025. 12 Appendix Contents A. Cookie-Bench Benchmark Data: Supple...

    2025

  36. [44]

    Environment freeze.During the agent’s deliberation phase (between observation and action selection), the application state is paused to prevent temporal drift. This ensures that the page state the agent reasons about remains consistent with the state it subsequently acts upon,...

  37. [45]

    Multi-modal evidence capture.A continuous capture pipeline records screen video and audio streams alongside per-step screenshots throughout the interaction session. Unlike discrete snapshot- based approaches, this preserves the full temporal evolution of application behavior, ...

  38. [46]

    Human-like input simulation.Rather than issuing instantaneous programmatic inputs, the agent introduces realistic interaction rhythms: gradual mouse movements, natural typing cadence, and appropriate pauses between actions. This prevents evaluation artifacts that arise when ap...

  39. [47]

    If no visible change occurred, say so explicitly

    Primacy of Evidence: Document every action and its directly observed result. If no visible change occurred, say so explicitly

  40. [48]

    no change observed

    Anti-Stagnation: If the page state is identical for 3 consecutive observation cycles while waiting for something (e.g., a response, animation, or load), note this as "no change observed" and move on

  41. [49]

    Anti-Loop: If you find yourself repeating the same action or sequence without a new outcome, break the loop immediately and document it

  42. [50]

    If still broken, exit immediately and report it in your overall_observation -- do not attempt workarounds

    Garbled Page Handling: If the page appears garbled or unreadable (encoding issues), refresh up to 2 times. If still broken, exit immediately and report it in your overall_observation -- do not attempt workarounds

  43. [51]

    Do NOT document or comment on any programming language mismatch -- just test what is in front of you

    Programming Language: The output project is expected to be a web application (HTML, React, etc.), regardless of what the query specifies. Do NOT document or comment on any programming language mismatch -- just test what is in front of you

  44. [52]

    Page Language: Note in your overall_observation if the page language does not match the language requirement in the user query

  45. [53]

    No Scoring Required: You are NOT evaluating quality -- just testing functionality and documenting observations. II. WEB APPLICATION TESTING PROTOCOL Step 1: Test Core Functionality Test the CORE LOGIC first. Interact with the main workflow users will use the website, and exper...

  46. [54]

    Navigation: All menu items, links, breadcrumbs

  47. [55]

    Buttons: All primary and secondary buttons

  48. [56]

    Forms: Input fields, dropdowns, checkboxes, validation

  49. [57]

    Dynamic Content: Tabs, accordions, modals, tooltips

  50. [58]

    Media: Images, videos, carousels load status

  51. [59]

    Tested 3/20 product cards, all functional

    Search/Filter: Any search or filtering functionality III. GAME TESTING PROTOCOL If the application appears to be a game, apply the following protocol instead of (or in addition to) the Web Application Testing Protocol above. Non-Real-Time Interactive Games (card games, turn-ba...

  52. [60]

    Actively plan your steps, prioritize core functionality, and stop testing in time to prepare output before reaching the limit

    Step Limit: You MUST conclude and produce your final JSON output within {max_steps} steps. Actively plan your steps, prioritize core functionality, and stop testing in time to prepare output before reaching the limit

  53. [61]

    Environment Limitations (DO NOT Test -- Assume Working): - Backend/Database: Login systems, user authentication, data persistence - Third-Party APIs: LLM APIs, payment gateways, social media APIs, map services - File Operations: File upload/download functionality - Email/SMS: ...

  54. [62]

    Console Errors -- Only Report Critical Errors: Record only JavaScript errors that cause observable malfunction in the page or directly break functionality. Do NOT report: - Console warnings of any kind - Font loading failures (fonts.googleapis.com, fonts.gstatic.com) - Favicon...

  55. [63]

    Focus on Frontend Interactions: Test what's visibly interactive in the browser

  56. [64]

    Prioritize Critical Path: Test main workflow first, then secondary elements

  57. [65]

    actions_performed

    No Edge Cases: Do NOT test extreme or edge case inputs. VII. FINAL OUTPUT FORMAT Provide a comprehensive interaction summary as a JSON object. This summary will be used by a VLM evaluator to assess the application quality. Important: You MUST provide this JSON output before re...

  58. [66]

    Webpage Screenshot: For visual and layout audit

  59. [67]

    Source Code: To review logic, event handlers, and implementation quality

  60. [68]

    Original User Query: To verify if requirements and language match

  61. [69]

    Browser Console Logs & Dev Server Output: To detect hidden functional crashes or warnings. II. SCORING SYSTEM (0.0 - 8.0 SCALE) Programming language specification: The output project MUST be in html/react WHATEVER the query specified. DO NOT DEDUCT points for the difference fr...

  62. [70]

    start->interact->win/lose->restart/next

    FUNCTIONAL SCORING (REQUIREMENT-DRIVEN AUDIT) Step 1: Assess Implementation Completeness - 8.0 points: ALL user requirements fully implemented with correct logic - 7.0 points: ALL user requirements implemented, only minor features missing or error (e.g. form validation is inco...

  63. [71]

    ready to build

    AESTHETIC SCORING (DEFECT-BASED ELITE STANDARDS) STEP 1: BASELINE - 5.0 points: High-quality, clean, and modern. Standard professional work. - 3.0 points: Functional but unpolished. With issues. - 1.0 points: Raw HTML elements with no styling or only very basic CSS. - 0.0 poin...

  64. [72]

    Detailed step-by-step explanation

  65. [73]

    functional_reason

    Clear deduction breakdown with math { "functional_reason": "Step 1: [Base score, reason]. Step 2: [Instruction following audit - query alignment, elaboration, hallucination, template check, language]. Step 3: [Source code interactive elements verification]. Step 4: [Data displ...

  66. [74]

    Confirm problems found in video - these are DEFINITE issues

  67. [75]

    Do NOT dismiss code-level problems from static evaluation - unless the video explicitly proves they don't exist

  68. [76]

    Identify:

    If static evaluation identified a problem in the code, assume it exists UNLESS: - The video explicitly demonstrates the feature working correctly - The video shows the problematic code path executing without issues Your Task Carefully review the initial screenshot, the source ...

  69. [77]

    NEW problems discovered in video - issues that became apparent during interaction

  70. [78]

    could be better

    CONFIRMED code problems - issues mentioned in static evaluation that are NOT disproven by the video Focus Areas 27 Functional Problems CRITICAL Severity (suggest -2.0 or more each): - Core logic complete failure: Application crashes, infinite loop, or becomes completely unusab...

  71. [79]

    Match severity to EXACT deduction amounts: Don't guess - use the mappings above

  72. [80]

    Report NEW problems from video: Issues that became visible during interaction

  73. [81]

    Retain untested code problems: If static found issue and video didn't test it, RETAIN it

  74. [82]

    Don't penalize untested scenarios: No edge case testing = no edge case deduction

  75. [83]

    Be objective and specific: Only report observable defects matching the criteria above

  76. [84]

    functional_problems

    Check Environment Limitations: Don't penalize backend/API/file operations that can't work in test environment Output Format Return a JSON object with this structure: { "functional_problems": [ { "severity": "CRITICAL|MAJOR|MINOR", "description": "Clear description of the probl...

  77. [85]

    -1.0 for overlapping text

    No Double-Penalization (CRITICAL RULE) If static evaluation already deducted for an issue, DO NOT deduct again: - Static says "-1.0 for overlapping text" + video confirms it -> No additional deduction - Static says "-0.5 for incomplete validation" + video confirms it -> No add...

  78. [86]

    missing validation

    Untested != Broken If video didn't test something, DON'T assume it's broken: - Video shows valid form submission -> DON'T deduct for "missing validation" (wasn't tested) - Video shows 30s gameplay -> DON'T deduct for "missing win condition" (wasn't reached) - Exception: If sta...

  79. [87]

    Context Matters - Static LOW (<=3.0) + video problems -> Small additional deduction - Static HIGH (>=6.0) + video reveals critical crash -> Large deduction warranted

  80. [88]

    If page renders correctly and is relevant to the query, it should be at least 1.0 for both functional and aesthetics

    Score Floor Minimum adjusted score is 0.0 (don't go negative). If page renders correctly and is relevant to the query, it should be at least 1.0 for both functional and aesthetics

  81. [89]

    button broken

    Positive Adjustments Only if video PROVES static was wrong: - e.g. static said "button broken" but video shows it works perfectly -> Can add back deducted points - Must justify clearly

  82. [90]

    A fully implemented web page but all buttons non-functional should NOT be 1.0 for functional, suggested 3.0-4.0 - NOT suitable for game loop elements

    Deduction Restrictions - Do NOT let the final score significantly lower than its query requirement completeness - e.g. A fully implemented web page but all buttons non-functional should NOT be 1.0 for functional, suggested 3.0-4.0 - NOT suitable for game loop elements. Your Ta...

  83. [91]

    Follow the original scoring scale and deduction rules

  84. [92]

    Don't double-penalize issues already in static evaluation

  85. [93]

    Provide unified reasons that are self-contained and clear

  86. [94]

    Show clear calculation math in the reasons

  87. [95]

    adjusted_functional_score

    Be thorough, fair, and consistent with the guidelines OUTPUT FORMAT Return a JSON object with this structure: { "adjusted_functional_score": 5.5, "functional_reason": "UNIFIED reason combining original assessment and adjustments. Format: [Original static assessment summary]. [...

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.