Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

T0 review · 3 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FronTalk establishes that in multi-turn front-end code generation, leading models systematically forget earlier instructions—with forgetting rates up to 44.6% and performance drops up to 46%—and that visual feedback is consistently harder t

desk verdict Useful, carefully built benchmark, but the headline visual-vs-textual gap is confounded by the paper's own 76% vs 98% intent-preservation measurement; treat those magnitudes as provisional. read the letter →

arxiv 2601.04203 v2 pith:SRYXP6R5 submitted 2025-12-05 cs.CL cs.CVcs.LGcs.SE

Xueqing Wu , Zihan Xue , Da Yin , Shuyan Zhou , Kai-Wei Chang , Nanyun Peng , Yeming Wen This is my paper
classification cs.CLcs.CVcs.LGcs.SE
keywords front-endcodegenerationmulti-turndialoguemulti-modalfeedbackvisualinstructionfollowingwebagentevaluationforgettingbenchmarkusability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FronTalk is built around a simple observation: building a website is a conversation, and the feedback in that conversation is often a picture. The paper constructs 100 real-world multi-turn dialogues—1,000 turns and 3,676 test cases—where each turn's intent is expressed once as text and once as a sketch with annotations, then evaluates 20 models with a web agent that clicks through the generated site. Its central findings are that every leading model overwrites previously implemented features at non-trivial rates (forgetting up to 44.6%, performance drops up to 46%), and that visual instructions are consistently harder than their textual equivalents, especially for open-source models. The proposed fix, AceCoder, runs the same web agent over the candidate site, collects critiques of all past instructions, and regenerates, cutting forgetting to near zero and improving the strongest model's pass rate from 56.0% to 65.3%. If these results hold, multi-turn front-end coding needs to be evaluated interactively, and 'check your earlier work' is a concrete, cheap remedy.

What carries the argument

The central mechanism is the paired instruction formulation plus an agent-based verification loop. For each conversational turn, FronTalk provides a textual instruction and a visual instruction aimed at the same user intent. Evaluation is performed by a web agent that receives annotated screenshots of the rendered website, can click, type, scroll, and hover, and is augmented with image-manipulation tools (cropping, comparing screenshots, viewing animations) to judge fine-grained visual behavior. The forgetting rate is defined as one minus the ratio of earlier-turn functionality still passing in the final output to earlier-turn functionality that passed at the time it was built. AceCoder reus

What would settle it

Take a subset of FronTalk turns and replace the simulator-generated visual instructions with human-created annotations that match textual-level intent preservation (about 98%); if the visual-vs-textual pass-rate gap and the 24.1% open-source gap persist under matched fidelity, the visual-hardness conclusion is real, but if the gap collapses, it was an artifact of the visual simulator. A cheaper check is to compute the correlation across turns between visual intent-preservation score and the size of the visual penalty: a strong negative correlation would indicate the gap is driven by annotation

Watch

Extended reading notes

Core claim

The paper's claim, stated sympathetically, is that conversational front-end development is its own evaluation regime, distinct from single-turn code generation: each turn carries user intent that can be communicated either as text or as an annotated screenshot, and the final website must satisfy every instruction, not just the last one. Measured this way, current models fail in a characteristic pattern: they lose earlier functionality when writing later code, and they read visual feedback less reliably than text. FronTalk quantifies both: forgetting rates between 4.3% and 44.6% across 20 models, open-source models trailing proprietary ones by 12.5% on text but 24.1% on image feedback, and a

Load-bearing premise

The load-bearing premise is that each turn's textual and visual instructions are equivalent renderings of the same user intent; if the visual channel is intrinsically noisier—and the paper's own 76% versus 98% intent-preservation numbers suggest it is—then the headline 'visual feedback is harder' gap partly measures simulator quality, not model capability.

Editorial extensions

If this is right

  • Benchmarks for conversational coding should report forgetting, not just final pass rate; a website can look complete while its earlier features are dead.
  • Interactive, agent-based evaluation is needed to catch dynamic functional bugs, such as a dropdown that filters or a button that increments, that screenshot or source-code checks miss.
  • Open-source vision-language models need qualitatively better image-to-code generalization: the gap to proprietary models nearly doubles when feedback is visual rather than textual.
  • A verify-and-regenerate loop over accumulated instructions is an effective, simple mitigation for forgetting, including when the same agent powers the critique.
  • For strong models, usability failures shift from broken functions to missing implicit design conventions, such as intuitive navigation or feedback after actions, so user-experience evaluation captures something beyond test-case pass rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own validity numbers suggest the visual-vs-text gap is partly an artifact of instruction fidelity: the visual simulator preserved only 76% of test-case intent versus 98% for text. If visual instructions were generated at textual fidelity, the visual penalty for models might shrink—this is a testable prediction, not something FronTalk claims.
  • The forgetting-rate metric counts only features that were fully correct in earlier turns; regressions in partially implemented or subtly broken features are not captured, so the measured 44.6% ceiling may understate the true overwrite problem.
  • AceCoder's cost grows with every past turn because each turn re-critiques all previous instructions; a natural extension is caching or diff-based verification to re-check only the components that changed.
  • A higher-fidelity visual simulator, or human-drawn annotations, would let the benchmark separate 'models can't see' from 'the annotation didn't say what we thought it said.'
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. FronTalk is a benchmark for multi-turn, multi-modal front-end code generation. It contains 100 dialogues (1,000 turns) derived from real websites, with each turn paired with a textual and a visual instruction intended to encode the same user intent, plus 3,676 manually refined test cases. The paper proposes an agent-based evaluation framework that measures pass rate through web-agent interaction and usability through simulated first-time-user trajectories with pairwise LLM judging. Experiments on 20 models identify two main findings: (1) models forget previously implemented features across turns, with forgetting rates up to 44.6% and performance degradation up to 46%; and (2) visual feedback is consistently harder than textual feedback, with the open-source gap widening to 24.1%. The paper also proposes AceCoder, an agent-critique baseline that re-generates each turn's website after an agent verifies all past and current instructions, reporting reductions of forgetting to near zero and a GPT-4o pass-rate gain from 56.0% to 65.3%.

Significance. If the findings hold, FronTalk addresses a genuinely underexplored setting—multi-turn code generation with visual feedback—and provides a reusable infrastructure: the benchmark is carefully documented, the test cases were manually refined (validity improving from 90.1% to 94.7%), the agent-based evaluator is validated against human annotators (Cohen's kappa 0.63 on 218 pass-rate cases and 0.67 on 50 usability pairs), and the code/data are released. The forgetting phenomenon and the AceCoder baseline are interesting and potentially useful for the community. However, the central visual-vs-textual comparison is weakened by a confound that the paper itself documents: the visual instruction simulator preserves only 76% of test-case intent versus 98% for the textual simulator. The current headline magnitudes therefore conflate model capability with simulator fidelity, and the paper should be revised to address this before the visual-feedback claims can be accepted.

major comments (3)
  1. [§2.3, Table 2] The visual-vs-textual comparison is confounded by unequal intent preservation. The paper reports that the textual simulator preserves 98% of test-case intent while the visual simulator preserves only 76%. Since the final pass rate is computed against all test cases in C(i) (Eq. 1), a model with perfect visual understanding could still only reach about 76% on the visual condition. Consequently, the per-model gaps in Table 2 (e.g., GLM-4.5V 35.2 textual vs. 7.9 visual) and the headline 24.1% open-source gap reflect not only model ability to interpret visual feedback but also the simulator's failure to encode roughly a quarter of the requirements. The manual error analysis in Figure 4 does control for this on a subset, but the main table and abstract claims do not. Please recompute the visual-vs-textual comparison on the subset of test cases whose intent is preserved in the visual instructi
  2. [Eq. (2), §3.1] The forgetting rate treats any loss of a previously passing test case as forgetting. In a multi-turn development dialogue, later instructions may legitimately supersede or revise earlier requirements (e.g., changing the design of a component). As written, FR=1 - ΣPC(o_T|i_t)/ΣPC(o_t|i_t) attributes all such losses to model forgetting, with no mechanism to distinguish intentional modification from forgetting. This is load-bearing for the claim that 'forgetting rates up to 44.6%' and performance degradation 'up to 46%' occur across all leading models. Please annotate a sample of FR failures to determine how many correspond to instructions that genuinely require the earlier feature to persist, and either refine the metric or restrict the forgetting claim to test cases whose requirements are not later revised.
  3. [§4, Tables 5–6] AceCoder's evaluation is coupled to the same agent family that produces its critiques. The benchmark evaluator is a GPT-4o-powered web agent (§2.4), and AceCoder uses an agent 'powered by the same LLM' to verify instructions and generate critiques (Algorithm 2). The reported gains—for example, GPT-4o pass rate 56.0 to 65.3 and FR 21.4 to 0.4—may therefore partly reflect alignment between the critique-generating agent and the evaluating agent rather than improvement in human-verifiable functionality. The human-alignment validation in Table 3 reduces but does not eliminate this concern, since it validates the evaluator on ordinary outputs, not on outputs generated after the same agent's critiques. Please report AceCoder results with an independent judge model, or with human annotation on a random subset, to confirm that the gains persist outside the GPT-4o-agent loop.
minor comments (3)
  1. [Abstract, §6] Minor typos and inconsistencies: Abstract has 'creenshots'; Conclusion says 'evaluation of 14 models' while the abstract and experiments report 20 models; Related Work contains 'is is' and future work should be spelled 'future'; Conclusion has 'employes'. Please proofread.
  2. [Table 2] The column formatting in Table 2 is hard to read: values such as '55.0↓3.28.0' and '52.5↓2.5' run together. Please add explicit column separators and clarify whether the single-turn columns apply to both text and visual settings or only to text.
  3. [§3.2, §2.4] The usability win rate is described as 'win rate against a curated set of reference websites,' but the reference set and the pairwise comparison protocol are not fully specified in the main text. Please define the reference websites and how the win rate is aggregated, since this metric is used in Table 2 and in the AceCoder tables.

Circularity Check

1 steps flagged · score 4.0 of 10

Visual-vs-textual gap is partially forced by input construction: the paper's own simulator preserves only 76% of test-case intent in visual instructions vs 98% textual, yet the headline treats the gap as a model limitation.

  1. fitted input called prediction [Sec. 2.3 (User Simulation), Sec. 3.1 (Visual v.s. textual instructions), Eq. 1]
    "Visual simulators also exhibit high contextual faithfulness (97%) due to grounding user intents directly in rendered screenshots, but demonstrate much lower intent preservation (76%) owing to limited tool-use effectiveness in GPT-4o. ... While the two instruction types are designed to convey the same set of user intents, Table 2 shows a consistent performance degradation for most VLMs when processing visual instructions, especially for open-source VLMs."

    The headline 'visual feedback is consistently harder' is measured on inputs the paper itself shows are not informationally equivalent: the visual simulator preserves only 76% of the test-case intent, versus 98% for the textual simulator. Eq. 1 computes pass rate against the full C(i) test-case set regardless of how much of C(i) the visual instruction actually conveys, so roughly a quarter of visual-condition test cases are absent from the instruction by construction. The per-model visual gaps in Table 2 and the 24.1% open-source gap are therefore capped by input fidelity, not purely by the models' visual interpretation ability. The paper's Fig. 4 manual error analysis does control for this on a text-success/visual-fail subset, but the headline result is not reported on that controlled subs

full rationale

The benchmark is largely self-contained: test cases were manually refined, the agent evaluator was validated against human judgments (82.0% accuracy, Cohen's Kappa 62.7 for pass rate; 84.0% and 66.7 for usability), and AceCoder's gains are measured against those human-refined test cases. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result. The main circularity-adjacent issue is the visual-vs-textual comparison: the two channels are claimed to represent the same user intent, but the paper's own measurement shows the visual simulator lacks 24% of the test-case intent. This makes the visual condition information-deficient by construction, so the 'visual feedback is harder' finding and the visual AceCoder gains are partly an artifact of the simulator's fidelity rather than a pure model-capability measure. Because the paper acknowledges this limitation and because the error analysis partially controls for it, the issue is a partial confound rather than a fully forced circular result; hence a score of 4 rather than 6+.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

The benchmark rests on a chain of LLM-generated artifacts: GPT-4o proposes intents/test cases (human-refined), GPT-4o simulates users, and a GPT-4o web agent scores all models. Human validation exists but is thin relative to the dataset size (218/3,676 pass-rate cases; 50/100 usability pairs). The headline visual-vs-textual comparison is not corrected for the unequal intent preservation of the two simulators (76% vs 98%). Dataset design constants (T=10, step budgets, token limits) are hand-chosen and influence the results.

free parameters (3)
  • Turns per dialogue (T) = 10
    Hand-chosen dialogue length; sets the forgetting-context exposure and long-context load that drive the paper's main findings (Sec. 2.1, footnote 1).
  • Usability trajectory step limit = 15 steps
    The first-time-user agent is cut off at 15 actions (Appendix H, 'Step limit reached'), which determines whether a task counts as completed in the UX comparison.
  • Per-model maximum output tokens = 10,000-30,000
    Set 'empirically for each model family' (Appendix C); truncation of long outputs would depress pass rates for affected models and is not reported per model.
assumptions (5)
  • domain assumption Human-refined, GPT-4o-generated test cases are a correct decomposition of user intent
    Test cases are the ground truth for pass rate (Eq. 1). 73.4% were edited and 1.7% removed; validation on 224 cases improved validity from 90.1% to 94.7% (Sec. 2.2).
  • domain assumption A GPT-4o-driven web agent's verdict approximates human judgment for every test case
    The pass-rate evaluator is a GPT-4o agent (Sec. 2.4). Alignment was measured on only 218 of 3,676 test cases (kappa 0.63), so the remaining ~3,458 cases rest on the agent unvalidated.
  • ad hoc to paper Machine-drawn visual instructions are equivalent in quality to textual instructions
    The visual-vs-textual comparison requires this equivalence. It is contradicted by the paper's own measurement: 76% intent preservation for the visual simulator vs 98% for textual (Sec. 2.3).
  • domain assumption LLM-simulated first-time users and an LLM pairwise judge capture real user experience
    Usability win rates come from simulated users plus an LLM judge (Sec. 2.4); validated on only 50 trajectory pairs (kappa 0.67, Appendix H).
  • domain assumption The rendered browser environment faithfully executes generated HTML/CSS/JS
    All evaluation - agent interactions, screenshots, forgetting measurement - assumes a stable rendering stack; headless-browser details and viewport are not specified in the main text (Sec. 2.4, Alg. 1).
invented entities (3)
  • Visual instruction simulator (drawing-tool VLM)
    purpose: Produces the annotated-image instructions that define the paper's novel visual-feedback setting
    No external benchmark against human-drawn annotations; its own intent-preservation is only 76%, making the entity's output quality the confound in the headline visual finding (Sec. 2.3, Appendix D.2).
  • Agent-based evaluator with image-manipulation tools (Compare/ViewRaw/ViewAnimation) independent evidence
    purpose: Measures pass rate and usability by interacting with rendered websites and inspecting fine visual detail
    Has an external handle: human-alignment results are reported (acc 82.0, kappa 0.63 for pass rate) and the added value of image tools is quantified in an ablation (Table 3).
  • AceCoder agent-critique loop
    purpose: Verifies past instructions on the rendered site and augments prompts to reduce forgetting
    Claimed gains (56.0 to 65.3 on GPT-4o) are measured by the same GPT-4o agent family that generates the critiques; ablations cover only GPT-4o plus two open models (Tables 5-6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback." pith.science (2026). https://pith.science/paper/SRYXP6R5

@misc{pith2026260104203,
  author       = {Pith},
  title        = {Pith review of: FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRYXP6R5}},
  note         = {Machine review of arXiv:2601.04203}
}
read the original abstract

We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback. In front-end development, visual artifacts such as sketches, mockups and annotated creenshots are essential for conveying design intent, yet their role in multi-turn code generation remains largely unexplored. To address this gap, we focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues derived from real-world websites across diverse domains such as news, finance, and art. Each turn features both a textual instruction and an equivalent visual instruction, each representing the same user intent. To comprehensively evaluate model performance, we propose a novel agent-based evaluation framework leveraging a web agent to simulate users and explore the website, and thus measuring both functional correctness and user experience. Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature: (1) a significant forgetting issue where models overwrite previously implemented features, resulting in task failures, and (2) a persistent challenge in interpreting visual feedback, especially for open-source vision-language models (VLMs). We propose a strong baseline to tackle the forgetting issue with AceCoder, a method that critiques the implementation of every past instruction using an autonomous web agent. This approach significantly reduces forgetting to nearly zero and improves the performance by up to 9.3% (56.0% to 65.3%). Overall, we aim to provide a solid foundation for future research in front-end development and the general interaction dynamics of multi-turn, multi-modal code generation. Code and data are released at https://github.com/shirley-wu/frontalk

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good h...

  2. LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks

    cs.AI 2026-03 conditional novelty 6.0 of 10

    Expert-authored dual-use skills make LLM judges more reliable on subjective long-horizon enterprise agent tasks than LLM-authored rubrics, with human preferences confirming the main ranking boundary.

Reference graph

Works this paper leans on

68 extracted references · cited by 2 Pith papers

  1. [1]

    Subplot organization: Tools to create and manage multiple subplots usingmatplotlib’s subplot functionality

  2. [2]

    This process generates additional artifacts: the coordinates of every element on the new layout page

    Layout visualization: Renders a layout sketch, represented by minimal HTML code, into an image. This process generates additional artifacts: the coordinates of every element on the new layout page. These coordinates are provided to the model in the next turn to assist with drawing

  3. [3]

    Shape drawing: Tools for drawing various shapes usingmatplotlib

  4. [4]

    screenshot_blog-post.png

    Text annotation: A custom text annotation tool. We constrain the VLM to call our tool instead of matplotlib’s native text function. Our tool automatically organizes text annotations to minimize overlapping. The VLM will generate python scripts to draw a visual instruction image using these tools. If the generated code has errors during execution, we allow...

    2024

  5. [5]

    Local Attractions

    Started on the homepage and navigated to the "Local Attractions" page

  6. [6]

    Add to Itinerary

    Attempted to add attractions to a personalized itinerary using the "Add to Itinerary" buttons, but these actions were not executable

  7. [7]

    My Itinerary

    Accessed the "My Itinerary" page, which displayed an empty itinerary with only a "Print Itinerary" button, and no options to add notes or preferences

  8. [8]

    Add to Itinerary

    Returned to the homepage to verify if itinerary details persist across navigation, but there was no indication of any preserved itinerary details. Analysis: - The website does not currently support the full functionality of creating a personalized itinerary as instructed. - The "Add to Itinerary" buttons do not function, and there is no way to add persona...

    2025

Show all 68 references
  1. [9]

    Home," "Live,

    **Navigation**: The main navigation options ("Home," "Live," "Weather") are clear and intuitive, allowing users to easily switch between different sections of the website

  2. [10]

    **Live Section**: The live section offers video content and schedules for upcoming broadcasts, enhancing user engagement with live updates and reminders

  3. [11]

    **Sports Highlights**: This section provides accessible video highlights for major sporting events, adding value for sports enthusiasts

  4. [12]

    It also features a location-based search function, which is practical for users seeking specific weather information

    **Weather Section**: The weather section includes current conditions, forecasts, and alerts. It also features a location-based search function, which is practical for users seeking specific weather information

  5. [13]

    Overall, the website is user-friendly, with a clean layout and logical structure that facilitates easy access to news content

    **Subscription Feature**: The option to subscribe for breaking news alerts is straightforward, with clear feedback via a confirmation message upon successful subscription. Overall, the website is user-friendly, with a clean layout and logical structure that facilitates easy ac...

  6. [14]

    Each screenshot is named as screenshot_x (x=1,2,...)

    An annotated screenshot - with **numerical labels** placed in the **top-left corner** of each web element. Each screenshot is named as screenshot_x (x=1,2,...)

  7. [15]

    Your goal is to generate **ACTIONS** to perform the evaluation

    A simplified **textual representation** of the page -- including the tag names and texts for every element annotated in the screenshot. Your goal is to generate **ACTIONS** to perform the evaluation. If the information is sufficient to make a judgment, your ACTION should be: ‘...

  8. [21]

    For example, compare the screenshots before (screenshot_x) and after (screenshot_y) a hover action to test hover effect

    Use ‘Compare‘ to compare the same element’s visual display across two **different** screenshots. For example, compare the screenshots before (screenshot_x) and after (screenshot_y) a hover action to test hover effect

  9. [22]

    Use ‘ViewRaw‘ to retrieve the high-fidelity raw screenshot without annotations

  10. [23]

    To focus on a specific element, provide its numerical label

    Use ‘ViewAnimation‘ to view animated behavior when loading screenshot_x. To focus on a specific element, provide its numerical label. ### Evaluation Guidelines

  11. [24]

    Use them **only as context** to help understand the website, but **always** rely on **real interactions** with the website to make the final evaluation

    You are also given the instructions for building the website, which **MAY NOT** align with the actual website structure. Use them **only as context** to help understand the website, but **always** rely on **real interactions** with the website to make the final evaluation

  12. [25]

    If it is missing (e.g., the test condition refers to a header, but the website does not have a header), the evaluation should be FAIL

    First verify whether the feature to be tested is present on the website. If it is missing (e.g., the test condition refers to a header, but the website does not have a header), the evaluation should be FAIL

  13. [26]

    **Extensively** explore the website to locate the feature, such as by scrolling down, checking menus, or following links, until you confirm whether the feature exists

    If the feature to be tested is not immediately visible on the homepage but can be accessed from it, you must first navigate from the homepage to find it. **Extensively** explore the website to locate the feature, such as by scrolling down, checking menus, or following links, u...

  14. [27]

    After navigating to the feature, if the test condition is based on **purely static** visual features such as color and layout, do not use any more actions -- directly output ‘ANSWER‘ based on the screenshot

  15. [28]

    When a test involves **multiple steps or questions**, use ‘ANSWER‘ only **after** addressing all of them

  16. [29]

    1", "2",

    **Extensively** interact with the website to trigger behaviors relevant to the test condition. For example, when testing a search bar, test it with multiple inputs including both reasonable pseudo-data and generic entries like "1", "2", "a", "b"

  17. [30]

    To evaluate the visual design, **always** rely on additional actions to gather visual details before making the judgment

    The annotated screenshot is **NOT** what end users actually see. To evaluate the visual design, **always** rely on additional actions to gather visual details before making the judgment

  18. [31]

    Do **not** judge pass/fail outcomes based on whether the displayed data reflects real-world data

    The website uses placeholder data. Do **not** judge pass/fail outcomes based on whether the displayed data reflects real-world data

  19. [32]

    Do **not** judge pass/fail outcomes based on the **content** of these files

    The website also uses placeholder content for media (images, videos, audio, PDFs). Do **not** judge pass/fail outcomes based on the **content** of these files. However, you should evaluate their display and interactive behavior when determining pass/fail. ### Your Reply Format...

  20. [33]

    An annotated screenshot - with **numerical labels** placed in the **top-left corner** of each web element

  21. [34]

    A simplified **textual representation** of the page -- including the tag names and texts for every element annotated in the screenshot. At each step, you can choose one of the following valid action formats: ### Valid Actions Action should **STRICTLY** follow the format: - Cli...

  22. [35]

    Execute only one action per iteration

  23. [36]

    **Avoid repeating** the same action if the page does not change -- you may have chosen the wrong element or label

  24. [37]

    Pressing ‘ENTER‘ is handled automatically

    To input text, you do **not** need to click the textbox first -- just use the ‘Type‘ action. Pressing ‘ENTER‘ is handled automatically. However, you may still need to click a search button afterward to apply a filter

  25. [38]

    If no textbox is visible, consider clicking a search button to reveal it

    Clearly distinguish between textboxes and buttons -- do **not** type into a button. If no textbox is visible, consider clicking a search button to reveal it

  26. [39]

    The filename **MUST** be chosen from: ‘placeholder.png‘, ‘placeholder.mp4‘, ‘placeholder.mp3‘, or ‘placeholder.pdf‘

    To upload a file, you do **not** need to click the upload button first -- just use the ‘Upload‘ action and specify the filename. The filename **MUST** be chosen from: ‘placeholder.png‘, ‘placeholder.mp4‘, ‘placeholder.mp3‘, or ‘placeholder.pdf‘

  27. [40]

    The website uses placeholder for data and media (images, videos, audio, PDFs). ### Your Reply Format Thought: {Describe image content, then perform step-by-step reasoning} Action: {One properly formatted action} Listing 5Prompt for performingpair-wise comparisonin the evaluati...

  28. [41]

    **Journey Success:** Did a goal emerge from the user’s browsing? If so, how well did the website support their journey from discovery to potential action? Did they successfully navigate to a satisfying endpoint or did they abandon their exploration?

  29. [42]

    **Efficiency & Path:** How much effort (cognitive load, number of actions, backtracking) was required? Was the user’s path logical and fluid, or was it disjointed and filled with unnecessary steps?

  30. [43]

    **Satisfaction & Friction:** How did the user feel? Identify specific moments of frustration, confusion, delight, or ease, as indicated by their thoughts

  31. [44]

    ### Instructions Follow these steps and structure your response exactly as follows

    **Discoverability & Clarity:** How easy was it for the user to understand the site’s structure, find what’s available, and learn what actions are possible? Is the value proposition clear from the start? This is especially critical for **Exploratory** tasks. ### Instructions Fo...

  32. [45]

    **Identify Emergent Goal(s) (if any):** Briefly state the primary goal(s) the user developed during each trajectory

  33. [46]

    **Analyze Trajectory A:** Summarize the user’s experience on Website A, citing specific evidence for each of the four evaluation criteria

  34. [47]

    **Analyze Trajectory B:** Do the same for Website B

  35. [48]

    State which website performed better and why

    **Comparative Analysis:** Directly compare Website A and Website B on each of the criteria. State which website performed better and why

  36. [49]

    Formulate your final verdict as follows: VERDICT: [WIN, LOSE, or TIE] Listing 6Prompt fortextual instruction simulatorfor functionality-type user intents

    **Final Verdict & Justification:** Conclude with a verdict for Website A and a concise justification highlighting the most critical factors. Formulate your final verdict as follows: VERDICT: [WIN, LOSE, or TIE] Listing 6Prompt fortextual instruction simulatorfor functionality-...

  37. [50]

    add a button to the main menu

    **Verify component or functionality references.** These references typically involve **where** to implement the new feature, or **how** to navigate to the new feature from the homepage. * If a referenced component/functionality exists, refine vague references to be more precis...

  38. [52]

    Your refined instructions should be a short paragraph (3-6 sentences)

    **Avoid code-level details.** Do not reference class names, HTML attributes, color hex codes, or other implementation-specific identifiers. Your refined instructions should be a short paragraph (3-6 sentences). Start your refined instructions with **Response:** # Instructions ...

  39. [53]

    Use a two-column layout

    **Check for already implemented instructions.** Compare each instruction to the current state of the website. If an instruction is already implemented, remove it from the refined list. * Example: If the instruction says "Use a two-column layout" but the site already has a two-...

  40. [54]

    the button

    **Verify component or functionality references.** * If a referenced component/functionality exists, refine vague references to be more precise. * Example: Change "the button" to "the green submit button in the bottom left". * If it does **not** exist, clearly note that it must...

  41. [55]

    **Clarify vague instructions.** If an instruction is not specific enough, make it more concrete based on the actual website

  42. [56]

    screenshot_name.png

    **Avoid code-level details.** Do not reference class names, HTML attributes, color hex codes, or other implementation-specific identifiers. Your refined instructions should be a short paragraph (~5 sentences). Start your refined instructions with **Response:** # Instructions t...

  43. [57]

    Subplot organization

  44. [58]

    Layout visualization

  45. [59]

    Your code should be wrapped within ‘‘python ‘‘‘

    Text annotation You’ll generate python code to call these tools - I’ll explain the tools one by one. Your code should be wrapped within ‘‘python ‘‘‘. Make sure to import necessary tools and libraries you want to use. ## Subplot Organization When visualizing the instructions, y...

  46. [60]

    Use a dark background color with white text for the menu items

    **First subplot - always the screenshot of relevant page**. Use shapes and text annotations to link each instruction to its corresponding component. For example: * If the instruction states, "Use a dark background color with white text for the menu items", add annotations to t...

  47. [61]

    **Revised layout subplot(s)** - for layout changes, if necessary

  48. [62]

    In the first subplot, highlight the components to be moved and indicate their intended movement

  49. [63]

    In the later subplot(s), show the components in their **new positions** with shapes and text annotations - using the **same annotation color** for each component in both the original screenshot and the updated layout views for clarity

  50. [64]

    (2) For major changes, use the HTML layout visualization tool to create a simplified updated layout

    To show the updated layout: (1) If the UI changes are minor, reuse the screenshot and mark it with shapes showing the new positions. (2) For major changes, use the HTML layout visualization tool to create a simplified updated layout

  51. [65]

    **State transition subplot(s)** - for interaction-driven changes, if necessary

  52. [66]

    In the first subplot, highlight the interactive component(s) that trigger the transition and annotate interaction type

  53. [67]

    In the new subplot, show the resulting end state and clearly indicate what changed

  54. [68]

    Current Layout

    **Titles**: Each subplot should have a **clear and descriptive title** to clearly indicate the relationship and transitioning between subplots (e.g., "Current Layout", "Proposed Layout", "After Clicking ‘Submit‘")

  55. [69]

    Annotations for each page will be based on its own screenshot

    **Multiple pages**: If the instructions involve multiple pages, you can allocate one or multiple subplots for each page following the instructions above. Annotations for each page will be based on its own screenshot. You may create subplots **only in your first drawing turn**....

  56. [70]

    current_screenshot.png

    Screenshot background: If you are visualizing the existing page or interaction flow, use the screenshot: ‘ax.imshow(Image.open("current_screenshot.png"))‘

  57. [71]

    ""<HTML code here>

    HTML layout background: If you are visualizing a new layout or interface, use the layout visualization tool: ‘layout_visualization(html_code, ax)‘. After this call, wait for coordinate data before adding shapes or annotations. ## Layout Visualization To visualize the component...

  58. [72]

    **Consistent color for pairing**: Use the same ‘edgecolor‘ for the shape and ‘color‘ for the text annotation to visually link them as a pair

  59. [73]

    add shadow effect

    **Shared annotation across components**: If a single annotation (e.g., "add shadow effect") applies to multiple components, avoid repeating the same annotation for each one. Instead, draw shapes around **all relevant components** using the same distinctive color, and annotate ...

  60. [74]

    **Distinct colors for distinct meanings**: If you are applying **multiple different annotations**, use **different colors** for each group of shape/text pairs to clearly distinguish between them. ## Text Annotation To annotate text, **NEVER use ‘ax.text‘ or ‘ax.annotate‘ direc...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.