{"id":"d87d5165-3d51-45b3-9079-19352804d3d8","arxiv_id":"2607.17050","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EvoGUI-Bench uses logged web trajectories to test GUI agents on three visual state-transition tasks, and the strongest of 28 VLMs reaches only 60.4% normalized gain.","lead":"GUI agents have to understand how screens change after clicks and typing, but current tests mix that skill with many others. This paper builds a benchmark that turns recorded web sessions into three visual reasoning questions, and shows the best AI model still scores only 60 out of 100.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label-fidelity assumption in §2.1 is unvalidated; replaying sampled transitions would determine whether EvoGain's headroom is real or an artifact of noisy logged trajectories.","rationale":"I agree with the reader that logged trajectory fidelity is the load-bearing assumption. The paper's own Limitations concede dependence on existing human/expert trajectories and note that T3 cannot prove unreachability; A.14 acknowledges the 'reachable' wording is stronger than the logged-adjacency interpretation. But the deeper issue is that even logged adjacency itself is not verified: source datasets were not collected for this diagnostic purpose, and screenshot capture timing relative to action execution is outside EvoGUI's control. The manual QC in §2.4 specifically targets T3 visible-change artifacts, but T1 ordering and T2 action/value labels inherit the same risk, and the QC's exclusion rules are not quantitative. The 'no additional task-label annotation' slogan is accurate only in the narrow sense that retained labels are mechanically assigned; the filtering step is a human judgment that can remove exactly the hard cases. A replay-based audit on a sample would directly test whether the assumed ground truth holds. If it fails, EvoGain's headroom and cross-model conclusions are not interpretable. The authors have been transparent about limitations, so I would keep the reader's CONDITIONAL rather than reject; the missing piece is a concrete validation experiment.","tokens_in":15142,"tokens_out":6101,"duration_ms":60118,"concrete_test":"Stratified sample of ~100 T2 and ~100 T3 instances. Reconstruct each s_t in a browser (Playwright) using source logs, execute the logged action at the recorded target (or as close as determinable), capture the resulting screen, and compare with s_{t+1} using region-level similarity plus human verification. For each T3 distractor, attempt every single-step action visibly available from s_t (clicks, typing in a focused field, select, scroll, back) and record whether it yields the distractor. If >5% of transitions fail replay or >5% of distractors are one-step reachable, the annotation-free/headroom claims require revision; if error rates are below this threshold, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"EvoGUI's central claim — automated, annotation-free labels and 60.4 EvoGain representing substantial headroom — rests on the §2.1 assumption that each logged transition (s_t, a_t, v_t, s_{t+1}) is a faithful before/after pair caused by a_t. Mind2Web/WebLINX captures are not guaranteed to be temporally tight around the action: page loads, ads, async refreshes, or ineffective actions can make the visible change unrelated to the logged operation. §2.4's human QC filters only T3 pairs whose visible difference is dominated by such artifacts; T1 and T2 are not similarly audited, and the filter itself is a subjective exclusion step that weakens the 'no additional task-label annotation' claim. If even a few percent of transitions are mislabeled, T2/T3 accuracies are not clean measures of transition understanding, and the 'far from saturation' conclusion could be inflated by label noise rather than reflecting genuine VLM limits. Bootstrap CIs do not address this because resampling preserves the same mislabels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EvoGUI, a trajectory-derived benchmark for GUI state-transition understanding. It normalizes Mind2Web and WebLINX logs into three VQA-style probes: T1 temporal ordering, T2 inverse action/value prediction, and T3 contrastive one-step successor discrimination. Labels are generated mechanically from trajectory order and logged actions, with human QC applied to T3 pairs; no new task-label annotation is claimed after normalization. The authors evaluate 28 zero-shot VLM configurations and introduce the EvoGain index, a chance-normalized macro-average. The strongest model reaches 60.4 EvoGain, and the paper argues that model scale and GUI specialization do not reliably predict performance. The main contributions are the benchmark construction pipeline, the 3,000-instance split over 120 domains, and the systematic model comparison with bootstrap CIs, no-vision controls, and resolution ablations.","tokens_in":15358,"tokens_out":8841,"duration_ms":83172,"significance":"If the label-fidelity assumptions hold, EvoGUI is a valuable scalable diagnostic complement to end-to-end GUI-agent evaluation. The paper has several genuine strengths: no fitted quantities enter label construction, bootstrap CIs are reported, no-vision controls demonstrate dependence on screenshots, prompt templates are published verbatim, and the code is released. However, two construct-validity issues currently temper the central claims: the unvalidated assumption that logged transitions are faithful before/after pairs for T1/T2, and the explicit mismatch between T3's 'reachable' prompt language and its logged-adjacency labels, which the appendix itself concedes. The 'model scale does not predict performance' conclusion also needs stronger statistical support than selected examples.","major_comments":[{"comment":"The entire benchmark rests on the §2.1 assumption that each logged transition (s_t, a_t, v_t, s_{t+1}) is a faithful before/after pair caused by a_t. Human QC in §2.4 audits only T3 pairs; T1 and T2 inherit source-log noise without audit. If even a small fraction of transitions contain ineffective actions, async page changes, or non-contiguous screenshots, the reported T2/T3 accuracies and the 'far from saturation' conclusion (Table 2: EvoGain 60.4) are not clean measures of transition understanding. Bootstrap resampling (A.4) cannot correct systematic mislabels. Please add a validation study: replay or re-execute a random sample of source transitions, or have independent annotators judge whether the visible change is caused by the logged action, and report scores conditioned on validated versus noisy subsets.","section":"§2.1, §2.4, §3.2"},{"comment":"The T3 prompt tells the model 'Exactly one of A and B is reachable ... by a SINGLE action' and 'The other is not reachable in one step,' but the labels are logged adjacency, not execution-validated reachability. The appendix concedes: 'The T3 prompt uses “reachable” language, whereas our revised interpretation is restricted to identifying the logged adjacent successor against the sampled distractor.' This is an internal inconsistency in a load-bearing task: a model that judges true one-step reachability can be marked wrong when the distractor is actually reachable in one step from another state, and an unskilled model can exploit the prompt's assertion. Please re-run T3 with a prompt that asks for the logged immediate successor (or validate reachability), and report whether the ranking and EvoGain change.","section":"§A.14/Table 12 vs §2.2"},{"comment":"The claim that 'model scale and GUI specialization do not reliably predict performance' is supported only by selected examples (UI-TARS-1.5-7B at 25.4 EvoGain; Qwen non-monotonicity in Figure 2). With 28 configurations, a formal analysis is needed — e.g., Spearman correlation between log parameters and EvoGain within model families, or a regression with family fixed effects. As written, the examples demonstrate non-monotonicity but not the absence of reliable prediction; the abstract's phrasing overstates the evidence.","section":"§3.2 and Abstract"}],"minor_comments":[{"comment":"The T2 joint random baseline (15.0%) is an empirical average of four random runs, not the theoretical chance level (33.3% if a click with empty value is the only way to be correct). Please report the theoretical chance and the random-run variance; EvoGain values depend on this baseline.","section":"§3.1, Table 2"},{"comment":"The phrase 'no additional task-label annotation' is accurate only if human QC is understood as item filtering, not label assignment. Consider stating this explicitly in the abstract or first mention to avoid overstatement.","section":"§2.4/A.1"},{"comment":"Only top-10 bootstrap CIs are shown; for reproducibility, provide the full table or an electronic supplement.","section":"A.4"},{"comment":"The SELECT slice contains only 78 instances; per-class recall values should be flagged more visibly as high-variance.","section":"Table 5"},{"comment":"The revised interpretation of T3 should also be reflected in the task name ('One-step Reachability Discrimination') and Figure 1, not only in the appendix.","section":"A.14, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The T3 prompt mismatch is explicitly admitted in the appendix, which is a good-faith disclosure but undermines the current results. The benchmark idea is sound and the label-fidelity concern can be addressed empirically; I would not reject, but the revision needs a validation study and a corrected T3 evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on EvoGUI.\n\nWhat's new: three complementary VQA probes — temporal ordering, inverse action/value, contrastive successor discrimination — mined mechanically from Mind2Web/WebLINX trajectories. No task-label annotation after normalization. The EvoGain normalization against random baselines gives a clean cross-model ranking. The paper is careful: no-vision controls show the tasks need the screenshots, bootstrap CIs separate the top model, and the long-skip T3 slice genuinely tests one-step discrimination against plausible future states. The 28-model evaluation is thorough. I was also impressed by the limitations section — they flag the T3 'reachable' wording mismatch, the small SELECT slice, and the weak OSWorld correlation themselves.\n\nSoft spots: The biggest is the fidelity of logged transitions. T2/T3 treat (s_t, a_t, s_{t+1}) as a clean before/after pair. If source logs contain page-loads, async updates, or ineffective actions, labels inherit the noise. The human QC only filters T3 and only the most obvious cases; T1 and T2 are not audited. The paper acknowledges this in A.14 and the limitations, but I'd push for a replay or manual audit on a random sample of T2/T1 transitions to quantify how often the visible change actually matches the logged action. If a few percent are noisy, the headroom claim (60.4 EvoGain) could be slightly inflated, though not fatally — the ranking and the long-skip gap would likely survive.\n\nThe other caveat is the T3 prompt/interpretation mismatch: the model-facing text says 'reachable by a single action,' while the label is just logged adjacency. The paper is explicit about this in A.14, but it means EvoGain shouldn't be read as a clean reachability measure. It's a successor-discrimination score against sampled distractors. That's fine as long as users read the appendix.\n\nThe OSWorld comparison (5 models, mixed protocols) is honestly labeled as indicative. It's not a strong validity argument, but it's not oversold.\n\nVerdict: This is a solid, reproducible benchmark that fills a real gap — a scalable complement to end-to-end GUI agent evaluation. The central claims hold up after accounting for the acknowledged caveats. Worth a serious referee; I'd recommend acceptance with minor revisions asking for a label-fidelity audit and a prompt-wording fix.","headline":"A solid, honestly-scoped GUI state-transition benchmark with real diagnostic value; the label-fidelity caveats are disclosed and the core ranking holds up.","tokens_in":15835,"tokens_out":1586,"would_cite":true,"duration_ms":15141,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a benchmark that derives three diagnostic tasks from logged GUI trajectories without extra annotation, and shows the strongest vision-language model reaches only 60.4 EvoGain, with scale and GUI specialization failing","keywords":["GUI state-transition understanding","vision-language models","trajectory mining","temporal ordering","inverse action prediction","reachability discrimination","benchmark","visual question answering"],"falsifier":"Replay a random sample of logged transitions in a live browser or a controlled environment and check whether executing the recorded action on the before-screenshot actually yields the after-screenshot; if even a modest fraction of transitions (say more than 10%) fails this replay test, the annotation-free label derivation is unreliable. A complementary check: have human annotators view only the before and after screenshots and predict the logged action; if their accuracy is no higher than the best model's, the visual evidence does not determine the label.","tokens_in":15021,"feed_emoji":"🖥️","tokens_out":8364,"duration_ms":71866,"temperature":0.7,"pith_summary":"The paper argues that GUI agents' understanding of how actions change screens can be measured separately from end-to-end task success. It converts existing logged web-navigation trajectories into three visual question-answering tasks — reordering shuffled screenshots, predicting the action and value behind a screen change, and selecting the logged immediate successor against distractors — with all labels taken from trajectory order and recorded actions, so no extra task-level annotation is needed. Applying this to 3,000 instances across 120 domains, the strongest of 28 vision-language models reaches only 60.4 percent on a chance-adjusted index, EvoGain, and neither model scale nor GUI-specific training reliably predicts performance. If right, this gives the field a scalable diagnostic that isolates state-transition reasoning from perception, grounding, planning, and recovery.","feed_headline":"Top vision model scores just 60.4 on new GUI transition test","feed_subtitle":"No model saturates the probes; scale and GUI specialization do not predict scores, so state-transition skill needs direct measurement.","key_machinery":"The load-bearing object is the normalized trajectory record, tau = [(s_t, a_t, v_t, s_{t+1}, m_t)], which stores consecutive screenshots, the logged user action, its optional value, and metadata. From this, the miner generates all labels: T1 samples K screenshots from one trajectory and uses their chronological order as the ground truth; T2 uses the recorded action and exact typed or selected value as the answer for an adjacent screenshot pair; T3 uses the logged adjacent screenshot as the correct successor and samples one distractor from cross-trajectory, same-domain, or long-skip pools. An EvoGain index normalizes each task accuracy against random-choice baselines so the three probes can b","core_discovery":"The paper's central claim is that the three diagnostics, T1, T2, and T3, have labels that are mechanically determined by logged trajectory adjacency and action vocabulary after a normalization step, making annotation-free benchmark construction possible. It further claims that current vision-language models are far from saturating these probes: the best configuration attains 60.4 EvoGain, with 58.3 percent exact temporal ordering, 65.4 percent joint action/value prediction, and 83.6 percent pairwise successor discrimination. The paper reports that larger models and GUI-specialized models do not consistently outperform smaller or generalist counterparts, so transition-level ability is not a s","pith_inferences":["One could separate the causal-attribution component of T2 from OCR by building value-free versions (for example, click-only transitions), making the diagnostic more purely about state-change understanding.","The same probe structure could generate training data for state-transition models — not just evaluation — by using logged trajectories as weak supervision for action-conditioned next-screen prediction.","Replacing sampled distractors in T3 with execution-validated counterfactuals would upgrade it from a discrimination test to a true reachability test, resolving the limitation the paper concedes.","Because scale and specialization do not predict EvoGain, a testable extension is to train models with explicit transition-level objectives (e.g., action-conditioned frame reconstruction) and measure whether EvoGain rises more than it does with scaling."],"forward_implications":["Direct evaluation of state-transition understanding becomes a routine part of GUI-agent development, complementing end-to-end success rates.","The framework can be transferred to new GUI corpora — mobile, desktop, OS-level — by mapping operations into the normalized action vocabulary, without collecting new task labels.","The consistent gap between action-only and joint action/value accuracy identifies value recovery (OCR, binding) as a measurable bottleneck distinct from action attribution.","The difficulty of long-skip distractors shows that workflow plausibility is not the same as one-step transition knowledge, and that models should be tested on the latter directly.","A positive rank association with an end-to-end desktop benchmark across five models, if replicated on larger samples, would support using offline transition scores as cheap indicators during model development."],"fun_headline_variants":["New GUI benchmark: best model 60.4, scale doesn't help","EvoGUI: 3 probes expose GUI transition gaps, top score 60.4","Why bigger GUI models don't get smarter at state changes","Annotation-free test shows GUI agents still miss transitions","EvoGUI: top vision model 60.4, scale and specialization don't predict"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire label-generation pipeline trusts that each logged before-action-after screenshot triple is an accurate, contiguous record with no unlogged asynchronous changes (ads, page loads, refreshes) accounting for the visible difference; if the source logs contain noise, all three tasks inherit that noise.","fun_headline_variants_meta":{"raw":{"variants":["New GUI benchmark: best model 60.4, scale doesn't help","EvoGUI: 3 probes expose GUI transition gaps, top score 60.4","Why bigger GUI models don't get smarter at state changes","Annotation-free test shows GUI agents still miss transitions","EvoGUI: top vision model 60.4, scale and specialization don't predict"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1080,"prompt_tokens":709,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":453,"tokens_out":371,"duration_ms":3899,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:09:34.419524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay a random sample of logged transitions in a live browser or a controlled environment and check whether executing the recorded action on the before-screenshot actually yields the after-screenshot; if even a modest fraction of transitions (say more than 10%) fails this replay test, the annotation-free label derivation is unreliable. A complementary check: have human annotators view only the before and after screenshots and predict the logged action; if their accuracy is no higher than the best model's, the visual evidence does not determine the label.","supporting_citations":[],"review_version":1}