{"id":"43b2c266-479f-47e3-868c-33b1e75289eb","arxiv_id":"2602.00575","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A tool-augmented verifier agent that progressively inspects trajectory screenshots and probes the live environment judges GUI-agent task completion more accurately than passive LLM judges.","lead":"This paper introduces VAGEN, a verifier agent that does not just watch a GUI agent's screenshots but actively pokes the live computer to check whether a task was really completed. The method improves judging accuracy on OSWorld-Verified and AndroidWorld, and the authors use it to pick better trajectories at test time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-eval ground truth in Table 2 relabels only script-false-negatives; un-audited script-successes could include false positives, so the reported VAGEN accuracy advantage is not yet an unbiased estimate.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree with it. The paper is otherwise well-engineered: progressive verification mechanism, ablations, test-time scaling, code release, and a falsifiable theorem. The theorem in Appendix C is internally consistent for a symmetric accuracy model, though its applicability depends on the same ground truth. I do not see a correctness error in the derivation; the soft read-only constraint is a secondary concern for the scaling experiment, but the headline comparison does not depend on it. The one-sided human evaluation is the load-bearing gap because every headline number passes through it. A re-audit would either restore or remove the advantage; until then CONDITIONAL (i.e., no change from the reader) is appropriate.","tokens_in":20698,"tokens_out":4925,"duration_ms":59003,"concrete_test":"Randomly sample at least 80 script-success trajectories per actor from OSWorld-Verified (stratified by domain), anonymize screenshots/actions, and run the same dual-annotator protocol as Appendix A.2 (with disagreement defaulting to script). Compute the false-positive rate among script-successes. Then recompute Table 2 human-eval accuracy for VAGEN and all baselines using the expanded ground truth. If the VAGEN-minus-best-baseline gap changes by more than ~2 accuracy points, the headline claim is not robust; if the gap persists, the objection is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim is Table 2's human-evaluation accuracy (VAGEN 92.9% vs FullTrajEval 84.7% class-balanced; 93.4% vs 85.3% class-imbalanced). Appendix A.2 states: \"ground truth unreliability affects only those trajectories predicted as failures by the test script... we conduct manual evaluation exclusively on these trajectories.\" This one-sided audit assumes OSWorld-Verified's test scripts have no false positives among their successes. That assumption is load-bearing and unverified. If a script-marked success is actually a failure, the human ground truth keeps it as a positive; a verifier that (correctly) labels it failure is scored as an error. The direction of the bias depends on the error pattern of VAGEN versus the baselines on the unaudited script-successes, so the reported 8-point gap is not established as an unbiased accuracy estimate. The issue is not hypothetical: A.2's own examples show test-script incompleteness (e.g., task@21ab7b40, where the script only checks column C) and such incomplete checks can also yield false positives (e.g., checking existence rather than exact content). The paper provides no false-positive audit, and all downstream claims—test-time scaling, reward-guided scaling—use this ground truth (or script labels) without error bars. A concrete re-annotation of script-successes is required before the headline accuracy can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VAGEN, a verifier-agent framework for GUI task evaluation. Instead of relying on static trajectory inspection as in LLM-as-a-Judge methods, VAGEN equips a verifier LLM with tools (check screenshot, execute shell, execute python, computer use) and a progressive verification mechanism that first assesses the final screenshot, then revisits intermediate screenshots, and finally actively probes the environment for latent state evidence. The method is evaluated on OSWorld-Verified and AndroidWorld with multiple actor backbones, reporting accuracy gains over LLM-as-a-Judge baselines. The paper also proposes read-only test-time scaling for the verifier and a reward-guided Best-of-N scaling for the actor, and it derives a closed-form expression for the expected success rate under such scaling.","tokens_in":21047,"tokens_out":4613,"duration_ms":52834,"significance":"The work addresses a real bottleneck in RLVR for GUI agents: partial observability of trajectory-based verification. The proposed paradigm of agentic interactive verification is conceptually sound and, if the results hold, would be a useful contribution to reward modeling. The paper ships several concrete assets: an explicit derivation of the scaling law in Theorem 3.1, open-source code, and falsifiable claims in the form of accuracy numbers across two benchmarks and multiple backbones. The consistent improvement over the strongest LLM-as-a-Judge baseline (e.g., 83.1 vs 78.1 on OSWorld test-script accuracy) suggests that the core direction is not a mere artifact of the benchmark's known false negatives. However, the headline human-evaluation numbers are weakened by a one-sided relabeling protocol that requires careful scrutiny.","major_comments":[{"comment":"The human-evaluation ground truth used in Table 2 is constructed by annotating only trajectories that the benchmark test script predicted as failures; script-predicted successes are assumed to be correct without manual inspection. This one-sided audit is load-bearing for the headline claim (e.g., 92.9% vs 84.7% class-balanced). If the test script also has false positives among its successes—which is plausible given the paper's own examples of incomplete checks (e.g., task@21ab7b40 checks only column C)—then the human ground truth is biased. A verifier that correctly labels such a script-success as failure would be scored as an error, and the reported accuracy gap is not an unbiased estimate. I request a two-sided audit: manual evaluation of a random sample of script-successes (or at least a sensitivity analysis) to bound the potential bias. Without this, the human-evaluation numbers shou","section":"Appendix A.2, Table 2"},{"comment":"The read-only guarantee during test-time scaling is enforced only by prompt instruction and a secondary LLM check, not by a hard permission boundary. The claim that 'the environment state remains invariant' is essential for the valid serial execution of N verification samples on a single environment instance without resets. A verifier action that writes state (e.g., creates a file or changes a configuration) would silently invalidate the independence of the N samples. The paper should either hard-block write operations at the tool/environment level, reset the environment between samples, or empirically demonstrate that write attempts did not occur. At minimum, the number of observed write attempts in the scaling experiments should be reported.","section":"Section 3.6.1 and Appendix A.3"},{"comment":"All accuracy comparisons are reported as point estimates without confidence intervals or significance tests. With 361 tasks in OSWorld-Verified and the class-imbalanced setting containing relatively few positive examples, the 4–8 point gaps could be within noise. The text repeatedly uses 'significantly', but no statistical support is provided. I request that the authors report confidence intervals (e.g., Wilson intervals or bootstrap CIs) and, where appropriate, a paired test (e.g., McNemar) for the headline comparisons in Tables 2 and 3. This does not change the direction of the results, but it is necessary to assess the strength of the claim.","section":"Section 4.2, Tables 2–3"}],"minor_comments":[{"comment":"The column 'SR' for the verifier agent is ambiguous: for an actor, SR is naturally the task success rate, but for a verifier it should be defined as the verification accuracy (or something else). Please clarify in the caption or text.","section":"Table 1"},{"comment":"The prompt text says 'where the red text is applied exclusively during read-only scaling', but in the displayed prompt the restriction is not distinguished in red. This makes it hard to see which parts were used only in the scaling experiments.","section":"Appendix E.2"},{"comment":"The Progressive Verification Mechanism is described as a staged protocol, but the stages are enforced only through the prompt; the verifier agent is not constrained from skipping Stage 1 or jumping directly to probing. The ablation shows that the mechanism helps, but its implementation should be clarified—is it a prompt-level guideline or a hard control flow?","section":"Section 3.5"},{"comment":"Theorem 3.1 assumes the reward model's accuracy a is symmetric for positive and negative samples. This assumption is stated but not discussed; in practice, verifier precision and recall can differ substantially (as shown in Table 2). A brief comment on how asymmetric accuracy would affect the theorem would be useful.","section":"Appendix C"},{"comment":"The annotators are described as 'researchers in the field of GUI agents'—please specify whether they are authors or external annotators, and report their agreement beyond the Jaccard coefficient (e.g., Cohen's kappa).","section":"Appendix A.2"},{"comment":"There appears to be a LaTeX artifact in the author block: 'envel⌢pe' after the last affiliation. Please fix.","section":"Authors/affiliations"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the script-label results already provide moderate support for the central claim. The main obstacle is the one-sided human-evaluation protocol, which directly affects the headline accuracy numbers. The read-only enforcement also needs to be made rigorous. The paper is above the threshold for rejection, but below the standard for acceptance without these issues being addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid, useful paper. The core move—replacing the passive LLM judge with a tool-augmented verifier that runs a staged protocol (static check, visual retrospection, then environment probing)—is a genuine design contribution, and the experiments show consistent gains on OSWorld-Verified and AndroidWorld. Even on raw script labels, VAGEN beats the best LLM-as-a-Judge baseline by roughly 4–6 accuracy points, so the central direction is supported. The behavioral analysis is informative and the authors are candid about limitations.\n\nThe soft spot is in the human-evaluation ground truth. Appendix A.2 states that manual evaluation was performed only on trajectories the test script marked as failures. That assumes script successes are all true positives. The paper's own examples of script incompleteness make that assumption unsafe (e.g., a script that checks existence rather than content can yield false positives). If there are false positives among script-successes, the headline numbers—92.9 vs 84.7 in the class-balanced setting—are not unbiased. The bias could inflate or deflate the gap, so the size of the claimed improvement is uncertain. This is the main thing I would want fixed: a two-sided audit, or at least a sensitivity analysis that treats script-successes as uncertain.\n\nMinor issues: no error bars or significance tests are reported, so 'significantly' is doing more work than the data supports. The read-only guarantee for scaling is prompt-enforced with a secondary LLM check, not a hard permission boundary; the unrestricted results are close, so this is a caveat, not a fatal flaw. And the paper mentions active-interaction methods in the abstract but does not clearly position VAGEN against MagicGUI-RMS beyond a citation—worth a sentence in the related work.\n\nTheorem 3.1 is a straightforward closed-form derivation with assumptions stated; no issue there.\n\nBottom line: this paper deserves peer review, not a desk rejection. It addresses an important problem for GUI-agent RLVR, and the method is plausible and well engineered. I would recommend conditional acceptance, with required re-audit of the human ground truth and error bars. The core idea will survive, I think, but the numbers need to be properly supported.","headline":"Solid agentic-verifier extension, but the human-eval headline rests on a one-sided audit and should be re-checked before the numbers are trusted.","tokens_in":21515,"tokens_out":4011,"would_cite":true,"duration_ms":40787,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For GUI agents, VAGEN claims that reward modeling should be done by an interactive verifier agent that probes the environment, not a passive judge that only reads the trajectory.","keywords":["GUI agents","reward modeling","LLM-as-a-Judge","verifiable rewards","agentic verification","OSWorld-Verified","AndroidWorld","test-time scaling"],"falsifier":"Annotate every trajectory that the benchmark test script marked as successful. If a material share of those are human-judged failures, VAGEN's human-evaluation accuracy advantage over baselines would shrink. A second check: run the verifier unrestricted inside a copy-on-write snapshot and count state-changing operations; if write operations are common, the read-only serial-scaling assumption fails.","tokens_in":20568,"feed_emoji":"🤖","tokens_out":3924,"duration_ms":42308,"temperature":0.7,"pith_summary":"This paper argues that rewards for GUI agents—the signal that tells a training or evaluation pipeline whether a task was completed—should come from an interactive verifier agent that can inspect the environment directly, not from a passive judge reading the actor's trajectory. Because GUI tasks are often 'easy to verify but hard to solve,' a verifier with tools can check the final state of the system (files, settings, shopping carts) instead of guessing from screenshots. VAGEN implements this with a progressive protocol: first judge from the final screen and an operation summary, then look back at key screenshots, and if still uncertain, actively probe the live environment. On OSWorld-Verified, it raises evaluation accuracy from about 85% to roughly 93% against human ground truth, in both balanced and imbalanced success-failure settings, and reaches about 92–93% on AndroidWorld. A read-only test-time scaling variant and a Best-of-N actor-selection strategy both improve success rates, suggesting the same verifier can power both evaluation and training.","feed_headline":"Proactive verifier agents lift GUI evaluation to 93%","feed_subtitle":"Agentic checking that inspects screenshots and probes system state beats passive judging on two GUI benchmarks.","key_machinery":"The Progressive Verification Mechanism: a staged protocol (static assessment → visual retrospection → proactive probing) that lets the verifier agent escalate from the cheapest evidence (final screenshot plus consolidated operations) to the most expensive (shell, Python, and direct GUI interaction) only when needed. The load-bearing pieces are the four verifier tools—check screenshot, execute shell, execute python, and computer use—and the trajectory memory consolidation that strips the actor's subjective reasoning, leaving only factual operations for the verifier to audit.","core_discovery":"The central claim is that partial state observability, not model capability, is the main bottleneck in LLM-as-a-Judge reward models for GUI agents: screenshots and reasoning traces often do not contain the decisive evidence of task completion, which lives in latent system state. VAGEN turns the reward model into a tool-augmented verifier agent—able to re-inspect screenshots, run shell and Python checks, and perform GUI actions—and guides it with a Progressive Verification Mechanism from cheap static assessment to costly proactive probing. On OSWorld-Verified, with human-evaluation labels, VAGEN reaches 92.9% accuracy in the class-balanced setting and 93.4% in the class-imbalanced setting, ve","pith_inferences":["If the benchmark's test script also contains false positives among its success labels, the reported human-evaluation accuracy advantage is optimistic, since the paper only re-labels script failures and leaves script successes untouched.","Because 'read-only' is enforced by prompt instruction and a secondary LLM check rather than a hard permission boundary, a deployed verifier could perform state-altering actions, so production use would need mandatory sandboxing or filesystem-level read-only enforcement.","The same verification trajectories (tool choices, probes, final judgments) could be used as training data to distill a cheaper specialized verifier, something the paper lists as future work.","VAGEN's successful interactive verification suggests that outcome-level verification can serve as a foundation for step-level credit assignment by locating which actions changed the checked evidence."],"forward_implications":["Reward signals for RLVR-style training of GUI agents become more reliable, since true task completion is checked against the environment rather than inferred from partial visual traces.","On OSWorld-Verified and AndroidWorld, VAGEN raises evaluation accuracy above 90% across different actor models, including a weaker 7B actor, in both class-balanced and imbalanced settings.","The read-only scaling result implies a verifier can be sampled multiple times on one environment instance without state resets, making test-time majority voting practical.","The Best-of-N theorem and experiments show that actor success rate improves with verifier accuracy, and VAGEN reaches baseline performance with fewer samples (N=5 vs N=8)."],"fun_headline_variants":["GUI reward models get proactive: VAGEN hits 93% accuracy","Verifier agent probes hidden state, beats passive judging","From screenshots to system state: VAGEN verifies GUI tasks","Progressive verification lifts GUI agent evaluation to 93%","VAGEN: reward model that inspects beyond the trajectory"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported accuracy gains rest on the belief that the benchmark's test script only ever wrongly marks real successes as failures, so fixing only those labels is enough; if script-marked successes also contain failures, the advantage shrinks.","fun_headline_variants_meta":{"raw":{"variants":["GUI reward models get proactive: VAGEN hits 93% accuracy","Verifier agent probes hidden state, beats passive judging","From screenshots to system state: VAGEN verifies GUI tasks","Progressive verification lifts GUI agent evaluation to 93%","VAGEN: reward model that inspects beyond the trajectory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3154,"prompt_tokens":719,"completion_tokens":2435,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":2362}},"tokens_in":463,"tokens_out":2435,"duration_ms":18588,"temperature":1.0,"reasoning_tokens":2362,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:56:52.155475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate every trajectory that the benchmark test script marked as successful. If a material share of those are human-judged failures, VAGEN's human-evaluation accuracy advantage over baselines would shrink. A second check: run the verifier unrestricted inside a copy-on-write snapshot and count state-changing operations; if write operations are common, the read-only serial-scaling assumption fails.","supporting_citations":[],"review_version":1}