{"id":"fe0c3adc-a9ff-4d10-b576-c5f23b3bba68","arxiv_id":"2608.09278","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.","lead":"This paper reviews 336 papers on GUI agents and argues that the field is growing fast but lacks mature software engineering, with recovery, safety, human oversight, maintenance, and observability underdeveloped. It offers a lifecycle-based research agenda linking capability to dependable deployment.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey's gap statistics hinge on coding absence-as-evidence; without released codes or inter-rater checks, 'underdeveloped' claims may overstate real gaps.","rationale":"The reader identified the same load-bearing assumption I do: the absence-as-evidence coding rule in Section 11. The central claim that capability improvements alone cannot ensure deployment readiness is supported by statistics that count papers lacking explicit engineering signals. This is exactly the kind of interpretive coding where false negatives are likely, especially in a multi-community literature with inconsistent terminology. The survey itself concedes that lifecycle concerns 'may remain implicit in system descriptions.' The absence of released codes and inter-rater reliability makes the statistics unverifiable. My proposed check directly tests whether the 'underdevelopment' conclusion survives a more generous reading of implicit support. If it does not, the conclusion stands but with reduced force; if it does, the concern is refuted. The reader's verdict of CONDITIONAL is appropriate: the central claim is plausible and well-argued, but its empirical foundation depends on a coding decision that is not independently checkable. I see no reason to move the verdict; the conditional status already reflects this. I set verdict_should_be to UNCHANGED because my analysis does not change the reader's verdict — it reinforces the condition under which the paper would be accepted. The paper's transparency about the coding rule is a credit, but transparency about a limitation does not remove the limitation; it makes it addressable, which is precisely why the paper should remain CONDITIONAL pending release of the coding artifacts or an inter-rater study.","tokens_in":1052,"tokens_out":937,"duration_ms":25427,"concrete_test":"Release the coded corpus and coding manual, then have two independent coders re-code a random sample of 30–50 papers from the 327-paper SE-parseable subset, specifically for the signals driving RQ4: recovery/retry, human oversight, maintainability, observability, and privacy mitigation. Coders should be instructed to flag both explicit mentions and clearly implied mechanisms (e.g., a described 'ask user before irreversible action' counts as human oversight, a 'reusable module library' counts as a maintainability signal). Compute Cohen's kappa and recompute the reported prevalence percentages using the union of the original and re-coded signals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'capability improvements alone cannot ensure deployment readiness' (Abstract, Section 12) rests heavily on RQ2/RQ4 statistics showing that recovery, human escalation, maintainability, observability, and oversight are 'underdeveloped.' The coding rule, stated in Section 11, treats 'absent signals as a lack of explicit evidence under the coding scheme.' This is the load-bearing assumption: every 'missing' count (e.g., 279/327 papers without a maintenance-process signal, 251/327 without monitoring/audit, 17/327 with runtime human oversight) is interpreted as the system lacking that support. But a paper can omit explicit discussion of a mechanism while still implementing it implicitly or under different terminology. For instance, a framework that describes 'retrying a failed action' may not use the word 'recovery,' and a system with a 'confirm before sending' prompt may not be coded as human-in-the-loop. If implicit support is widespread, the reported gap percentages are inflated and the paper's headline conclusion is weakened. This concern is not merely hypothetical: the coding is interpretive, no inter-rater reliability is reported, the coded corpus is not released, and Section 11 acknowledges that lifecycle concerns 'may remain implicit in system descriptions.' The authors' internal-coherence checks do not substitute for independent verification. Because the central conclusion is a negative claim about the field's maturity, the burden is on the survey to demonstrate that absence of explicit signals corresponds to absence of engineering support, not just absence of terminology.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys 336 GUI-agent papers published or posted between January 2018 and April 2026 from a software-engineering perspective. It poses five research questions covering the research landscape, system architectures, evaluation protocols, software-lifecycle concerns, and future opportunities. The central claim is that GUI agents are closed-loop software systems whose deployment readiness depends on lifecycle engineering, and that capability improvements alone cannot ensure dependable, maintainable, secure, and deployable systems. The survey reports multi-label, functionally interpreted coding of the corpus and finds that recovery, human escalation, safety enforcement, maintainability, observability, and oversight are underdeveloped relative to perception, planning, and task-capability work, and that evaluation remains centered on task success with limited cross-protocol comparability. The paper closes with a research agenda built around execution contracts, trajectory-based testing, reproducible evaluation, and governed human oversight.","tokens_in":49764,"tokens_out":5421,"duration_ms":56619,"significance":"If its findings are accepted, this survey provides a valuable corrective to benchmark-centric assessments of GUI agents and a useful map for software-engineering research on these systems. The paper has clear strengths: a large and explicitly bounded corpus; a transparent RQ structure; consistent reporting of per-RQ denominators; multi-label coding and an explicit functional-interpretation rule for mechanisms such as verifiers, critics, and reward models; and a candid threats-to-validity section. The synthesis connecting architecture, evaluation, and lifecycle concerns is genuinely useful and goes beyond existing capability-oriented surveys. However, the headline negative claims are built on counts that treat the absence of explicit textual signals as absence of engineering support, and the paper does not currently demonstrate the reliability of that coding. The significance of the survey is therefore conditional on coding transparency and validation, which are not yet provided.","major_comments":[{"comment":"The central negative findings are load-bearing on the coding rule stated in §11: 'we treat absent signals as a lack of explicit evidence under the coding scheme.' For example, §7.3 reports that 279/327 papers lack a maintenance-process signal and 251/327 lack a monitoring or audit mechanism, and §7.4 reports runtime human oversight in only 47/327 papers. These counts are then used to conclude that lifecycle concerns are 'underdeveloped' and that 'capability improvements alone cannot ensure deployment readiness.' But a paper can implement recovery, confirmation, or oversight without using the survey's terminology; the functional-coding step in §3.3 mitigates this, yet §11 itself acknowledges that such concerns 'may remain implicit in system descriptions.' Without a sensitivity analysis that re-codes absent signals as unknown, or a validation sample checked against author statements or system repositories, the size of the reported gaps may be inflated. I would like to see either (a) a demonstration that absent signals correlate with absent mechanisms in a sample, or (b) rewording of the statistics as 'explicitly discussed under this coding scheme' rather than 'lacking support.'","section":"§11, §3.3, §7.3, §7.4"},{"comment":"Section 3.3 states that the coding scheme was developed iteratively and that records were checked for internal coherence, but Section 11 concedes that these 'internal-coherence checks... do not provide independently measured inter-rater agreement from multiple coders.' Because the RQ2 and RQ4 statistics are the quantitative backbone of the paper, the absence of any inter-rater agreement measure for the interpretive dimensions is a significant gap. The paper should report agreement per coding dimension—especially for human oversight, maintainability, and observability—on a sample of papers, and describe how disagreements were resolved. Without this, counts such as 95/327 versus 86/327 for related runtime-control signals are not auditable, and the precision of the gap percentages cannot be assessed.","section":"§3.3, §11"},{"comment":"The RQ4 denominator of 327 out of 336 records is never explained. Section 3.3 says 'RQ4 uses the 327 records with complete software-engineering coding,' and Table 6 uses '327 SE-parseable papers,' but the reader is not told which nine papers are excluded or what 'SE-parseable' means. This matters because all RQ4 shares use the 327 denominator, and a reader cannot reproduce the subsets or check for selection effects. Please state the exclusion rule, provide the identifiers or at least the contribution types and years of the excluded records, and release the coding instrument with the paper so that the reported counts can be independently verified.","section":"§3.3, Table 6, §7"}],"minor_comments":[{"comment":"Table 7 mixes denominators within one table: the first three rows use the 327-paper SE-parseable set, while the last row ('Explicit human-in-the-loop flag') uses the 145-paper framework set. The caption notes this, but it would be clearer to split the table or to add a denominator column to each row.","section":"Table 7"},{"comment":"The right panel of Figure 3 reports contribution types as multi-label counts, and the text correctly notes that counts do not sum to the corpus size; the figure caption would benefit from stating this explicitly to avoid reader confusion.","section":"Figure 3"},{"comment":"Table 1 shows that 60.7% of the corpus consists of preprints. The paper would be strengthened by a brief robustness discussion of whether the RQ1 growth trends and the RQ4 gap counts change when the analysis is restricted to peer-reviewed records.","section":"§3.2, §4.1"},{"comment":"The term 'SE-parseable' is used at first in Section 3.3 and then repeatedly in the RQ4 analysis, but it is only loosely defined. Please provide a precise definition at first use, even if the full list of excluded records appears in an appendix or supplementary artifact.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"This is a well-organized survey that makes a timely and plausible argument. My recommendation of major revision is driven by coding transparency and reliability concerns rather than by disagreement with the thesis. Please ask the authors to provide the coding instrument, an inter-rater agreement study, and a precise definition of the SE-parseable subset before acceptance. If the authors add a sensitivity analysis around the absence-as-evidence rule, the article would be considerably stronger. The survey is within scope for a software-engineering venue, but its contribution depends on making the coding evidence auditable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the first survey that takes a software-engineering lifecycle lens to GUI agents rather than just organizing capabilities, platforms, or benchmarks. It codes 336 papers across requirements, testing, maintainability, observability, and governance, and makes a coherent case that capability gains alone don't imply deployment readiness. The five RQs are well chosen and the analysis is transparent about denominators, multi-label coding, and its own limitations. The framing of GUI agents as closed-loop software systems with execution contracts, trajectory-based testing, and governed oversight is a useful reframing for the field.\n\nThe main value is the synthesis: RQ2's module counts (verification 64%, memory 61%, planning 55% of framework papers) and RQ4's governance gaps (50% safety risk recognition but 15% defenses, 65% privacy risk but 10% mitigations) together give a plausible picture of a field that builds capability faster than lifecycle support.\n\nNow the soft spots. The load-bearing claim rests on interpreting absent signals as absent support. The paper states this rule explicitly in the threats section and acknowledges that lifecycle concerns may be implicit. Multi-label functional coding (coding verifier/critic/reward model all as verification) reduces the terminology problem, and retry/replanning is captured as a separate signal. But the deeper issue stands: a system that implements recovery or maintainability without naming them is counted as lacking them. Without an inter-rater agreement check and without releasing the coded corpus, the specific percentages (e.g., 279/327 without maintenance process, 251/327 without monitoring) are not independently verifiable. That's a genuine proportionality problem for the RQ4 conclusion, not a fatal one.\n\nThere's also a concrete inconsistency: RQ4 uses 327 as the denominator throughout, but the corpus is 336 and the paper never explains why 9 papers lack complete SE coding. This is minor to fix but needs fixing. The paper does self-report its own limitations, which is credit due, but the reported limitations do not remove the burden on the gap statistics.\n\nBottom line: directionally correct, useful, and worth engaging. I'd send it to a serious referee, with a request for code release, inter-rater reliability, and a fix for the denominator. If you work on GUI agents or agentic SE, you'll want to cite it; just don't quote the gap percentages as exact.","headline":"First SE-lifecycle map of GUI agents that reads the literature as systems work; the gap counts are directionally right but the absence-as-evidence rule makes the magnitudes unverifiable.","tokens_in":50254,"tokens_out":3078,"would_cite":true,"duration_ms":29646,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GUI agents are closed-loop systems that benchmarks alone cannot validate","keywords":["GUI agents","software engineering","closed-loop systems","benchmarks","evaluation","human oversight","software lifecycle","large language models"],"falsifier":"Re-code a random sample of papers coded as lacking maintenance and observability signals, searching for implicit versioning, log, replay, escalation, or permission mechanisms; if a substantial fraction describes such mechanisms without the coded keywords, the survey's gap statistics would not hold.","tokens_in":1468,"feed_emoji":"🖥️","tokens_out":1558,"duration_ms":40741,"temperature":0.7,"pith_summary":"This review of 336 papers argues that GUI agents have outgrown the framing of a model that clicks: they are now closed-loop software systems coupling perception, reasoning, execution, feedback, recovery, and human oversight. Because of that, the paper claims, capability improvements alone cannot guarantee deployment readiness. It finds the field has expanded sharply since 2024 while recovery, human escalation, safety enforcement, auditability, maintainability, observability, and privacy controls remain underdeveloped. The conclusion is that progress depends on lifecycle engineering, meaning explicit execution contracts, trajectory-based testing, reproducible evaluation, and governed human oversight, rather than on benchmark scores alone.","feed_headline":"Benchmark wins will not make GUI agents deployable","feed_subtitle":"A 336-paper review finds recovery, oversight, and lifecycle engineering missing from today's agent success.","key_machinery":"The analytical object is the closed-loop perceive-decide-execute architecture treated as a software system, coded across 327 SE-parseable papers for signals such as verification, memory, reflection, recovery, execution contracts, observability, and governance. The central proposed mechanism is the execution contract: a specification of valid actions, acceptable state changes, retry and stop rules, rollback conditions, and permission boundaries that would make recovery and human oversight testable. The coding scheme's rule that absent signals count as lack of explicit evidence carries the survey's gap statistics.","core_discovery":"The paper establishes that GUI agents should be analyzed as closed-loop software systems whose behavior emerges from the interaction between a foundation model, an orchestration layer, the target application, and the user. Reviewing 336 papers from 2018 to April 2026, it finds that modular perceive-decide-act architectures dominate but that recovery, human escalation, safety enforcement, and auditability are far less developed than perception and planning. On evaluation, task success dominates as a metric while protocol comparability, reproducibility, and risk-awareness lag. Across the lifecycle, testing beyond benchmarks, maintainability, observability, privacy controls, and systematic human oversight are sparse. The paper concludes that capability improvements alone cannot ensure deployment readiness and that future work must connect dependable execution with lifecycle-centered testing, reproducible evaluation, and cost-aware, human-centered governance.","pith_inferences":["If the coding rule is correct, published task-success rates likely overstate operational maturity, since many systems implement recovery or oversight implicitly without reporting it as an engineering artifact.","The execution-contract proposal could be made directly testable by expressing contracts as preconditions and postconditions over UI states and measuring violation rates under interface perturbations.","A natural extension is to treat benchmarks themselves as versioned software artifacts with reproducibility budgets, an idea the paper opens but does not fully formalize.","Merging early reinforcement-learning web agents with post-2023 LLM agents in one corpus may smooth over generational differences; separating those cohorts could sharpen the reported architectural and evaluation trends."],"forward_implications":["Benchmark task success should no longer be treated as sufficient evidence of deployment readiness; a reported score describes the whole agent stack and protocol, not just the model.","Evaluation protocols must disclose observation access, action spaces, environment resets, success oracles, retry policies, and intervention rules before results can be meaningfully compared across benchmarks.","Research should invest in explicit execution contracts that define valid actions, stop conditions, escalation points, and reversible versus irreversible operations.","A GUI-agent test pyramid spanning component tests, integration tests, trajectory regression, and adversarial system-level stress tests is needed to replace benchmark-only validation.","Permissions, privacy controls, audit logs, and risk-adaptive human oversight must be designed as runtime components of the system rather than as post-hoc safeguards."],"supporting_citations":[{"why":"Agent S supplies a representative modular framework that the survey uses to characterize closed-loop architecture.","marker":"[2]"},{"why":"SeeClick provides the GUI-grounding training evidence used to describe screenshot-centered perception.","marker":"[38]"},{"why":"Mind2Web grounds web-agent evaluation in action prediction and task decomposition, shaping RQ3 analysis.","marker":"[48]"},{"why":"CogAgent represents GUI-specialized pre-training and fine-tuning that the survey cites as a key enabling model.","marker":"[79]"},{"why":"VisualWebArena supplies one of the interactive web benchmarks that define modern evaluation protocols.","marker":"[99]"},{"why":"VeriSafe exemplifies logic-based action verification, the kind of recovery and safety mechanism RQ4 finds underdeveloped.","marker":"[104]"},{"why":"STEVE shows trajectory-level step verification, used to argue that verification and recovery must analyze intermediate states.","marker":"[138]"},{"why":"AndroidWorld anchors the mobile interactive benchmark family and the paper's claim that evaluation became more realistic.","marker":"[176]"},{"why":"OSWorld establishes the desktop cross-application evaluation setting the survey relies on for closed-loop evidence.","marker":"[260]"},{"why":"WebArena defines the realistic web environment that the survey treats as a turning point for interactive benchmarks.","marker":"[331]"}],"fun_headline_variants":["GUI agents lack recovery, oversight, and lifecycle engineering","Benchmark success won't fix GUI agent brittleness","GUI agents need software engineering, not just benchmarks","Review of 336 papers: GUI agents are not deployable","Capability alone won't make GUI agents dependable"],"cache_read_input_tokens":52480,"weakest_assumption_plain":"The survey treats the absence of an explicitly coded engineering signal as evidence that the underlying system lacks that support, so if many agents implement recovery, maintenance, or human oversight under different terminology, the reported gaps would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["GUI agents lack recovery, oversight, and lifecycle engineering","Benchmark success won't fix GUI agent brittleness","GUI agents need software engineering, not just benchmarks","Review of 336 papers: GUI agents are not deployable","Capability alone won't make GUI agents dependable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2272,"prompt_tokens":982,"completion_tokens":1290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1224}},"tokens_in":598,"tokens_out":1290,"duration_ms":9321,"temperature":1.0,"reasoning_tokens":1224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:15:44.179173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-code a random sample of papers coded as lacking maintenance and observability signals, searching for implicit versioning, log, replay, escalation, or permission mechanisms; if a substantial fraction describes such mechanisms without the coded keywords, the survey's gap statistics would not hold.","supporting_citations":[{"cited_title":"Manmatha, and Shabnam Ghadar","cited_arxiv_id":null,"evidence_quote":"AndroidWorld anchors the mobile interactive benchmark family and the paper's claim that evaluation became more realistic."}],"review_version":1}