{"id":"3cd093bf-4b40-4ae5-bfde-4960987ac9bb","arxiv_id":"2609.04869","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An online skill-evolution framework for GUI agents is evaluated and shown to improve mean task scores across four desktop applications, though with domain-dependent effects and revision churn.","lead":"The paper presents a framework that lets computer-use agents build and revise a library of reusable skills from their own interactions without changing the underlying models. It finds that such skill libraries can improve mean task scores across four desktop applications, but the benefits are domain-dependent and repeated revisions do not guarantee success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Pre-existing warm-up differences obscure the causal contribution of skill availability to post-warm-up scores; the single-run design cannot rule out persistent stochastic variation.","rationale":"The reader correctly identified the weakest assumption as the inability to attribute post-warm-up differences to skill availability rather than pre-existing stochastic differences. This is the single most load-bearing concern because the paper’s central claim hinges on the comparison between Full and Empty. Without controlling for pre-existing differences, the post-warm-up advantage could be an artifact of run-to-run variation, especially in Writer where the pre-difference is large. A concrete difference-in-differences analysis with a permutation test would directly address this concern by isolating the incremental effect of skills relative to the warm-up period. The paper’s own limitation paragraph acknowledges this issue, and the reader’s CONDITIONAL verdict appropriately flags it. My stress test does not uncover a new concern beyond what the reader already identified; therefore, the verdict should remain unchanged. The non-finding here reinforces the need for multiple runs or a DiD adjustment, which the paper itself lists as future work.","tokens_in":10136,"tokens_out":3055,"duration_ms":29196,"concrete_test":"Compute a difference‐in‐differences (DiD) estimator per domain: DiD = (Full_post_mean − Empty_post_mean) − (Full_pre_mean − Empty_pre_mean). Perform a one‐sided permutation test: for each of 10,000 permutations, independently flip the condition labels (Full vs. Empty) within each domain, recompute the average DiD across the four domains, and record the null distribution. If the observed average DiD falls above the 95th percentile, the skill contribution is statistically significant beyond pre‐existing variation. If not, the post‐warm‐up advantage is consistent with run‐specific noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the evolving-library system outperforms the empty‐library control—rests on the post‐warm‐up mean difference across four domains. However, the Full and Empty runs already diverge during the five‐iteration empty‐library warm‐up: the pre‐period difference ranges from −6.7 to +9.1 percentage points (Table 2). This indicates that stochastic factors (e.g., randomness in model outputs, environment initialization, or task order) can produce substantial performance gaps even when no skill library is available. Without multiple independent runs, the post‐warm‐up advantage cannot be uniquely attributed to skill availability rather than to the persistence or amplification of these pre‐existing differences. The Writer domain is particularly problematic: the pre‐warm‐up difference of +9.1 pp accounts for 77% of the post‐warm‐up difference of +11.8 pp, so the incremental effect of skills could be as small as 2.7 pp. The paper acknowledges this limitation but does not provide any quantitative adjustment. Therefore, the claim that the skill library adds measurable value is not adequately supported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes an online skill-evolution framework for computer-use agents. Executor and GUI-grounding models are kept fixed, while interaction trajectories and evaluator feedback are abstracted into structured facts that drive versioned creation, revision, and deletion of reusable skill documents. Each iteration executes against a frozen library snapshot, so accepted skill mutations only affect later sweeps. The evaluation compares a Full evolving-library condition with an Empty-library control in four OSWorld domains (GIMP, VLC, Writer, Thunderbird), reporting that after a five-iteration empty-library warm-up, Full attains a higher post-warm-up mean evaluator score in all four observed runs, with differences of 5.7 to 18.6 percentage points. A GIMP-focused provenance analysis reports substantial cross-origin retrieval and a revision-churn case in which repeated accepted edits fail to recover the originating task. The manuscript is explicit that each condition–domain pair is a single run and that the warm-up periods already differ between conditions.","tokens_in":10660,"tokens_out":3744,"duration_ms":37237,"significance":"If the findings are confirmed with replications, the paper would make a useful contribution to external memory for GUI agents: it separates execution from evolution, versiones skill changes, and provides provenance machinery that goes beyond aggregate endpoint scores. The frozen-snapshot design and the telemetry-based provenance tracking are sensible methodological choices, and the release of code is a concrete asset. The paper is also admirably candid about the observational nature of its comparisons. However, the empirical core rests on four single-run comparisons in which pre-existing warm-up imbalances are not controlled for, so the headline claim that the evolving library 'can improve a fixed computer-use stack' is currently supported only as a descriptive observation, not as an established causal effect.","major_comments":[{"comment":"The Writer domain is the clearest example of why the post-warm-up differences cannot be read as evidence of skill value: the Full-minus-Empty difference during the empty-library warm-up is already +9.1 percentage points, while the post-warm-up difference is +11.8, so 77% of the post-warm-up gap is present before any skill is available. The paper acknowledges this in §4.2 and §5, but the abstract and Contribution 2 still assert that the system 'can add measurable value' and 'demonstrate the value' of the evolving library; the Writer observation does not support that inference without an adjustment for the warm-up offset, such as a difference-in-differences or change-score analysis.","section":"§4.2, Table 2"},{"comment":"With one independently executed longitudinal run per condition–domain pair, there is no within-domain variance estimate, no confidence interval, and no significance test. The statement that Full has a higher post-warm-up mean 'in all four observed domain runs' is descriptively accurate, but the abstract's stronger claim that the library 'can improve a fixed computer-use stack' and Contribution 2's claim to 'demonstrate the value' require replication. At minimum, the authors should report multiple independent seeds per condition–domain pair, or a bootstrap over task-level scores within a run, and a paired or permutation test on the post-warm-up means or on the change from the warm-up period.","section":"§4.2, Figure 2"},{"comment":"The cross-origin retrieval share (43.3% of recorded skill calls) supports only the modest claim that skills are consulted beyond their originating tasks; it does not measure whether the Executor followed the returned procedure or whether the retrieval improved the outcome. The authors explicitly disclaim causal attribution for telemetry, and the revision-churn analysis is likewise observational. The abstract and Conclusion should therefore not present these results as evidence that the library constitutes 'shared procedural memory that can improve' the stack without the same conditional language used in §4.3 and §5.","section":"§4.3"}],"minor_comments":[{"comment":"The notation S^Full_t = ∅ for t = 0,...,4 is clear, but the equation does not define how the library evolves for t ≥ 5; a compact definition of S^Full_{t+1} in terms of accepted mutations would make the protocol easier to follow.","section":"Equation (2)"},{"comment":"The column heading 'Pre diff. (pp)' could be misread as a performance score; consider renaming it 'Warm-up diff. (pp), t = 0–4' so it is clear that this is the Full-minus-Empty difference during the period when both libraries are empty.","section":"Table 2"},{"comment":"The statistic '82.4% of valid rollouts invoke get_skill at least once' needs a definition of 'valid rollout' (e.g., rollouts with complete telemetry) before the denominator is interpretable.","section":"§4.3"},{"comment":"The abstract gives the code URL as 'https://github.com/LongtaoHu/Skill-Evo4GUI', while the full text gives 'https://github.com/LongtaoHu/Skill-Evo4GUI.git'; please harmonize.","section":"Abstract and §1"},{"comment":"The serial decision mode is described as order-dependent, which is correct and acknowledged, but it would help to state explicitly in Table 2 or the protocol description that the reported results use the serial mode only, since a parallel mode could change the revision dynamics and hence the outcome.","section":"§3.4"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Two things to know. First, the real value here is the framework and the provenance analysis, not the headline numbers. Second, the headline claim — Full beats the empty-library control in all four domains — is a descriptive comparison the authors themselves decline to call a significance test or a causal estimate. On the evidence as presented, that restraint is correct.\n\nThe framework is cleanly designed. Tasks execute against a frozen library snapshot; accepted mutations are committed to a versioned git-backed library and only reach the next snapshot; retrieval is progressive; all model parameters stay fixed. The empty-library control is configuration-matched, including the catalog prompt and get_skill interface, which is the right way to isolate the library's contribution. Code is released.\n\nThe genuinely new material is the GIMP provenance analysis. 43.3% of retrievals target skills created by a different task — real evidence that the library operates as shared procedural memory, not per-task scratch storage. The revision-churn case is a useful negative result: gimp-add-alpha-channel receives repeated accepted edits while its originating task succeeds in only 2 of 35 iterations. That's the kind of observation the skill-library literature mostly does not print.\n\nThe soft spot is the performance comparison, and it is soft in the way the stress test says. Writer already differs by +9.1 pp during the empty-library warm-up against a +11.8 pp post-warm-up difference, so most of Writer's margin is pre-existing. With one run per condition–domain pair, stochastic persistence cannot be ruled out.\n\nBut the stress test overstates the damage. Across the other three domains the pattern is friendlier to the paper than the Writer cell alone suggests: GIMP and Thunderbird reverse from negative to positive exactly when the library activates (−4.0→+5.7, −6.7→+12.4), and VLC's gap quadruples (+4.7→+18.6). A difference-in-differences reading supports the skill-availability hypothesis in three of four domains; the paper itself flags Writer and declines to claim causality. So the central claim is plausible but under-supported, not refuted.\n\nThis paper deserves a serious referee. The path forward is replicated runs with significance testing, or at minimum a difference-in-differences adjustment for the warm-up imbalance, plus held-out tasks. The framework and the qualitative findings stand regardless of how replications land. My recommendation: accept for peer review with a major-revision ask focused on the statistics, not the system, and engage with it — it is an honest, reproducible artifact with a finding the community should see.","headline":"Honest, well-scoped system paper whose real value is the versioned skill-library framework and the GIMP provenance findings; the four-domain performance claim is descriptive and under-powered, exactly as the authors admit.","tokens_in":11099,"tokens_out":5723,"duration_ms":51350},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"An evolving skill library improves computer-use agents by converting interaction traces into reusable procedures, though benefits are conditional and revision churn can occur.","keywords":["computer-use agents","skill evolution","procedural memory","GUI automation","online adaptation","skill library","revision churn","provenance"],"falsifier":"Run multiple independent longitudinal trials (e.g., 10 per condition–domain pair) and test whether the Full system's post-warm-up mean evaluator score consistently exceeds Empty by more than the warm-up offset. If the advantage is not replicable or does not exceed pre-existing differences, the claim that skill evolution provides incremental value is unsupported.","tokens_in":11777,"feed_emoji":"🤖","tokens_out":2581,"duration_ms":24179,"temperature":0.7,"pith_summary":"This paper aims to establish that a computer-use agent can improve its performance by converting its own interaction trajectories and evaluator feedback into a persistent, versioned library of reusable skills, without modifying the underlying action-generation or GUI-grounding models. The authors compare two conditions across four OSWorld application domains: a Full system that accumulates and retrieves skills, and an Empty control that retains no skills. They find that Full achieves higher post-warm-up mean evaluator scores in all four domains, with differences ranging from 5.7 to 18.6 percentage points. However, these results are descriptive: each condition–domain pair uses only one independently executed run, and warm-up imbalances (most notably in Writer) limit causal attribution. In GIMP, provenance analysis reveals that 43.3% of skill retrievals are cross-origin, meaning skills are reused beyond the tasks that created them, but also that repeated accepted edits can fail to recover the originating task—a pattern the paper terms revision churn. The core contribution is a longitudinal, provenance-rich characterization of skill evolution as a shared, auditable adaptation layer whose value is domain-dependent and not guaranteed.","feed_headline":"Skill libraries boost GUI agents by up to 18.6 points, with caveats","feed_subtitle":"Online evolution framework converts interaction traces into reusable skills, but domain dependence and revision churn limit gains.","key_machinery":"The central mechanism is an iterative pipeline: (1) iteration-frozen execution under a read-only library snapshot $S_t$, (2) trace abstraction converting rollouts into structured facts, (3) evidence-guided proposal generation (create/edit/delete/no-op), (4) versioned commit of accepted mutations to form $S_{t+1}$, and (5) progressive retrieval at runtime via a lightweight catalog and on-demand skill body retrieval. The comparison uses the Full condition (catalog populated) against an Empty condition (identical interface, empty catalog), with all other components (Executor, Grounding model, prompt templates, task sets) held fixed.","core_discovery":"The paper presents a framework where a fixed computer-use agent (Executor and Grounding models unchanged) repeatedly executes tasks under a frozen skill snapshot, then converts trajectories and evaluator feedback into structured facts that guide skill creation, revision, or preservation. Accepted mutations form the next snapshot, separating execution from evolution. Through a matched longitudinal comparison across four OSWorld domains, the Full system (with evolving library) achieves a higher mean evaluator score than the Empty control in all observed runs after the warm-up period. In VLC the difference reaches 18.6 percentage points; in Writer a 9.1-point pre-existing warm-up offset accounts for most of the 11.8-point post-warm-up gap. The paper also provides provenance evidence from GIMP that skills are retrieved across task boundaries (56.7% same-task, 43.3% cross-origin) and that revision churn can persist—the background-transparency task succeeds only 2 of 35 times despite repeated edits.","pith_inferences":["The single-run design means the observed performance advantages could be stochastic; replication with multiple independent runs is essential to assess robustness and quantify effect sizes.","The warm-up imbalance in Writer (Full already +9.1 points before any skills exist) suggests that factors other than skill availability (e.g., random success on earlier tasks) can account for much of the later gap, weakening causal claims about skill effectiveness in that domain.","Revision churn in GIMP implies that the Proposer's failure-driven edits may introduce noise rather than convergence; this points to a need for reinforcement learning over library-mutation policies, as the paper itself suggests.","The framework's reliance on a fixed Executor–Grounding stack means that gains are bottlenecked by those components; combining skill evolution with improvements in the base agent might yield larger and more consistent benefits."],"forward_implications":["Computer-use agents can accumulate procedural knowledge without updating model parameters, enabling continual adaptation through external memory alone.","Skill libraries can serve as auditable shared memory: version history and provenance allow tracing a skill's origin, edits, and retrieval contexts.","Cross-origin retrieval (43% in GIMP) suggests that libraries facilitate transfer of procedures across related tasks, offering a path toward generalizable skill reuse.","Revision churn (repeated accepted edits without task recovery) indicates that current evidence-guided evolution policies can be counterproductive; smarter policies (e.g., validation or rollback) may be needed.","The framework's separation of execution from evolution allows independent replication and ablation of the library component, making it a test bed for studying procedural memory dynamics."],"supporting_citations":[{"why":"Provides the OSWorld environment and task sets used for evaluation.","marker":"Xie et al., 2024"},{"why":"Supplies the EvoCUA-32B Executor that generates actions under both conditions.","marker":"Xue et al., 2026"},{"why":"Supplies the MVP grounding model that maps natural-language element descriptions to pixel coordinates.","marker":"Zhang et al., 2025"},{"why":"Serves as the Extractor (both conditions) and as the Proposer and Skill Builder (Full condition).","marker":"Kimi Team, 2026"},{"why":"Establishes the prior paradigm of externalizing agent feedback as verbal reflection, which the skill-evolution approach extends.","marker":"Shinn et al., 2023"},{"why":"Introduces Voyager, a code-skill framework for embodied agents, which the paper builds on for GUI domains.","marker":"Wang et al., 2023"}],"fun_headline_variants":["Evolving skill libraries lift GUI agents 5.7 to 18.6 points","Skill libraries add up to 18.6 points, but only conditionally","Online skill evolution turns interaction traces into reusable procedures","Fixed GUI stack improves with persistent skills, yet revision churn persists"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The post-warm-up performance differences are attributed to skill availability, but the separately executed Full and Empty conditions already show non-trivial warm-up differences (e.g., Writer +9.1), so not all of the later gap can be uniquely credited to the library.","fun_headline_variants_meta":{"raw":{"variants":["Evolving skill libraries lift GUI agents 5.7 to 18.6 points","Skill libraries add up to 18.6 points, but only conditionally","Online skill evolution turns interaction traces into reusable procedures","Fixed GUI stack improves with persistent skills, yet revision churn persists"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"cost":0.00028952,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00028952,"upstream_inference_prompt_cost":0.00016744,"upstream_inference_completions_cost":0.00012208},"total_tokens":1632,"prompt_tokens":1196,"completion_tokens":436,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":371}},"tokens_in":1196,"tokens_out":436,"cost_source":"openrouter_usage","duration_ms":4939,"temperature":1.0,"reasoning_tokens":371,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-08T18:10:29.014264+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run multiple independent longitudinal trials (e.g., 10 per condition–domain pair) and test whether the Full system's post-warm-up mean evaluator score consistently exceeds Empty by more than the warm-up offset. If the advantage is not replicable or does not exceed pre-existing differences, the claim that skill evolution provides incremental value is unsupported.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}