{"id":"86f17438-6d5b-4f64-8998-394d96b26b91","arxiv_id":"2608.10775","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Visual Skill Cards improve GUI action prediction for frozen and distilled agents by retrieving reusable, state-conditioned visual procedures.","lead":"The paper introduces Visual Skill Cards, a memory format that stores reusable GUI procedures with screenshots and verification cues, and a retrieval layer that feeds them to a frozen visual-language agent. On web and desktop benchmarks, the method lifts the agent's action accuracy and also distills the same knowledge into a smaller student model that runs without the memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim assumes VSC after-state and verification cues are reusable procedure memory, but the paper's audit is only asserted; without a disjoint-source control, the +11.6/+2.9 gains may reflect benchmark-episode answer lookup.","rationale":"The reader's weakest assumption identifies exactly the same gap: VSC evidence may encode the expected outcome from benchmark episodes used to build the library, and the paper's audit is asserted rather than demonstrated. This is the single most load-bearing concern because it directly targets the interpretation of the headline improvements. If it lands, the central claim reduces to retrieval of memorized after-states, which would not constitute reusable visual procedural memory and would not generalize to new workflows. If it does not land, the claim is much stronger. The paper does include useful controls: negative controls show that random or irrelevant VSCs do not reproduce the gains, and the w/o visual ablation indicates the contribution is not uniformly visual. But neither control establishes provenance. The proposed disjoint-source rerun would settle the issue without requiring code release, though artifact release would also help. Since the concern is real but addressable and the current evidence is insufficient to confirm or refute it, the reader's conditional verdict should remain unchanged.","tokens_in":14358,"tokens_out":5051,"duration_ms":49493,"concrete_test":"Re-run the Table 1 main evaluation with a VSC library built only from source traces that provably exclude the evaluation episodes: hold out all Mind2Web/WebLINX tasks used in evaluation, or build Mind2Web cards from OSWorld/WebLINX traces only, keeping executors and budgets fixed. If the GPT-5.4-mini +11.6 Step SR and WebLINX-BG +2.9 Overall gains collapse or become insignificant, the after-state and verification cues are acting as episode-specific answer lookup rather than reusable visual procedure memory. If the gains persist under the disjoint library, the leakage concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that VSC evidence is reusable procedural memory rather than episode-specific answer lookup. In the method, Trace-to-VSC constructs VSCs from Mind2Web and WebLINX traces, and the VSC schema (Eq. 7) stores before-target-after visual evidence plus verification cues. When evaluated on the same benchmarks, the after-state screenshot can be exactly the ground-truth next observation from the episode that generated the card, so the frozen executor could be matching the live screen to a stored answer rather than learning a reusable procedure. The only stated guard is the audit sentence in the Trace-to-VSC section: 'held-out evaluations do not expose answer coordinates as templates.' That is an assertion, not a described mechanism or a reported measurement. The negative controls in Table 5 replace retrieved cards with random or irrelevant cards, which controls for prompt-length effects but not for provenance: a card built from the same evaluation episode would be highly relevant and would carry the answer. Table 4 complicates the picture because 'w/o visual' sometimes outperforms Full SkillLens on Mind2Web, suggesting the beneficial signal may be procedural text; nevertheless, the leakage path is not eliminated because procedure text could also encode episode-specific steps. Without a library constructed from sources provably disjoint from the evaluation episodes, the central claim that VSCs provide reusable visual procedural memory is not fully supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Visual Skill Cards (VSCs), a memory representation that binds a reusable procedure with applicability cues, visual before/target/after evidence, and verification signals. Trace-to-VSC converts heterogeneous interaction traces into this schema; SkillLens retrieves and selectively expands VSC evidence for a frozen VLM executor; CardDistill uses the same evidence as privileged teacher context for on-policy distillation into a student that runs without retrieval. Experiments on Multimodal-Mind2Web, WebLINX-BG, and OSWorld-G report consistent improvements over no-skill baselines across several frozen executors, plus student-only gains for Qwen3-VL-2B, with negative controls and modality ablations.","tokens_in":14635,"tokens_out":8449,"duration_ms":80740,"significance":"If the results hold, this is a useful contribution: VSCs offer a common, auditable unit for heterogeneous GUI experience; the separation of low-cost retrieval from high-resolution evidence expansion is pragmatic; and CardDistill comes with confidence intervals and a shuffled-VSC control. The negative-control experiments in Table 5 and the retrieval-coverage analysis in Table 3 are genuine strengths. However, because the main quantitative claims hinge on a single table without error bars and on the assumption that the VSC library does not contain evaluation-episode answers, the current evidence does not yet establish that the gains come from reusable visual procedural memory rather than from episode-specific answer lookup or from text-only procedures.","major_comments":[{"comment":"The VSC schema stores after-state screenshots and verification cues, and the executor is conditioned on this evidence before action selection (Eq. (14)). The paper evaluates on the same benchmarks used to build the library and only asserts that 'held-out evaluations do not expose answer coordinates as templates.' This does not rule out the after-state screenshot or verification cue acting as an episode-specific answer lookup: for a card built from a Mind2Web or WebLINX trace, the after-state can be the ground-truth next observation of that very episode. Please specify exactly how the construction/evaluation split is managed, and add a control in which the library is built from sources provably disjoint from the evaluation episodes (or from the training split with a statement that no evaluation episode contributed). Without such a control, the +11.6 and +2.9 headline gains could be explained by answer lookup rather than reusable procedural memory.","section":"Trace-to-VSC Construction / Eq. (7)"},{"comment":"Table 4's modality ablation directly undercuts the 'visual' half of the central claim. On Mind2Web, removing visual evidence improves over Full SkillLens for both Qwen3-VL-2B (Step SR 14.0 vs 10.3; Elem. Acc. 67.5 vs 62.2) and Gemini 2.5 Flash (Step SR 75.0 vs 66.2; Elem. Acc. 86.0 vs 77.7); for Qwen3-VL-2B, image-only cards (w/o procedure) give Step SR 4.5, essentially identical to the 4.6 no-skill baseline. The Step SR gain on Mind2Web is therefore carried by procedure text, not visual evidence. Please either explain this pattern or reframe the contribution as mixed procedural memory, and report the headline result with the text-only variant disambiguated.","section":"Table 4"},{"comment":"The main results in Table 1 are single-run percentages with no repeated-seed or error-bar information, while the CardDistill results are reported with 95% CIs and p-values. Since the abstract's headline numbers come from Table 1, please report variance estimates (e.g., multiple evaluation subsets or seeds) or state why the evaluation is deterministic and why variance is negligible. This is needed to assess whether the +11.6, +2.9, +12.0, and +3.2 deltas are statistically meaningful.","section":"Table 1 / Experimental Setup"}],"minor_comments":[{"comment":"The objective is written as KL(P_theta || P_phi) and called 'reverse KL'; the usual reverse-KL direction is KL(teacher || student). Please clarify the intended direction or adjust the terminology.","section":"Eq. (15)"},{"comment":"All selectors report Oracle Cov. 1.00 on every benchmark. Please discuss whether this means retrieval recall is saturated in these settings and whether the observed gains should be attributed primarily to evidence expansion rather than to retrieval.","section":"Table 3"},{"comment":"The construction pipeline (trace normalization, segmentation, summarization, evidence selection, audit) is described at a high level. Please provide concrete adapter details for Mind2Web and WebLINX, such as which LLM or VLM performs summarization and what segmentation rules are used, so the method is reproducible.","section":"Trace-to-VSC Construction"},{"comment":"The latency numbers (1.5s vs 4.0s) are reported without hardware details. Please specify the inference stack and hardware used for the timing comparison.","section":"Table 2"},{"comment":"The caption mentions 'training dynamics' but the text does not describe the plotted curves. Either describe the figure in the body or point to the supplementary material for a full explanation.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The main risk is the provenance and leakage issue. If the authors can provide a disjoint-source or train-split provenance control and address the Table 4 visual-evidence inconsistency, the paper could become acceptable. The current version's central claim is defensible but not yet fully supported. I would also note that the paper does not compare empirically against prior skill-library methods (e.g., MMSkills, VISUALSKILL, CUA-Skill); such a comparison would increase the impact if it is feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SkillLens is a clean, well-ablated memory interface for GUI agents, and the gains look real. The one thing I need before fully trusting the headline numbers is an explicit statement that no VSC was constructed from an evaluation-episode trace. The paper's audit sentence is too thin, but I don't see evidence of a leak.\n\nWhat's genuinely new: the VSC binds procedure text, applicability cues, visual evidence, and verification signals into a single card, and the runtime cleanly separates cheap retrieval from selective high-resolution expansion. CardDistill reusing the same cards as privileged teacher context is a nice trick. The negative controls (random/irrelevant cards, shuffled VSC in distillation) are the right tests, and they pass. Table 4 is honest—it shows w/o visual sometimes beats Full SkillLens on Mind2Web, so the visual component is not uniformly the driver, but the paper doesn't hide that.\n\nSoft spots, in order: (1) After-state screenshots/verification cues. If any card came from the evaluation split, it could carry the answer. The audit sentence 'held-out evaluations do not expose answer coordinates as templates' is an assertion, not a mechanism. They need to state the data split and, ideally, run a disjoint-source control. My guess is they used standard train splits, which would make this a non-issue, but the paper must say so. (2) Table 1 has no error bars; the CardDistill results have CIs, so it's inconsistent. (3) No code/data release—especially important for a memory-system paper where the value is in the library.\n\nThese are all fixable. The central design is sound, the ablations are careful, and the negative controls are convincing. Send it to peer review with a request for the split clarification, repeated-seed CIs, and artifact release. If the split is clean, this is a solid contribution.","headline":"A well-ablated, plausible memory interface for GUI agents; the headline numbers need a clearer statement that evaluation episodes are excluded from the card library.","tokens_in":15159,"tokens_out":3765,"would_cite":true,"duration_ms":34698,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual Skill Cards give frozen GUI agents reusable visual procedural memory and lift step accuracy by up to 11.6 points on Multimodal-Mind2Web.","keywords":["Visual Skill Cards","GUI action prediction","retrieval-augmented inference","on-policy distillation","visual procedural memory","computer-use agents","visual-language models","GUI grounding"],"falsifier":"Build the card library from episodes on one set of sites and evaluate on a disjoint set of sites with the same task types, or strip every after-state and verification crop from the cards: if the improvement mostly collapses, the cards are leaking benchmark-specific outcome templates rather than supplying transferable visual procedures.","tokens_in":14163,"feed_emoji":"🖱️","tokens_out":7509,"duration_ms":64453,"temperature":0.7,"pith_summary":"This paper argues that computer-using agents lack visual procedural memory: they can see controls but cannot tell which familiar workflow is active, which control matters next, or what evidence confirms progress. It introduces Visual Skill Cards (VSCs), a state-conditioned memory object that bundles a reusable procedure with applicability cues, visual evidence, and verification signals. The paper claims that retrieving and selectively expanding the relevant card lets a frozen visual-language model executor choose grounded GUI actions more accurately, and that the same cards can serve as privileged teacher context to distill that behavior into a smaller student that runs without retrieval. If true, external visual skill libraries would offer a practical, auditable upgrade path for existing GUI agents without retraining them.","feed_headline":"Skill cards lift frozen GUI agents by up to 11.6 points","feed_subtitle":"Visual before/after skill cards make frozen GUI models pick the right control and distill into a small student.","key_machinery":"The load-bearing object is the Visual Skill Card, written $s_i = (p_i, z_i, v_i, \\kappa_i)$: a procedure $p_i$, state cues $z_i$ (applicability and verification), visual evidence views $v_i$, and optional auxiliary fields $\\kappa_i$. Trace-to-VSC converts heterogeneous interaction traces into this common schema, segmenting them into reusable units and auditing that held-out evaluations do not expose answer coordinates as templates. At inference, SkillLens uses a lightweight context-aware selector that scores cards by token overlap with the task query, then reranks and expands only the top candidates' evidence, so retrieval decides where to look while expansion controls how much visual evidence the frozen executor sees. For distillation, CardDistill gives the teacher the card bundle and the student only the benchmark-native context, optimizing a teacher-confidence-weighted reverse KL over student-generated action prefixes.","core_discovery":"The central discovery is that a VSC — a card pairing a procedure with when-it-applies cues, before/after visual evidence, and verification signals — works both as external runtime memory and as training-time privilege. On Multimodal-Mind2Web and WebLINX-BrowserGym, adding retrieved VSCs to a frozen GPT-5.4-mini executor raises Step SR by +11.6 points and Overall by +2.9 points, and the same evidence used in CardDistill raises the student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points. The paper further shows that the gains require relevance: random or irrelevant cards do not reproduce them and can actively hurt grounding. The design separates a lightweight retrieval step from selective high-resolution evidence expansion, keeping the live screen as the final grounding source and bounding runtime evidence.","pith_inferences":["A direct cross-site transfer test — build cards from site A, evaluate on site B — would sharpen the claim that the cards store reusable procedures rather than site-specific layouts; the paper does not report such a test.","If the card library is the only source of task knowledge, the same framework could in principle be pointed at new domains (mobile UI, design tools, enterprise software) by swapping in new traces, without retraining the executor.","The teacher-confidence-weighted reverse KL objective could be applied to other privileged signals beyond VSCs, such as ground-truth element boxes or future-state crops, to test how general the distillation recipe is.","The board-based selector diagnostics suggest an alternative route: a VLM reading a rendered candidate board can improve retrieval-only Hit@1 even though it underperforms in end-to-end execution, so a better board layout or hybrid scoring might convert that retrieval gain into downstream accuracy."],"forward_implications":["Frozen GUI executors can be upgraded without parameter updates: the paper shows positive Step SR / Overall gains across Qwen, Gemini, and GPT models when relevant VSCs are retrieved and expanded.","CardDistill shows the retrieved behavior can be internalized: a Qwen3-VL-2B student improves by +12.0 Step SR on Mind2Web and +3.2 Overall on WebLINX-BG while running without any runtime card retrieval.","Relevance is necessary: negative controls with random or irrelevant cards fail to reproduce the gains and can reduce grounding accuracy, so the effect is not simply extra images or longer prompts.","The retrieve-then-expand cost split bounds runtime evidence while preserving high-resolution visual detail, making the approach tractable for step-by-step interactive agents.","Verification cues give a lightweight contract for checking progress after an action, which could be used for self-monitoring as well as action prediction."],"supporting_citations":[{"why":"Supplies the Mind2Web benchmark that Multimodal-Mind2Web builds on and provides one of the trace sources for card construction.","marker":"Deng et al. 2023"},{"why":"Introduces the grounded-web-agent setting that Multimodal-Mind2Web extends to multimodal inputs used in the evaluation.","marker":"Zheng et al. 2024"},{"why":"Provides the WebLINX conversational web-navigation benchmark used as WebLINX-BG.","marker":"Lù, Kasner, and Reddy 2024"},{"why":"Defines the BrowserGym ecosystem that wraps WebLINX into the interactive BrowserGym evaluation.","marker":"de Chezelles et al. 2025"},{"why":"Supplies the OSWorld-G diagnostic grounding setting and the scaling grounding methodology it is based on.","marker":"Xie et al. 2025a"},{"why":"Serves as a diagnostic grounding reference (GUI-Actor) against which CardDistill's center-in-target-box score is contextualized.","marker":"Wu et al. 2025a"},{"why":"Serves as a diagnostic grounding reference (UGround) for the same center-in-target comparison.","marker":"Gou et al. 2025"},{"why":"Serves as a diagnostic grounding reference (UI-TARS) and represents a native trained GUI agent baseline.","marker":"Qin et al. 2025"}],"fun_headline_variants":["Visual skill cards lift frozen GUI agents by 11.6 points","Skill cards raise GUI accuracy and distill into small students","SkillLens cards boost GUI grounding and student distillation","VSC memory improves frozen GUI action prediction","Visual skill cards boost frozen GUI agent scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on the before/after visual evidence in a card being reusable procedural memory rather than a stored picture of the correct answer; the paper does not say how after-state screenshots and verification cues are kept from encoding the target outcome for the episodes that were used to build the library.","fun_headline_variants_meta":{"raw":{"variants":["Visual skill cards lift frozen GUI agents by 11.6 points","Skill cards raise GUI accuracy and distill into small students","SkillLens cards boost GUI grounding and student distillation","VSC memory improves frozen GUI action prediction","Visual skill cards boost frozen GUI agent scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000855,"raw_usage":{"total_tokens":3723,"prompt_tokens":963,"completion_tokens":2760,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2685}},"tokens_in":579,"tokens_out":2760,"duration_ms":17887,"temperature":1.0,"reasoning_tokens":2685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:34:42.775807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build the card library from episodes on one set of sites and evaluate on a disjoint set of sites with the same task types, or strip every after-state and verification crop from the cards: if the improvement mostly collapses, the cards are leaking benchmark-specific outcome templates rather than supplying transferable visual procedures.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Mind2Web benchmark that Multimodal-Mind2Web builds on and provides one of the trace sources for card construction."}],"review_version":1}