{"arxiv_id":"2606.15017","title":"Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents","abstract":"Online web agents often augment a base actor with memory, workflow, or skill modules. These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. Across four WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents.","primary_cat":"cs.CL","cross_cats":[],"submitted_at":"2026-06-12T23:30:14+00:00","latest_version":2,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":false,"has_source":false,"verdict":null,"confidence":null,"one_line_summary":null,"novelty_score":null,"correctness_risk":null,"formal_verification":null,"upvotes":0,"downvotes":0,"pith_score":0.0,"inbound_count":5,"_links":{"html":"https://pith.science/paper/2606.15017","verdict":"/api/v1/verdicts/2606.15017"}}