Computation and Language · arXiv:2606.15017 v2 · submitted 12 Jun 2026
Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
The paper says
“Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Online web agents often augment a base actor with memory, workflow, or skill modules. These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. Across four WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23 · no language-model review
How the main claim is stated: hedged possibility 0.99
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.58 · human participants 0.03 · controlled comparison 0.98 · randomized assignment 0.02 · states quantitative result 0.29 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Online web agents often augment a base actor with memory, workflow, or skill modules. | background 1.00 | 0.04 | 0.01 | asserted 0.92 |
| These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. | background 0.51 | 0.10 | 0.03 | hedged possibility 0.61 |
| We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. | contribution 0.93 | 0.77 | 0.04 | asserted 0.87 |
| We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. | method 0.95 | 0.87 | 0.04 | asserted 0.87 |
| Across four WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. | result 1.00 | 0.96 | 0.70 | asserted 1.00 |
| We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. | result 0.96 | 0.94 | 0.06 | hedged possibility 0.73 |
| Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. | interpretation 1.00 | 0.90 | 0.85 | hedged possibility 0.99 |
| We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents. | result 0.49 | 0.84 | 0.07 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,269 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 4 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Hippo: Training-Free Agent Self-Evolution via Hippocampal Replay cs.AI · Oct 2026
- Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents cs.AI · Oct 2026
- Learning What to Remember: Resource-Efficient Self-Improvement for Lifelong Agents cs.AI · Oct 2026
- Component Replacement in LLM Agents: A Critical Scoping Review of Task Benefit and Attribution cs.AI · Sep 2026
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 6 sentences without a confident label
Cites as background 4 papers
-
A line of work explores structurally different paradigms: API-based agents substantially outperform browser agents on WebArena, with hybrid agents reaching even higher performance [ref] , and terminal-based agents operating through code and programmatic interfaces can match or outperform GUI-based agents on enterprise automation tasks [ref] .
Terminal Agents Suffice for Enterprise Automation · Related Work · 0.89
-
WALT [ref] learns stable tool abstractions reverse-engineered from website structure prior to evaluation, and WebXSkill [ref] mines parameterized action programs from synthetic agent trajectories.
WebXSkill: Skill Learning for Autonomous Web Agents · Related Work · 0.98
-
SkillWeaver [ref] discovers and synthesizes Python API skills in an offline phase before deployment.
SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills · Related Work · 0.96
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment