Pith.

Computation and Language · arXiv:2606.15017 v2 · submitted 12 Jun 2026

Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents

Sina Hajimiri, Masih Aminbeidokhti, Jose Dolz, Ismail Ben Ayed, Issam H. Laradji, Spandana Gella, Nicolas Gontier

The paper says

“Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor.”

from the abstract · picked out as the main claim by a classifier

Pith says

Pith has not reviewed this paper.

Get a Pith Review, $5

Reviews

Reviews by people

No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Online web agents often augment a base actor with memory, workflow, or skill modules. These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. Across four WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents.

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23 · no language-model review

How the main claim is stated: hedged possibility 0.99

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.58 · human participants 0.03 · controlled comparison 0.98 · randomized assignment 0.02 · states quantitative result 0.29 · compares with prior results 0.97 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Online web agents often augment a base actor with memory, workflow, or skill modules. background 1.00 0.04 0.01 asserted 0.92
These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. background 0.51 0.10 0.03 hedged possibility 0.61
We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. contribution 0.93 0.77 0.04 asserted 0.87
We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. method 0.95 0.87 0.04 asserted 0.87
Across four WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. result 1.00 0.96 0.70 asserted 1.00
We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. result 0.96 0.94 0.06 hedged possibility 0.73
Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. interpretation 1.00 0.90 0.85 hedged possibility 0.99
We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents. result 0.49 0.84 0.07 asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,269 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 4 Pith papers
How the literature uses it

1 arXiv paper cite this in 1 sentence

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 6 sentences without a confident label

Cites as background 4 papers
  • A line of work explores structurally different paradigms: API-based agents substantially outperform browser agents on WebArena, with hybrid agents reaching even higher performance [ref] , and terminal-based agents operating through code and programmatic interfaces can match or outperform GUI-based agents on enterprise automation tasks [ref] .

    Terminal Agents Suffice for Enterprise Automation · Related Work · 0.89

  • WALT [ref] learns stable tool abstractions reverse-engineered from website structure prior to evaluation, and WebXSkill [ref] mines parameterized action programs from synthetic agent trajectories.

    WebXSkill: Skill Learning for Autonomous Web Agents · Related Work · 0.98

  • SkillWeaver [ref] discovers and synthesizes Python API skills in an offline phase before deployment.

    SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills · Related Work · 0.96

Challenges and discussion

Know something about this paper? Leave a comment.

No comments yet.

Sign in to comment