IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents

Rongqian Chen; Sizhe Tang; Tian Lan; Weidong Cao; Yu Li; Zeyu Fang

A lightweight reward model trained on multi-OS GUI trajectories can re-rank computer-use agent actions and raise success on an unseen desktop benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 09:32 UTC pith:Q4HAQCFS

load-bearing objection Solid systems paper: intent-in-action dual encoder + dual losses give a real offline signal, and +6.9 on OSWorld is useful but rests on one constrained unreplicated config. the 3 major comments →

arxiv 2604.05157 v3 pith:Q4HAQCFS submitted 2026-04-06 cs.AI

IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents

Rongqian Chen , Yu Li , Zeyu Fang , Sizhe Tang , Weidong Cao , Tian Lan This is my paper

classification cs.AI

keywords computer-use agentsGUI reward modelintent-conditioned scoringcontrastive alignmentmargin rankingoffline multi-OS trajectoriesaction re-rankingOSWorld

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Computer-use agents often emit a single GUI action without checking whether it is the right one, so one wrong click can derail a long task. This paper argues that a small, plan-aware reward model can fix that by scoring candidate actions before they run. IntentScore learns from about 398K offline GUI steps across Windows, Mac, and Ubuntu, using contrastive alignment to capture which actions fit a screen and margin ranking to separate correct from incorrect ones. Critically, each candidate’s planning intent is embedded with the action so similar surface moves with different rationales get different scores. On held-out data the model reaches 97.5% pairwise discrimination; as a re-ranker for an existing agent on OSWorld—an environment never seen in training—it lifts task success by 6.9 points. The claim is that offline multi-environment reward learning can transfer to new agents and task distributions without extra language-model calls or environment copies.

Core claim

Reward estimation learned from heterogeneous offline GUI trajectories spanning three operating systems generalizes to an unseen agent and task distribution: when IntentScore re-ranks candidate actions for Agent S3 on OSWorld, task success rises by 6.9 points to 52.1%, while offline pairwise discrimination reaches 97.5%.

What carries the argument

IntentScore: a ~13M-parameter dual-encoder reward model that places each candidate’s planning intent in the action encoder and trains with InfoNCE (state–action relevance) plus margin ranking (correctness), scoring candidates by temperature-scaled cosine similarity.

Load-bearing premise

Abstract GUI patterns and per-step correctness labels from the offline multi-OS dataset are similar enough to free-form agent outputs on a new environment that the offline scorer can still rank candidates usefully despite format and distribution shift.

What would settle it

Train the same dual-objective, intent-conditioned scorer on the multi-OS offline data, deploy it as a re-ranker for the same constrained Agent S3 setup on OSWorld, and check whether task success still rises by roughly 6–7 points over the no-scorer baseline; a null or negative lift would falsify the transfer claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid systems paper: intent-in-action dual encoder + dual losses give a real offline signal, and +6.9 on OSWorld is useful but rests on one constrained unreplicated config.

read the letter

The useful takeaway is practical, not theoretical: a 13M dual-encoder scorer that puts planning intent in the action encoder, trains InfoNCE for relevance plus margin ranking for correctness on 398K multi-OS AgentNet steps, and as a re-ranker for Agent S3 lifts OSWorld success by 6.9 points under a matched, deliberately constrained setup.

What is actually new is the empirical split, not the ingredients. Dual encoders, InfoNCE, margin ranking, and Best-of-N re-ranking are familiar. The paper shows alignment alone collapses the correct–incorrect score gap to ~0.005 while still looking decent on pairwise accuracy, and that margin ranking plus intent-in-action (not state) is what opens the gap and drives Hard accuracy to 97.5%. Cross-OS pretrain then Ubuntu finetune adds a few points. Ablations (Tables 1–3) isolate those pieces cleanly; offline metrics include score-gap fractions that map to the deployment threshold; OSWorld is execution-script judged and claimed unseen. That is careful empirical work.

Soft spots are real but proportional. The headline transfer claim is one point estimate (Table 5) under N=3, horizon 50, GPT-5-mini, coding disabled—no multi-seed intervals. Mean unique candidates ~2.19 and 46.6% override rate mean the scorer is intervening a lot on a thin candidate set. The authors themselves document the offline-to-online gap (format shift, free-form candidates, cascading errors) and field mapping in Appendix E. So 97.5% pairwise on AgentNet-style pairs does not by itself secure ranking quality under Agent S3’s distribution. Free parameters (δ, λ, τ, history length, merge radius) are tuned; that is normal for this genre if reported. Circularity is low: labels are offline, eval is held-out tasks plus a different benchmark.

This is for people building GUI agents who need a cheap alternative to LLM judges or world-model rollouts. Math is standard contrastive/ranking; citations cover CUA, OSWorld, and ICRL without obvious padding. I would send it to peer review. Engage if you care about action selection reliability; treat +6.9 as provisional until multi-seed and less constrained runs exist.

Referee Report

3 major / 0 minor

Summary. The paper proposes IntentScore, a lightweight (~13M) plan-aware dual-encoder reward model that scores candidate GUI actions for Computer-Use Agents. It is trained on 398K offline AgentNet steps across Ubuntu/Windows/Mac with two objectives—InfoNCE for state–action relevance and margin ranking for correctness—and places each candidate’s planning intent in the action encoder so that similar surface actions with different rationales can be separated. Offline, it reports 97.5% adjacent-step pairwise discrimination and 99.7% real-incorrect detection on held-out Ubuntu task splits. Deployed as a re-ranker for Agent S3 on OSWorld (stated to be entirely unseen), it raises task success rate from 45.2% to 52.1% (+6.9) under a constrained configuration (N=3, horizon 50, GPT-5-mini, coding actions disabled). Ablations isolate intent encoding, dual losses, and cross-OS pretraining; deployment includes override/defer analysis and case studies.

Significance. Action-selection quality is a genuine bottleneck for CUAs: irreversible GUI errors cascade, and existing remedies (LLM self-judges, environment rollouts, world models) are heavy. A small, independently trained scorer that re-ranks candidates without extra LLM calls or environment copies is practically valuable if it transfers. The dual-objective analysis (alignment alone yields near-zero correct–incorrect gap; margin alone hurts cross-state discrimination) and the intent-in-action-encoder design are clear, falsifiable contributions. Thorough staged ablations (Tables 1–3), score-gap reporting tied to a deployment threshold, and explicit offline-to-online gap discussion strengthen the empirical case. If the multi-OS offline → unseen-agent transfer result holds under stronger evaluation, the work is a useful step toward lightweight inference-time scaling for GUI agents.

major comments (3)

§4.3 and Table 5: The central transfer claim (+6.9 SR to 52.1% on OSWorld for an unseen agent/task distribution) rests on a single point estimate under a deliberately constrained Agent S3 setup (N=3, horizon 50, GPT-5-mini, coding disabled), with no multi-seed runs, confidence intervals, or variance. Table 6 further shows limited candidate diversity (mean 2.19 unique candidates/step) and a 46.6% override rate. Without replication or error bars, it is unclear whether the gain is stable; the headline generalization result is therefore only weakly secured. Please report multi-seed means±std (or bootstrap CIs) for baseline and IntentScore under the same protocol, and state whether the gain persists under a less constrained Agent S3 configuration if feasible.
§4.3 offline-to-online gap and Appendix E: The paper correctly documents distribution shift (Agent S3 multi-statement code, structured plans, free-form phrasing vs. AgentNet single-action schema; field mapping required). Offline Hard accuracy (97.5%, Table 1) is pairwise on AgentNet-style pairs and does not measure ranking quality under Agent S3’s output distribution. The abstract and contribution list still state transfer as demonstrated by the +6.9 result. Please either (i) add a quantitative bridge (e.g., scorer accuracy on held-out Agent S3–style candidates, or calibration of score gaps on deployment logs) or (ii) soften the transfer claim to match the evidence: offline multi-OS reward learning helps under a constrained re-ranking setup, with residual shift limiting online gains.
§3.5 deployment threshold τ=0.10 and free parameters: Override decisions depend on τ (and on merge radius 20 px, N, λ, δ). Table 1 reports Gap≥0.10 on Hard pairs as a deployment-linked metric, but there is no sensitivity analysis of end-to-end SR to τ, nor justification that τ was fixed before OSWorld evaluation rather than tuned on it. Because selective override is part of the claimed mechanism (46.6% override, 6.7% defer), please report SR vs. τ (or a small grid) and clarify selection protocol for τ and related hyperparameters so the online gain is not confounded by post-hoc threshold choice.

Circularity Check

1 steps flagged

No load-bearing circularity: offline labels train the scorer; OSWorld success is an independent execution-based metric on an unseen environment.

specific steps

self citation load bearing [§3.1 Problem Formulation (transfer motivation)]
"Theoretical results support such transfer: Zhang et al. (2026a) show that Q-value estimation error between two MDPs � and � � is bounded by their structural distance �(�� ). GUI environments across operating systems exhibit small inter-MDP distance..."

The multi-OS transfer premise is partly justified by Zhang et al. (2026a), whose author list overlaps the present paper (Tian Lan). This is a mild self-citation for theoretical framing only: the paper’s headline result (+6.9 on OSWorld) is measured independently and does not reduce to that bound. Not load-bearing for the empirical claim.

full rationale

IntentScore’s central chain is empirical and non-circular. The model is trained on AgentNet per-step correctness labels with InfoNCE + margin ranking, evaluated on task-level held-out AgentNet pairs (97.5% Hard accuracy), then deployed as a re-ranker for Agent S3 on OSWorld, which the paper states is entirely unseen (tasks, OS images, configs). End-to-end success is judged by OSWorld’s execution-based final-state scripts, not by the scorer’s training objective or by reusing fitted AgentNet scores as the reported outcome. That is standard supervised transfer evaluation, not a prediction forced by construction. Self-citations (e.g., Zhang et al. 2026a/b with overlapping authors Lan/Fang) appear only as theoretical motivation for multi-MDP transfer and geometric structure; they do not define the reward, force the architecture, or substitute for the OSWorld measurement. Residual mild burden is ordinary self-citation for framing, not a reduction of the +6.9 claim to its inputs. Score 1 reflects that minor non-load-bearing self-citation only.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

The central transfer claim rests on treating multi-OS GUI trajectories as a family of related MDPs with shared abstract structure, on AgentNet correctness labels as reward, and on frozen vision/text encoders plus cosine similarity as a sufficient value estimator. Many training knobs (margin, λ, τ, history length, task weights) are free parameters chosen for validation/deployment behavior rather than derived. No new physical entity is postulated; the invented object is the IntentScore architecture and intent-conditioned scoring interface.

free parameters (7)

margin δ = 0.20
Target separation in margin ranking; set to 0.20 in both stages (Appendix C).
margin loss weight λ = 2.0 then 3.0
Balances InfoNCE vs margin ranking; raised from 2.0 (pretrain) to 3.0 (finetune).
deployment override threshold τ = 0.10
Minimum score gap/score for overriding Agent S3; set to 0.10 and used in offline gap reporting.
temperature τ_sim (learnable cosine scale) = init 0.07
Scales cosine similarity into reward scores; initialized at 0.07, clipped to [0.01, 1.0].
history length H = 3
Number of prior steps in GRU history encoder; fixed at 3 after architecture search.
task-completion sample weights = 1.0 / 0.3 / 0.7
Weights completed/failed/unknown tasks in InfoNCE (1.0 / 0.3 / 0.7).
candidate merge radius = 20 px
Clicks within 20 px and identical code are merged before scoring at deployment.

axioms (5)

domain assumption GUI MDPs across Windows/Mac/Ubuntu have small structural distance, so multi-OS offline reward learning transfers to unseen desktop environments.
Stated in §3.1 via inter-MDP distance and used to justify Stage-1 pretraining and OSWorld transfer.
domain assumption AgentNet per-step correctness labels are a valid offline reward signal for learning action quality.
Dataset D in §3.1 and margin ranking construction in §3.3 depend on r_t ∈ {0,1}.
domain assumption Frozen SigLIP screenshot embeddings and MPNet text embeddings plus a small dual encoder suffice to represent GUI state and actions for scoring.
Architecture §3.2 / Appendix A; vision ablations note near-identical consecutive screenshots.
ad hoc to paper Temperature-scaled cosine similarity between state and intent-aware action embeddings is an adequate action-value estimator for re-ranking.
Reward definition r(s,a)=s·a/τ in §3.2; not derived from Bellman optimality, chosen as a lightweight scoring form.
ad hoc to paper InfoNCE and margin ranking are complementary and jointly necessary for relevance plus correctness.
Motivated and tested in §3.3 and Table 3; treated as design principle rather than theorem.

invented entities (2)

IntentScore (intent-conditioned dual-encoder reward model) independent evidence
purpose: Score candidate CUA actions from (state, history, action, planning intent) without extra LLM calls or environment copies.
The paper’s main proposed artifact; evaluated offline and as Agent S3 re-ranker.
Intent-conditioned action-value estimator Q̂_θ(s, a | H, intent) no independent evidence
purpose: Formal target that recontextualizes history by candidate intent for candidate selection (Eq. 2).
Modeling construct introduced in §3.1; not a physical entity, but a new scoring interface for CUAs.

pith-pipeline@v1.1.0-grok45 · 21960 in / 3852 out tokens · 37862 ms · 2026-07-13T09:32:49.198049+00:00 · methodology

0 comments

read the original abstract

Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore, a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate's planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions.

Figures

Figures reproduced from arXiv: 2604.05157 by Rongqian Chen, Sizhe Tang, Tian Lan, Weidong Cao, Yu Li, Zeyu Fang.

**Figure 1.** Figure 1: Architecture of IntentScore. The state encoder is computed once per step; the intention-aware action encoder is computed per candidate. Reward estimation is temperaturescaled cosine similarity. Training uses a dual objective: state-action alignment (InfoNCE) plus reward learning (margin ranking on hard negatives). where diagonal entries are positives and all off-diagonal entries serve as in-batch negative… view at source ↗

**Figure 2.** Figure 2: Deployment inference pipeline. The CUA generates multiple candidate actions for the current state. The state and action encoders map inputs into a shared latent space, where cosine similarity determines action quality. We deploy IntentScore as a rewardguided re-ranker within Agent S3 on OSWorld, an environment entirely unseen during training. Agent S3 uses GPT-5-mini for planning and UI-TARS-1.5-7B (Q… view at source ↗

**Figure 3.** Figure 3: Decision timeline for a complete OSWorld trajectory (27 steps, task: “write gram [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

**Figure 4.** Figure 4: Case study 1 (step 7): IntentScore overrides Alt+Tab in favor of the “Window” menu for reliable document switching. The score gap of 0.236 reflects the intent-aware encoder’s ability to distinguish navigation strategies. Case 2: Consistent preference for deterministic navigation (step 10). Three steps later, the agent is back in “Answer.docx” and needs to switch to “Grammer test 2.docx” to read its questio… view at source ↗

**Figure 5.** Figure 5: Case study 2 (step 10): IntentScore again overrides a navigation hotkey (Ctrl+F6) in favor of the “Window” menu. The screenshot shows Answer.docx with the Window menu open, listing both documents. demonstrates that the intent-aware encoder distinguishes candidates with nearly identical coordinates but different spatial reasoning. # Action (intent summary) Score 1 Click below “Grammar test 2:” (“blank area … view at source ↗

**Figure 6.** Figure 6: Case study 3 (step 16): Three click candidates target the same line at slightly [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

**Figure 7.** Figure 7: Case study 4 (step 17): Three type candidates with identical content but different [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

Review history (3 revisions) →

discussion (0)

Reference graph

Works this paper leans on

6 extracted references

[1]

Agent s: An open agentic framework that uses computers like a human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Wang. Agent s: An open agentic framework that uses computers like a human. In�� , volume 2025, pp. 22924–22946, 2025a. Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-spec...

2025
[2]

When right meets wrong: Bilateral context conditioning with reward-confidence correction for grpo.�� , 2026a

Yu Li, Tian Lan, and Zhengling Qi. When right meets wrong: Bilateral context conditioning with reward-confidence correction for grpo.�� , 2026a. Yu Li, Rui Miao, Zhengling Qi, and Tian Lan. Arise: Agent reasoning with intrinsic skill evolution in hierarchical reinforcement learning.�� , 2026b. Yu Li,...

2024
[3]

Androidworld: A dynamic benchmarking environment for autonomous agents

Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. In�� , volume 2025, pp. 406–441,

2025
[4]

Webarena: A realistic web environment for building autonomous agents

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In�� , volume 2024, pp. 15585–15606,

2024
[5]

MPNet + SigLIP2 + larger model

�� ...

1920
[6]

Adding incorrect-step negatives (labeled �� =

Negative type matters more than quantity.Adjacent-step negatives ( �±1) are the most effective training signal for Hard test performance, as they require distinguishing temporally close actions that share nearly identical UI context—a challenge shared by offline RL methods that must learn from suboptimal demonstrations without environment interaction (Fan...

2024

[1] [1]

Agent s: An open agentic framework that uses computers like a human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Wang. Agent s: An open agentic framework that uses computers like a human. In�� , volume 2025, pp. 22924–22946, 2025a. Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-spec...

2025

[2] [2]

When right meets wrong: Bilateral context conditioning with reward-confidence correction for grpo.�� , 2026a

Yu Li, Tian Lan, and Zhengling Qi. When right meets wrong: Bilateral context conditioning with reward-confidence correction for grpo.�� , 2026a. Yu Li, Rui Miao, Zhengling Qi, and Tian Lan. Arise: Agent reasoning with intrinsic skill evolution in hierarchical reinforcement learning.�� , 2026b. Yu Li,...

2024

[3] [3]

Androidworld: A dynamic benchmarking environment for autonomous agents

Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. In�� , volume 2025, pp. 406–441,

2025

[4] [4]

Webarena: A realistic web environment for building autonomous agents

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In�� , volume 2024, pp. 15585–15606,

2024

[5] [5]

MPNet + SigLIP2 + larger model

�� ...

1920

[6] [6]

Adding incorrect-step negatives (labeled �� =

Negative type matters more than quantity.Adjacent-step negatives ( �±1) are the most effective training signal for Hard test performance, as they require distinguishing temporally close actions that share nearly identical UI context—a challenge shared by offline RL methods that must learn from suboptimal demonstrations without environment interaction (Fan...

2024