Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

Anji Liu; Kaichen He; Muyao Li; Yitao Liang; Zihao Wang

REVIEW 4 major objections 6 minor 30 references

A single vision-language agent trained with a three-stage reinforcement-learning pipeline can choose the right action interface at every step, and that ability transfers from 30 training tasks to more than 800 evaluation tasks in Minecraft.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:24 UTC pith:HZXLF2PJ

load-bearing objection Interesting training recipe for step-level action-space switching, but Tables 1 and 2 are arithmetically inconsistent on the two headline categories, so the central generalization claim is not currently supported by the reported numbers. the 4 major comments →

arxiv 2512.09706 v2 pith:HZXLF2PJ submitted 2025-12-10 cs.LG

Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

Kaichen He , Zihao Wang , Muyao Li , Anji Liu , Yitao Liang This is my paper

classification cs.LG

keywords agentic AIreinforcement learningaction-space selectionMinecraftvision-language modelGRPOgeneralizationstep-level decision

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the optimal action interface for an agent is not fixed but shifts from step to step, and that a single vision-language model can learn to choose the right interface on its own. CrossAgent is trained in three stages: cold-start supervised fine-tuning on mixed action types, single-turn reinforcement learning that rewards semantic correctness regardless of surface form, and multi-turn reinforcement learning that optimizes episodic success minus token cost. Trained on only 30 Minecraft tasks, it evaluates on over 800 and reports higher average success rates than every fixed-action-space baseline, with particularly large gains in crafting. The contribution is a recipe: action-space selection treated as a learnable, context-dependent decision rather than a design-time choice.

Core claim

The central claim is that dynamic, step-level action-space selection in a single policy can be learned from experience rather than specified by hand, and that this learned switching transfers to novel tasks. The authors demonstrate a three-stage pipeline — mixed-space SFT, single-turn GRPO with an action-space-agnostic reward, and multi-turn GRPO with an episodic success reward plus a token-length penalty — and show that the resulting model, CrossAgent, outperforms fixed-action-space baselines across more than 800 Minecraft tasks despite RL training on only 30. Qualitative rollouts indicate the model uses coarse motion primitives for exploration, grounding actions for visual tracking, and ra

What carries the argument

The central mechanism is a composite action space with an implicit router: rather than a separate switching module, the same policy emits actions from any subspace, and the router is the policy's own choice. The parser g maps every action string to a canonical raw representation, so the single-turn reward (Eq. 2) is surface-form-agnostic; this lets GRPO reinforce whichever interface yields the correct raw action. The multi-turn stage (Eq. 6) then adds a token-length penalty, pushing the policy to prefer high-level interfaces when they suffice and reserve low-level actions for precision. The staged curriculum — cold start, single-turn, multi-turn — is what the claims rest on.

Load-bearing premise

Every action-space choice in a trajectory gets the same reward as the final task outcome, so the training signal cannot tell the model which interface decisions actually helped.

What would settle it

Annotate test trajectories with the action space an oracle would have chosen at each step (knowing future success and execution cost). If the model's per-step choices match the oracle only at chance level, its claimed step-level switching is not supported by the data.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If correct, agent designers no longer need to fix one interface; a single post-trained model can cover APIs, GUI events, and primitives, simplifying system design.
Training on a small task set can transfer to a large benchmark, suggesting RL-tuned switching may generalize across task distributions.
The token-cost penalty yields more efficient trajectories without explicit hand-crafted rules.
Action-space selection becomes part of the policy, so it can adapt within a trajectory, e.g., switching from search API to GUI manipulation when a webpage blocks automation.
The approach positions RL as the mechanism for learning interface choice, not just action content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's qualitative evidence does not yet show per-step router accuracy; a direct test would be to compare each step's chosen action space against an oracle that knows which interface is optimal for that step. If the model only matches task-level biases, the step-level claim weakens.
The parser g is a semantic-equivalence oracle; deploying this in less formal environments (e.g., real GUIs, robotics) requires a similarly reliable mapping, whose availability is a practical bottleneck.
The token-length penalty conflates efficiency with token count; in domains where low-level actions are cheap and high-level actions are expensive, the learned preference may flip.
A natural extension would be to vary the cost coefficient per action space and measure how the policy's switching behavior adapts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Interesting training recipe for step-level action-space switching, but Tables 1 and 2 are arithmetically inconsistent on the two headline categories, so the central generalization claim is not currently supported by the reported numbers.

read the letter

The novel piece here is the three-stage pipeline: mixed-space cold-start SFT, single-turn GRPO with an action-space-agnostic reward on parsed raw actions, then multi-turn GRPO with a token penalty. That's a real contribution — no one else has trained dynamic per-step selection among five heterogeneous action spaces with RL, and the recipe is clearly described. The STRL ablation is sensible and points in the right direction, and the environment is third-party. I also appreciate that they ship code and models.

But there's a load-bearing problem. Table 1 reports Mine Blocks ASR 40.0±48.3 and Craft Items 78.8±41.0 for CrossAgent; Table 2 reports exactly the same values for OOD tasks, even though Table 1 is supposed to include the 30 training tasks and the ID means for those categories are 70.7 and 83.7. Arithmetic makes that impossible unless the pools are not what they're described to be. The same identity holds for the w/o-STRL variant. Either the OOD evaluation includes training tasks, or Table 1 copied the OOD column. Either way, the central claim — that 30-task RL transfers to 800+ tasks — rests on numbers that don't yet stand up. This is not a minor typo; it's the headline result.

Other issues are smaller but real. The "emergent efficiency" language is overreach: Eq. 6 has a hand-set λ·l(τ) token penalty, so efficiency is shaped, not emergent, and λ is never reported. "State-of-the-art across all categories" is also too strong: on Finished Tasks, OpenHA beats CrossAgent in 3 of 4 columns. And the central behavioral capability — step-level switching — is evidenced only by density plots from 20 episodes of 3 tasks, with no per-step router accuracy and no oracle-switch baseline. Given that multi-turn GRPO assigns the same episodic reward to all steps, the model may simply have learned task-level interface biases rather than context-appropriate per-step switching. The qualitative rollouts are suggestive but don't rule that out.

The idea is worth engaging with, and the pipeline deserves serious scrutiny — but as written, the quantitative foundation is compromised. A serious referee should be asked to check the tables, re-run the OOD evaluation with released code, and require seed-level variance and disclosed λ before this can be accepted. Send it to review, but flag the data integrity issue prominently; the authors need to correct or explain Tables 1 and 2 before the headline claim is credible.

Referee Report

4 major / 6 minor

Summary. The paper proposes CrossAgent, a 7B vision-language policy for Minecraft that can emit actions from three heterogeneous action spaces (motion, grounding, raw). Training proceeds in three stages: mixed-space SFT, single-turn GRPO over one-step action prediction with a semantic-equivalence parser, and multi-turn GRPO with an episodic success reward and a token penalty. The authors claim that this yields step-level, context-dependent action-space selection and state-of-the-art average success rate on over 800 OpenHA tasks after training on only 30 tasks, with improved out-of-distribution generalization relative to fixed-action baselines. The paper includes ablations removing the STRL stage, single-space MTRL baselines, and qualitative case studies of three tasks.

Significance. The direction is timely: training a single policy to choose among multiple action granularities is a meaningful step toward generalist agents, and the three-stage pipeline with cold-start SFT, STRL, and MTRL is a concrete, reproducible recipe. The paper also provides code/model links and detailed hyperparameters, which is a strength. If the reported numbers are correct, the ASR gains are substantial. However, the central quantitative support is compromised by the apparent identity between Table 1 and Table 2 category-level ASRs, and the step-level selection claim is supported only by qualitative evidence. The wording about 'emergent efficiency' is also inconsistent with the explicit token penalty in Eq. (6). These issues are fixable but need to be addressed before the claims can be accepted.

major comments (4)

[§4.4, Table 2; §4.1, Table 1] The OOD ASR values for Mine Blocks and Craft Items in Table 2 are numerically identical to the corresponding all-task ASRs in Table 1, for both CrossAgent and the w/o-STRL variant: CrossAgent Mine 40.0±48.3 and Craft 78.8±41.0; w/o-STRL Mine 39.0±46.5 and Craft 58.0±48.4. §4.1 states that MTRL trains on 10 tasks per category, and Table 2 reports ID ASRs for CrossAgent of 70.7 (Mine) and 83.7 (Craft). Excluding 10 training tasks with these means cannot leave the category mean and standard deviation unchanged to two decimal places. The All Tasks row differs (49.1 vs 54.6), so the identity is category-specific, but it is still arithmetically impossible under the stated split. If the OOD evaluation actually included the training tasks, the central 'trained on 30 tasks, generalizes to 800+' claim is unsupported; if it is a copy-paste error, the table must be recomputed. This is load-bearing f
[§3.3, Appendix C.2 Eq. (8)] The MTRL objective assigns the same episodic reward r(τ) to every intermediate response in a trajectory. This does not by itself provide per-step credit for action-space choice; the model can only learn step-level switching if the policy-gradient signal reliably distinguishes which action-space decisions contributed to success. The direct evidence for step-level selection is qualitative: density curves aggregated over 20 episodes of 3 tasks (§4.5, Figs. 5–6). There is no per-step router accuracy, no oracle-switch baseline, no seed-level variance, and no check against the possibility that the model learned only task-level interface biases. Please add a quantitative evaluation of action-space selection, or soften the central claim to task-adaptive rather than step-adaptive selection.
[Eq. (6), §1] The objective explicitly includes −λ·l_θ(τ), a hand-set token penalty, and the problem formulation in §3 includes λ_x·cost(a_t). The abstract and introduction nevertheless credit the model with 'emergent' efficiency and 'without human-specified rules.' Efficiency is directly rewarded, so it is not emergent; what is learned is the trade-off induced by the penalty. Please revise the wording to avoid overclaiming, or provide evidence that the observed efficiency improvements exceed what the penalty alone would induce.
[§3.2, Eqs. (2)–(4)] The STRL reward and the relabeling rule in Eq. (4) both depend on a deterministic parser g that maps action strings from all spaces to a canonical raw representation. The paper does not describe how g handles high-level, ambiguous, or partially specified actions, nor does it report any validation of g's semantic equivalence. If g is not an exact oracle, the STRL reward is noisy and the relabeled dataset D_strl can contain incorrect labels. Please provide details of g, including its coverage across the three action spaces, and an error analysis.

minor comments (6)

[§4.1 vs Appendix D.1] Model initialization is described as Qwen2-VL-7B-Instruct in §4.1 but as the open-source OpenHA model in Appendix D.1. Clarify which base model is actually used and how the two statements are consistent.
[Table 1] The caption states that three metrics are reported (representative task success, FT, ASR), but the column headers only show FT and ASR. Add explicit headers/labels for the representative-task column and explain the icon.
[§4.5] Figure numbering appears inconsistent: the text says 'Figure 5 further illustrates some key frames', but Figure 5 is the density-curve figure and Figure 6 is the example rollouts. Swap or renumber.
[Abstract] The abstract's 'significantly outperforms fixed-action baselines' is only fully true for ASR. For FT, OpenHA is higher on All Tasks (62.8 vs 58.7) and on Mine Blocks (67.3 vs 45.2). Qualify the claim to avoid overstatement.
[§4.1, Appendix A.1] Observation resolution is given as 360×640×3 in §4.1 and 640×360 in Appendix A.1. Use one consistent order.
[§4.1] The baseline list mentions RawHA, but Table 1 has no RawHA row; RawHA-RL appears only in Table 2. Clarify the relationship or add the row.

Circularity Check

1 steps flagged

Central RL pipeline is self-contained; the only true circularity is the 'emergent efficiency' claim, which relabels an explicit hand-set token penalty in Eq. (6) as an autonomous discovery.

specific steps

fitted input called prediction [Section 3.3, Eq. (6); Abstract; Section 1]
"J(θ)=𝔼_{x∼T} 𝔼_{τ∼πθ(·|x)}[r(τ)−λ l_θ(τ)] ... where l_θ(τ) denotes the total number of tokens produced by the model throughout the trajectory. This penalty term encourages the model to prefer concise action spaces (e.g., high-level APIs) over verbose ones (e.g., raw primitive commands) when both yield success. ... This approach enables the agent to learn adaptive action switching—balancing high-level efficiency with low-level precision—without human-specified rules."

The reported efficiency is not an emergent byproduct of RL: Eq. (6) explicitly contains a hand-set token-length penalty λ·l_θ(τ), so shorter trajectories are the direct optimization target. Claiming that the model exhibits 'emergent behavior' for efficiency and balances efficiency 'without human-specified rules' re-describes the imposed reward-shaping term as an autonomous discovery. The efficiency result is therefore equivalent to the input objective by construction, not a separate prediction.

full rationale

Most of the derivation chain is self-contained and non-circular. STRL and MTRL define direct RL objectives (Eqs. 2, 3, 6, 8); the parser g is a stated semantic-equivalence assumption rather than a result smuggled from the conclusion; and the evaluation includes multiple external baselines (VPT, ROCKET-1, STEVE-1, JARVIS-VLA, UI-TARS, Game-TARS), so the central 'dynamic action-space switching beats fixed-action spaces' claim has independent empirical content. Self-citations to OpenHA are provenance for the base model and benchmark, not load-bearing proofs. The one genuine circularity is the 'emergent efficiency' narrative: Eq. (6)'s explicit λ·l_θ(τ) token penalty directly encodes the efficiency objective, which the abstract and Section 1 then present as emergent and rule-free. That is a minor relabeling of an input term, not a collapse of the central action-space-selection claim. I also note the Table 2 vs Table 1 numerical overlap as a data-integrity concern rather than a circularity: the paper labels Table 1 itself as OOD, so identical OOD category numbers would be expected, though the All-Tasks mismatch (54.6 vs 49.1) is unresolved and should be corrected; this does not affect the circularity score.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The ledger is dominated by domain assumptions about oracle annotators: the parser g (Eq. 2), the SAM/MineCLIP annotation pipeline, and the success(τ) oracle are all treated as error-free, with no accuracy reported. The RL objectives add two undisclosed weights: λ in Eq. 6 and the conceptual λ_x·cost(·) of §3. The most novel mechanism — step-level credit for interface selection — rests entirely on the ad hoc reward decomposition of Eq. 8, which is never validated. No invented entities are introduced: the Router and Mixture of Action Spaces (Fig 1) are architectural components of the trained policy, not claimed entities with independent falsifiable handles.

free parameters (2)

λ (token-penalty weight in MTRL objective) = not reported
Eq. 6: J(θ) = E[r(τ) − λ·l_θ(τ)]. This encodes the preference for concise, high-level action spaces that the paper credits to the model as 'emergent efficiency'; the value is never disclosed.
per-space cost term λ_x·cost(a_t) = not operationalized
Introduced in the composite objective in §3 but silently replaced by raw token count l_θ(τ) in Eq. 6; the trade-off weight between success and cost is a modeling choice, not a derived quantity.

axioms (6)

domain assumption Parser g: A → R is an exact, deterministic oracle for semantic equivalence of action strings across all action spaces (Eqs 2-4).
STRL reward r(â,a*) = 1{g(â)=g(a*)} and the M_cs2 relabeling (Eq. 4) assume g never errs; parser accuracy is not reported.
ad hoc to paper Assigning the final episodic reward to every intermediate response (Eq. 8) provides valid step-level credit for action-space choice.
This is the sole mechanism in MTRL that could shape per-step switching; no ablation or metric confirms credit actually propagates to individual action-space decisions.
domain assumption The SAM-based grounding and fine-tuned MineCLIP motion modules produce correct ground-truth actions a* used to reward STRL.
§4.1 admits annotations cannot be generated for every trajectory; error rates of these oracles are unreported, yet they define the STRL reward (Eq. 2).
domain assumption Controllers C_x execute abstract Motion/Grounding/Language actions with reliability comparable across spaces.
§1 motivates the work by controller brittleness; cross-space performance comparisons are meaningful only if controller reliability is not the dominant driver of differences.
domain assumption OpenHA's 800+ task suite is a valid and comprehensive proxy for open-world generalization.
All evaluation claims and most baselines originate from the authors' own CraftJarvis line; external rows in Table 1 use unstated, different protocols.
standard math GRPO surrogate objective with clipped ratios (Eq. 1) behaves as in DeepSeekMath (Shao et al., 2024).
Standard trust-region RL objective; accepted from cited literature.

pith-pipeline@v1.3.0-alltime-deepseek · 18549 in / 31070 out tokens · 301421 ms · 2026-08-03T17:24:43.572469+00:00 · methodology

0 comments

read the original abstract

The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models. However, existing agents are typically confined to static, predefined action spaces-such as exclusively using APIs, GUI events, or robotic commands. This rigidity limits their adaptability in dynamic environments where the optimal granularity of interaction varies contextually. To bridge this gap, we propose CrossHA, a unified agentic model that masters heterogeneous action spaces and autonomously selects the most effective interface for each step of a trajectory. We introduce a comprehensive training pipeline that integrates cold-start supervised fine-tuning with a Multi-Turn Group Relative Policy Optimization (GRPO) algorithm. This approach enables the agent to learn adaptive action switching-balancing high-level efficiency with low-level precision-without human-specified rules. Extensive experiments on over 800 tasks in the open-world Minecraft environment demonstrate that CrossHA achieves state-of-the-art performance. By dynamically leveraging the strengths of diverse action spaces, our model significantly outperforms fixed-action baselines, exhibiting superior generalization and efficiency in long-horizon reasoning. All code and models are available at https://github.com/CraftJarvis/OpenHA.

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 26 linked inside Pith

[1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

Pith/arXiv arXiv
[3]

Brohan, N

A. Brohan, N. Brown, J. Carbajal, Y. Cheb- otar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023a. A. Brohan, N. Brown, J. Carbajal, Y. Cheb- otar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al....

Pith/arXiv arXiv
[7]

J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong. Re- tool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536,

Pith/arXiv arXiv
[8]

D. Guo, F. Wu, F. Zhu, F. Leng, G. Shi, H. Chen, H. Fan, J. Wang, J. Jiang, J. Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025a. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. 13 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Deepseek-r1: Ince...

Pith/arXiv arXiv 1907
[10]

H. Jia, J. Liao, X. Zhang, H. Xu, T. Xie, C. Jiang, M. Yan, S. Liu, W. Ye, and F. Huang. Osworld-mcp: Benchmarking mcp tool invo- cation in computer-use agents.arXiv preprint arXiv:2510.24563, Nov

arXiv
[11]

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open- source vision-language-action model. arXiv preprint arXiv:2406.09246,

Pith/arXiv arXiv
[12]

J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, et al. Mol- moact: Action reasoning models that can rea- son in space.arXivpreprintarXiv:2508.07917,

Pith/arXiv arXiv
[13]

K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui groundingforprofessionalhigh-resolutioncom- puter use. arXiv preprint arXiv:2504.07981, 2025a. M. Li, Z. Wang, K. He, X. Ma, and Y. Liang. Jarvis- vla: Post-training large-scale vision language models to play visual games with keyboards and mouse.arXiv preprint a...

Pith/arXiv arXiv
[14]

F. Lin, R. Nai, Y. Hu, J. You, J. Zhao, and Y. Gao. Onetwovla: A unified vision-language-action model with adaptive reasoning.arXiv preprint arXiv:2505.11917,

arXiv
[15]

H. Lin, Z. Wang, J. Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft.arXiv preprint arXiv:2310.08367,

Pith/arXiv arXiv
[16]

O’Neill, A

A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, et al. Open x- embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864,

Pith/arXiv arXiv
[17]

com/blog/chatgpt/

URLhttps://openai. com/blog/chatgpt/. OpenAI. Introducing chatgpt agent: bridging research and action, 2025a. URL https://openai.com/index/ introducing-chatgpt-agent. OpenAI. Introducing deep research, 2025b. URL https://openai.com/index/ introducing-deep-research/. openai. Operator,2025. URLhttps://openai. com/index/introducing-operator/. Y. Qin, Y. Ye, ...

Pith/arXiv arXiv 2025
[18]

Khedr, R

N.Ravi, V.Gabeur, Y.-T.Hu, R.Hu, C.Ryali, T.Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714,

Pith/arXiv arXiv
[19]

14 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models. arXiv preprint arXiv:2402.03300,

Pith/arXiv arXiv
[20]

L. Song, Y. Dai, V. Prabhu, J. Zhang, T. Shi, L. Li, J. Li, S. Savarese, Z. Chen, J. Zhao, R. Xu, and C. Xiong. Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923, Aug

arXiv
[21]

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

Pith/arXiv arXiv
[22]

S. T. Team. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learn- ing. arXiv preprint arXiv: 2509.02544,

Pith/arXiv arXiv
[23]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.- A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

Pith/arXiv arXiv
[25]

Z. Wang, S. Cai, A. Liu, Y. Jin, J. Hou, B. Zhang, H. Lin, Z. He, Z. Zheng, Y. Yang, et al. Jarvis- 1: Open-world multi-task agents with memory- augmented multimodal language models.IEEE Transactionson PatternAnalysis and Machine Intelligence, 2024b. Z. Wang, S. Cai, Z. Mu, H. Lin, C. Zhang, X. Liu, Q. Li, A. Liu, X. Ma, and Y. Liang. Om- nijarvis: Unifie...

Pith/arXiv arXiv
[26]

Y. Yan, S. Wang, J. Du, Y. Yang, Y. Shan, Q. Qiu, X. Jia, X. Wang, X. Yuan, X. Han, M. Qin, Y. Chen, C. Peng, S. Wang, and M. Xu. Mcp- world: A unified benchmarking testbed for api, gui, and hybrid computer use agents.arXiv preprint arXiv:2506.07672,

Pith/arXiv arXiv
[27]

A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

Pith/arXiv arXiv
[28]

J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable au- tomated software engineering.arXiv preprint arXiv:2405.15793,

Pith/arXiv arXiv
[29]

Zhong, F

15 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Y. Zhong, F. Bai, S. Cai, X. Huang, Z. Chen, X. Zhang, Y. Wang, S. Guo, T. Guan, K. N. Lui, et al. A survey on vision-language-action mod- els: An action tokenization perspective.arXiv preprint arXiv:2507.01925, 2025a. Y. Zhong, X. Huang, R. Li, C. Zhang, Y. Liang, Y. Y...

Pith/arXiv arXiv 2022
[30]

Inembodied mode, these displacements control the camera orienta- tion via pitch (Δ𝑦) and yaw (Δ𝑥) adjustments; in GUI mode, they correspond to 2D cursor move- ments on the screen

Mouse Control..Mouse displacements are dis- cretized into 1800 bins, and their semantic mean- ing is context-dependent. Inembodied mode, these displacements control the camera orienta- tion via pitch (Δ𝑦) and yaw (Δ𝑥) adjustments; in GUI mode, they correspond to 2D cursor move- ments on the screen. Mouse clicks are encoded as dedicated tokens representing...

2024
[2017]

P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K.Chen,X.Liu,J.Wang,W.Ge,etal. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024a. Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, Y. Liang, and T. CraftJarvis. Describe, ex- plain, plan and select: interactive planning with large language ...

Pith/arXiv arXiv
[2019]

Huang, Y

Y. Huang, Y. Chen, H. Zhang, K. Li, H. Zhou, M. Fang, L. Yang, X. Li, L. Shang, S. Xu, et al. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096,

Pith/arXiv arXiv
[2022]

Belkhale, T

S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024a. S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:24...

Pith/arXiv arXiv
[2023]

org/abs/2306.06070

URL https://arxiv. org/abs/2306.06070. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

Pith/arXiv arXiv
[2024]

J. Deng, Z. Wang, S. Cai, A. Liu, and Y. Liang. Open-world skill discovery from unsegmented demonstrations.arXiv preprint arXiv:2503.10684,

Pith/arXiv arXiv
[2025]

S. Cai, Z. Wang, K. Lian, Z. Mu, X. Ma, A. Liu, and Y.Liang. Rocket-1: Masteringopen-worldinter- action with visual-temporal context prompting. arXiv preprint arXiv:2410.17856,

Pith/arXiv arXiv

[1] [1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

Pith/arXiv arXiv

[2] [3]

Brohan, N

A. Brohan, N. Brown, J. Carbajal, Y. Cheb- otar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023a. A. Brohan, N. Brown, J. Carbajal, Y. Cheb- otar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al....

Pith/arXiv arXiv

[3] [7]

J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong. Re- tool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536,

Pith/arXiv arXiv

[4] [8]

D. Guo, F. Wu, F. Zhu, F. Leng, G. Shi, H. Chen, H. Fan, J. Wang, J. Jiang, J. Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025a. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. 13 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Deepseek-r1: Ince...

Pith/arXiv arXiv 1907

[5] [10]

H. Jia, J. Liao, X. Zhang, H. Xu, T. Xie, C. Jiang, M. Yan, S. Liu, W. Ye, and F. Huang. Osworld-mcp: Benchmarking mcp tool invo- cation in computer-use agents.arXiv preprint arXiv:2510.24563, Nov

arXiv

[6] [11]

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open- source vision-language-action model. arXiv preprint arXiv:2406.09246,

Pith/arXiv arXiv

[7] [12]

J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, et al. Mol- moact: Action reasoning models that can rea- son in space.arXivpreprintarXiv:2508.07917,

Pith/arXiv arXiv

[8] [13]

K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui groundingforprofessionalhigh-resolutioncom- puter use. arXiv preprint arXiv:2504.07981, 2025a. M. Li, Z. Wang, K. He, X. Ma, and Y. Liang. Jarvis- vla: Post-training large-scale vision language models to play visual games with keyboards and mouse.arXiv preprint a...

Pith/arXiv arXiv

[9] [14]

F. Lin, R. Nai, Y. Hu, J. You, J. Zhao, and Y. Gao. Onetwovla: A unified vision-language-action model with adaptive reasoning.arXiv preprint arXiv:2505.11917,

arXiv

[10] [15]

H. Lin, Z. Wang, J. Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft.arXiv preprint arXiv:2310.08367,

Pith/arXiv arXiv

[11] [16]

O’Neill, A

A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, et al. Open x- embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864,

Pith/arXiv arXiv

[12] [17]

com/blog/chatgpt/

URLhttps://openai. com/blog/chatgpt/. OpenAI. Introducing chatgpt agent: bridging research and action, 2025a. URL https://openai.com/index/ introducing-chatgpt-agent. OpenAI. Introducing deep research, 2025b. URL https://openai.com/index/ introducing-deep-research/. openai. Operator,2025. URLhttps://openai. com/index/introducing-operator/. Y. Qin, Y. Ye, ...

Pith/arXiv arXiv 2025

[13] [18]

Khedr, R

N.Ravi, V.Gabeur, Y.-T.Hu, R.Hu, C.Ryali, T.Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714,

Pith/arXiv arXiv

[14] [19]

14 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models. arXiv preprint arXiv:2402.03300,

Pith/arXiv arXiv

[15] [20]

L. Song, Y. Dai, V. Prabhu, J. Zhang, T. Shi, L. Li, J. Li, S. Savarese, Z. Chen, J. Zhao, R. Xu, and C. Xiong. Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923, Aug

arXiv

[16] [21]

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

Pith/arXiv arXiv

[17] [22]

S. T. Team. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learn- ing. arXiv preprint arXiv: 2509.02544,

Pith/arXiv arXiv

[18] [23]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.- A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

Pith/arXiv arXiv

[19] [25]

Z. Wang, S. Cai, A. Liu, Y. Jin, J. Hou, B. Zhang, H. Lin, Z. He, Z. Zheng, Y. Yang, et al. Jarvis- 1: Open-world multi-task agents with memory- augmented multimodal language models.IEEE Transactionson PatternAnalysis and Machine Intelligence, 2024b. Z. Wang, S. Cai, Z. Mu, H. Lin, C. Zhang, X. Liu, Q. Li, A. Liu, X. Ma, and Y. Liang. Om- nijarvis: Unifie...

Pith/arXiv arXiv

[20] [26]

Y. Yan, S. Wang, J. Du, Y. Yang, Y. Shan, Q. Qiu, X. Jia, X. Wang, X. Yuan, X. Han, M. Qin, Y. Chen, C. Peng, S. Wang, and M. Xu. Mcp- world: A unified benchmarking testbed for api, gui, and hybrid computer use agents.arXiv preprint arXiv:2506.07672,

Pith/arXiv arXiv

[21] [27]

A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

Pith/arXiv arXiv

[22] [28]

J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable au- tomated software engineering.arXiv preprint arXiv:2405.15793,

Pith/arXiv arXiv

[23] [29]

Zhong, F

15 Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning Y. Zhong, F. Bai, S. Cai, X. Huang, Z. Chen, X. Zhang, Y. Wang, S. Guo, T. Guan, K. N. Lui, et al. A survey on vision-language-action mod- els: An action tokenization perspective.arXiv preprint arXiv:2507.01925, 2025a. Y. Zhong, X. Huang, R. Li, C. Zhang, Y. Liang, Y. Y...

Pith/arXiv arXiv 2022

[24] [30]

Inembodied mode, these displacements control the camera orienta- tion via pitch (Δ𝑦) and yaw (Δ𝑥) adjustments; in GUI mode, they correspond to 2D cursor move- ments on the screen

Mouse Control..Mouse displacements are dis- cretized into 1800 bins, and their semantic mean- ing is context-dependent. Inembodied mode, these displacements control the camera orienta- tion via pitch (Δ𝑦) and yaw (Δ𝑥) adjustments; in GUI mode, they correspond to 2D cursor move- ments on the screen. Mouse clicks are encoded as dedicated tokens representing...

2024

[25] [2017]

P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K.Chen,X.Liu,J.Wang,W.Ge,etal. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024a. Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, Y. Liang, and T. CraftJarvis. Describe, ex- plain, plan and select: interactive planning with large language ...

Pith/arXiv arXiv

[26] [2019]

Huang, Y

Y. Huang, Y. Chen, H. Zhang, K. Li, H. Zhou, M. Fang, L. Yang, X. Li, L. Shang, S. Xu, et al. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096,

Pith/arXiv arXiv

[27] [2022]

Belkhale, T

S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024a. S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:24...

Pith/arXiv arXiv

[28] [2023]

org/abs/2306.06070

URL https://arxiv. org/abs/2306.06070. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

Pith/arXiv arXiv

[29] [2024]

J. Deng, Z. Wang, S. Cai, A. Liu, and Y. Liang. Open-world skill discovery from unsegmented demonstrations.arXiv preprint arXiv:2503.10684,

Pith/arXiv arXiv

[30] [2025]

S. Cai, Z. Wang, K. Lian, Z. Mu, X. Ma, A. Liu, and Y.Liang. Rocket-1: Masteringopen-worldinter- action with visual-temporal context prompting. arXiv preprint arXiv:2410.17856,

Pith/arXiv arXiv