PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning

Bin Cui; Chong Chen; Keer Lu; Wentao Zhang; Yunhuai Liu

REVIEW 4 major objections 6 minor 1 cited by

PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training LLM agents with adaptive global plans and progressive reinforcement learning lets open 8B models outperform GPT-4o on six agent benchmarks.

desk verdict A solid three-stage RL recipe for plan-guided agents, but the headline 'beats GPT-4o' claim rests on a DeepSeek-V3-simulated world rather than the real benchmark environments. read the letter →

arxiv 2508.00344 v5 pith:24T2PMM6 submitted 2025-08-01 cs.CL

Keer Lu , Chong Chen , Bin Cui , Yunhuai Liu , Wentao Zhang This is my paper · ORCID

classification cs.CL

keywords AdaPlanPilotRLadaptiveglobalplanningprogressivereinforcementlearningLLMagentsGRPOlong-horizondecisionmakingLLM-as-Judge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PilotRL is a training recipe that teaches an open-weights language model to solve long-horizon agent tasks by first following an adaptive global plan, then writing better plans, and finally coordinating planning with execution inside one model. The paper's central claim is that this progressive reinforcement-learning pipeline fixes the short-horizon blindness of the ReAct paradigm and the memorization tendencies of supervised fine-tuning, producing a more generalizable agent. A sympathetic reader would care because, if the claim holds, a single 8B-parameter open model can rival a much larger proprietary model in agent problem-solving. The authors report that LLaMA3.1-8B-Instruct + PilotRL surpasses GPT-4o by 3.60% and GPT-4o-mini by 55.78% across six benchmarks, with PilotRL-trained open models averaging 2.35% above GPT-4o and 53.90% above GPT-4o-mini.

What carries the argument

The load-bearing objects are AdaPlan and the three-stage PilotRL objective. AdaPlan keeps a plan $P^{(t)} = [p^{(t)}_1, \ldots, p^{(t)}_{N_t}]$; at time $t$ it freezes steps up to $t$ and regenerates the rest as $p^{(t)}_i \sim \pi(\cdot \mid G, C^{(t)}, P^{(t-1)}, i)$, so high-level guidance is always aligned with accumulated evidence. Stage 1 rewards the executor for format, adherence to a frontier-model plan, and end-to-end efficiency; Stage 2 adds a global-plan-quality reward covering correctness, executability, and standardization, each scored 1 to 5; Stage 3 trains planner and executor jointly on format plus end-to-end performance. GRPO is the policy-optimization algorithm, and a frontier model supplies initial plans, simulates environment observations, and scores the reward terms.

What would settle it

Re-run the trained LLaMA3.1-8B-Instruct + PilotRL model on the original native simulators (TextWorld-based ALFWorld, the real BabyAI grid, the native Wordle and MAZE engines) with human-scored success rates; a large drop from the reported 70.43 average would show the method was tuned to the simulated world and its judge rather than to the environments themselves.

Watch

Extended reading notes

Core claim

The discovery is that a model which generates its own step-by-step global plan and rewrites that plan after every environment observation solves agent tasks substantially better than an untrained ReAct-style reasoner or a model trained by supervised fine-tuning or vanilla reinforcement learning alone. The staged order matters: building plan-following before plan-generation before joint optimization beats merging all reward signals at once and beats swapping the first two stages. The paper also shows that hosting the planner and executor in the same model outperforms an isolated planner-executor pairing, and that updating the global plan every step outperforms less frequent replanning.

Load-bearing premise

The reported numbers depend on DeepSeek-V3 faithfully simulating the six environments and judging task success; if its observations and scores differ from the real benchmarks, the headline advantages may not appear in actual deployments.

Editorial extensions

If this is right

A plain open-weight instruct model can be turned into a competitive planner-agent using only task instructions and final answers for reinforcement learning, without expert action trajectories.
Because the unified planner-executor architecture beats an isolated pairing by 5.63%, future agent designs should co-train planning and execution rather than bolt a planner onto a frozen executor.
Plan-update frequency is part of the design: replanning every step improves success by 5.67 points over every-two-step replanning and by 14.96 points over every-five-step replanning on BabyAI.
Progressive staging is worth more than the sum of its rewards: merging all three reward signals at once drops average performance by 3.32%.
The three-stage recipe produces gains on Qwen2.5-7B-Instruct, LLaMA3.1-8B-Instruct, and Qwen3-8B, suggesting the mechanism transfers across open-weight backbones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the trained model is judged and rewarded by the same frontier model family that also simulates the environments, reported gains may partly reflect alignment with that judge; a native-environment or human-scored pass would show how much transfers.
The AdaPlan recipe should extend to other long-horizon domains with available reference trajectories, such as web navigation or API tool use, even though all six benchmarks here are text games and embodied-style tasks.
A testable consequence of the cost analysis is that the per-step token overhead of planning should be offset by fewer total steps on complex tasks, keeping end-to-end cost at or below ReAct.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, what's worth your time. The paper's contribution is a concrete training recipe: adaptive global planning (AdaPlan) with a unified planner-executor, trained in three progressive RL stages—first executor adherence, then planner quality, then joint optimization. That specific curriculum is new, and the ablations are genuinely useful. Removing stages or reordering them produces consistent, interpretable drops; the unified vs isolated architecture comparison is clean; the plan-update-frequency sensitivity check is a nice touch. They also publish full prompts, hyperparameters, and a resource budget, and they ran a human meta-evaluation of the judge (30 cases per dataset, ~98% agreement). That is real work, and it shows care.

The soft spot is exactly where the reader put it. Every headline number in Table 1 comes from an environment simulated by DeepSeek-V3, and DeepSeek-V3 also scores the end-to-end performance and supplies training rewards. The judge is given the reference interaction alongside the agent's trajectory, which can make the evaluation more lenient toward models that imitate reference-like behavior. The human meta-eval confirms the judge's classification of trajectories is good, but it does not validate the fidelity of the simulated observations or state transitions. So the claim that 8B open models beat GPT-4o on ALFWorld and BabyAI is really a claim about the Zerosearch-style proxy environment. It may transfer, but it hasn't been shown.

Also, the paper has no error bars, no multiple seeds, and no code release. Differences of 2-3% in average score could be noise. The authors do disclose the dependency in their Limitations section, so this is not hiding anything, but it does cap the significance.

On balance, I think the method is plausible and the internal evidence is consistent. The paper is honest about its evaluation pipeline, and the recipe is detailed enough for others to reimplement. What it isn't is a demonstrated SOTA result on the actual benchmarks. If a referee asks for validation on the official ALFWorld/BabyAI/TextCraft simulators, plus seeds and code, the paper could turn into something solid. I'd send it to review rather than desk-reject; the training recipe and ablations deserve referee time, and the validity gap is explicitly addressable.

Referee Report

4 major / 6 minor

Summary. The paper introduces AdaPlan, an agent paradigm with a global planner and executor integrated in a single model, and PilotRL, a three-stage progressive reinforcement learning framework (executor adherence, planner cultivation, joint end-to-end optimization). Experiments on six agent benchmarks (ALFWorld, IQA, TextCraft, Wordle, BabyAI, MAZE) report that open-weights 8B models trained with PilotRL outperform GPT-4o and GPT-4o-mini, with LLaMA3.1-8B-Instruct+PilotRL averaging 70.43 versus GPT-4o's 67.98. Evaluation is conducted through a DeepSeek-V3-based environment simulator and LLM-as-Judge protocol following Sun et al. (2025).

Significance. If the results transfer to the official benchmark environments, PilotRL would be a valuable demonstration that progressive RL can materially improve the agentic capabilities of open-weights models, and the paper provides a clear, ablated account of the training design. Strengths include the staged reward design, the ablation of stage order and architecture, the inference-cost analysis, and the human meta-evaluation of the judge. However, the central comparative claim rests on an unvalidated proxy environment: because DeepSeek-V3 both simulates the environment and judges success, the reported SOTA numbers have not been shown to hold on the actual ALFWorld, BabyAI, TextCraft, MAZE, Wordle, or IQA environments. The human meta-evaluation (Table 13) validates only the judge's classification of given trajectories, not the fidelity of the simulated observations or outcomes. The achievable contribution is thus currently conditional, and the load-bearing evidence is missing.

major comments (4)

[§3.1, Table 1, Appendix B.5] The headline results are produced in a proxy environment where DeepSeek-V3 simulates environmental feedback and scores end-to-end performance, following Sun et al. (2025). No validation is reported against the official ALFWorld, BabyAI, TextCraft, MAZE, Wordle, or IQA simulators. The meta-evaluation in Table 13 (30 instances per dataset) tests only whether the judge classifies given interaction trajectories correctly; it does not verify that the simulated observations, object placements, action validity, or task outcomes match the real environments. If the simulator is more permissive than the true benchmarks, the claimed 3.60% gain over GPT-4o (LLaMA3.1-8B-Instruct+PilotRL 70.43 vs. 67.98) may not transfer to actual agent deployments. This is the main load-bearing gap and should be addressed by running the trained models on the official environments or by providing a systematic fidelity check of the simulator.
[Table 1, Tables 2–6] No error bars, multiple seeds, or significance tests are reported for any of the main or ablation results. Differences such as 70.43 vs. 69.77 (LLaMA3.1-8B-Instruct+PilotRL vs. Qwen3-8B+PilotRL) or the 2.35% average improvement over GPT-4o could be within run-to-run variance for RL training. Please report at least three independent runs with standard deviations and, for the GPT-4o comparison, a paired significance test or a confidence interval. Without this, the relative ranking of methods and of backbones is not statistically grounded.
[§2.2.1–2.2.3, §3.1, Appendix B.5] DeepSeek-V3 is used as the plan generator in Stage 1, as the reward model for adherence and plan quality (Eqs. 3 and 5), and as the final judge for E2E performance. Consequently, the trained policy is optimized against the same model family that later evaluates it, so the relative gains over GPT-4o could partly reflect judge self-preference or alignment with DeepSeek-V3's planning style rather than general task-solving ability. I am not claiming this circularity is present, but the current design cannot rule it out. A concrete test would be to re-evaluate all methods on the official environments with rule-based success metrics, or to swap the judge to an independent model family and confirm that the ranking of methods is preserved.
[Section 7 (Limitations), Abstract, §3.2] The Limitations section honestly acknowledges the dependency on frontier LLMs for plan generation and evaluation, but the abstract and Section 3.2 state the state-of-the-art result without this qualification. The claims should be explicitly scoped to the LLM-simulated evaluation protocol until validation on official environments is supplied. As written, the paper's central claim overstates what the evidence supports.

minor comments (6)

[Table 1, Qwen3-8B row] Numbers are run together in several cells ('70.2967.35', '72.5169.06'), making the table hard to read. Please fix the formatting.
[Appendix B.6.1] There is a typo: 'transfomrers' should be 'transformers'. Also, 'tqdm=4.40.0' appears to be an invalid version; if this is intentional, please clarify.
[Prompts in Appendix C] In the ALFWorld global-plan prompt, the action list includes 'cool obj with rece', which is missing a 'p'; elsewhere it appears as 'recep'. Please make the terminology consistent across prompts.
[Table 1 caption] The caption says 'The best and second best of each model are in bold and underlined', but it is unclear whether the best is bold and the second best underlined, or whether both markings are combined. Please clarify the marking scheme.
[Appendix B.2, DeepResearcher-7B] Please state whether the DeepResearcher-7B numbers are taken from the original paper or recomputed under the same LLM-as-Judge protocol; the same clarification would help for the other open-sourced agent-specific baselines.
[Figure 3] The y-axis is unlabeled, and the normalization of Stage 1 rewards versus Stage 2/3 rewards (described only in Appendix B.3.3) should be signaled in the figure caption, since the planner reward is not active in Stage 1.

Circularity Check

2 steps flagged · score 6.0 of 10

The reported 'surpassing GPT-4o' numbers are computed by the same DeepSeek-V3 E2E judge that supplied the training reward, so the headline result is partially self-validating rather than an external benchmark measurement.

self definitional [Section 3.1 (evaluation protocol); Section 2.2.1 and 2.2.3 (reward design)]
"We adopt the unified LLM-as-Judge (Zheng et al., 2023; Gu et al., 2024) paradigm based on DeepSeek-V3 (Liu et al., 2024) to verify the model’s end-to-end (E2E) performance in a fair comparison protocol, including (1) the task completion rates, and (2) the efficiency of interaction trajectories, and then calculate the average scores as the evaluation metric. ... The reward function at this stage is the sum of normalized format and end-to-end (E2E) performance."

The headline numbers in Table 1 are the average DeepSeek-V3 E2E judgment. In training, DeepSeek-V3 supplies the E2E reward RE2E (Equation 4) in Stage 1 and again in Stage 3, where the reward is 'the sum of normalized format and end-to-end (E2E) performance.' The policy is thus explicitly optimized to maximize the same score that is later reported as benchmark 'task completion rates.' The claim that LLaMA3.1-8B-Instruct + PilotRL surpasses GPT-4o by 3.60% is a comparison in a metric that is in-distribution for PilotRL and out-of-distribution for GPT-4o; the score is partly the training objective itself, so it is not independent evidence of superiority on the official environments.
other [Appendix B.5.1; Appendix C, Prompt: Environmental Feedback]
"the frontier model simulates real-world environmental behaviors ... when the frontier model evaluates task completion, we additionally provide reference trajectories sourced from Song et al. (2024) and Xi et al. (2024), which further supply a concrete reference standard to guide and calibrate its judgments, including task completion and solution efficiency."

The 'environment' is not the official ALFWorld, BabyAI, MAZE, TextCraft, Wordle, or IQA simulator; it is DeepSeek-V3 generating text observations, prompted with a reference successful interaction, and the same model family decides when 'Task Completed!' occurs. The agent's entire experience is produced and adjudicated by DeepSeek-V3, the same model that also generates the training rewards and the final E2E scores. The reported ID and OOD numbers therefore measure traversal of a self-consistent text world rather than performance in the real benchmark environments, so the absolute beating-GPT-4o claim is not established by the paper's data.

full rationale

The technical framework (AdaPlan, three-stage GRPO, ablations) is internally coherent: the relative comparisons among methods are all measured under the same protocol, and the ablation results are consistent with the paper's claims. There is no load-bearing self-citation; the simulation approach is credited to Sun et al. (2025), an external prior work. The circularity is concentrated in the evaluation loop. The metric supporting the central claim--that open-weights 8B models 'surpass GPT-4o by 3.60%'--is the DeepSeek-V3 LLM-as-Judge E2E score, which is also the reward signal used to train PilotRL in Stages 1 and 3, and the environment is a DeepSeek-V3 text simulator conditioned on reference trajectories. The paper's Limitations section acknowledges the dependency ('relies on advanced large language models. This introduces a dependency and may lead to the propagation of biases') but does not remove the fact that the headline result is measured by the same model that shaped the policy. Relative gains over baselines remain informative; the absolute SOTA claim is entangled with the training signal and thus partially circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced; AdaPlan is an architectural pattern, not an invented entity. The central claim relies on the listed domain assumptions about DeepSeek-V3's fidelity as environment and judge, and on hand-chosen training hyperparameters.

free parameters (3)

Stage epoch allocation = 1/2/1 (Stage1:1, Stage2:2, Stage3:1)
Chosen by hand; ablation shows sequential order matters but no search over allocations.
Plan update frequency = every step
Chosen by hand; sensitivity analysis shows every-step outperforms every-2 or every-5 steps on BabyAI.
Reward component weights = equal after normalization
Each stage sums normalized format, adherence/E2E/plan-quality scores without learned weights.

assumptions (4)

domain assumption DeepSeek-V3 as judge provides valid E2E task-success and efficiency scores
Invoked in Section 3.1 and Appendix B.5; central to all reported scores.
domain assumption DeepSeek-V3 simulated environment faithfully reproduces real benchmark state transitions and observations
Invoked in Section 3.1 and Appendix B.5.1; observations are generated from reference trajectories.
domain assumption Reference trajectories from Song et al. (2024) and Xi et al. (2024) are correct and sufficient for judging agent behavior
Used inside the judge and environment prompts in Appendix C.
standard math GRPO optimizes the policy as assumed
Standard RL algorithm used without modification (Appendix A).

how reviews work

0 comments

Cite this review

Pith. "Pith review of PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning." pith.science (2026). https://pith.science/paper/24T2PMM6

@misc{pith2026250800344,
  author       = {Pith},
  title        = {Pith review of: PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24T2PMM6}},
  note         = {Machine review of arXiv:2508.00344}
}

read the original abstract

Large Language Models (LLMs) have shown remarkable advancements in tackling agent-oriented tasks. Despite their potential, existing work faces challenges when deploying LLMs in agent-based environments. The widely adopted agent paradigm ReAct centers on integrating single-step reasoning with immediate action execution, which limits its effectiveness in complex tasks requiring long-term strategic planning. Furthermore, the coordination between the planner and executor during problem-solving is also a critical factor to consider in agent design. Additionally, current approaches predominantly rely on supervised fine-tuning, which often leads models to memorize established task completion trajectories, thereby restricting their generalization ability when confronted with novel problem contexts. To address these challenges, we introduce an adaptive global plan-based agent paradigm AdaPlan, aiming to synergize high-level explicit guidance with execution to support effective long-horizon decision-making. Based on the proposed paradigm, we further put forward PilotRL, a global planning-guided training framework for LLM agents driven by progressive reinforcement learning. We first develop the model's ability to follow explicit guidance from global plans when addressing agent tasks. Subsequently, based on this foundation, we focus on optimizing the quality of generated plans. Finally, we conduct joint optimization of the model's planning and execution coordination. Experiments indicate that PilotRL could achieve state-of-the-art performances, with LLaMA3.1-8B-Instruct + PilotRL surpassing closed-sourced GPT-4o by 3.60%, while showing a more substantial gain of 55.78% comparing to GPT-4o-mini at a comparable parameter scale.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

APPO: Agentic Procedural Policy Optimization
cs.LG 2026-06 unverdicted novelty 6.0 of 10

APPO refines branching and credit assignment in agentic RL via a Branching Score and procedure-level scaling, improving baselines by nearly 4 points on 13 benchmarks.

Reference graph

Works this paper leans on

47 extracted references · 41 canonical work pages · cited by 1 Pith paper

[2]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others

Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agent- bench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688. Pattie Maes. 1995. Agents that reduce work and infor- mation overload. InReadings in hum...

arXiv 2023
[3]

There are no redundant, repetitive, or off-topic actions

work page
[4]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. Re...

arXiv 2024
[5]

Step 1: ... Step 2:

answer [True]/[False] where obj correspond to objects. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json 20 [" Step 1: ... Step 2: ... ... ", ...] ''' Prompt: Global Plan Generation - TextCraft You are given a few useful craf...

work page
[6]

Order” is the sequential order of Stage 1, 2, and 3 during training. Specifically, “1 & 2 & 3

to assess the upper bound of the model performance on agent tasks. • Open-Sourced Agent-Specific Models: These models refer to models that were trained specif- ically on agent-task datasets. We have se- lected Agent-FLAN-7B (Chen et al., 2024), LLaMA-xLAM-2-8B-fc-r (Zhang et al., 2024a) and DeepResearcher-7B (Zheng et al., 2025) to represent the open-sour...

work page arXiv 2025
[8]

Step 1: ... Step 2:

toggle: there is a closed or locked door right in front of you and you can toggle it. where <obj> and <id> correspond to ob- jects and index number. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json [" Step 1: ... Step 2: ......

work page
[9]

Step 1: ... Step 2:

cool obj with rece where obj and recep correspond to objects and receptacles. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json [" Step 1: ... Step 2: ... ... ", ...] ''' Prompt: Global Plan Generation - IQA Based on the task...

work page
[13]

Step 1: ... Step 2:

move right For instance, given the current environment state: The goal is at position 8, 6. Your current position is at position 1, 1. There are walls to your left, above you, below you. The index of already executed steps is 0. The possible global plans could be: [“ Step 1: move right (from 1, 1 to 1, 2) Step 2: move right (from 1, 2 to 1, 3) Step 3: mov...

work page

Show all 47 references

[14]

clean obj with recep
[15]

Reminder:

cool obj with rece where obj and recep correspond to objects and receptacles. Reminder:
[17]

Thought” or “Action

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Task {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] Prompt: Execution Generation - IQA Imagine you are an intelligent agen...
[18]

Reminder:

answer [True]/[False] where obj correspond to objects. Reminder:
[20]

craft [target object] using [input ingredients]

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Task {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] 25 Prompt: Execution Generation - TextCraft You are given a few useful...
[22]

s h i r e

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Crafting Commands and Goal {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] Prompt: Execution Generation - Wordle You are an...
[23]

s h i r e

The output format of the action should be a sequence of 5 individual letters, each separated by a space, such as “s h i r e”. Any other formats are consid- ered invalid
[24]

Thought” or “Action

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Task {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] Prompt: Execution Generation - BabyAI You are an exploration master th...
[25]

go through <door> <id>: <door> must be an open door
[26]

If you want to open a locked door, you need to carry a key that is of the same color as the locked door

toggle and go through <door> <id>: <door> can be a closed door or a locked door. If you want to open a locked door, you need to carry a key that is of the same color as the locked door
[27]

where <obj> and <id> correspond to ob- jects and index number

toggle: there is a closed or locked door right in front of you and you can toggle it. where <obj> and <id> correspond to ob- jects and index number. Reminder:
[29]

Thought” or “Action

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Task {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] Prompt: Execution Generation - MAZE You are an expert maze solver. You...
[30]

move right Reminder:
[31]

Actions not included in the provided list are considered invalid

The action is restricted to those listed as available. Actions not included in the provided list are considered invalid
[32]

score": xxx,

Think when necessary, but prioritize di- rect action wherever possible through- out the process. # Example {example} # Task {task} # Global Plan {global_plan} # Previous Observation {observation} [optional] Prompt: Adherence Degree Judgment (for Equation (3)) You are an expert...
[33]

The agent successfully completes the task in a direct and efficient manner
[34]

1 Point Answer Criteria:

There are no unnecessary steps or re- dundant actions in the interaction tra- jectory. 1 Point Answer Criteria:
[35]

The task is ultimately completed, but the process includes some level of re- dundancy or unintended topic drift
[36]

0 Point Answer Criteria:

While the final objective is met, there may be deviations from the optimal path. 0 Point Answer Criteria:
[37]

The agent fails to achieve the final task objective
[38]

score": xxx,

Contains significant deviations, errors, or inability to progress towards the goal. # Task {task} # Agent-Environment Interaction {accumulated_context} # Reference Interaction {ref_interaction} Output Format: ```json { "score": xxx, "reason": "..." } ''' Prompt: E2E Performanc...
[39]

The final task objective is fully and cor- rectly completed
[40]

The interaction trajectory is direct and logically structured
[42]

70–89: Successful but Inefficient

Any minor errors (e.g., phrasing) do not impede progress. 70–89: Successful but Inefficient. The task is completed, but with non-critical inefficiencies. At least one of the following applies:
[43]

The agent takes unnecessary steps or detours before completing the task
[44]

There are minor errors or invalid ac- tions that require recovery
[45]

Brief topic drift or redundant reasoning occurs but is self-corrected
[46]

29 50–69: Partially Successful

The overall strategy works but is sub- optimal in efficiency. 29 50–69: Partially Successful. Significant progress is made, but the task is not fully completed or requires excessive effort. At least one of the following applies:
[47]

The agent fails to reach the final goal, but completes most subtasks
[48]

Completion requires major detours, re- peated failures, or external hints
[49]

The agent shows inconsistent planning or weak strategy
[50]

0–49: Failure or Severely Flawed

Some actions are irrelevant or counter- productive, though not fatal. 0–49: Failure or Severely Flawed. The agent fails to make meaningful progress. At least one of the following applies:
[51]

The agent fails to complete the core objective
[52]

There is complete task misunderstand- ing or topic deviation
[53]

The interaction contains repeated in- valid actions or infinite loops
[54]

score": xxx,

The agent shows no coherent strategy and cannot recover from errors. # Task {task} # Agent-Environment Interaction {accumulated_context} # Reference Interaction {ref_interaction} Output Format: ```json { "score": xxx, "reason": "..." } ''' D Case Studies For agent tasks involv...

2023
[2023]

Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu

Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforce...

2024 arXiv
[2024]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others

A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement le...

2025 arXiv
[2025]

Michael Wooldridge and Nicholas R Jennings

Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599. Michael Wooldridge and Nicholas R Jennings. 1995. In- telligent agents: Theory and practice.The knowledge engineering review, 10(2):115–152. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong,...

1995 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others

Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agent- bench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688. Pattie Maes. 1995. Agents that reduce work and infor- mation overload. InReadings in hum...

arXiv 2023

[2] [3]

There are no redundant, repetitive, or off-topic actions

work page

[3] [4]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. Re...

arXiv 2024

[4] [5]

Step 1: ... Step 2:

answer [True]/[False] where obj correspond to objects. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json 20 [" Step 1: ... Step 2: ... ... ", ...] ''' Prompt: Global Plan Generation - TextCraft You are given a few useful craf...

work page

[5] [6]

Order” is the sequential order of Stage 1, 2, and 3 during training. Specifically, “1 & 2 & 3

to assess the upper bound of the model performance on agent tasks. • Open-Sourced Agent-Specific Models: These models refer to models that were trained specif- ically on agent-task datasets. We have se- lected Agent-FLAN-7B (Chen et al., 2024), LLaMA-xLAM-2-8B-fc-r (Zhang et al., 2024a) and DeepResearcher-7B (Zheng et al., 2025) to represent the open-sour...

work page arXiv 2025

[6] [8]

Step 1: ... Step 2:

toggle: there is a closed or locked door right in front of you and you can toggle it. where <obj> and <id> correspond to ob- jects and index number. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json [" Step 1: ... Step 2: ......

work page

[7] [9]

Step 1: ... Step 2:

cool obj with rece where obj and recep correspond to objects and receptacles. # Task {task} # Previous Global Plan {global_plan} [optional] # Execution Step Index {execution_step_index} # Accumulated Observation {observation} [optional] Output Format: ```json [" Step 1: ... Step 2: ... ... ", ...] ''' Prompt: Global Plan Generation - IQA Based on the task...

work page

[8] [13]

Step 1: ... Step 2:

move right For instance, given the current environment state: The goal is at position 8, 6. Your current position is at position 1, 1. There are walls to your left, above you, below you. The index of already executed steps is 0. The possible global plans could be: [“ Step 1: move right (from 1, 1 to 1, 2) Step 2: move right (from 1, 2 to 1, 3) Step 3: mov...

work page