{"id":"2bc8634e-2cac-4b46-97cf-e546b2bd796a","arxiv_id":"2508.00344","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PilotRL trains LLM agents in three progressive reinforcement-learning stages (plan-following, plan generation, joint coordination) and reports state-of-the-art scores on six text-based agent benchmarks.","lead":"This paper trains open-source language models to act as agents by first teaching them to follow a global plan, then improving the plans themselves, and finally coordinating planning with execution. The authors report that an 8B-parameter model trained this way outscores the closed-source GPT-4o on six simulated agent benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation uses DeepSeek-V3 as both simulator and judge (§3.1, §B.5); absent validation on the official ALFWorld/BabyAI environments, the reported overperformance over GPT-4o is unestablished.","rationale":"I agree with the reader's weakest assumption. The paper is internally coherent: the ablations show each training stage contributes, the unified planner-executor architecture outperforms the isolated one, and the human meta-evaluation genuinely checks judge calibration on a small sample. However, none of this validates the external benchmark claim because the test environment itself is not the benchmark. The specific risk is not merely LLM-as-Judge bias but the coupling of training and evaluation simulators: the same frontier model generates plans, simulates observations, scores adherence, plans, and E2E performance, and then judges the final results. This closes the loop in a way that makes the relative numbers untrustworthy for real deployment. The reported 3.6% margin over GPT-4o may reflect the trained policy exploiting patterns in DeepSeek-V3's text simulation rather than general agent competence. A direct evaluation on the official environments is necessary before accepting the SOTA claim. The reader's conditional verdict is appropriate, and I do not see a reason to move it.","tokens_in":31335,"tokens_out":3284,"duration_ms":30805,"concrete_test":"Run the final LLaMA3.1-8B-Instruct + PilotRL and the GPT-4o/ReAct baselines on the official ALFWorld test split via the TextWorld/ALFWorld API with the standard success-rate metric, and on the BabyAI gym environment; if the PilotRL model's lead over GPT-4o falls below the reported ~3.6-point margin (or reverses), the central claim is an artifact of the simulated judge and environment. A secondary check: sample 100 ALFWorld tasks and compare DeepSeek-V3's simulated transitions and outcomes against the official environment for identical agent actions; a disagreement rate above 10% would demonstrate simulator infidelity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The pivotal claim is that open-weights 8B models trained with PilotRL beat GPT-4o. Every number in Table 1 comes from DeepSeek-V3 acting as both the environment simulator (Section 3.1, following Sun et al., 2025) and the LLM-as-Judge that assigns E2E scores (Appendix C, E2E evaluation prompt). Training rewards in Stages 1–3 are also produced by DeepSeek-V3 (Sections 2.2.1–2.2.3 and B.5.1). Thus the policy is optimized and evaluated in the same text-simulated world with the same judge family. If that simulator is more permissive than the real ALFWorld, BabyAI, TextCraft, MAZE, Wordle, or IQA environments — for example, accepting invalid actions, hallucinating object placements, or generating lenient feedback — the trained agent may be exploiting simulator artifacts. Conversely, the judge may be systematically biased toward models trained on its own outputs. The human meta-evaluation (Table 13, 30 instances per dataset) validates only the judge's classification of given interaction trajectories, not the fidelity of the simulated observations or outcomes. The SOTA claim — LLaMA3.1-8B-Instruct + PilotRL averaging 70.43 vs. 67.98 for GPT-4o — is therefore not shown to hold on the actual benchmark environments. This closed training-evaluation loop is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdaPlan, an agent paradigm with a global planner and executor integrated in a single model, and PilotRL, a three-stage progressive reinforcement learning framework (executor adherence, planner cultivation, joint end-to-end optimization). Experiments on six agent benchmarks (ALFWorld, IQA, TextCraft, Wordle, BabyAI, MAZE) report that open-weights 8B models trained with PilotRL outperform GPT-4o and GPT-4o-mini, with LLaMA3.1-8B-Instruct+PilotRL averaging 70.43 versus GPT-4o's 67.98. Evaluation is conducted through a DeepSeek-V3-based environment simulator and LLM-as-Judge protocol following Sun et al. (2025).","tokens_in":31603,"tokens_out":3850,"duration_ms":39104,"significance":"If the results transfer to the official benchmark environments, PilotRL would be a valuable demonstration that progressive RL can materially improve the agentic capabilities of open-weights models, and the paper provides a clear, ablated account of the training design. Strengths include the staged reward design, the ablation of stage order and architecture, the inference-cost analysis, and the human meta-evaluation of the judge. However, the central comparative claim rests on an unvalidated proxy environment: because DeepSeek-V3 both simulates the environment and judges success, the reported SOTA numbers have not been shown to hold on the actual ALFWorld, BabyAI, TextCraft, MAZE, Wordle, or IQA environments. The human meta-evaluation (Table 13) validates only the judge's classification of given trajectories, not the fidelity of the simulated observations or outcomes. The achievable contribution is thus currently conditional, and the load-bearing evidence is missing.","major_comments":[{"comment":"The headline results are produced in a proxy environment where DeepSeek-V3 simulates environmental feedback and scores end-to-end performance, following Sun et al. (2025). No validation is reported against the official ALFWorld, BabyAI, TextCraft, MAZE, Wordle, or IQA simulators. The meta-evaluation in Table 13 (30 instances per dataset) tests only whether the judge classifies given interaction trajectories correctly; it does not verify that the simulated observations, object placements, action validity, or task outcomes match the real environments. If the simulator is more permissive than the true benchmarks, the claimed 3.60% gain over GPT-4o (LLaMA3.1-8B-Instruct+PilotRL 70.43 vs. 67.98) may not transfer to actual agent deployments. This is the main load-bearing gap and should be addressed by running the trained models on the official environments or by providing a systematic fidelity check of the simulator.","section":"§3.1, Table 1, Appendix B.5"},{"comment":"No error bars, multiple seeds, or significance tests are reported for any of the main or ablation results. Differences such as 70.43 vs. 69.77 (LLaMA3.1-8B-Instruct+PilotRL vs. Qwen3-8B+PilotRL) or the 2.35% average improvement over GPT-4o could be within run-to-run variance for RL training. Please report at least three independent runs with standard deviations and, for the GPT-4o comparison, a paired significance test or a confidence interval. Without this, the relative ranking of methods and of backbones is not statistically grounded.","section":"Table 1, Tables 2–6"},{"comment":"DeepSeek-V3 is used as the plan generator in Stage 1, as the reward model for adherence and plan quality (Eqs. 3 and 5), and as the final judge for E2E performance. Consequently, the trained policy is optimized against the same model family that later evaluates it, so the relative gains over GPT-4o could partly reflect judge self-preference or alignment with DeepSeek-V3's planning style rather than general task-solving ability. I am not claiming this circularity is present, but the current design cannot rule it out. A concrete test would be to re-evaluate all methods on the official environments with rule-based success metrics, or to swap the judge to an independent model family and confirm that the ranking of methods is preserved.","section":"§2.2.1–2.2.3, §3.1, Appendix B.5"},{"comment":"The Limitations section honestly acknowledges the dependency on frontier LLMs for plan generation and evaluation, but the abstract and Section 3.2 state the state-of-the-art result without this qualification. The claims should be explicitly scoped to the LLM-simulated evaluation protocol until validation on official environments is supplied. As written, the paper's central claim overstates what the evidence supports.","section":"Section 7 (Limitations), Abstract, §3.2"}],"minor_comments":[{"comment":"Numbers are run together in several cells ('70.2967.35', '72.5169.06'), making the table hard to read. Please fix the formatting.","section":"Table 1, Qwen3-8B row"},{"comment":"There is a typo: 'transfomrers' should be 'transformers'. Also, 'tqdm=4.40.0' appears to be an invalid version; if this is intentional, please clarify.","section":"Appendix B.6.1"},{"comment":"In the ALFWorld global-plan prompt, the action list includes 'cool obj with rece', which is missing a 'p'; elsewhere it appears as 'recep'. Please make the terminology consistent across prompts.","section":"Prompts in Appendix C"},{"comment":"The caption says 'The best and second best of each model are in bold and underlined', but it is unclear whether the best is bold and the second best underlined, or whether both markings are combined. Please clarify the marking scheme.","section":"Table 1 caption"},{"comment":"Please state whether the DeepResearcher-7B numbers are taken from the original paper or recomputed under the same LLM-as-Judge protocol; the same clarification would help for the other open-sourced agent-specific baselines.","section":"Appendix B.2, DeepResearcher-7B"},{"comment":"The y-axis is unlabeled, and the normalization of Stage 1 rewards versus Stage 2/3 rewards (described only in Appendix B.3.3) should be signaled in the figure caption, since the planner reward is not active in Stage 1.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the proposed training framework is plausible, but the evidence for the central SOTA claim is currently conditional on the fidelity of the DeepSeek-V3 simulator and judge. The authors cite Sun et al. (2025) for the simulation approach but provide no evidence that the simulated environments reproduce the original benchmark distributions or scoring. I would ask for official-environment results, multiple seeds with variance estimates, and code release before acceptance. If the official-environment validation is infeasible, the claims need to be substantially rephrased as results in an LLM-simulated setting. Please also confirm that the version under review (arXiv v5, dated 2026) matches the submitted manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's worth your time. The paper's contribution is a concrete training recipe: adaptive global planning (AdaPlan) with a unified planner-executor, trained in three progressive RL stages—first executor adherence, then planner quality, then joint optimization. That specific curriculum is new, and the ablations are genuinely useful. Removing stages or reordering them produces consistent, interpretable drops; the unified vs isolated architecture comparison is clean; the plan-update-frequency sensitivity check is a nice touch. They also publish full prompts, hyperparameters, and a resource budget, and they ran a human meta-evaluation of the judge (30 cases per dataset, ~98% agreement). That is real work, and it shows care.\n\nThe soft spot is exactly where the reader put it. Every headline number in Table 1 comes from an environment simulated by DeepSeek-V3, and DeepSeek-V3 also scores the end-to-end performance and supplies training rewards. The judge is given the reference interaction alongside the agent's trajectory, which can make the evaluation more lenient toward models that imitate reference-like behavior. The human meta-eval confirms the judge's classification of trajectories is good, but it does not validate the fidelity of the simulated observations or state transitions. So the claim that 8B open models beat GPT-4o on ALFWorld and BabyAI is really a claim about the Zerosearch-style proxy environment. It may transfer, but it hasn't been shown.\n\nAlso, the paper has no error bars, no multiple seeds, and no code release. Differences of 2-3% in average score could be noise. The authors do disclose the dependency in their Limitations section, so this is not hiding anything, but it does cap the significance.\n\nOn balance, I think the method is plausible and the internal evidence is consistent. The paper is honest about its evaluation pipeline, and the recipe is detailed enough for others to reimplement. What it isn't is a demonstrated SOTA result on the actual benchmarks. If a referee asks for validation on the official ALFWorld/BabyAI/TextCraft simulators, plus seeds and code, the paper could turn into something solid. I'd send it to review rather than desk-reject; the training recipe and ablations deserve referee time, and the validity gap is explicitly addressable.","headline":"A solid three-stage RL recipe for plan-guided agents, but the headline 'beats GPT-4o' claim rests on a DeepSeek-V3-simulated world rather than the real benchmark environments.","tokens_in":32165,"tokens_out":3550,"would_cite":false,"duration_ms":32465,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training LLM agents with adaptive global plans and progressive reinforcement learning lets open 8B models outperform GPT-4o on six agent benchmarks.","keywords":["AdaPlan","PilotRL","adaptive global planning","progressive reinforcement learning","LLM agents","GRPO","long-horizon decision making","LLM-as-Judge"],"falsifier":"Re-run the trained LLaMA3.1-8B-Instruct + PilotRL model on the original native simulators (TextWorld-based ALFWorld, the real BabyAI grid, the native Wordle and MAZE engines) with human-scored success rates; a large drop from the reported 70.43 average would show the method was tuned to the simulated world and its judge rather than to the environments themselves.","tokens_in":31113,"feed_emoji":"🤖","tokens_out":9730,"duration_ms":84695,"temperature":0.7,"pith_summary":"PilotRL is a training recipe that teaches an open-weights language model to solve long-horizon agent tasks by first following an adaptive global plan, then writing better plans, and finally coordinating planning with execution inside one model. The paper's central claim is that this progressive reinforcement-learning pipeline fixes the short-horizon blindness of the ReAct paradigm and the memorization tendencies of supervised fine-tuning, producing a more generalizable agent. A sympathetic reader would care because, if the claim holds, a single 8B-parameter open model can rival a much larger proprietary model in agent problem-solving. The authors report that LLaMA3.1-8B-Instruct + PilotRL surpasses GPT-4o by 3.60% and GPT-4o-mini by 55.78% across six benchmarks, with PilotRL-trained open models averaging 2.35% above GPT-4o and 53.90% above GPT-4o-mini.","feed_headline":"Adaptive global planning plus staged RL lets 8B agents beat GPT-4o","feed_subtitle":"PilotRL's three-stage training lifts LLaMA3.1-8B 3.60% over GPT-4o and far beyond GPT-4o-mini across six agent benchmarks.","key_machinery":"The load-bearing objects are AdaPlan and the three-stage PilotRL objective. AdaPlan keeps a plan $P^{(t)} = [p^{(t)}_1, \\ldots, p^{(t)}_{N_t}]$; at time $t$ it freezes steps up to $t$ and regenerates the rest as $p^{(t)}_i \\sim \\pi(\\cdot \\mid G, C^{(t)}, P^{(t-1)}, i)$, so high-level guidance is always aligned with accumulated evidence. Stage 1 rewards the executor for format, adherence to a frontier-model plan, and end-to-end efficiency; Stage 2 adds a global-plan-quality reward covering correctness, executability, and standardization, each scored 1 to 5; Stage 3 trains planner and executor jointly on format plus end-to-end performance. GRPO is the policy-optimization algorithm, and a frontier model supplies initial plans, simulates environment observations, and scores the reward terms.","core_discovery":"The discovery is that a model which generates its own step-by-step global plan and rewrites that plan after every environment observation solves agent tasks substantially better than an untrained ReAct-style reasoner or a model trained by supervised fine-tuning or vanilla reinforcement learning alone. The staged order matters: building plan-following before plan-generation before joint optimization beats merging all reward signals at once and beats swapping the first two stages. The paper also shows that hosting the planner and executor in the same model outperforms an isolated planner-executor pairing, and that updating the global plan every step outperforms less frequent replanning.","pith_inferences":["Because the trained model is judged and rewarded by the same frontier model family that also simulates the environments, reported gains may partly reflect alignment with that judge; a native-environment or human-scored pass would show how much transfers.","The AdaPlan recipe should extend to other long-horizon domains with available reference trajectories, such as web navigation or API tool use, even though all six benchmarks here are text games and embodied-style tasks.","A testable consequence of the cost analysis is that the per-step token overhead of planning should be offset by fewer total steps on complex tasks, keeping end-to-end cost at or below ReAct."],"forward_implications":["A plain open-weight instruct model can be turned into a competitive planner-agent using only task instructions and final answers for reinforcement learning, without expert action trajectories.","Because the unified planner-executor architecture beats an isolated pairing by 5.63%, future agent designs should co-train planning and execution rather than bolt a planner onto a frozen executor.","Plan-update frequency is part of the design: replanning every step improves success by 5.67 points over every-two-step replanning and by 14.96 points over every-five-step replanning on BabyAI.","Progressive staging is worth more than the sum of its rewards: merging all three reward signals at once drops average performance by 3.32%.","The three-stage recipe produces gains on Qwen2.5-7B-Instruct, LLaMA3.1-8B-Instruct, and Qwen3-8B, suggesting the mechanism transfers across open-weight backbones."],"supporting_citations":[{"why":"Defines the ReAct paradigm that AdaPlan replaces and serves as the main single-step reasoning baseline.","marker":"Yao et al., 2023"},{"why":"Supplies ALFWorld, one of the four in-domain training and evaluation environments.","marker":"Shridhar et al., 2021"},{"why":"Supplies IQA, the interactive question-answering benchmark used for in-domain evaluation.","marker":"Gordon et al., 2018"},{"why":"Supplies TextCraft, the decomposable crafting benchmark used for in-domain evaluation.","marker":"Prasad et al., 2024"},{"why":"Supplies Wordle and MAZE, the puzzle benchmarks used for in-domain and out-of-domain evaluation.","marker":"Abdulhai et al., 2023"},{"why":"Supplies BabyAI, the out-of-domain grid-world benchmark.","marker":"Chevalier-Boisvert et al., 2019"},{"why":"DeepSeek-V3 is the frontier model that generates Stage 1 plans, simulates environment observations, and scores rewards and evaluations.","marker":"Liu et al., 2024"},{"why":"Provides GRPO, the reinforcement-learning algorithm underlying every PilotRL stage and the Vanilla RL baseline.","marker":"Shao et al., 2024"},{"why":"Provides the approach of using a frontier model to simulate real-world environment behaviors during training.","marker":"Sun et al., 2025"},{"why":"MPO is the external-planner baseline whose separate planner-executor design motivates the unified architecture.","marker":"Xiong et al., 2025"}],"fun_headline_variants":["PilotRL: plan-guided RL lets 8B agents beat GPT-4o","Adaptive global planning plus staged RL lifts 8B models to SOTA","Step-by-step replanning: 8B LLaMA tops GPT-4o","Three-stage RL: plan-follow, then plan, then win","PilotRL replans every step, beats GPT-4o with 8B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported numbers depend on DeepSeek-V3 faithfully simulating the six environments and judging task success; if its observations and scores differ from the real benchmarks, the headline advantages may not appear in actual deployments.","fun_headline_variants_meta":{"raw":{"variants":["PilotRL: plan-guided RL lets 8B agents beat GPT-4o","Adaptive global planning plus staged RL lifts 8B models to SOTA","Step-by-step replanning: 8B LLaMA tops GPT-4o","Three-stage RL: plan-follow, then plan, then win","PilotRL replans every step, beats GPT-4o with 8B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1452,"prompt_tokens":967,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":379}},"tokens_in":583,"tokens_out":485,"duration_ms":4960,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:11:51.401262+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the trained LLaMA3.1-8B-Instruct + PilotRL model on the original native simulators (TextWorld-based ALFWorld, the real BabyAI grid, the native Wordle and MAZE engines) with human-scored success rates; a large drop from the reported 70.43 average would show the method was tuned to the simulated world and its judge rather than to the environments themselves.","supporting_citations":[],"review_version":1}