{"id":"1d8c58b2-13e2-413a-9f8c-b495c001e474","arxiv_id":"2502.01600","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Reinforcement learning with a simple PPO variant (LOOP) makes a 32B model the best interactive digital agent on AppWorld, beating OpenAI o1 without ground-truth action sequences.","lead":"The authors train a 32-billion-parameter AI agent directly inside a simulated multi-app environment using a memory-efficient reinforcement learning variant called LOOP. The trained agent outperforms much larger commercial models like OpenAI o1 on the AppWorld benchmark, suggesting reinforcement learning can replace hand-built workflows for interactive software agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing concern: the OpenAI o1 baseline protocol is unreported and could invalidate the headline SOTA comparison; the reader's stochasticity concern is less threatening because AppWorld is deterministic.","rationale":"The reader's weakest assumption is technically not the main threat because the paper's empirical setting is the deterministic AppWorld environment; policy-independent environmental stochasticity would cancel in the importance ratio. The absence of a detailed o1 evaluation protocol is a concrete, unreported variable that directly touches the abstract's headline claim. The rest of the empirical apparatus (controlled comparisons, multiple runs in Table 2, checkpoint selection protocol) is credible. Releasing artifacts and documenting the o1 harness would settle this; hence the CONDITIONAL verdict is appropriate and unchanged.","tokens_in":38195,"tokens_out":12381,"duration_ms":112420,"concrete_test":"Reconstruct the exact NFT evaluation harness from the released code/checkpoints, or ask the authors for the o1 evaluation script: same ReAct prompt (including the in-context example), same API-doc retrieval, same 50-interaction limit, and same output parser. Run OpenAI o1 with 5 independent seeds on Test-N and Test-C. If the mean TGC is within 2 pp of the reported 61.9/36.7, the comparison is fair; if it exceeds LOOP's 71.3/45.7 or deviates by more than the reported 9 pp gap, the headline SOTA claim would not survive. Also verify that o1 was not given ground-truth task solutions or extra turns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (71.3 TGC on Test-N, 45.7 on Test-C, outperforming o1 by 9pp) depends on the o1 comparison being measured under the same harness as LOOP: identical ReAct-style prompt with the in-context example, identical API-doc access, identical 50-interaction cap, and identical parsing of model output into code cells. Section 5.3 lists o1 only as an NFT baseline with no protocol specification, no number of seeds, and no transcripts; since o1 is a closed model with a different interaction style, even a small mismatch (e.g., no in-context example, different code-extraction, or a weaker system prompt) could account for the reported gap. This is more load-bearing than the reader's identified stochasticity assumption: AppWorld is explicitly deterministic (Section 6 lists non-determinism as a missing feature; footnote 1 describes deterministic replay), and for a POMDP with policy-independent environment transitions, the environment's transition probabilities cancel in the importance ratio even when stochastic, so Eq. 8 would still hold. The theoretical concern affects generality, not the reported numbers. The missing o1 protocol and lack of released evaluation code are the real soft spot.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LOOP, a proximal-policy-optimization variant for fine-tuning interactive digital agents with LLMs in a stateful API environment. The method formalizes agent-environment interaction as a POMDP, replaces the learned critic with a leave-one-out Monte Carlo baseline, and uses per-token importance weights and clipping to allow off-policy reuse. Training a 32B Qwen2.5-Instruct model with LOOP on 24 AppWorld scenarios with a unit-test-fraction reward is reported to achieve 71.3 TGC on test-normal and 45.7 TGC on test-challenge, outperforming OpenAI o1 by 9 percentage points and all examined fine-tuning and RL baselines. The paper also provides behavioral analyses and full rollouts in the appendix.","tokens_in":38410,"tokens_out":5872,"duration_ms":52192,"significance":"If the results hold, this is a meaningful advance: it is the first reported RL training of an IDA in a stateful, multi-domain, multi-app environment via direct API calls, and it shows that a simple, memory-efficient policy-gradient method can substantially improve long-horizon interactive behavior from only 24 training scenarios. The paper's strengths include controlled comparisons with shared hyperparameters across RL methods, multiple training seeds and evaluation runs with reported standard deviations, full rollout transcripts, and an external unit-test reward without fitted constants. The main load-bearing weakness is the under-specified OpenAI o1 baseline protocol, which directly supports the headline SOTA claim, plus a few reproducibility gaps that should be addressed.","major_comments":[{"comment":"The evaluation protocol for the OpenAI o1 baseline is not specified: the paper lists o1 only as an NFT baseline, with no API model version, sampling temperature, number of evaluation runs, interaction budget, or example transcript, and it does not explicitly confirm that o1 received the same ReAct prompt with the same in-context example, the same API-doc access, the same 50-interaction cap, and the same code-cell parsing as LOOP. Since the abstract's central claim is a 9 pp / 15% improvement over o1, this comparison is load-bearing and currently not independently verifiable. Please provide the complete o1 evaluation protocol and, ideally, release the evaluation harness so that the comparison can be reproduced.","section":"Section 5.3, Table 1"},{"comment":"The algorithm specification in Algorithm 1 (line 4) assumes exactly K i.i.d. rollouts per task, but Appendix D states that rollout collection is early-stopped once at least 4 rollouts per task and 90% of the total rollouts have been collected. The paper does not state whether the leave-one-out baseline in Eq. 3 is computed with the actual collected group size or with the nominal K=6; if the nominal K is used after early stopping, the baseline is no longer the mean of the other rollouts and the stated unbiasedness of Eq. 3 no longer holds. Please clarify how the effective group size is used in the advantage computation and, if necessary, correct the estimator or the algorithm description.","section":"Algorithm 1, Appendix D"},{"comment":"The derivation of the importance-weight cancellation in Eq. 8 assumes that the environment is deterministic and policy-independent, formalized by the indicator I(s0,x) in Eq. 6. This matches AppWorld and is acknowledged in Section 6 as a missing feature, but the paper presents the POMDP formulation and LOOP derivation in a general way. The text should state explicitly that in stochastic or policy-dependent environments the leave-one-out baseline in Eq. 3 is biased with respect to the sampling policy and that Eq. 8 no longer holds; currently this limitation is only implicit in a footnote and in the discussion of AppWorld's scope.","section":"Eqs. 6-8, Section 6"}],"minor_comments":[{"comment":"Section 5.2 says '24 out of 30 scenarios' while Section 5.1 reports a train split of 35 scenarios; please clarify whether 30 refers to the number of available difficulty-1/2 scenarios after filtering or whether one of these numbers is a typo.","section":"Section 5.2, Appendix D.1"},{"comment":"The headline 71.3 TGC and 9 pp o1 comparison come from the best checkpoint of the best training run, while the multi-run average for LOOP (token) is 66.4 TGC on Test-N. The tables do disclose this selection protocol, but the abstract and Section 5.4 should state it more prominently so that the best-run result is not read as expected performance.","section":"Table 1 vs. Table 2, Section 5.4"},{"comment":"There are minor typographical issues in the appendix, including 'at at gradient step' in Appendix G.3 and the missing space in 'K = 6rollouts' in Section 5.2; these should be corrected in a final pass.","section":"Appendix G"},{"comment":"No code or checkpoints are released. Given the closed-source nature of the o1 baseline and the complexity of the AppWorld harness, releasing the evaluation and training code, or at least the exact o1 evaluation script, would substantially strengthen the reproducibility of the central claims.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LOOP is a credible empirical result: a 32B model fine-tuned with a PPO variant that uses a leave-one-out baseline and per-token importance weights, trained on 24 AppWorld scenarios, beats o1 and GPT-4o on AppWorld. The controlled comparisons in Table 2 — three training runs per method, shared hyperparameters — are the strongest part of the paper. The algorithm itself is not new in its parts; the paper says so, citing Gupta et al. (2025). What is new is the application to a stateful multi-app API environment and the clean comparison of token/turn/trajectory importance weighting. The behavioral analysis (fewer assumptions, fewer dummy values, more doc lookups, persistence after API errors) is concrete and matches the full rollouts in the appendix.\n\nThe soft spots are mostly about verification. The o1 baseline protocol is the load-bearing one: Section 5.3 lists o1 with no prompt, no harness details, no interaction cap, no seeds, no transcripts. The headline \"beats o1 by 9pp\" depends on o1 running under the same ReAct prompt and 50-interaction limit. A mismatch of even a few details could account for or erase that gap. The stress-test note is right that this matters more than the reader's stochasticity concern: AppWorld is deterministic (Section 6, footnote 1), so the i.i.d. rollout assumption holds here, and the importance ratio derivation in Eq. 8 does not depend on environment stochasticity for the reported numbers. That concern is about generality, not correctness.\n\nSecond, no code, checkpoints, or evaluation harness are released, and a few hyperparameters (clipping epsilon, exact advantage pruning threshold) are left underspecified. For a paper whose central claim is an empirical SOTA, that matters. It is fixable.\n\nWho should read this: anyone doing RL for LLM agents. It deserves a serious referee. My recommendation is conditional acceptance: release the artifacts and pin down the o1 evaluation protocol before publication. The core result — per-token LOOP beats on-policy baselines with the same data and hyperparameters — is well supported by Table 2.","headline":"LOOP is a credible empirical result—a 32B model trained on 24 AppWorld scenarios with per-token PPO beats o1 and GPT-4o—but the unreported o1 baseline protocol and missing artifacts keep the SOTA claim from being fully verifiable.","tokens_in":39012,"tokens_out":2547,"would_cite":true,"duration_ms":22043,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a 32-billion-parameter interactive agent trained with LOOP, a PPO variant using leave-one-out advantages and per-token importance weights, outperforms o1 on AppWorld after training on just 24 scenarios.","keywords":["reinforcement learning","interactive digital agents","proximal policy optimization","leave-one-out advantage","AppWorld","long-horizon tasks","large language models","policy gradient"],"falsifier":"Run the same LOOP training on a version of AppWorld in which API responses are nondeterministic (e.g., random delays, shuffled query results, or transient errors) while rewards and tasks stay identical; if task goal completion drops sharply or training diverges compared with the deterministic environment, the i.i.d.-rollout assumption is the cause. A cheaper check is to compare LOOP against an oracle on-policy advantage estimator in the standard environment and look for a systematic gap that grows as the policy drifts off-policy.","tokens_in":37962,"feed_emoji":"🤖","tokens_out":7950,"duration_ms":55213,"temperature":0.7,"pith_summary":"The paper sets out to show that reinforcement learning can train interactive digital agents inside the stateful environments they are meant to operate in, rather than relying on instruction-tuned models that have never practiced in those environments. It introduces LOOP, a proximal policy optimization variant that estimates advantages by leaving one rollout out of a group and uses per-token importance weights to allow off-policy updates, all without a value network or extra model copies. Trained on 24 AppWorld scenarios with a simple unit-test reward, a 32-billion-parameter agent reaches 71.3 task goal completion on the test-normal split and 45.7 on test-challenge, surpassing the much larger o1 reasoning model by 9 percentage points. The result matters because prior methods completed fewer than half of AppWorld tasks, and the paper shows that RL with tiny data and a task-completion reward can teach agents to consult API documentation, avoid assumptions, and recover from failed calls.","feed_headline":"RL-trained agent beats o1 on AppWorld by 9 points","feed_subtitle":"LOOP trains a 32B model on just 24 scenarios, with no value network, to outscore far larger reasoning models.","key_machinery":"The load-bearing object is the LOOP update, which pairs the leave-one-out advantage estimator of Eq. (3) with the per-token PPO objective of Eq. (5). The leave-one-out baseline gives an unbiased, low-variance Monte Carlo advantage without a value network, while per-token importance weights keep off-policy gradient updates stable by clipping each token's probability ratio independently. This combination lets the algorithm reuse K=6 rollouts per task across multiple gradient epochs, making it sample-efficient and memory-efficient, since only one LLM copy is in memory.","core_discovery":"LOOP combines PPO with a leave-one-out advantage estimate: for each task, K rollouts are sampled i.i.d., and each rollout's advantage is its return minus the average return of the other K-1 rollouts. The PPO objective is applied with per-token importance weights, so the policy can be updated multiple times over reused rollouts while staying inside a trust region, and no critic network is needed. The paper formalizes the agent-environment interaction as a POMDP and assumes the environment response is a deterministic function of initial state and agent tokens, which makes the trajectory distribution factor into policy token probabilities. Under this model, LOOP trains Qwen2.5-32B-Instruct with LoRA on 24 scenarios and 6 rollouts per task, maximizing the fraction of unit tests passed, and achieves state-of-the-art results on both AppWorld splits. The paper also reports that per-token importance weighting outperforms per-turn and per-trajectory variants and that normalizing rewards by group standard deviation hurts performance.","pith_inferences":["A direct consequence the authors leave implicit is that LOOP should transfer to other executable environments with unit-test-style rewards, such as tool-use benchmarks or code repair, whenever rollouts are cheap enough to sample K per task.","The method's reliance on i.i.d. rollouts implies that stochastic environments would require an environment-aware baseline or a learned critic, potentially reintroducing the memory cost LOOP avoids.","The observed rollout diversity (many distinct solution strategies per task) suggests a testable extension: deliberately encouraging diversity, e.g., via temperature schedules or entropy bonuses, may improve generalization further.","One could quantify how much of the gain comes from the leave-one-out baseline versus the per-token clipping by ablating each component independently on the same training runs."],"forward_implications":["RL with a task-completion reward can be a practical training signal for long-horizon interactive agents, even with only a few dozen training scenarios.","LOOP's memory footprint equals fine-tuning a single LLM, so the method scales to agents where a separate critic or reference model would be prohibitive.","The trained agent generalizes beyond training scenarios, and its behavior shifts toward reading API docs, avoiding assumptions, and persisting after failed calls.","Per-token importance weighting is the key design choice; per-turn and per-trajectory variants are less stable and clip more often.","Reward normalization by group standard deviation, as used in GRPO, is not beneficial in this setting and lowers performance."],"supporting_citations":[{"why":"Supplies the leave-one-out advantage estimator used in Eq. (3).","marker":"(Kool et al., 2019)"},{"why":"Provides the PPO trust-region objective that LOOP adapts with per-token importance weights.","marker":"(Schulman et al., 2017)"},{"why":"Shows RLOO works without a critic and is the on-policy special case of LOOP.","marker":"(Ahmadian et al., 2024)"},{"why":"Defines AppWorld, its tasks, unit tests, and the TGC and SGC evaluation metrics.","marker":"(Trivedi et al., 2024)"},{"why":"Provides Qwen2.5-32B-Instruct, the base LLM used for all fine-tuning experiments.","marker":"(Yang et al., 2024)"},{"why":"Defines GRPO, the main comparison baseline with group reward normalization.","marker":"(Shao et al., 2024)"},{"why":"Defines Agent Q and the DPO-MCTS method used as a comparison baseline.","marker":"(Putta et al., 2024)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method applied to the base model.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["RL agent beats o1 on AppWorld by 9 points","No critic, no value net: RL agent tops o1 on AppWorld","Trained on 24 scenarios, 32B RL agent outdoes o1","LOOP RL: 32B model, no critic, beats o1 on AppWorld","RL without critic: 32B agent surpasses o1 on AppWorld"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The leave-one-out baseline is unbiased only if the K rollouts for a task are independent samples from the same policy, which requires the environment's responses to be deterministic and unaffected by which policy produced the tokens.","fun_headline_variants_meta":{"raw":{"variants":["RL agent beats o1 on AppWorld by 9 points","No critic, no value net: RL agent tops o1 on AppWorld","Trained on 24 scenarios, 32B RL agent outdoes o1","LOOP RL: 32B model, no critic, beats o1 on AppWorld","RL without critic: 32B agent surpasses o1 on AppWorld"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001095,"raw_usage":{"total_tokens":4599,"prompt_tokens":998,"completion_tokens":3601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":3513}},"tokens_in":614,"tokens_out":3601,"duration_ms":20716,"temperature":1.0,"reasoning_tokens":3513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:51:08.439896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LOOP training on a version of AppWorld in which API responses are nondeterministic (e.g., random delays, shuffled query results, or transient errors) while rewards and tasks stay identical; if task goal completion drops sharply or training diverges compared with the deterministic environment, the i.i.d.-rollout assumption is the cause. A cheaper check is to compare LOOP against an oracle on-policy advantage estimator in the standard environment and look for a systematic gap that grows as the policy drifts off-policy.","supporting_citations":[{"cited_title":"Buy 4 reinforce samples, get a baseline for free! In ICLR 2019 Workshops, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the leave-one-out advantage estimator used in Eq. (3)."},{"cited_title":"Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLMs","cited_arxiv_id":null,"evidence_quote":"Shows RLOO works without a critic and is the on-policy special case of LOOP."},{"cited_title":"A pp W orld: A controllable world of apps and people for benchmarking interactive coding agents","cited_arxiv_id":null,"evidence_quote":"Defines AppWorld, its tasks, unit tests, and the TGC and SGC evaluation metrics."}],"review_version":1}