Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rewarding the reasoning process rather than only the final outcome drives a 7-billion-parameter agent to state-of-the-art results on unseen long-horizon tasks, including an 83.6% success rate on the hardest held-out split.

desk verdict Solid empirical recipe; the causal story about improved meta-reasoning overshoots the evidence. read the letter →

arxiv 2507.22844 v1 pith:FQ54GI46 submitted 2025-07-30 cs.LG cs.AI

Zijing Zhang , Ziyang Chen , Mingxiao Li , Zhaopeng Tu , Xiaolong Li This is my paper
classification cs.LGcs.AI
keywords reinforcementlearninglong-horizonagentsmeta-reasoningrewardsprocess-levelsupervisiongeneralizationinefficientexplorationlanguagesparse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a particular failure mode—not lack of model capacity—is what keeps reinforcement-trained language agents from generalizing on long-horizon tasks. The authors call it inefficient exploration: when only final task success is rewarded, agents learn to succeed through repetitive, invalid, or illogical action sequences, and those brittle strategies break on unseen tasks. Their proposed fix, RLVMR, makes the agent tag each reasoning step as planning, exploration, reflection, or monitoring, and then hands out dense, rule-based rewards for tag-compliant behavior that genuinely helps (new states explored, corrective actions after failures, well-formed output), on top of the final outcome reward. On two long-horizon benchmarks, RLVMR reports new state-of-the-art results, including an 83.6% success rate on the hardest unseen split with a 7-billion-parameter model, while cutting repetitive actions to about a third of the outcome-only baseline's rate on that split. The conclusion is that process-level supervision, not more parameters or more outcome signal, is the lever that makes agents robust to novel tasks.

What carries the argument

The load-bearing machinery is the meta-reasoning tag system plus the grouped relative advantage that assigns credit to it. Four XML-style tags (<planning>, <explore>, <reflection>, <monitor>) turn otherwise invisible cognitive steps into parseable, rule-checkable events, so that a lightweight program can reward each step without a learned reward model or critic. The GRPO-MR update then computes two advantage signals: a trajectory-level outcome advantage and a tag-group-level meta-reasoning advantage, and combines them with a weighting parameter. The tags make reasoning auditable, the rules make reward dense, and the grouped normalization lets the policy compare steps of the same cognitive type against each other, which is what steers behavior away from loops and toward corrective, exploratory, and plan-following action.

What would settle it

If a version of RLVMR whose dense rewards are attached to arbitrary surface features (for example, tag diversity or never repeating an action name) matched RLVMR's success-rate gains on the L2 split, the claim that meta-reasoning specifically causes the improvement would be refuted. A more direct test is to inspect whether the tags in held-out trajectories align with actual behavior—for instance, whether a <reflection> tag is followed by a different action after a failure—and whether removing that alignment by shuffling tags preserves the gains.

Watch

Extended reading notes

Core claim

The central claim is that outcome-only reinforcement learning entrenches flawed reasoning habits in long-horizon agents, and that this can be reversed by rewarding verifiable meta-reasoning steps during training. RLVMR has the agent emit one of four XML-style tags—planning, exploration, reflection, or monitoring—before each action, and programmatic rules convert those tags into dense rewards: planning is rewarded when the trajectory ends in success, exploration when the action targets a new object or location, reflection when a corrective action follows a run of failures, and malformed output is penalized. These step-level rewards are normalized within groups of the same tag, blended with the trajectory-level outcome advantage, and optimized with a critic-free policy gradient algorithm (GRPO-MR). The authors report state-of-the-art success on ALFWorld and ScienceWorld across seen and unseen splits; the 7-billion-parameter model reaches 83.6% on ALFWorld's hardest unseen split, up from 52.3% for outcome-only RL, and on that same split the repetitive action rate falls from 31.2% to 11.7%. Ablations show that removing the meta-reasoning advantage, the outcome signal, or the cold-start supervised phase each costs a substantial share of the gain, so the method's power is attributed to the combination rather than any single reward.

Load-bearing premise

The whole method rests on the assumption that the programmatic reward rules—rewarding planning only on successful trajectories, rewarding exploration only for new objects or locations, rewarding reflection only when corrective actions follow failures, and penalizing malformed output—faithfully reward genuine good reasoning rather than shallow surface behaviors that satisfy the rules.

Editorial extensions

If this is right

  • Outcome-only RL baselines are leaving large generalization gains on the table: on the hardest unseen ALFWorld split, RLVMR's 7B model beats the strongest competing method by 16.4 percentage points.
  • Process-level supervision compresses the efficiency burden of exploration: on unseen tasks, RLVMR keeps repetitive actions at 11.7% where outcome-only RL rises to 31.2%, and cuts average solution length by about 28% relative to outcome-only RL on the hardest split.
  • Smaller models can overtake much larger ones when reasoning is supervised: RLVMR's 1.5B model reaches 87.9% on an unseen ALFWorld split, above the 66.0% of a much larger model used with ReAct prompting.
  • Each component is load-bearing: ablations show dropping the meta-reasoning advantage drops L2 success from 56.3% to 45.3%, dropping the outcome signal collapses it to 12.5%, and dropping the cold-start phase lowers it to 40.6%.
  • Training becomes more stable and data-efficient: RLVMR converges within 100 RL epochs where baselines get 150, and its trajectory lengths decline steadily rather than growing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal claim holds, the same recipe should transfer to any domain where agent steps can be parsed and rule-checked (code edits, tool calls, robot actions), and the efficiency gains may be largest in settings where invalid steps are expensive.
  • A stress test left implicit in the paper: after RL, the tags may drift toward satisfying the rules superficially; decoding held-out trajectories and checking whether a <reflection> tag actually accompanies a change in behavior would separate genuine meta-reasoning from reward hacking.
  • The L2 gains suggest that out-of-distribution robustness in long-horizon agents is primarily a process-quality problem rather than a capacity problem; a direct corollary is that future scaling efforts should invest in reward design and verifiable intermediate signals along with parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes RLVMR, a GRPO-style RL method for LLM agents that adds dense rewards for four meta-reasoning tags (planning, explore, reflection, monitor) plus a format penalty, after a 200-trajectory cold-start SFT. On ALFWorld and ScienceWorld, the method reports large gains over SFT, GRPO, ETO, and GiGPO, including 83.6% on the author-defined L2 unseen split with Qwen2.5-7B, and the paper attributes these gains to improved reasoning quality (fewer repetitive and invalid actions). The key technical novelty is replacing a learned process reward model with programmatic rules and using a group-relative advantage per tag.

Significance. The paper is potentially useful: it offers a lightweight, rule-based way to inject process-level supervision into long-horizon agent RL, and the reported gains are large on a hard, held-out split. The paper also includes ablations of the outcome advantage, the meta-reasoning advantage, and the cold-start phase, and it makes a code URL available. However, the current evidence does not support the strong causal claim that the gains come from coherent meta-reasoning: the process rewards are partly outcome-contingent or syntactic, the efficiency metrics are partly definitional, and the empirical results are single-seed without variance. These issues are fixable with additional experiments and analysis.

major comments (5)
  1. [§3.4.1, Eq. (3)] The planning reward is granted only when the trajectory ultimately succeeds, so after the per-tag normalization in Eq. (3) it becomes a reweighted version of the trajectory outcome advantage. A policy can increase its planning-channel advantage by emitting a <planning> tag early in a trajectory that is already on a successful path, without producing a better plan. This confound undermines the paper's claim that dense process rewards, rather than outcome reweighting, drive the generalization gains. Please either define r_planning without the success gate or show that the L2 results are unchanged when the success-conditioned component is removed.
  2. [§3.4.1 and §4.2] The exploration reward ('targets a new object or location') and the reflection reward ('corrective action after a sequence of failures') are syntactic heuristics; for example, in ALFWorld's finite discrete space, cycling through object names in the latest observation can earn the exploration bonus without coherent exploration. The reported reductions in repetitive and invalid actions are therefore partly definitional, because the format penalty and the novelty bonus directly penalize exactly those behaviors. To support the causal claim, please add an ablation that isolates the format penalty, and a control with non-semantic dense rewards delivered at the same frequency.
  3. [Tables 1-2 and Figures 3-6] No variance or number of seeds is reported; each number appears to come from a single training run. With stochastic RL training for 1.5B and 7B models and with margins as small as 4.1 points on ScienceWorld L2 (RLVMR 32.2% vs ETO 28.1%), the SOTA claim is not statistically supported. Please report at least three seeds with mean and standard deviation, and state how many episodes are used for each evaluation split.
  4. [§2.1 and Appendix A.1] The L2 split is author-constructed by holding out entire task categories, but the paper does not give the exact task IDs, the number of tasks per category, or the procedure used to ensure no category leakage between training and evaluation. Because the paper claims a new SOTA on this split, the split must be released and documented; otherwise the results cannot be reproduced or compared with future work.
  5. [§4.4, Table 3] The ablation 'w/o AMC' removes both the meta-reasoning advantage and the format penalty, so the gap between the full model and that variant conflates two interventions. A separate variant that keeps the format penalty but removes the tag-conditional advantage is needed to determine whether the gain comes from meta-reasoning rewards or from output-format shaping. In addition, the ablation is reported only for the 1.5B model on L2, not for the main 7B result.
minor comments (6)
  1. [§4.1 and Figure 6] The text says RLVMR trains for 100 RL epochs while RL baselines train for 150, but Figure 6 is labeled 'Step' and appears to compare up to 150 steps; please clarify the unit and ensure the comparison uses matched environment-interaction budgets.
  2. [Eq. (5)] The KL penalty uses λ_KL and π_ref that are introduced only after the equation; please define them in the text.
  3. [References] The entries 'Feng et al. 2025a' and 'Feng et al. 2025b' both point to arXiv:2505.10978; one citation is incorrect.
  4. [Figures 3 and 5] The panel labels contain a typo ('Invlaid Actions'), and the first page duplicates the paper title; please fix these presentation issues.
  5. [§4.1] The comparison to GPT-4o, DeepSeek-V3 and DeepSeek-R1 uses those systems with ReAct prompting only; 'state-of-the-art' should be qualified so that the reader understands the comparison is to RL-trained 1.5B/7B baselines plus prompted larger models.
  6. [§3.3] The cold-start annotations are generated by GPT-4 and described as 'the most likely cognitive step', but no annotation-quality validation is reported; a small human-agreement check would strengthen the claim that the tags are meaningful.

Circularity Check

2 steps flagged · score 6.0 of 10

The 'improved reasoning quality' evidence is the same behavior the exploration reward is defined to enforce, and the planning reward is the outcome reward re-attributed; the benchmark scores themselves are not circular.

  1. self definitional [Section 3.4.1 (Meta-Reasoning-Aware Reward Shaping); Section 4.2 (Exploration Efficiency)]
    "• Exploration Reward (rexplore): Awarded if the current action targets a new object or location, discouraging redundancy. ... This confirms that our verifiable meta-reasoning rewards—such as the format penalty and the reward for exploring new states—successfully guide the agent away from flawed or redundant steps, leading to more direct and effective problem-solving."

    The paper's evidence for 'improved reasoning quality' is the Repetitive Action Rate, defined as 'the percentage of steps where the agent executes a meaningless repeated action.' The exploration reward is paid exactly when the action targets a new object or location, i.e., when the action is not a repetition of the previously targeted object or location. Maximizing r_explore therefore reduces the repetitive-action metric by construction. The paper explicitly credits the reward for this reduction ('This behavior is a direct outcome of our process-level rewards'), so using the same metric as independent confirmation that meta-reasoning quality improved is circular: the observed reduction is built into the training objective rather than being a free prediction of coherent reasoning.

  2. other [Section 3.4.1; Eq. (3) in Section 3.4.2]
    "• Planning Reward (rplanning): Awarded for a <planning> step if the trajectory ultimately succeeds. ... The core of GRPO-MR is the computation of a context-aware advantage. We group all steps within a batch that share the same meta-reasoning tag (e.g., all <explore> steps) and normalize their rewards within that group."

    For any <planning> step, rplanning is 1 iff the trajectory's final outcome R(τ) is success, and 0 otherwise; the reward definition contains no check of planning content. After grouping by tag and normalizing, the planning advantage is exactly a standardized success indicator among planning steps. Equation (4) then adds this to the trajectory-level advantage, which is itself the standardized outcome reward. Hence the 'planning' process signal is the outcome signal re-attributed to a tag; it carries no verifiable information about planning quality, so the claim that the planning reward provides independent process-level supervision reduces by construction.

full rationale

RLVMR's headline success rates (e.g., 83.6% on ALFWorld L2) are external empirical outcomes and are not derived from the reward definitions, so the main benchmark claim is not circular. No load-bearing self-citation chain exists: the cited RLVER and Sentient Agent-as-a-Judge works are prior work by overlapping authors but do not supply a uniqueness theorem or a premise the method depends on. The circularity is in the paper's explanatory claim: the behavior metrics used to 'confirm' improved reasoning quality (repetitive-action rate, invalid-action rate, action-length reduction) are directly optimized by the exploration reward and format penalty, so observing those reductions is partly definitional. The planning reward is also the outcome reward conditioned on a tag, so it adds no independent process signal. This makes the causal attribution 'gains stem from improved reasoning quality' partially circular, even though the empirical gains and ablations are real and the method could still work for the stated reason. Score 6 reflects partial, not total, circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on hand-set hyperparameters, a set of rule-based reward definitions whose faithfulness is assumed rather than measured, and a new evaluation split constructed by the authors. No new physical entities are introduced; the only invented construct is the tag taxonomy itself.

free parameters (5)
  • alpha (α) = 0.5
    Balances trajectory-level and meta-reasoning advantages in Eq. 4; set by hand without sensitivity study.
  • lambda_format = 0.1
    Penalty for malformed output, chosen without reported tuning; can dominate rare format errors.
  • lambda_KL = 0.01
    KL penalty coefficient in Eq. 5, hand-set.
  • max_steps_per_episode = 30
    Cap on episode length for both benchmarks; influences success metrics and the repetitive-action metric.
  • cold_start_trajectories = 200
    Number of GPT-4-annotated expert trajectories for SFT; stated as lightweight without a sweep.
assumptions (5)
  • standard math The clipped surrogate objective with KL penalty is a valid policy-gradient update (Eq. 5).
    Standard PPO/GRPO result, not proved in the paper.
  • domain assumption ALFWorld and ScienceWorld provide a terminal reward usable as R(τ).
    Sections 2.1 and A.1; the agent uses only final success, discarding intermediate ScienceWorld rewards.
  • domain assumption GPT-4 annotations of the 200 cold-start trajectories assign the true cognitive step preceding each action.
    Section 3.3 step 2; no inter-annotator agreement or validation is reported.
  • domain assumption The rule-based rewards (new object/location, corrective action after failures) are faithful and non-gameable proxies for good reasoning.
    Section 3.4.1; central to the causal claim, never tested against independent reasoning-quality metrics.
  • domain assumption The L2 split (held-out task categories) is a clean measure of generalization, with no leakage through the prompt template.
    Sections 2.1 and A.1; splits created by authors, leakage not checked.
invented entities (1)
  • Meta-reasoning tag taxonomy (planning, explore, reflection, monitor)
    purpose: Gives the policy a structured action space for cognitive steps and enables per-tag reward normalization.
    No external validation that these tags correspond to distinct or real cognitive states; the only evidence is the performance gain and selected case study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents." pith.science (2026). https://pith.science/paper/FQ54GI46

@misc{pith2026250722844,
  author       = {Pith},
  title        = {Pith review of: RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQ54GI46}},
  note         = {Machine review of arXiv:2507.22844}
}
read the original abstract

The development of autonomous agents for complex, long-horizon tasks is a central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term inefficient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, a novel framework that integrates dense, process-level supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps, such as planning, exploration, and reflection, and provides programmatic, rule-based rewards for actions that contribute to effective problem-solving. These process-centric rewards are combined with the final outcome signal and optimized using a critic-free policy gradient method. On the challenging ALFWorld and ScienceWorld benchmarks, RLVMR achieves new state-of-the-art results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.

Figures

Figures reproduced from arXiv: 2507.22844 by the authors.

Figure 1
Figure 1. Reinforcement learning with outcome-only rewards (e.g., GRPO) improves performance [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Performance of SFT and GRPO on ALFWorld. While SFT excels on seen tasks (L0) but [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. A schematic diagram of the RLVMR framework, which consists of two training phases: [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Exploration efficiency of RLVMR compared to SFT and GRPO baselines on ALFWorld. RLVMR consistently and significantly reduces both invalid and repetitive actions across all general￾ization levels and model sizes, demonstrating its effectiveness at mitigating inefficient…
Figure 6
Figure 6. Figure 6: The step count curves of different methods on two datasets during the RL training process. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Skill entropy, a reference-model-based measure of skill-switching difficulty, calibrates a new cross-skill benchmark and serves as an RL reward, more than doubling small models' scores.

  2. STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Normalized entropy locates trajectory-neglect steps; STAPO selectively optimizes them with a trajectory-aware reward and trajectory-independent penalty, beating group-based RL baselines on agent tasks.

  3. RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.

  4. TAPO: Transition-Aware Policy Optimization for LLM Agents

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Interleaving action-conditioned next-observation supervision with group RL on a shared LLM backbone consistently lifts long-horizon agent success over pure policy optimization.

  5. RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A 3B VLM trained with SFT plus GRPO and an LCS-based reward reaches 55.3% on EmbodiedBench's EB-ALFRED, beating GPT-4o-mini and the 7B REBP planner.

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems

    Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032, 2024

    arXiv 2024

  3. [3]

    Agent s: An open agentic framework that uses computers like a human

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164, 2024

    arXiv 2024

  4. [4]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 12461--12495, 2024

    2024

  5. [5]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025

    arXiv 2025

  6. [7]

    Group-in-group policy optimization for llm agent training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978, 2025 b

    arXiv 2025

  7. [8]

    Agentrefine: Enhancing agent generalization through refinement tuning

    Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint arXiv:2501.01702, 2025

    arXiv 2025

  8. [9]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025

    arXiv 2025

Show all 40 references
  1. [10]

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation

    Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023

    2023 arXiv

  2. [11]

    Metacognition: A literature review

    Emily R Lai. Metacognition: A literature review. 2011

    2011

  3. [12]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  4. [13]

    What is metacognition? Phi delta kappan, 87 0 (9): 0 696--699, 2006

    Michael E Martinez. What is metacognition? Phi delta kappan, 87 0 (9): 0 696--699, 2006

    2006

  5. [14]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  6. [15]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  7. [16]

    Agent planning with world knowledge model

    Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Advances in Neural Information Processing Systems, 37: 0 114843--114871, 2024

    2024

  8. [17]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In ICLR, 2024

    2024

  9. [18]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023

    2023

  10. [19]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  11. [20]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 8634--8652, 2023

    2023

  12. [21]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C \^o t \'e , Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020

    2010 arXiv

  13. [22]

    Trial and error: Exploration-based trajectory optimization for llm agents

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502, 2024

    2024 arXiv

  14. [23]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023

    2023 arXiv

  15. [24]

    Rlver: Reinforcement learning with verifiable emotion rewards for empathetic agents, 2025 a

    Peisong Wang, Ruotian Ma, Bang Zhang, Xingyu Chen, Zhiwei He, Kang Luo, Qingsong Lv, Qingxuan Jiang, Zheng Xie, Shanyi Wang, Yuan Li, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, and Xiaolong Li. Rlver: Reinforcement learning with verifiable emotion rewards for empathetic age...

    2025

  16. [25]

    Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022

    2022 arXiv

  17. [26]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025 b

    2025 arXiv

  18. [27]

    Agentgym: Evolving large language model-based agents across diverse environments

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.04151, 2024

    2024 arXiv

  19. [28]

    Watch every step! llm agent learning via iterative step-level process refinement

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! llm agent learning via iterative step-level process refinement. In EMNLP, 2024

    2024

  20. [29]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36: 0 71995--72007, 2023

    2023

  21. [30]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  22. [31]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  23. [32]

    Steptool: A step-grained reinforcement learning framework for tool learning in llms

    Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. Steptool: A step-grained reinforcement learning framework for tool learning in llms. 2024

    2024

  24. [33]

    Agent-r: Training language model agents to reflect via iterative self-training

    Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025

    2025 arXiv

  25. [34]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 3053--3077, 2024

    2024

  26. [35]

    Sentient agent as a judge: Evaluating higher-order social cognition in large language models, 2025

    Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, and Xiaolong Li. Sentient agent as a judge: Evaluating higher-order social cognition in large language models, 2025. URL https://ar...

    2025 arXiv

  27. [36]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the A...

    2024 doi

  28. [37]

    You only look at screens: Multimodal chain-of-action agents

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 3132--3149, 2024

    2024

  29. [38]

    Archer: training language model agents via hierarchical multi-turn rl

    Yifei Zhou and Andrea Zanette. Archer: training language model agents via hierarchical multi-turn rl. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 62178--62209, 2024

    2024

  30. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  31. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  32. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.