Pith. sign in

REVIEW 23 cited by

Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.07773 v4 pith:ZCMT7GW5 submitted 2025-05-12 cs.AI

Xinji Mai , Haotian Xu , Zhong-Zhi Li , Xing W , Weinong Wang , Jian Hu , Yingying Zhang , Wenqiang Zhang This is my paper
classification cs.AI
keywords codeexecutionreasoningtrainingagentmathematicalasyncfindings
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.

Discussion (0). Sign in to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    ReSum trains LLMs via RLVR to self-summarize reasoning trajectories, yielding 4% average performance gains and 18.6% shorter rollouts through contrastive rollout branches.

  2. PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

    cs.AI 2026-05 reject novelty 7.0 of 10

    Training an LLM as a multi-turn agent that runs and repairs solver code raises verified optimization solve rates, with the 4B PEARL model outperforming DeepSeek-V3.2-685B in aggregate.

  3. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

    cs.AI 2025-11 unverdicted novelty 7.0 of 10

    CostBench is a new travel-planning benchmark showing that leading LLM agents often fail to select cost-optimal tool sequences and drop sharply in performance when facing dynamic events like tool failures or cost changes.

  4. rStar2-Agent: Agentic Reasoning Technical Report

    cs.CL 2025-08 conditional novelty 7.0 of 10

    A 14B model trained with agentic RL and a resample-on-correct rollout strategy scores 80.6% on AIME24 and 69.8% on AIME25, nearly matching DeepSeek-R1 (671B) in one week on 64 GPUs.

  5. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 conditional novelty 6.0 of 10

    ReSum's contrastive RL branching on self-summarization points improves LLM math reasoning accuracy by about 4% and shortens rollouts by about 18.6% across tested backbones.

  6. PruneTIR: Inference-Time Tool Call Pruning for Effective yet Efficient Tool-Integrated Reasoning

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    PruneTIR prunes erroneous tool-call trajectories during LLM inference via three trigger-based components to raise Pass@1 accuracy and efficiency while shortening context.

  7. Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    MAPO improves multimodal chain-of-thought reasoning by requiring explicit textual descriptions of visual tool results and using a novel advantage estimator that combines semantic alignment with task rewards.

  8. Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    MARL-Rad trains region-specific and global agents with reinforcement learning on clinical rewards to produce more accurate radiology reports than prior methods on MIMIC-CXR and IU X-ray datasets.

  9. Making Expert Reasoning Learnable with Self-Distillation

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A self-distillation method (DAIL) converts compressed expert solutions into in-distribution reasoning traces and trains with a contrastive objective, yielding pass@k gains on hard math benchmarks with under 1,000 examples.

  10. Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.

  11. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

    cs.AI 2025-11 conditional novelty 6.0 of 10

    Current LLM agents often fail to find cost-optimal tool-use plans and degrade sharply when tool costs or availability change mid-task.

  12. ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards

    cs.CL 2025-10 unverdicted novelty 6.0 of 10

    ReSeek adds self-correction via a JUDGE action and a dense instructive reward (correctness plus utility) to RL training of search agents, yielding higher success and faithfulness on a new contamination-resistant benchmark.

  13. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.

  14. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  15. Latent Visual States for Efficient Multimodal Reasoning

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    EVA generates adaptive Latent_slot tokens as internal visual thoughts, trained end-to-end with text tokens via D-GSPO on the EVA-230K dataset, claiming performance gains and better inference efficiency.

  16. STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    STARE applies surprisal-guided token-level advantage reweighting plus a target-entropy gate to stabilize entropy in GRPO RL for LLMs, yielding stable training and 4-8% gains on AIME24/25 over baselines.

  17. IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    IAPO is an RL method that aligns model input attributions with a teacher to improve tool-calling in multimodal SLMs, reporting 3% average VQA accuracy gains on Qwen2.5-VL-3B across six tests.

  18. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0 of 10

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.

  19. Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search

    cs.CV 2025-09 unverdicted novelty 5.0 of 10

    Mini-o3 scales visual search reasoning to tens of interaction turns via a new probe dataset, iterative trajectory collection, and over-turn masking in RL, claiming SOTA performance while training only up to six turns.

  20. AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    AutoTIR applies GRPO with a hand-designed action reward so a 7B instruct model learns to mix search and code tools, beating tool-using baselines on ten benchmarks.

  21. Rethinking Agentic Reinforcement Learning In Large Language Models

    cs.AI 2026-04 unverdicted novelty 3.0 of 10

    The paper reviews conceptual foundations, methodological innovations, effective designs, critical challenges, and future directions for LLM-based Agentic Reinforcement Learning.

  22. Rethinking Agentic Reinforcement Learning In Large Language Models

    cs.AI 2026-04 unverdicted novelty 2.0 of 10

    The paper surveys the conceptual foundations, methodological innovations, challenges, and future directions of agentic reinforcement learning frameworks that embed cognitive capabilities like meta-reasoning and self-r...

  23. Rethinking Agentic Reinforcement Learning In Large Language Models

    cs.AI 2026-04 unverdicted novelty 2.0 of 10

    This review synthesizes conceptual foundations, methods, challenges, and future directions for agentic reinforcement learning in large language models.

Pith tools