Pith. sign in

REVIEW 12 cited by

Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.07773 v4 pith:ZCMT7GW5 submitted 2025-05-12 cs.AI

Xinji Mai , Haotian Xu , Zhong-Zhi Li , Xing W , Weinong Wang , Jian Hu , Yingying Zhang , Wenqiang Zhang This is my paper · ORCID
classification cs.AI
keywords codeexecutionreasoningtrainingagentmathematicalasyncfindings
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    ReSum trains LLMs via RLVR to self-summarize reasoning trajectories, yielding 4% average performance gains and 18.6% shorter rollouts through contrastive rollout branches.

  2. PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

    cs.AI 2026-05 reject novelty 7.0 of 10

    Training an LLM as a multi-turn agent that runs and repairs solver code raises verified optimization solve rates, with the 4B PEARL model outperforming DeepSeek-V3.2-685B in aggregate.

  3. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

    cs.AI 2025-11 unverdicted novelty 7.0 of 10

    Current LLM agents often fail to find cost-optimal tool-use plans and degrade sharply when tool costs or availability change mid-task.

  4. rStar2-Agent: Agentic Reasoning Technical Report

    cs.CL 2025-08 conditional novelty 7.0 of 10

    A 14B model trained with agentic RL and a resample-on-correct rollout strategy scores 80.6% on AIME24 and 69.8% on AIME25, nearly matching DeepSeek-R1 (671B) in one week on 64 GPUs.

  5. Making Expert Reasoning Learnable with Self-Distillation

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A self-distillation method (DAIL) converts compressed expert solutions into in-distribution reasoning traces and trains with a contrastive objective, yielding pass@k gains on hard math benchmarks with under 1,000 examples.

  6. Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.

  7. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.

  8. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  9. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0 of 10

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.

  10. AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    AutoTIR applies GRPO with a hand-designed action reward so a 7B instruct model learns to mix search and code tools, beating tool-using baselines on ten benchmarks.

  11. CoRT: Code-integrated Reasoning within Thinking

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Inserting targeted hints into a few training examples teaches reasoning models to compute with Python instead of text, improving accuracy and cutting token use by 30 to 50 percent.

  12. Towards Effective Code-Integrated Reasoning

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A reinforcement learning recipe with precise code-block matching, masked tool feedback, and a progressive interaction budget improves the accuracy and token efficiency of code-integrated math reasoning.

Pith tools