pith. sign in

arxiv: 2606.19047 · v1 · pith:SIIZPKUKnew · submitted 2026-06-17 · 💻 cs.AI

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Ruishan Fang , Siyuan Lu , Chenyi Zhuang , Tao Lin This is my paper

Pith reviewed 2026-06-26 20:53 UTC · model grok-4.3

classification 💻 cs.AI
keywords multi-turn tool-usereinforcement learningonline data synthesisreward variancecapability boundarydynamic replay buffertool-use agentsGRPO
0
0 comments X

The pith

RODS detects shifting capability boundaries via rollout reward variance and synthesizes matching multi-turn variants to train tool-use agents from a small dynamic pool.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Static datasets for multi-turn tool-use reinforcement learning lose value quickly because the agent's improving policy moves the set of informative tasks, leaving mostly easy or impossible examples. The paper shows that the largest policy gradients come from tasks where rollout rewards have high variance, meaning successes and failures occur in roughly equal measure. RODS turns this observation into a practical loop: it re-uses the variance already computed during training to pick boundary samples, generates new variants that preserve the original tasks' API structure and dependency depth, and keeps a small active pool of roughly 800 examples drawn from 400 human seeds. If the approach holds, agents can reach performance comparable to training on 17 000 fixed trajectories while computing roughly twenty times fewer total rollouts and outperforming both fixed-data RL and simple environment augmentation.

Core claim

RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies samples near the agent's capability boundary, synthesizes new multi-turn variants matching their structural complexity via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of approximately 800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories.

What carries the argument

The reward-driven online data synthesis loop that uses observed reward variance from training rollouts to select boundary samples and then applies skill-aligned resampling to produce new multi-turn tasks of matching API topology and dependency depth.

If this is right

  • A small initial set of human demonstrations can seed a training process that expands the data pool in step with the policy's progress.
  • The total number of trajectories needed for training drops by a factor of roughly twenty compared with a large static dataset while reaching similar final performance.
  • The method outperforms both purely fixed-data reinforcement learning and non-adaptive environment augmentation under the same controlled conditions.
  • The active pool size stays near 800 samples even as the policy improves, because new boundary tasks replace those that have become too easy or too hard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same variance-driven selection could be applied to other reinforcement-learning settings where task difficulty shifts, such as multi-step planning or interactive environments.
  • If the resampling preserves complexity distributions accurately, the approach could lower the human annotation burden for training agents that use external tools.
  • Integration with algorithms other than GRPO that also emphasize high-variance samples might further reduce the data required for policy improvement.

Load-bearing premise

Reward variance observed in training rollouts reliably marks the moving capability boundary, and the resampling step can produce new tasks whose structural complexity matches the originals without adding bias or extra inference cost.

What would settle it

An experiment that replaces the variance-based selection with random selection of the same number of tasks and shows no performance difference, or that measures structural complexity metrics and finds the synthesized tasks deviate systematically from the human seeds.

Figures

Figures reproduced from arXiv: 2606.19047 by Chenyi Zhuang, Ruishan Fang, Siyuan Lu, Tao Lin.

Figure 1
Figure 1. Figure 1: Current limitations in static data training vs. RODS dynamic synthesis paradigm. (a) Data scarcity (C1): High-quality multi-turn tool-use datasets require massive human annotation effort. (b) Capability vs. static Data (C2): As the model learns, static data becomes mastered, leading to a loss of gradient signal. (c) Semantic disjointedness (C3): Naively stitching single-turn queries creates disjointed inte… view at source ↗
Figure 2
Figure 2. Figure 2: The RODS closed-loop RL-data synthesis architecture. (Top) Reward Calculation & Agent Training: The agent trains on a mixed dataset via GRPO; the Progress Reward (RP) identifies boundary seeds and feeds them back to the data engine. (Bottom-left) Plan & Execute: The Planner Agent selects a function sequence from the API graph; the Execution Orchestrator instantiates it on a simulation environment with envi… view at source ↗
Figure 3
Figure 3. Figure 3: The RODS dynamic synthesis mechanism during stage 3 training. (Left) Evolution of the data space partition. RODS breaks the static capacity limit (400 tasks) by continuously synthesizing active boundary data and evicting mastered tasks, effectively expanding the training curriculum without exploding memory. (Middle) Reward trajectory of newly synthesized variants upon injection. The continuous data generat… view at source ↗
Figure 4
Figure 4. Figure 4: Data scaling analysis. Performance on BFCL V3 as a function of the generated pool cap Pmax. Pmax = 0 corresponds to the static baseline (static pool of 400 items). (Left) Performance across sub-splits (Base, Missing Functions, Missing Parameters, Long-Context). (Right) The overall average performance (solid orange) steadily improves as Pmax increases, with specific average scores annotated. concentrates ne… view at source ↗
read the original abstract

Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent's capability boundary -- where successes and failures are roughly balanced -- contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ~800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes RODS (Reward-Driven Online Data Synthesis) for multi-turn tool-use RL agents. It observes that GRPO gradients concentrate on high reward-variance tasks near the capability boundary (via the Popoviciu upper bound), which depletes in static datasets as the boundary shifts. RODS repurposes rollout variance as a zero-cost detector to identify boundary samples, synthesizes new multi-turn variants via skill-aligned resampling that matches structural complexity (API topology, dependency depth), and maintains a dynamic replay buffer. Starting from 400 human seeds and an active pool of ~800 samples, it claims performance parity with a 17K-sample offline pipeline using ~20x fewer trajectories while outperforming fixed-data RL and environment augmentation.

Significance. If the central empirical claims are substantiated, the work offers a practical mechanism to mitigate data depletion in multi-turn tool-use RL by dynamically targeting informative samples without additional inference cost. The reuse of existing variance statistics for boundary detection is a low-overhead contribution. The approach could reduce reliance on large curated datasets, but its validity rests on unverified properties of the synthesis step.

major comments (2)
  1. [Abstract] Abstract: The headline claim of comparable performance to a 17K-sample offline pipeline with ~20x fewer trajectories (while maintaining an active pool of ~800 samples) is presented without any experimental details, ablation results, error bars, baseline definitions, or controls for confounders such as task difficulty or data distribution shifts. This renders the quantitative result impossible to evaluate.
  2. [Abstract] Abstract: The skill-aligned resampling pipeline is stated to produce new multi-turn variants whose structural complexity (API topology and dependency depth) matches the human seeds, yet no quantitative comparison, statistics, or validation measurements are supplied. This assumption is load-bearing for the claim that gains arise from accurate boundary targeting rather than from changes in the data distribution induced by synthesis.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below. We agree that the abstract would benefit from greater self-containment and will revise it to reference key experimental details and validation results from the main text.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The headline claim of comparable performance to a 17K-sample offline pipeline with ~20x fewer trajectories (while maintaining an active pool of ~800 samples) is presented without any experimental details, ablation results, error bars, baseline definitions, or controls for confounders such as task difficulty or data distribution shifts. This renders the quantitative result impossible to evaluate.

    Authors: We agree that the abstract presents the headline claim without sufficient experimental context. The main manuscript details the experimental setup, including baseline definitions (fixed-data RL, environment augmentation, and the 17K offline pipeline), ablation studies, error bars from multiple random seeds, and controls for task difficulty and distribution shifts (Sections 4–5). To address the concern, we will revise the abstract to briefly note that the result holds under controlled experiments with reported error bars and ablations, directing readers to the main text for full specifications. revision: yes

  2. Referee: [Abstract] Abstract: The skill-aligned resampling pipeline is stated to produce new multi-turn variants whose structural complexity (API topology and dependency depth) matches the human seeds, yet no quantitative comparison, statistics, or validation measurements are supplied. This assumption is load-bearing for the claim that gains arise from accurate boundary targeting rather than from changes in the data distribution induced by synthesis.

    Authors: We acknowledge that the abstract does not supply quantitative validation for the structural matching. The full manuscript provides these measurements, including distributional comparisons of API topology and dependency depth between human seeds and synthesized samples (Section 3.2, with accompanying tables and figures showing close alignment). We will revise the abstract to include a concise reference to this validation (e.g., noting that complexity metrics were matched within reported tolerances), thereby clarifying that the performance gains are not attributable to unintended distribution shifts. revision: yes

Circularity Check

0 steps flagged

No significant circularity; variance detector reuses existing rollout statistics without self-referential fitting or definition

full rationale

The paper's chain begins with an observation about GRPO gradients concentrating on high-variance tasks (via Popoviciu bound), then directly repurposes the same per-sample variance computed during training rollouts as a boundary detector. This reuse is zero-cost and non-fitted; no parameter is optimized against the target performance metric. The subsequent skill-aligned resampling pipeline is presented as an independent generation step whose structural-matching claim is external to the variance equations. No self-citation chain, ansatz smuggling, or renaming of known results is load-bearing. The headline efficiency result is therefore an empirical outcome, not a definitional tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that reward variance serves as an effective boundary detector and that the resampling pipeline preserves task difficulty without extra cost; no free parameters or invented entities are mentioned in the abstract.

axioms (1)
  • domain assumption The gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound.
    Invoked as the starting observation that motivates the boundary detector.

pith-pipeline@v0.9.1-grok · 5760 in / 1313 out tokens · 25356 ms · 2026-06-26T20:53:48.272507+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references

  1. [1]

    Loading the latesttracker.jsonto recover PromptTracker historical windows

  2. [2]

    Replaying allexpanded_*.jsonlfiles to rebuild the list of generated data

  3. [3]

    This guarantees zero data loss across restart boundaries and lets the curriculum resume where it was interrupted

    Deterministically recomputing the data retirement logic from stored metrics. This guarantees zero data loss across restart boundaries and lets the curriculum resume where it was interrupted. B.5 Motivating Analysis: Gradient Variance at the Capability Boundary We present the analysis motivating our boundary-targeting design heuristic. Specifically, we est...

  4. [4]

    Select functions that test similar skills to the seed task (e.g., if the seed requires multi-step parameter passing, your plan should also require it)

  5. [5]

    HIGH-LEVEL functions are preferred when available – they produce richer, multi-step call sequences after decomposition

  6. [6]

    Each turn should have 1-3 functions from the SAME class

  7. [7]

    For multi-class seeds, alternate classes across turns (e.g., Turn 1: ClassA, Turn 2: ClassB, Turn 3: ClassA)

  8. [8]

    Output 2-5 turns total

  9. [9]

    Closed”→“Open

    Ensure the sequence is logically coherent (e.g., authenticate before posting, fill fuel before driving) 14 Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents # Output Format First, analyze the seed task and plan your approach inside <reason></reason> tags. Then output a brief narrative scenario (2-3 sentences) inside<narrative>tags. Finall...

  10. [10]

    Accurately describes what the GT function calls do (without mentioning function names)

  11. [11]

    Sounds like a real user talking to an AI assistant

  12. [12]

    Is semantically coherent with the narrative scenario

  13. [13]

    Does NOT contain special characters that could cause parsing issues

  14. [14]

    IMPORTANT: Output exactly{num_turns}queries, one for each turn

    For turns with multiple function calls, the query should naturally imply all of them # Output Format Output each query inside<query>tags, one per turn: <query>Turn 1 user query here</query> <query>Turn 2 user query here</query> . . . IMPORTANT: Output exactly{num_turns}queries, one for each turn. C.4 Quality Judge Agent The Quality Judge validates synthes...

  15. [15]

    Query-GT Alignment: Does each user query accurately describe what the GT function calls do?

  16. [16]

    State Consistency: Do parameter values reflect the actual environment config?

  17. [17]

    Cross-Turn Coherence: Is there logical state progression across turns?

  18. [18]

    Query Naturalness: Do queries sound like real user requests?

  19. [19]

    Thought Process

    Structural Correctness: Are deliberate ambiguities (e.g., missing parameters) properly reflected? 17 Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents # Automatic Rejection Patterns If the query contains any of these patterns, REJECT immediately: • “Thought Process”, “Construct Query”, “Step 1:”, “Step 2:” • Function names or parameter na...

  20. [20]

    It naturally and accurately describes what the GT function calls actually do

  21. [21]

    It sounds like a real user talking to an AI assistant

  22. [22]

    It does NOT mention function names, parameter names, or technical details

  23. [23]

    It does NOT contain special characters that could cause parsing issues

  24. [24]

    The pipeline processes each turn sequentially through the following stages:

    It fixes the specific issue described above # Output Format Output ONLY the rewritten query inside<answer>tags: <answer>your rewritten query here</answer> D Deterministic Execution Pipeline Internals The Stage II execution pipeline instantiates the abstract plan within a Python-based sandbox environment that faithfully replicates the training environment’...

  25. [25]

    HIGH-LEVEL functions are automatically decomposed into sequences of BOTTOM-LEVEL API calls

    Function Sampling:Given the Planner’s output specifying which functions to call per turn, the pipeline samples concrete function instances from the available catalog. HIGH-LEVEL functions are automatically decomposed into sequences of BOTTOM-LEVEL API calls

  26. [26]

    Parameter Generation:For each selected function, an LLM generates concrete parameter values condi- tioned on the function schema, the current environment state Ct, and any dependencies from previous turns

  27. [27]

    The VM maintains a complete environment state (file systems, account balances, database entries, etc.) and returns execution results or error messages

    VM Execution:The parameterized function call is executed against the sandbox VM. The VM maintains a complete environment state (file systems, account balances, database entries, etc.) and returns execution results or error messages

  28. [28]

    Query Generation:A per-class LLM prompt generates a natural user query that describes the function call’s intent without exposing function names or parameters

  29. [29]

    Use BOTTOM-LEVEL functions only

    Query Verification:A verification LLM checks whether the generated query’s semantics align with the GT function calls, rejecting queries that are misaligned or contain data generation artifacts. If any stage fails, a structured error is recorded with the error type, failing function, turn number, and diagnostic detail. This structured error feeds directly...

  30. [30]

    Patches are accumulated across retries via recursive deep-merge, where later patches override earlier ones for the same field but coexist for different fields

    Environment Config Patching:If the error type is patchable (param_gen_failed,vm_exec_failed,func_- sample_failed, or decompose_failed), the Config Patch Agent (Appendix C.2) analyzes the initial envi- ronment configuration and outputs structured XML patches to update the state. Patches are accumulated across retries via recursive deep-merge, where later p...

  31. [31]

    Use BOTTOM-LEVEL functions only

    Action Space Pruning:Simultaneously, the names of all functions involved in failures are extracted and added to a cumulative blocklist. On re-invocation, the Planner Agent receives: (a) the full failure history with structured error descriptions, (b) a list of specifically blocked function names, and (c) error- type-specific guidance (e.g., “Use BOTTOM-LE...

    2025