Recursive Agent Optimization

Apurva Gandhi; Aviral Kumar; Graham Neubig; Satyaki Chakraborty; Xiangjun Wang

arxiv: 2605.06639 · v1 · pith:YF6Q6WFEnew · submitted 2026-05-07 · 💻 cs.LG · cs.AI· cs.CL· cs.MA

Recursive Agent Optimization

Apurva Gandhi , Satyaki Chakraborty , Xiangjun Wang , Aviral Kumar , Graham Neubig This is my paper

Pith reviewed 2026-05-08 12:06 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CLcs.MA

keywords recursive agentsreinforcement learningtask delegationdivide-and-conquerinference-time scalinglanguage model agentsmulti-agent systems

0 comments

The pith

Reinforcement learning trains agents to recursively delegate sub-tasks to copies of themselves for divide-and-conquer scaling.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Recursive Agent Optimization, a reinforcement learning technique that trains agents to spawn new instantiations of themselves and delegate sub-tasks recursively. This creates an inference-time scaling method based on divide-and-conquer that lets agents handle problems too long for their context window. A sympathetic reader would care because the resulting agents train more efficiently, generalize to tasks much harder than their training distribution, and complete work in less wall-clock time than single-agent baselines.

Core claim

Recursive Agent Optimization is a reinforcement learning method for training recursive agents that implement inference-time scaling by delegating sub-tasks to new instantiations of themselves. The training teaches agents when and how to delegate and communicate, allowing natural divide-and-conquer on complex problems.

What carries the argument

Recursive Agent Optimization (RAO), the reinforcement learning approach that optimizes agents for spawning and delegating to recursive copies of themselves, which carries the argument by supplying the reward signal for effective delegation and communication rules.

Load-bearing premise

Reinforcement learning can teach agents reliable delegation and communication without creating overhead or failures that erase the benefits of breaking tasks into smaller pieces.

What would settle it

A controlled experiment comparing recursive agents against single-agent baselines on tasks exceeding the context window that finds no accuracy gain or increased errors from delegation failures would disprove the scaling and generalization claims.

Figures

Figures reproduced from arXiv: 2605.06639 by Apurva Gandhi, Aviral Kumar, Graham Neubig, Satyaki Chakraborty, Xiangjun Wang.

**Figure 1.** Figure 1: Example of recursive agent inference on a deep research or travel-planning task view at source ↗

**Figure 2.** Figure 2: RAO Reward Design. Each node receives a local reward from its own success and a delegation bonus from the success rate of its children. The example uses λ = 0.4. A recursive agent should learn both to solve its assigned task and, when useful, to delegate productively. Ideally, we can take advantage of node-local credit assignment: a sub-agent should receive signal from whether its own assigned task and i… view at source ↗

**Figure 3.** Figure 3: Figure adapted from Prasad et al. (2024) visualizing a TEXTCRAFT crafting tree. In order to craft the target beehive, the agent must first craft oak planks from oak logs. To study the properties of recursive-agent training in a controlled setting, we introduce TEXTCRAFT-SYNTH, inspired by the TEXTCRAFT benchmark from Prasad et al. (2024). In TEXTCRAFT, an agent is given an initial inventory and a target … view at source ↗

**Figure 4.** Figure 4: TEXTCRAFT-SYNTH training curvess (moving average; window size 10). constrained-context and unconstrained-context settings, with especially large gains on hard tasks ( view at source ↗

**Figure 5.** Figure 5: OOLONG-REAL training curves (moving average; window size 10). recursive agents learn substantially faster. On TEXTCRAFT-SYNTH , view at source ↗

**Figure 6.** Figure 6: DEEPDIVE training curves (moving average; window size 10) view at source ↗

**Figure 7.** Figure 7: Maximum delegation depth on TEXTCRAFT-SYNTH. Recursive agents adapt their delegation depth to the task. RAO also teaches agents when to delegate and how much to delegate view at source ↗

**Figure 8.** Figure 8: Ablation of RAO design choices on TEXTCRAFT-SYNTH view at source ↗

**Figure 9.** Figure 9: ART-E (Email Search) training curves for Qwen-3-14B (moving average; window size 10). A.2 Unbiased Baseline for RAO Lemma 1 (Unbiased leave-one-out baseline). Let τ (g) be any trajectory in rollout tree T (g) — root or sub-agent — and let b−g be the leave-one-out baseline defined in Eq. (3). Then E h (R(τ (g) ) − b−g) ∇θ log πθ (τ (g) ) i = E h R(τ (g) ) ∇θ log πθ (τ (g) ) i , i.e., subtracting b−g does no… view at source ↗

read the original abstract

We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents to scale to longer contexts and generalize to more difficult problems via divide-and-conquer. RAO provides a method to train models to best take advantage of such recursive inference, teaching agents when and how to delegate and communicate. We find that recursive agents trained in this way enjoy better training efficiency, can scale to tasks that go beyond the model's context window, generalize to tasks much harder than the ones the agent was trained on, and can enjoy reduced wall-clock time compared to single-agent systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

RAO trains agents with RL to recursively delegate to self-copies for inference scaling, but the stability of that delegation under harder tasks remains the untested core assumption.

read the letter

The main point is that this paper takes recursive agent spawning and turns it into something you can train with reinforcement learning. Agents learn when to create new instances of themselves, hand off sub-tasks, and pass results back, which in theory gives a divide-and-conquer path to longer contexts and harder problems than a flat agent can manage. They also claim this yields better training efficiency and lower wall-clock time than single-agent baselines. Framing recursion as a learned inference-time scaling method rather than a prompting trick is the concrete step forward here. It directly addresses the practical problem of getting models to use delegation productively instead of hoping they do it right out of the box. The motivation and high-level design read cleanly. The soft spot is exactly where the stress test points: the claims rest on RL producing stable delegation policies without excessive depth, failed hand-offs, or token overhead. Standard success-based rewards do not automatically penalize those failure modes, and the abstract gives no reward shaping details, state features for delegation decisions, or termination safeguards. If those are not handled, the scaling and generalization benefits on out-of-distribution tasks disappear. Without the experiments, baselines, or ablations in view, it is impossible to judge how well the training actually controls the risks. This is for people working on agent architectures and inference scaling in language models. Readers interested in hierarchical or multi-agent RL will see a fresh angle in the self-recursive formulation. It deserves a serious referee because the idea is distinct and the potential payoff for complex tasks is real, even if the current evidence is thin on the training mechanics. Send it to review, but expect the referees to press hard on reward design and empirical validation of the stability claims.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Recursive Agent Optimization (RAO), a reinforcement learning method for training recursive agents that spawn and delegate sub-tasks to new instantiations of themselves. This implements an inference-time divide-and-conquer scaling algorithm claimed to allow scaling beyond the model's context window, generalization to harder tasks than those seen in training, improved training efficiency, and reduced wall-clock time relative to single-agent baselines.

Significance. If the empirical claims are substantiated, RAO would offer a novel inference-time scaling paradigm for agentic systems that leverages learned recursion rather than larger models or longer contexts. This addresses a core limitation in current LLM agents and could influence research on multi-agent coordination and test-time compute allocation.

major comments (2)

[Methods] Methods section: No reward function, state features for delegation decisions, or termination safeguards are specified. Standard RL objectives supply only sparse task-success rewards and do not inherently constrain recursion depth or communication overhead; without explicit shaping or depth limits, the stability of delegation policies on harder out-of-distribution tasks cannot be assessed and directly threatens the claimed generalization and wall-clock benefits.
[Experiments] Experiments section: The manuscript provides no description of baselines (e.g., single-agent or non-recursive multi-agent variants), quantitative metrics for training efficiency or wall-clock time, task construction for context-exceeding cases, or statistical significance of results. These omissions make it impossible to verify the reported improvements or rule out confounds such as increased total compute.

minor comments (2)

[Abstract] Abstract: The phrase 'better training efficiency' is used without reference to the specific baseline or the magnitude of improvement.
[Introduction] Notation: The terms 'recursive agents' and 'new instantiations of themselves' are introduced without a precise definition of the agent state or communication protocol.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback on our paper 'Recursive Agent Optimization'. We have carefully considered the major comments and revised the manuscript to provide the requested details on methods and experiments.

read point-by-point responses

Referee: [Methods] Methods section: No reward function, state features for delegation decisions, or termination safeguards are specified. Standard RL objectives supply only sparse task-success rewards and do not inherently constrain recursion depth or communication overhead; without explicit shaping or depth limits, the stability of delegation policies on harder out-of-distribution tasks cannot be assessed and directly threatens the claimed generalization and wall-clock benefits.

Authors: We agree that additional details are necessary to fully substantiate our claims. In the revised manuscript, we have included a precise specification of the reward function, which combines a sparse task-success signal with dense shaping rewards for efficient delegation and penalties for high communication overhead. The state features used for delegation decisions encompass the current recursion depth, estimated sub-task difficulty, and available context length. Termination safeguards consist of a configurable maximum recursion depth and an automatic termination condition when overhead exceeds a threshold. These enhancements allow for a thorough assessment of policy stability on out-of-distribution tasks and support the reported generalization and efficiency benefits. revision: yes
Referee: [Experiments] Experiments section: The manuscript provides no description of baselines (e.g., single-agent or non-recursive multi-agent variants), quantitative metrics for training efficiency or wall-clock time, task construction for context-exceeding cases, or statistical significance of results. These omissions make it impossible to verify the reported improvements or rule out confounds such as increased total compute.

Authors: We have revised the Experiments section to address these omissions. We now describe the baselines in detail, including the single-agent RL baseline and a non-recursive multi-agent variant where delegation is not recursive. Quantitative metrics for training efficiency (e.g., episodes to reach 90% success rate) and wall-clock time (measured in seconds per task on standardized GPUs) are provided in new tables. Task construction for context-exceeding cases is explained as the concatenation of sub-problems designed to surpass the model's context window while maintaining logical coherence. Statistical significance is evaluated using 5 independent seeds with reported p-values from t-tests to rule out confounds like increased compute. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical outcomes of RL training, not derived by construction

full rationale

The paper introduces RAO as an RL-based training procedure for recursive agents and reports empirical benefits (training efficiency, context scaling, generalization to harder tasks, wall-clock gains) as measured results from that procedure. No equations, uniqueness theorems, fitted parameters renamed as predictions, or self-citation chains are invoked to derive these outcomes; they are presented as experimental findings. The derivation chain is therefore self-contained against external benchmarks and does not reduce to tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no equations, parameters, or explicit assumptions; the approach implicitly assumes that recursive delegation is learnable via standard RL without additional invented mechanisms.

pith-pipeline@v0.9.0 · 5425 in / 1052 out tokens · 41021 ms · 2026-05-08T12:06:44.950664+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

m0_i1": 2,

def craft(ingredients: dict, target: tuple[str, int]) -> str Craft items using ingredients from your inventory. - ingredients: Dict of item_name: count to consume - target: (item_name, total_count) where total_count must be divisible by recipe result_count - Example: craft({"m0_i1": 2, "m1_i1": 1}, ("m2_i2", 2))

work page
[2]

item": str,

def get_info(items: list) -> list[dict] Get recipe information for items. - Returns: List with {"item": str, "can_craft": bool, "is_base": bool, "in_inventory": int, "crafting_depth": int, "recipes": [...]} - crafting_depth indicates complexity: 0=base item, 1=direct craft, 2+=needs intermediate steps - Each recipe shows {"ingredients": {...}, "result_cou...

work page
[4]

Successfully crafted all required items

def finish(message: str) -> str Complete the task. - Example: finish("Successfully crafted all required items") TextCraft-Synth Recursive Agent Action Space

work page
[5]

m0_i1": 2,

def craft(ingredients: dict, target: tuple[str, int]) -> str Craft items using ingredients from your inventory. - ingredients: Dict of {item_name: count} to consume - target: (item_name, total_count) where total_count must be divisible by recipe result_count - Example: craft({"m0_i1": 2, "m1_i1": 1}, ("m2_i2", 2))

work page
[6]

item": str,

def get_info(items: list) -> list[dict] Get recipe information for items. - Returns: List with {"item": str, "can_craft": bool, "is_base": bool, "in_inventory": int, "crafting_depth": int, "recipes": [...]} - crafting_depth indicates complexity: 0=base item, 1=direct craft, 2+=needs intermediate steps - Each recipe shows {"ingredients": {...}, "result_cou...

work page
[7]

- Returns: Dict of {item_name: count} - Example: inv = view_inventory()

def view_inventory() -> dict View your current inventory. - Returns: Dict of {item_name: count} - Example: inv = view_inventory()

work page
[8]

Successfully crafted all required items

def finish(message: str) -> str Complete the task. - Example: finish("Successfully crafted all required items")

work page
[9]

m0_i2": 4}, 20) - Parallel: results = await asyncio.gather( launch_subagent({

async def launch_subagent(targets: dict, num_steps: int, context: str = "") -> str Launch a subagent to craft specific targets (shares your inventory). - targets: Dict of {item_name: count} to craft - num_steps: Budget for subagent * Use crafting_depth from get_info() to estimate: depth×8-10 steps * Example: depth=4 needs ˜32-40 steps, depth=8 needs ˜64-8...

work page
[10]

22 Oolong-Real Recursive Agent Action Space Available Actions (python functions): Pre-loaded variable: - context (str): The full text context to analyze

def finish(message: str) -> str Complete the task with your answer. 22 Oolong-Real Recursive Agent Action Space Available Actions (python functions): Pre-loaded variable: - context (str): The full text context to analyze

work page
[11]

, context=context_chunk ) print(f

async def launch_subagent(goal: str, context: str) -> Any Launch a subagent to process a chunk of the context. Returns the result of the subagent's execution with return type specified in the goal (if specified, else str). - goal: The goal/instruction for the subagent. Tell the subagent what information you want and specify the exact format and type in wh...

work page
[12]

Oolong-Real LLM Judge Prompt for sub-agent rewards We need to judge the performance of an agent on a task

def finish(result: Any) -> Any Complete the task with your result. Oolong-Real LLM Judge Prompt for sub-agent rewards We need to judge the performance of an agent on a task. The task relates to aggregating some information from a potentially very large context

work page
[13]

Read the context carefully and see if the agent’s answer is accurate

work page
[14]

reason":

Do not mark the agent as successful unless it prints out the context and reads it manually or alternatively uses subagents to answer the question. For example, if the agent uses regex or string matching/contains logic to answer the question, this is a heuristic that may not be reliable in general and thus should not be marked as successful. Using subagent...

work page
[15]

query": str,

async def search_web(query: str, max_results: int = 5) -> dict Search the web for information related to the query. - query: The query to search for. - max_results: Optional maximum number of results to return. Must be between 1 and 20. Defaults to 5. Returns a dictionary containing: { "query": str, "follow_up_questions": list[str], "answer": str, "images...

work page
[16]

- url: The URL of the webpage to view

async def view_webpage_content(url: str) -> str 25 View the content of a webpage. - url: The URL of the webpage to view. Returns a string containing the webpage content. This may be very long, so it is often useful to inspect its size before printing the full text

work page
[17]

This is synchronous and should not be awaited

def finish(message: str) -> str Complete the task with your answer. This is synchronous and should not be awaited. DeepDive Recursive Agent Action Space Available Actions (python functions):

work page
[18]

Find the price range of a PS5 across sony, bestbuy, amazon and gamestop. Return the answer as a string of the form'$$$ - $$$$'

async def launch_subagent(goal: str) -> Any Launch a subagent to solve a subtask. - goal: The instruction for the subagent. This can be a simple or compound task. Subagents can recursively delegate tasks to other subagents. - Specify the format and type of answer you expect from the subagent. Example: ps5_price_range = launch_subagent( "Find the price ran...

work page
[19]

- query: The query to search for

async def search_web(query: str, max_results: int = 5) -> dict Search the web for information related to the query. - query: The query to search for. - max_results: Optional maximum number of results to return. Must be between 1 and 20. Defaults to 5

work page
[20]

Returns a string containing the webpage content

async def view_webpage_content(url: str) -> str View the content of a webpage. Returns a string containing the webpage content. This may be very long

work page
[21]

reason":

def finish(message: str) -> str Complete the task with your answer. This is synchronous and should not be awaited. 26 DeepDive LLM Judge Prompt (Root Task) We need to judge the performance of a deep research agent on a task. The task requires searching the web for information across various sources and synthesizing information together to answer a questio...

work page

[1] [1]

m0_i1": 2,

def craft(ingredients: dict, target: tuple[str, int]) -> str Craft items using ingredients from your inventory. - ingredients: Dict of item_name: count to consume - target: (item_name, total_count) where total_count must be divisible by recipe result_count - Example: craft({"m0_i1": 2, "m1_i1": 1}, ("m2_i2", 2))

work page

[2] [2]

item": str,

def get_info(items: list) -> list[dict] Get recipe information for items. - Returns: List with {"item": str, "can_craft": bool, "is_base": bool, "in_inventory": int, "crafting_depth": int, "recipes": [...]} - crafting_depth indicates complexity: 0=base item, 1=direct craft, 2+=needs intermediate steps - Each recipe shows {"ingredients": {...}, "result_cou...

work page

[3] [4]

Successfully crafted all required items

def finish(message: str) -> str Complete the task. - Example: finish("Successfully crafted all required items") TextCraft-Synth Recursive Agent Action Space

work page

[4] [5]

m0_i1": 2,

def craft(ingredients: dict, target: tuple[str, int]) -> str Craft items using ingredients from your inventory. - ingredients: Dict of {item_name: count} to consume - target: (item_name, total_count) where total_count must be divisible by recipe result_count - Example: craft({"m0_i1": 2, "m1_i1": 1}, ("m2_i2", 2))

work page

[5] [6]

item": str,

def get_info(items: list) -> list[dict] Get recipe information for items. - Returns: List with {"item": str, "can_craft": bool, "is_base": bool, "in_inventory": int, "crafting_depth": int, "recipes": [...]} - crafting_depth indicates complexity: 0=base item, 1=direct craft, 2+=needs intermediate steps - Each recipe shows {"ingredients": {...}, "result_cou...

work page

[6] [7]

- Returns: Dict of {item_name: count} - Example: inv = view_inventory()

def view_inventory() -> dict View your current inventory. - Returns: Dict of {item_name: count} - Example: inv = view_inventory()

work page

[7] [8]

Successfully crafted all required items

def finish(message: str) -> str Complete the task. - Example: finish("Successfully crafted all required items")

work page

[8] [9]

m0_i2": 4}, 20) - Parallel: results = await asyncio.gather( launch_subagent({

async def launch_subagent(targets: dict, num_steps: int, context: str = "") -> str Launch a subagent to craft specific targets (shares your inventory). - targets: Dict of {item_name: count} to craft - num_steps: Budget for subagent * Use crafting_depth from get_info() to estimate: depth×8-10 steps * Example: depth=4 needs ˜32-40 steps, depth=8 needs ˜64-8...

work page

[9] [10]

22 Oolong-Real Recursive Agent Action Space Available Actions (python functions): Pre-loaded variable: - context (str): The full text context to analyze

def finish(message: str) -> str Complete the task with your answer. 22 Oolong-Real Recursive Agent Action Space Available Actions (python functions): Pre-loaded variable: - context (str): The full text context to analyze

work page

[10] [11]

, context=context_chunk ) print(f

async def launch_subagent(goal: str, context: str) -> Any Launch a subagent to process a chunk of the context. Returns the result of the subagent's execution with return type specified in the goal (if specified, else str). - goal: The goal/instruction for the subagent. Tell the subagent what information you want and specify the exact format and type in wh...

work page

[11] [12]

Oolong-Real LLM Judge Prompt for sub-agent rewards We need to judge the performance of an agent on a task

def finish(result: Any) -> Any Complete the task with your result. Oolong-Real LLM Judge Prompt for sub-agent rewards We need to judge the performance of an agent on a task. The task relates to aggregating some information from a potentially very large context

work page

[12] [13]

Read the context carefully and see if the agent’s answer is accurate

work page

[13] [14]

reason":

Do not mark the agent as successful unless it prints out the context and reads it manually or alternatively uses subagents to answer the question. For example, if the agent uses regex or string matching/contains logic to answer the question, this is a heuristic that may not be reliable in general and thus should not be marked as successful. Using subagent...

work page

[14] [15]

query": str,

async def search_web(query: str, max_results: int = 5) -> dict Search the web for information related to the query. - query: The query to search for. - max_results: Optional maximum number of results to return. Must be between 1 and 20. Defaults to 5. Returns a dictionary containing: { "query": str, "follow_up_questions": list[str], "answer": str, "images...

work page

[15] [16]

- url: The URL of the webpage to view

async def view_webpage_content(url: str) -> str 25 View the content of a webpage. - url: The URL of the webpage to view. Returns a string containing the webpage content. This may be very long, so it is often useful to inspect its size before printing the full text

work page

[16] [17]

This is synchronous and should not be awaited

def finish(message: str) -> str Complete the task with your answer. This is synchronous and should not be awaited. DeepDive Recursive Agent Action Space Available Actions (python functions):

work page

[17] [18]

Find the price range of a PS5 across sony, bestbuy, amazon and gamestop. Return the answer as a string of the form'$$$ - $$$$'

async def launch_subagent(goal: str) -> Any Launch a subagent to solve a subtask. - goal: The instruction for the subagent. This can be a simple or compound task. Subagents can recursively delegate tasks to other subagents. - Specify the format and type of answer you expect from the subagent. Example: ps5_price_range = launch_subagent( "Find the price ran...

work page

[18] [19]

- query: The query to search for

async def search_web(query: str, max_results: int = 5) -> dict Search the web for information related to the query. - query: The query to search for. - max_results: Optional maximum number of results to return. Must be between 1 and 20. Defaults to 5

work page

[19] [20]

Returns a string containing the webpage content

async def view_webpage_content(url: str) -> str View the content of a webpage. Returns a string containing the webpage content. This may be very long

work page

[20] [21]

reason":

def finish(message: str) -> str Complete the task with your answer. This is synchronous and should not be awaited. 26 DeepDive LLM Judge Prompt (Root Task) We need to judge the performance of a deep research agent on a task. The task requires searching the web for information across various sources and synthesizing information together to answer a questio...

work page