{"arxiv_id":"2611.16409","title":"Experience-Handoff Reinforcement Learning for Recursive Self-Improvement","abstract":"It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \\acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \\acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.","primary_cat":"cs.LG","cross_cats":["cs.CL"],"submitted_at":"2026-11-12T07:18:30+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"CONDITIONAL","confidence":"HIGH","one_line_summary":"ACE-RL trains an LLM to act, reflect, and curate playbooks by rewarding each role from a downstream rollout, achieving higher test scores than actor-only or end-to-end training across four benchmarks.","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0,"inbound_count":0,"_links":{"html":"https://pith.science/paper/2611.16409","verdict":"/api/v1/verdicts/2611.16409"}}