{"id":"dd88d84b-4642-4d5e-b4dd-707246067eb0","arxiv_id":"2611.16409","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ACE-RL trains an LLM to act, reflect, and curate playbooks by rewarding each role from a downstream rollout, achieving higher test scores than actor-only or end-to-end training across four benchmarks.","lead":"This paper introduces ACE-RL, a reinforcement learning method that trains a large language model not only to solve tasks but also to write playbooks that improve its future attempts. By using a separate rollout to evaluate each candidate playbook and comparing outputs within each role (actor, reflector, curator), the method outperforms actor-only training on four agentic benchmarks, suggesting that teaching an agent to hand off experience in text form yields lasting gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Single training run per method limits statistical robustness; observed gains may not be reproducible across seeds.","rationale":"The reader's verdict is CONDITIONAL, with the primary requested condition being code release and 'ideally additional seeds.' This concern is the single most load-bearing: the paper's headline empirical claim (ACE-RL outperforms baselines) rests on comparisons from a single training run per method. While the paper acknowledges this limitation, it does not resolve it, and the reader's condition already targets this. My concern aligns with the reader's call for additional seeds but goes further in highlighting that seed-level variance could affect the central conclusion. I recommend UNCHANGED because the reader's condition already accounts for this risk. The concern is not that the paper is wrong, but that the evidence is incomplete without multi-seed verification.","tokens_in":48268,"tokens_out":2786,"duration_ms":25370,"concrete_test":"Run the main comparison of Table 2 (ACE-RL vs. GRPO vs. PPO vs. ERL vs. Delayed-joint PPO) on at least one benchmark (e.g., Mastermind or MBPP) with 5 independent random seeds per method, keeping all other hyperparameters fixed. Report the mean and standard deviation of test reward (RSI@3 for ACE-RL, pass@3 for GRPO) across seeds. If ACE-RL's mean is above GRPO's by at least 0.05 with non-overlapping standard deviations, the claim is robust; otherwise the advantage may be seed-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper reports only one training run per method (Section 4, Table 2, Figure 4). All error bars are standard errors over held-out tasks within that single run, not over training seeds. The training process involves stochasticity in policy initialization, rollout sampling, group composition, and playbook-reset timing. Without multiple independent training runs, it is possible that ACE-RL's advantage over baselines (e.g., GRPO) is partially driven by a favourable seed. This concern is acknowledged in Appendix H ('Our evidence concerns … one training run per method'), but the central claim that ACE-RL outperforms all baselines at equal compute would be weakened if another seed yielded smaller or reversed gains. The paired comparison over tasks within a single run does not address seed-level variance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes ACE-RL, a method that trains a single language model to act, reflect, and curate task-local playbooks within a fixed ACE harness, using verifiable rewards for handoff artifacts and per-role comparison groups. The central claim is that ACE-RL achieves higher held-out reward than baselines (actor-only RL, ERL, delayed-joint PPO) at equal training compute across four benchmarks (Mastermind, MBPP, AppWorld, MiniSWEBench). The results suggest that training should optimize not only actions but also the model-written experience that future rollouts inherit.","tokens_in":48524,"tokens_out":2740,"duration_ms":24155,"significance":"If the results hold, ACE-RL provides a practical method for training agents to improve their own experience handoffs, combining outcome-based RL with rich textual artifacts. The experimental design is careful: matched training compute, multiple baselines, a control (edits-discarded), and evaluation beyond the training horizon. The method is grounded in a verifiable reward for handoff artifacts, which avoids learned reward models. However, the central empirical claim rests on a single training run per method, which weakens the statistical robustness of the conclusions across random seeds.","major_comments":[{"comment":"Only one training run is reported per method. The paper acknowledges this in Appendix H (\"Our evidence concerns… one training run per method\"), but the central claim that ACE-RL outperforms all baselines is directly supported by this single run. Error bars over held-out tasks address within-run variability but not seed-level variance. The authors should either provide additional training runs (e.g., 3 seeds for one benchmark) to demonstrate stability, or add a prominent caveat in the main text that the results are from a single training run and may be sensitive to seed. This is load-bearing because the ranking could change with different random initializations, and the current presentation overstates the strength of the evidence.","section":"Section 4, Table 2, Figure 4, and Appendix H","authors_acknowledge":"Measuring that variance directly, and establishing with additional training runs when it pays off, is left to future work. 17 Experience-Handoff Reinforcement Learning for Recursive Self-Improvement C ROLEPROMPTS The listings below are the actor, reflector, and curator prompts used for training on each benchmark.","authors_acknowledge_p":0.72},{"comment":"The derivation of the gradient estimator is careful, but the implementation uses several approximations: a single evaluation per candidate playbook (L=1), mixing seed and candidate rollouts in the same actor group (different inputs (q,P) vs (q,P')), and group filtering that drops constant-reward groups. These approximations are not guaranteed to preserve the expected gradient direction. While Appendix D.1 discusses these issues, the main text presents the method as a straightforward application of GRPO. The authors should add a brief statement in Section 3.2 that the update is a sparse, variance-reduced approximation to the full-tree estimator, and reference the appendix for details.","section":"Appendix D.1 and Section 3.2","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The term 'Experience-Handoff RL' is introduced but not formally defined until Section 2.2. Consider defining it earlier, after the first paragraph, to help the reader.","section":"Introduction (Section 1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The font size in the figure is small; consider enlarging labels for readability, especially the role-group annotations.","section":"Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'Advantage' column uses abbreviations ('group', 'critic (GAE)') that could be expanded ('group-relative', 'GAE from learned critic') for clarity.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that 'the final playbook is verifiable' relies on a single rollout (Eq. 6). While group normalization reduces variance, the paper should note explicitly that the verifier reward is a noisy estimate, not a ground-truth value, and that noise is handled by the per-role groups.","section":"Section 3.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper says 'the gain does not fade with more attempts', but on Mastermind the benefit of applying edits fades after K=3 and from K=5 onward the discarded run is slightly ahead. This should be stated in the text rather than only visible in the figure.","section":"Section 5.3, Figure 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The variance decomposition in Eq. (9) is insightful; consider citing it briefly in the main paper (e.g., after Section 3.2) to motivate the per-role grouping.","section":"Appendix B.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conclusion restates results without mentioning the single-run limitation. Even a short sentence (e.g., 'Results are based on one training run per method; see Appendix H for discussion') would be appropriate.","section":"Section 6 (Conclusion)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper introduces a real advance: training all three roles (actor, reflector, curator) in a harness using downstream rollout rewards and per-role comparison groups. The central claim—that ACE-RL outperforms actor-only RL, ERL, and chain-level credit methods—holds in the reported experiments. The experimental design is careful: matched compute, a control that discards edits, evaluation beyond the training horizon, and multiple baselines including a critic-based method that collapses. The paper is well written and transparent about its limitations.\n\nThe soft spot is the one you flagged. The authors ran one training seed per method. The error bars they report are over held-out tasks within that single run, not over training seeds. The stress-test note is right: with stochasticity in initialization, rollout sampling, group composition, and playbook resets, the observed advantage might not replicate under another seed. The authors acknowledge this in Appendix H, but it weakens the headline claim. A second concern: the method depends on a single rollout to score each candidate playbook. The paper says this works because the final playbook is verifiable, but in stochastic environments that signal can be noisy. The per-role groups mitigate variance, but they don't eliminate it.\n\nThe method also adds generation cost (reflector and curator calls) even though update FLOPs are matched. The code is promised but not yet released. The experiments are on models up to 4B parameters; scaling to frontier models is open.\n\nThis paper is for researchers in agentic RL and harness-based self-improvement. It deserves a serious referee and conditional acceptance. I would not desk reject it. My recommendation: ask for a second training seed as a robustness check, and release the code before final acceptance.","headline":"Single training run per method limits statistical robustness despite careful design.","tokens_in":49052,"tokens_out":1377,"duration_ms":12877},"pith_extraction":{"msc":["68T05","68T07","68T50"],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Training an agent to write its own future instructions outperforms only training its actions.","keywords":["reinforcement learning","LLM agents","recursive self-improvement","experience handoff","playbook curation","verifiable reward","per-role normalization","ACE harness"],"falsifier":"On a new benchmark with a reliable task verifier, train GRPO and ACE-RL with matched compute and evaluate both at RSI@3. If ACE-RL does not outperform GRPO (deployed in the same ACE harness) by a statistically significant margin, the central claim that experience-handoff training adds value beyond actor-only RL is falsified. Alternatively, if on the existing benchmarks the playbook adaptation does not increase RSI@K over independent pass@K beyond K=3 (the training horizon), the claim of extrapolation would be weakened.","tokens_in":49743,"feed_emoji":"🤖","tokens_out":3056,"duration_ms":26646,"temperature":0.7,"pith_summary":"Large language models trained with reinforcement learning for agentic tasks typically compress each trial into a scalar reward, discarding the rich environment feedback (error logs, test failures) that the agent experienced. Harnesses like ACE preserve that feedback as text artifacts (reflections, playbooks) that future attempts can read, but the model never learns to write better artifacts—the handoff is frozen. This paper introduces experience-handoff RL, in which a single model is trained from task outcomes alone to act, to reflect on its trajectory, and to curate a task-local playbook that improves future attempts. The method, ACE-RL, scores each candidate playbook by executing a fresh rollout with it and normalizes rewards within per-role comparison groups. Across four benchmarks (Mastermind, MBPP, AppWorld, MiniSWEBench), ACE-RL achieves higher held-out reward than actor-only RL, end-to-end chain training with a learned critic, and the common practice of running an actor-only checkpoint inside the same harness.","feed_headline":"Training agents to write their future playbooks boosts performance","feed_subtitle":"ACE-RL learns to act, reflect, and curate task tips from outcomes alone, beating actor-only RL on four benchmarks.","key_machinery":"The fixed Agentic Context Engineering (ACE) harness with three roles (actor, reflector, curator), the verifiable playbook reward (each candidate playbook P' is scored by a fresh rollout R_q(τ̃)), and per-role GRPO-style comparison groups (actions compared with other actor rollouts of the same task, reflections compared with other reflections of the same seed trajectory, edits compared with other edits of the same reflection). These allow joint optimization of all roles with one clipped surrogate update, without a learned critic.","core_discovery":"The paper claims that simultaneously training the actor, reflector, and curator roles of a fixed harness (ACE) from verifiable downstream rewards produces a model whose playbook edits actually help future attempts, while training only the actor does not. The key enablers are (i) a verifiable reward for each handoff artifact obtained by executing the candidate playbook in a fresh rollout, and (ii) per-role comparison groups that give each output (action, reflection, edit) an advantage normalized within its own group under a shared input, rather than sharing a chain’s final reward. On all four benchmarks the ACE-RL checkpoint beats GRPO, ERL, delayed-joint PPO, and the GRPO checkpoint run inside the ACE harness. The gain persists beyond the training horizon (F=3) up to K=8 attempts, and an edits-discarded control shows the playbook contributes beyond actor improvement alone.","pith_inferences":["If ACE-RL scales to larger models and more open-ended tasks, it could enable agents that iteratively refine their own strategies without manual prompt engineering or external memory management—a step toward autonomous recursive self-improvement.","The verifiable-playbook assumption is brittle: if the task verifier is noisy, stochastic, or expensive, the single-rollout estimate of a candidate's value may miscredit the handoff, and the paper's per-role groups cannot fix a biased reward signal.","The edits-discarded control shows the playbook itself drives performance gains beyond actor improvement; this implies that the gap between actor-only and experience-handoff training may widen as tasks require more structured knowledge (API dependencies, implied constraints) that a playbook can capture.","The periodic reset of playbooks (every F=3 encounters) bounds the age of retained experience; extending to cross-task or cross-episode transfer (longer-lived playbooks) could compound the benefit but also risk accumulating stale or misleading tips."],"forward_implications":["Agentic RL pipelines should optimize not only the actor's actions but also the model's ability to write useful experience (reflections, playbooks) for future attempts.","The per-role comparison group design can be applied to other harness architectures that support isolable handoff artifacts, potentially improving credit assignment in multi-step agentic systems.","The extrapolation beyond the training horizon suggests that learned playbook adaptation continues to provide benefits with more attempts, reducing the need for expensive re-evaluation of every candidate.","The method demonstrates that a single policy can be trained to serve multiple roles in a fixed scaffold, which may simplify deployment of self-improving agents."],"supporting_citations":[{"why":"Defines the ACE harness (actor, reflector, curator) that serves as the fixed scaffold for ACE-RL.","marker":"[Zhang et al., 2025b]"},{"why":"Provides the GRPO algorithm that ACE-RL extends with per-role comparison groups and verifiable rewards for handoff artifacts.","marker":"[Shao et al., 2024]"},{"why":"Supplies the clipped surrogate objective used by all compared methods, including the PPO and delayed-joint PPO baselines.","marker":"[Schulman et al., 2017]"},{"why":"Describes ERL, a baseline that hands off experience through weight updates (internalization) rather than through model-written artifacts.","marker":"[Shi et al., 2026]"},{"why":"Introduces CompactionRL, which forms the basis for the delayed-joint PPO baseline that trains actor, reflector, and curator with chain-level credit and a learned critic.","marker":"[Li et al., 2026]"}],"fun_headline_variants":["RL that trains agents to write their own playbooks from outcomes","Agent learns to curate task tips from past rolls, boosting future attempts","Playbook-writing RL beats actor-only training on four agentic benchmarks","Recursive self-improvement: model writes experience handoff that works","From outcome alone, model learns to act, reflect, and curate future help"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The final playbook is verifiable: a single downstream rollout of the candidate playbook yields a useful reward signal for crediting the reflections and edits that produced it.","fun_headline_variants_meta":{"raw":{"variants":["RL that trains agents to write their own playbooks from outcomes","Agent learns to curate task tips from past rolls, boosting future attempts","Playbook-writing RL beats actor-only training on four agentic benchmarks","Recursive self-improvement: model writes experience handoff that works","From outcome alone, model learns to act, reflect, and curate future help"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"cost":0.000270088,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000270088,"upstream_inference_prompt_cost":0.000106288,"upstream_inference_completions_cost":0.0001638},"total_tokens":1549,"prompt_tokens":964,"completion_tokens":585,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":490}},"tokens_in":964,"tokens_out":585,"cost_source":"openrouter_usage","duration_ms":5973,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":490,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:22:12.164731+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"On a new benchmark with a reliable task verifier, train GRPO and ACE-RL with matched compute and evaluate both at RSI@3. If ACE-RL does not outperform GRPO (deployed in the same ACE harness) by a statistically significant margin, the central claim that experience-handoff training adds value beyond actor-only RL is falsified. Alternatively, if on the existing benchmarks the playbook adaptation does not increase RSI@K over independent pass@K beyond K=3 (the training horizon), the claim of extrapolation would be weakened.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}